stDyer enables spatial domain clustering with dynamic graph embedding

Abstract

Spatially resolved transcriptomics (SRT) data provide critical insights into gene expression patterns within tissue contexts, necessitating effective methods for identifying spatial domains. We introduce stDyer, an end-to-end deep learning framework for spatial domain clustering in SRT data. stDyer combines Gaussian Mixture Variational AutoEncoder with graph attention networks to learn embeddings and perform clustering. Its dynamic graphs adaptively link units based on Gaussian Mixture assignments, improving clustering and producing smoother domain boundaries. stDyer’s mini-batch strategy and multi-GPU support facilitate scalability to large datasets. Benchmarking against state-of-the-art tools, stDyer demonstrates superior performance in spatial domain clustering, multi-slice analysis, and large-scale dataset handling.