Spatial transcriptomics protocols based on in situ sequencing or multiplexed RNA fluorescent hybridization can reveal detailed tissue organization. However, distinguishing the boundaries of individual cells in such data is challenging, and can hamper downstream analysis. Current methods generally approximate cells positions using nuclei stains. We describe a segmentation method, Baysor, which optimizes 2D or 3D cell boundaries considering joint likelihood of transcriptional composition and cell morphology. While Baysor can take into account segmentation based on co-stains, it can also perform segmentation based on the detected transcripts alone. To evaluate performance, we extend MERFISH to incorporate immuno-staining of cell boundaries. Using this and other benchmarks we show that Baysor segmentation can in some cases nearly double the number of cells, while reducing segmentation artifacts. Importantly, we demonstrate that Baysor performs well on data acquired using five different protocols, making it a useful general tool for analysis of imaging-based spatial transcriptomics.

All methods are described in detail in the accompanying paper.

Briefly, MERFISH was performed against cryosections of the mouse ileum using standard protocols. Imaging was performed on a purpose-built microscope and flow system. Identification of RNAs was performed with an open-source analysis package (github.com/ZhuangLab/MERFISH_analysis).  Baysor analysis and cellpose analysis of these data were performed as described in the accompanying paper. 

The provided data contain text files that describe the data organization and contents. 

Models included in the directory data_analysis/cellpose are binary files created by and readable by the cellpose software (https://github.com/MouseLand/cellpose).