DeepCAD-RT: Real-time denoising of fluorescence time-lapse imaging using deep self-supervised learning
1. Basic principle
Calcium imaging enables parallel recordings of large neuronal ensembles in living animals and offers a new possibility for deciphering information propagation, integration, and computation in neural circuits. However, calcium imaging is inherently susceptible to detection noise especially when imaging with high frame rate or under low excitation dosage. We previously presented DeepCAD, a deep self-supervised learning-based method for calcium imaging denoising. Using our method, detection noise can be effectively removed and the accuracy of neuron extraction and spike inference can be highly improved without requiring any high-SNR observations.
Paper
Nature Methods, 2021.
Citation
Li, X., Zhang, G., Wu, J. et al. Reinforcing neuron extraction and spike inference in calcium imaging using deep self-supervised denoising. Nat Methods (2021). https://doi.org/10.1038/s41592-021-01225-0
Noise is an ineluctable obstacle in scientific observation. For almost all fluorescence imaging technologies, the inherent shot-noise limit determines the upper bound of imaging SNR and restricts the imaging resolution, speed, and sensitivity. With advancements in methods and applications, we present DeepCAD-RT, a versatile model to denoise fluorescence images with rapid processing speed and improved performance. It can also be incorporated with the microscope acquisition system to achieve real-time denoising. We demonstrate the capability and generality of DeepCAD-RT on a series of photon-limited imaging experiments.
Much faster processing speed and much lower memory cost:
We constructed a lightweight network and compressed the model parameters by 94%, which consequently reduced 85% processing time and 70% memory consumption.
More stable performance:
We provide 16-fold data augmentation to alleviate the data dependency and make the method still tractable with a small amount of data.
Hardware deployment:
We optimized the hardware deployment and achieved an overall improvement of a 27-fold reduction in running memory and a 20-fold acceleration in inference speed, which supports real-time image denoising incorporated with the microscope acquisition system.
Real-time implementation
To achieve real-time processing during imaging acquisition, we made a program interface to incorporate DeepCAD-RT into our image acquisition software (Scanimage 5.7, Vidrio Technologies). For further acceleration and memory conservation, the inference of DeepCAD-RT was optimally deployed on GPU with TensorRT (NVIDIA), programmed in C++ for best hardware interaction, and then compiled in Matlab (MathWorks). Three parallel threads were designed for imaging, data processing, and display. The schedule for multi-thread programming is depicted in the following figure. The real-time implementation of DeepCAD-RT has been packaged as a free plugin with a user-friendly interface, and it could also be called by a Matlab script.
Paper
Nature Biotechnology, 2022.
Citation
Li, X., Li, Y., Zhou, Y. et al. Real-time denoising enables high-sensitivity fluorescence time-lapse imaging beyond the shot-noise limit. Nat Biotechnol (2022). https://doi.org/10.1038/s41587-022-01450-8
Animal models currently used in systems and evolutionary neuroscience are diverse that extend from jellyfish to monkeys. DeepCAD-RT is a versatile self-supervised method and we have demonstrated extensive experiments including calcium imaging in mice, zebrafish, and flies, cell migration observations, and the imaging of a new genetically encoded ATP sensor, covering both 2D single-plane imaging and 3D volumetric imaging. We recommend watching these videos with the highest quality. More details please refer to the companion paper.
DeepCAD-RT massively improves the imaging SNR of neuronal population recordings in the zebrafish brain
DeepCAD-RT reveals the 3D migration of neutrophils in vivo after acute brain injury
DeepCAD-RT reveals the ATP (Adenosine 5’-triphosphate) dynamics of astrocytes in 3D after laser-induced brain injury
For more information, you can visit following sub-pages:
