UDMT: Unsupervised Multi-animal Tracking for Quantitative Ethology

Introduction

Quantitative ethology necessitates accurate tracking of animal locomotion, especially for population-level analyses involving multiple individuals. However, current methods rely on laborious annotations for supervised training and have restricted performance in challenging conditions. Here, we present UDMT, the first unsupervised multi-animal tracking method that achieves state-of-the-art performance without requiring any human annotations. By synergizing a bidirectional closed-loop tracking strategy, a spatiotemporal transformer network, and three sophisticatedly designed modules for localization refining, bidirectional ID correction, and automatic parameter tuning, UDMT can track multiple animals accurately in various challenging conditions, such as crowding, occlusion, rapid motion, low contrast, and cross-species experiments.

New strategy, new architecture, and new functional modules

Paper | Code

Tracking in a completely unsupervised manner:

UDMT does not require any human annotations for training. The only thing users need to do is to click the animals in the first frame to specify the individuals they want to track.

Spatiotemporal transformer for better feature extraction:

To better capture the spatiotemporal evolution of animal features more effectively, we incorporated a spatiotemporal transformer network (ST-Net) to utilize self-attention and cross-attention mechanisms for feature extraction, leading to a threefold reduction in IDSW compared with convolutional neural networks (CNNs).

Three key functional modules for performance optimization:

We designed three critical functional modules to solve the inherent performance degradation by localization refining, bidirectional ID correction, and automatic parameter tuning.

A readily accessible and user-friendly GUI

We have released a user-friendly graphical user interface (GUI) of UDMT to make it an easily accessible tool for ethology and neuroethology.

Applications and results

We demonstrate the state-of-the-art performance of UDMT on five different kinds of model animals, including mice, rats, Drosophila, C. elegans, and Betta splendens. Combined with a head-mounted miniaturized microscope, we recorded the calcium transients synchronized with mouse locomotion to decipher the correlations between animal locomotion and neural activity.More details please refer to the companion paper.