Neural Quantification and Visualization

A User Interface for Automated Behavior Quantification and Multimodal Data Visualization

PI: David J. Anderson (Division of Biology and Biological Engineering)
SASE: Dave Rumph, Senior Software Engineer

The long-term goal of the Anderson lab is to understand the function of neural circuits involved in the control of naturalistic behaviors such as aggression, mating, and defense. Lab members use a combination of neural recording and perturbation techniques to investigate the activity of genetically targeted neural populations in freely interacting mice. To perform quantitative analysis relating neural activity to an animal’s behavior, the animal’s actions must be tracked and labeled. But because social behaviors in mice are complex and highly variable, the quantification of such behaviors is typically performed manually, on a frame-by-frame basis—a process that is slow, expensive, and highly subjective. When behavior is being analyzed alongside neural recording data, it is also usually unclear whether the set of behaviors that were chosen to annotate are a good fit for explaining the activity of a neural population, or whether other, unannotated behaviors with clearer neural correlates may have been missed.

Working in collaboration with the laboratory of Pietro Perona, the Anderson lab has developed an automated system for pose (posture) estimation and behavior classification in pairs of freely interacting mice, named the Mouse Action Recognition System (MARS). MARS extracts spatiotemporal features from the poses of two mice, which it uses to automatically detect social behaviors of experimenter interest with high accuracy. MARS is accompanied by the Behavior Ensemble and Neural Trajectory Observatory (Bento), an interface that allows researchers to quickly navigate and analyze MARS pose estimates alongside videos of animal behavior, recordings of neural activity, and data in other modalities such as audio recording. Together, MARS and Bento will allow researchers to visualize the activity of the brain in new ways, to discover unexpected correlations between neural activity and behavior, and to automate the detection of newly identified behaviors so they may be identified in other datasets. The Schmidt Academy is working with the Anderson lab to integrate MARS and Bento into a single software package that will be made available to a broad community of neuroscientists.