Decoding Natural Behavior from Neuroethological Embedding

Summary

This protocol gives an integrated framework based on advanced computational neuroethological methods to understand brain coding in naturalistic contexts.

Abstract

Animals engage with their natural environment through rich and dynamic brain activity. Understanding how neural population dynamics encode naturalistic behavior remains a fundamental challenge in systems neuroscience. Recent advances in deep learning-based behavior analysis and miniature fluorescence imaging have opened new avenues for investigating how the brain encodes natural behavior. Here, this study presents an integrated experimental and computational framework that combines the Social Behavior Atlas (SBeA), miniature Two-Photon Microscopy (mTPM), and Consistent EmBeddings of high-dimensional Recordings using Auxiliary variables (CEBRA) to decode complex behaviors from brain dynamics. This study uses naturalistic social interactions between freely moving mice as a model system, enabling high-resolution behavioral annotation alongside simultaneous neural imaging. This framework includes precise behavioral pose estimation, synchronized dual-mouse tracking, neural embedding alignment, and decoding of behavioral features directly from neural principal components. This study demonstrates that this approach achieves a decoding precision of 3. ± 1.5 pixels for posture and 89 ± 6% accuracy for motif decoding across animals, highlighting its robustness and generalizability. This method provides a powerful tool for exploring how brain activity reflects structured behavioral states, and it lays the groundwork for future studies of naturalistic neural coding principles.

Introduction

This framework is designed to capture and decode behavioral and neuroimaging data from freely moving animals in naturalistic experimental settings. It comprises three key components: deep learning-based pose estimation and behavior classification methods, SBeA¹, miniature fluorescence imaging techniques mTPM², and a contrastive learning-based neuroethological embedding algorithm, CEBRA³. Recent studies have highlighted the complexity of neuroethological processes in freely moving animals, which surpasses that observed in head-fixed experimental paradigms^4,5. However, technical limitations and variability have hindered the widespread application of these approaches to broader investigations of natural behavior. This protocol presents a stable and integrated framework that ensures the accessibility of behavioral and neural data collected in naturalistic contexts for a wide range of research laboratories.

Given that animals move freely in natural environments, this framework incorporates deep learning-based pose estimation to achieve precise tracking of postures^6,7. Traditional image processing-based tracking methods are insufficient for capturing fine-scale movements, such as limb and paw dynamics, compared to deep learning-based approaches⁸. The diverse and complex behaviors exhibited by freely moving animals pose challenges for supervised behavior classification methods⁹, as predefined behavioral categories often fail to encompass the full range of natural behavioral phenotypes¹⁰. Consequently, unsupervised learning-based classification methods are better suited for analyzing behavior in naturalistic settings¹. They can comprehensively decompose continuous behavior into discrete subsecond motifs according to their intrinsic structural similarities, and then their consistent definitions are given through data-driven clusters.

Brain imaging in freely moving animals requires capturing the extensive variability of single-neuron activity^4,5. Electrophysiological recordings in freely moving animals are limited in their ability to detect neurons with predominantly subthreshold activity¹¹. Additionally, single-photon microscopy suffers from low resolution and contrast, making it difficult to maintain consistent neuron identities across imaging sessions¹². mTPM offers superior resolution and contrast compared to single-photon microscopy, making it a more effective tool for investigating the neural coding of natural behaviors^2,13,14,15.

Establishing a robust mapping between behavior and neural data necessitates methods capable of revealing their shared informational structure¹⁶. Conventional dimensionality reduction techniques, such as Principal Component Analysis (PCA)¹⁷, t-Distributed Stochastic Neighbor Embedding (t-SNE)¹⁸, and Uniform Manifold Approximation and Projection (UMAP)¹⁹, cannot effectively embed behavioral and neural data within a common feature space. In contrast, deep learning-based embedding approaches, such as CEBRA, enable the integration of multiple data modalities in both supervised and self-supervised frameworks, generating high-quality latent representations³. While various alternative methods have emerged in recent years^20,21,22, this proposed framework prioritizes practical applications by incorporating well-established methods that are either commercially available or supported by comprehensive tutorials.

Compared to recent studies^4,5, this framework offers three key advancements. First, it eliminates human bias in behavior classification. Previous studies relied on manual behavior labeling, which is labor-intensive and prone to inconsistency, particularly as annotators experience fatigue^23,24,25. In contrast, this framework employs unsupervised behavior classification, which preserves the natural structure of behavioral patterns by objectively decomposing and clustering behavior motifs before assigning definitions^26,27. Second, the use of mTPM enables the capture of more intricate neuronal dynamics at the single-neuron level. This methodological advantage expands the applicability of this framework to decoding complex natural behaviors from diverse neural populations, including those involved in subthreshold coding²⁸. Third, this framework integrates behavioral and neural data into a unified representational space, rather than employing UMAP to embed each modality separately or using support vector machines to impose a rigid mapping between neural activity and behavior while disregarding their intrinsic dynamics^4,5. This joint embedding approach ensures a more comprehensive and biologically meaningful representation of the relationship between behavior and brain activity.

This framework is well-suited for research projects that involve the recording and decoding of behavioral and neural data from freely moving animals in naturalistic experimental conditions. While the current implementation is optimized for mouse studies, adapting it to other animal models may require additional development. As the hardware components utilized in this framework are commercially available, on the one hand, the overall cost may be relatively high. On the other hand, this commercial availability significantly reduces the time spent on troubleshooting logistical issues and ensures the acquisition of stable and reliable results in an efficient manner.

This protocol is designed to be reproducible and accessible to neuroscience laboratories equipped for small-animal imaging and behavior tracking. The complete system integrates a commercially available mTPM device with a multi-angle behavioral acquisition setup. Typical neural recordings are acquired at 4.84 Hz with 512 × 512 pixel resolution, and behavioral data are captured at 30 frames per second. Data synchronization is achieved through TTL pulse alignment during preprocessing. Training and decoding can be performed on a standard workstation with a GPU (e.g., NVIDIA RTX 3090 or equivalent), and the full pipeline requires approximately 100 GB of storage per experiment. While the current implementation is optimized for freely moving mice, the modular design of the workflow allows adaptation to other species by adjusting tracking calibration and imaging parameters based on the animal's size and mobility. These practical details support the protocol's adaptability and reproducibility across a range of experimental settings.

Protocol

The Animal Care and Use Committee at the Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, approved all husbandry and experimental procedures.

1. Platform establishment

NOTE: The platform consists of two primary components: the mTPM device and the 3D behavior device (Figure 1A). The mTPM device facilitates real-time synchronization of mTPM imaging with behavioral data, thereby enabling the efficient, stable, and continuous acquisition of high-quality data from freely moving animals. The 3D behavior device is equipped with four cameras to capture the full scene of animal behavior and an automatic calibration module for reconstructing 3D animal poses. Both devices are required to incorporate synchronization modules in their respective versions.

1. Device connection
   1. Remove the outer shell of the 3D behavior device. Place the remaining parts of the 3D behavior device, including four cameras and an automatic calibration module, inside the behavioral recording chamber of the mTPM device.
   2. Connect the Universal Serial Bus (USB) cable of the synchronization module of the 3D behavior device to the workstation of the 3D behavior device.
   3. Connect the synchronization module of the mTPM device to the controller of the mTPM device through one SubMiniature version A (SMA) cable.
   4. Connect the Transistor-Transistor Logic (TTL) output port of the synchronization module of the 3D behavior device to the TTL input port of the synchronization module of the mTPM device through one SMA-Bayonet Neill-Concelman (BNC) conversion cable.
2. Timestamp recording of mTPM frames
   1. Switch on every power supply of the mTPM device.
   2. Start the mTPM recording software and the mTPM synchronization software.
   3. Set the save paths of mTPM frames and synchronization timestamps.
      NOTE: Do not use any of the non-English letters and special characters to name paths.
   4. Set the number of recording frames of the mTPM recording software. The recommended number of frames is 6000, which is enough to finish the synchronization debug step.
   5. Select every channel of the mTPM synchronization software.
   6. Start the recording of mTPM through the mTPM recording software. Before starting the recording, switch off any of the lights inside the room to protect the mTPM from light overburning. The mTPM synchronization software will be automatically run when the user starts the recording.
   7. Check if the mTPM frame timestamps are captured by the mTPM synchronization software.
      NOTE: The mTPM frame timestamps shown in the mTPM synchronization software are sharp TTL pulses. Each frame of imaging triggers a sharp TTL pulse, which is 4.84 Hz by default. If there is no pulse shown on the software panel, there are 2 normal problems. The first one is that the driver of the controller of the mTPM device does not support timestamp capturing. Improving its driver version to support timestamp capturing can solve the first problem. The second one is the bad connection of the SMA cable. Ensure that the SMA connectors are securely tightened. Before checking the timestamps, return to 1.1.2, then try step by step.
   8. Wait for the recording to stop. Check if the .tif file of mTPM frames, one .tdms file, and one .tdms_index file of the timestamps are saved.
3. Timestamp recording of four cameras of the 3D behavior device
   1. Set the save path of the videos and behavior timestamps in the customized camera synchronization script.
      NOTE: The customized camera synchronization code is at https://github.com/YNCris/natural_behavior_device/blob/main/camera_code/mul_camera_save_video_event.py.
   2. Start the mTPM synchronization software. Set the save paths of mTPM synchronization timestamps.
   3. Start the recording of the mTPM synchronization software. Run the customized camera synchronization script.
   4. Check if the behavior timestamps are captured by the mTPM synchronization software.
      NOTE: The behavior timestamps shown in the mTPM synchronization software are sharp TTL pulses. One behavior timestamp is sent to the mTPM synchronization software for every 30 frames of the behavior video frame recording. The behavior timestamps will also be saved in the workstation of the 3D behavior device.
   5. Check if four .avi files of behavior videos, one .txt file of behavior timestamps in the 3D behavior device, and one .tdms file and one .tdms_index file of the behavior timestamps in the mTPM workstation are saved.
4. Camera calibration of the 3D behavior device
   1. Adjust the shooting angle of four cameras. The four cameras should comprehensively cover the entire base of the open field while also extending their field of view at least 20 cm above the farthest boundary of the open field to ensure that instances of the mouse rearing up are fully captured.
   2. Put the calibration module at the center of the shooting areas. Run the camera calibration software. Before running the camera calibration software, switch off all of the lights.
      NOTE: Four cameras will capture the frames of the moving checkboard displayed by the calibration screen. The camera calibration is based on Zhang's calibration method²⁹.
   3. After running the camera calibration software, one .mat file will be saved, which contains the camera projection matrix for animal 3D pose reconstruction. Because the behavior data indexing at the system synchronization step relies on the calibration .mat file, ensure the camera calibration step is finished before the synchronization step of the whole system.
5. Synchronization of the whole system
   NOTE: Before this step, make sure timestamps from mTPM frames and the 3D behavior device can be received by the mTPM synchronization software separately.
   1. Switch on every power supply of the mTPM device and the 3D behavior device.
   2. Start the mTPM recording software, the mTPM synchronization software, and the customized camera synchronization script.
   3. Set their path and parameters referring to step 1.3.1. Start the recording of mTPM through the mTPM recording software.
   4. Run the customized camera synchronization script. Ensure the start of recording of mTPM frames is ahead of running the behavior recording script so that the mTPM synchronization software can receive the timestamps from the 3D behavior device. Set the end time of recording of mTPM frames to longer than the time of behavior recording so that the mTPM synchronization software can capture the behavior timestamps without loss.
      NOTE: The recommendation of the recording time of mTPM frames is longer than 5 min of the behavior recording. For example, if the user wants to record 15 min behavior, the number of mTPM frames should be set to 4.84 x (15+5) x 60 = 5808 frames.
   5. Wait for the stop of the recording. After the recording, there is one mTPM frame .tif file, two mTPM synchronization .tdms files, two .tdms_index files, four behavior video .avi files, and one behavior timestamp .txt file.
   6. Run the customized synchronization code to align the mTPM frame and the behavior videos. The customized synchronization code is at https://github.com/YNCris/natural_behavior_device/blob/main/preprocess_code/step1_align_tpm_data.m.
      1. Before running this script, manually organize the files as follows:
         ----\Root path # The path to save all of the data
         --------\behavior_all # The path to save behavior data
         ------------\A-B-C-D-E-caliParas.mat # Camera calibration file
         ------------\A-B-C-D-E-camera-1.avi # behavior video from camera 1
         ------------\A-B-C-D-E-camera-2.avi # behavior video from camera 2
         ------------\A-B-C-D-E-camera-3.avi # behavior video from camera 3
         ------------\A-B-C-D-E-camera-4.avi # behavior video from camera 4
         ------------\A-B-C-D-E-event.txt # behavior timestamp from 3D behavior device
         ----\tpm_suite2p # The path to save mTPM data
         --------\sep # The path to save mTPM frames and timestamps
         ------------\A-B-C-D-E-event # The path to do timestamp alignment
         ----------------\beh.tdms # The behavior channel .tdms file
         ----------------\beh.tdms_index #The behavior channel .tdms_index file
         ----------------\tpm.tdms # The mTPM channel .tdms file
         ----------------\tpm.tdms_index # The mTPM channel .tdms_index file
         ------------\A-B-C-D-E-tpm # The path to save mTPM frames
         ----------------\F.tif # The mTPM frames of each recording
         --------\process # The path to extract neural traces
         ------------\C # The path of each mouse
         ----------------\F.tif # The mTPM frames of each mouse
         A to G are definitions of name fields, where
         A means experimental groups, for example, free,
         B means the sequence of videos, for example, seg1,
         C means the identity of animals, for example, 1tpmss,
         D means the interaction partners, for example, 1wt,
         E means the experimental date, for example, 20220226, and
         F means the mTPM recording frame chunk (5000 frames per chunk), for example, social 1.
         ​NOTE: The frame rate of the 3D behavioral tracking system is 30 Hz, while that of the mTPM is 4.84 Hz. Since the maximum achievable synchronization precision is constrained by the lowest frame rate (4.84 Hz), the temporal resolution of synchronization is approximately 206 ms. Behavioral timestamps serve as the reference, and each mTPM frame is aligned to the nearest behavioral time point. Intervals between successive mTPM frames with respect to the behavioral timeline are interpolated using a step-wise approach.

2. Neuroethological data recording

NOTE: The neuroethological data recording process consists of four key steps (Figure 1B).

1. Mounting mTPM
   NOTE: Because the details of mTPM mounting are contained in the tutorial of the mTPM, only the key steps are introduced here.
   1. Prepare the cranial window.
      NOTE: The preparation of the cranial window mainly includes the injection of the virus, the implantation of the cover glass, and the fixation of the metal plate. Because the imaging sites are different in various research, the details of the preparation of the cranial window are different. In this case, the preparation of the cranial window is performed by an outsourcing service. The imaging brain area in the example data is the primary somatosensory cortex (S1).
   2. Fix the mouse restrainer to the mTPM micromanipulator. Fix the head of the mouse to the restrainer through the metal plate.
   3. Find the fluorescence through mTPM. Switch off all of the lights before switching on the mTPM imaging. Fix the mTPM to the holder before the following steps.
      1. Add one drop of Carbomer eye gel to the top of the cranial window. Move the mouse through the motion platform when the cranial window is aligned beneath the mTPM objective.
      2. Move the micromanipulator vertically to find the imaging plane. Move the micromanipulator in-plane to center the imaging plane.
   4. Fix the upper base to the mTPM. Glue the lower base to the upper base and the cranial window.
      1. To ensure structural stability, fill the gap between the two bases and the metal plate bracket attached to the mouse's head and bond it with a high-performance acrylic structural adhesive. Cure the adhesive for 30 min before assessing the stability of the bond by gently probing the base with tweezers. If necessary, additional adhesive is applied until a secure fixation is achieved.
         NOTE: It is essential to prevent any direct adhesion between the base and the mTPM itself. The upper and lower bases are small aluminum frames. The upper base is custom-fabricated to snugly fit the lower contour of the mTPM housing. Lower bases with multiple height options are used to bridge the gap between the upper base and the cranial window. All lower bases have the same planar dimensions as the upper base to ensure mechanical compatibility, differing solely in height to match varying skull-to-upper-base distances.
   5. Add one drop of Carbomer eye gel inside the base chamber. Check the neuronal fluorescence through mTPM. If neuronal fluorescence is not clearly visible, remove the adhesive using a cranial drill, allowing for separation of the base, after which the above procedure is repeated until fluorescence clarity is achieved.
   6. Secure the aluminum foil with tape between the fiber of mTPM and the cranial window.
      NOTE: This step is to maintain proper light shielding throughout the recording process. The use of aluminum foil and adhesive tape was minimized to reduce overall weight.
   7. Switch on the room light and test the clarity of mTPM frames.
2. Putting the mouse into an open field
   NOTE: This step involves placing the mouse in the open field while ensuring proper weight balance for the fiber and mTPM.
   1. Inflate at least 10 helium balloons and separately tie them with cotton twine. Detach the metal plate from the mouse restrainer.
   2. Hold the mouse by its tail using one hand. Support the mTPM fiber using the other hand.
   3. Gently position the mouse into the open field. Suspend the helium balloons using cotton twine to the fiber. Adjust the number of balloons until the mouse can move freely and explore the open field without constraint.
      NOTE: The optimal number of balloons is determined when the mouse remains grounded, with its forelimbs not lifted by upward buoyancy, while still sufficiently counterbalancing the weight of the mTPM to allow the animal to maintain a natural head posture and spontaneously stand upright. The cotton twine should be tied at a height above the open field to ensure sufficient fiber length for unrestricted movement of the mouse. The cotton twine was tied at the height of the optical fiber, which was approximately aligned with the top edge of the open-field chamber. In the given example, once this step is complete, an untreated mouse is introduced into the open field to enable free social interaction.
   4. Close the door of the mTPM enclosure to minimize external disturbances.
3. Switching on mTPM recording.
   1. Start the mTPM recording software and the mTPM synchronization software. Set their path and parameters referring to the platform establishment step.
   2. Start the recording of mTPM through the mTPM recording software. Verify the presence of time markers corresponding to each two-photon frame within the synchronization software.
   3. Assess whether the contrast of the two-photon images remains unaffected and confirm that the mouse's locomotion does not compromise the stability of the recorded frames. If any issues are detected, repeat the synchronization procedure and the mTPM mounting process until the two-photon imaging produces clear frames with accurately aligned time markers.
4. Switching on behavior recording
   1. Start the customized camera synchronization script.
      NOTE: The customized camera synchronization code is at https://github.com/YNCris/natural_behavior_device/blob/main/camera_code/mul_camera_save_video_event.py.
   2. Set the path and parameters referring to the platform establishment step.
   3. Start the recording of behavior through the customized camera synchronization script. Verify the presence of time markers corresponding to each 30 behavior frames within the mTPM synchronization software.
   4. Check if the four video streams from the cameras are properly synchronized, and check the video capture parameters of the 3D behavioral tracking system.
      ​NOTE: In the above setup, the cameras use a frame resolution of 640 × 480 pixels, a frame rate of 30 frames per second, RGB frames, and automatic exposure. The automatic exposure allows the camera to dynamically adjust brightness based on ambient lighting conditions. Since exposure time directly affects the achievable frame rate, particularly under low-light conditions where longer exposures are required, the frame rate may decrease. To ensure stable frame acquisition at 30 Hz, it is important to control the background illumination to minimize exposure time. The camera gain and binning settings are kept at their default values throughout the recording.
   5. The behavioral recording will automatically stop once the predefined duration is reached. After completing the behavioral recording, manually switch off the mTPM recording and synchronization. By following these steps, a single trial of simultaneous neural and behavioral data acquisition is completed.

3. Neuroethological data preprocessing

NOTE: If all preceding steps are completed successfully, three categories of data files should be obtained: two-photon imaging frames (.tif), four behavioral video recordings (.avi) along with a camera calibration file (.mat), and two synchronization timestamp files (.tdms) for subsequent data preprocessing (Figure 1C). These data should be manually renamed and put into the folders referring to step 1.5.7.

1. Preprocessing mTPM data
   1. Extract neural signal trajectories from mTPM frames through suite2p³⁰. Keep the parameters of suite2p at their default values to ensure reproducibility. Only adjust the frame rate to match the mTPM acquisition settings.
      NOTE: Since subsequent steps include signal processing and quality control, there is no need to extensively fine-tune the suite2p parameters at this stage. Maintaining consistency across datasets is of greater importance.
   2. Run the customized code to align the timestamps between the mTPM frames and the behavior videos.
      NOTE: This script aligns behavioral event timestamps with corresponding mTPM acquisition times. It generates and saves index mappings for each recording, enabling synchronized analysis of behavior and neural data. This customized code is at https://github.com/YNCris/natural_behavior_device/blob/main/preprocess_code/step1_align_tpm_data.m.
   3. Run the customized code to convert the data format from the suite2p output.
      NOTE: This script extracts and reorganizes mTPM data for individual animals by identifying and mapping relevant frame indices. It selects the corresponding neural activity traces and restructures them into a standardized data format, enabling streamlined downstream analysis of segmented recordings. This customized code is at https://github.com/YNCris/natural_behavior_device/blob/main/preprocess_code/step2_separate_tpm_data.m.
   4. Run the customized code to resample neural signals to align with behavior frames.
      NOTE: This script performs temporal resampling of mTPM data to align neural activity traces with behavioral timestamps. It loads previously segmented neural data and synchronization indices, extracts resampled traces accordingly, and saves the output in a standardized format for further signal processing. The method of temporal resampling is stepwise interpolation. This customized code is at https://github.com/YNCris/natural_behavior_device/blob/main/preprocess_code/step3_resample_tpm_data.m.
   5. Run the customized code to refine neural trajectories.
      1. As stepwise interpolation can introduce ringing artifacts and high-frequency noise, design an equiripple low-pass filter with a 2 Hz passband frequency and a 2.2 Hz stopband frequency to mitigate these artifacts. The order of the filter is 61.
      2. After that, use Weijian Zong's denoising method to refine resampled mTPM calcium traces¹³. It estimates local baselines using percentile and local variance criteria and calculates the ΔF/F signals. ΔF/F is a dimensionless measure that represents the relative change in fluorescence intensity over baseline, commonly used to quantify neural activity in calcium imaging. Since both the numerator (ΔF) and the denominator (F) are in arbitrary fluorescence units, the resulting ΔF/F value has no physical unit and is expressed as a ratio or percentage. Cells with extremely flat or saturated signals are excluded based on signal range thresholds. The resulting high-quality neural activity traces are saved for downstream analysis ( Figure 2A, Figure 3A, and Figure 4A).
         NOTE: The customized code is available at https://github.com/YNCris/natural_behavior_device/blob/main/preprocess_code/step4_filter_tpm_data.m.
2. Preprocessing behavior data
   1. Extract behavioral poses from video recordings using an anti-drifted pose tracker (ADPT)⁷.
      NOTE: ADPT addresses the issue of frequent point drift observed in convolutional neural network-based methods such as DeepLabCut⁶ and SLEAP²². ADPT is particularly well-suited for scenarios involving neural recording devices, such as mTPM imaging in freely moving animals, as it effectively reduces point drift caused by the fiber movement. The repository of ADPT is at https://github.com/tangguoling/ADPT. 
      1. Given that the experimental setup involves two mice, with the mTPM serving as an identifier for one of them, train two independent single-mouse ADPT models to estimate the poses of each mouse separately.
      2. For each model, manually annotate 16 key points^1,10,15,31-including the nose, left ear, right ear, neck, left and right front limbs, left and right hind limbs, left and right front paws, left and right hind paws, back, tail base, mid-tail, and tail tip-for approximately 600 frames, with 150 frames manually labeled per camera view for a 15 min video.
      3. Train the ADPT models using default parameters. The details of the training and prediction parameters are at https://github.com/tangguoling/ADPT/blob/main/code/config.yaml and https://github.com/tangguoling/ADPT/blob/main/code/config_predict.yaml. 
   2. Reconstructing 3D animal poses. Once trained, apply the two models independently to predict the poses of each mouse from each of the videos captured by different cameras. Merge the resulting data into a single table file for further processing.
   3. Perform a 3D reconstruction of the behavioral trajectories using triangulation in combination with the camera calibration file³². Following reconstruction, obtain the movement trajectories of the subject mouse (the one carrying the mTPM, Figure 2B, Figure 3B, Figure 4B) and the object mouse (the interacting conspecific, Figure 2C, Figure 3C, Figure 4C) for subsequent analysis.
   4. In social interaction studies, inter-individual distances serve as critical indicators of social dynamics³³. Compute the relative body distances between the two mice using pairwise Euclidean distances between corresponding body points, providing quantitative measures for further social behavior analysis ( Figure 2D, Figure 3D, Figure 4D).
   5. Decompose and classify behavior motifs.
      NOTE: Even though the preceding steps yield fine-grained pose trajectories, they do not yet tell us what the animal is doing. Behavior refers to how an animal moves over a specific period, and assigning behavior involves giving each such period a meaningful, human-interpretable label. The normal way is to use human annotations.
      1. Because behavioral recordings are often too long to label manually in their entirety, use machine learning methods.
         NOTE: Supervised learning adheres to predefined labeling rules: human annotators assign labels to selected behavior segments, which are then used to train machine learning or AI models to predict labels across the full dataset. While this approach improves the efficiency of behavior classification, it remains constrained by the limited set of human-interpretable labels, an especially significant limitation when dealing with complex, naturalistic behaviors. To capture the rich variability of naturalistic behaviors, unsupervised behavior classification methods have been developed. These approaches identify intrinsic similarities among behavioral motifs across time and cluster them within low-dimensional feature spaces, enabling more efficient and unbiased human interpretation. In essence, supervised classification relies on predefined human labels to guide behavior recognition, whereas unsupervised classification uncovers latent behavioral structures without prior labeling, offering greater flexibility in capturing complex naturalistic dynamics.
      2. Within the framework for naturalistic behavior, identify the behavioral motifs of the two mice using Behavior Atlas (BeA)¹⁰ and Social Behavior Atlas (SBeA)¹ ( Figure 2E, Figure 3E, Figure 4E), both of which are unsupervised clustering methods designed for behavioral segmentation. These methods decompose and cluster animal behavior into subsecond motifs based on their intrinsic dynamic and hierarchical structures³⁴, subsequently associating them with defined behavioral categories.
   6. Extract behavioral motifs of individual mice using BeA.
      1. To reduce noise, apply a median filter with a 500 ms time window. Perform body alignment using the back and tail base key points to facilitate the decomposition of non-locomotor movements, while the right front limb, left front limb, right hind limb, and left hind limb key points are utilized for body size normalization.
      2. For behavior decomposition, use 39 aligned body coordinates, including: nose (x, y, z), left ear (x, y, z), right ear (x, y, z), neck (x, y, z), left front limb (x, y, z), right front limb (x, y, z), left hind limb (x, y, z), right hind limb (x, y, z), left front paw (x, y, z), right front paw (x, y, z), left hind paw (x, y, z), right hind paw (x, y, z), back (z), and tail base (x, z).
      3. Set the temporal reduction index to 5, with a clustering resolution of 3. Use spectral clustering for initialization, employing a Gaussian kernel with a bandwidth sigma of 30. Set the minimum and maximum segmentation lengths to 500 ms and 2000 ms, respectively.
      4. For behavior motif embedding, apply UMAP with 30 nearest neighbors and a minimum distance of 0.05. Cluster the motifs through hierarchical clustering using Ward's linkage and Euclidean pairwise distance to improve the stability of classification. The repository of BeA is at
      5. Identify social behavioral motifs using SBeA. Set the initialization parameters for SBeA to be consistent with those used in BeA, with the following modifications: the median filter window size is set to 1000 ms, the minimum segmentation length is increased to 2000 ms, and the maximum segmentation length is extended to 5000 ms for social behavior decomposition.
      6. For motif embedding, apply a Gaussian kernel with a bandwidth sigma of 2. The final number of social behavior clusters ranges from 16 (lower boundary) to 280 (upper boundary). In the presented examples, only the 16 largest clusters are utilized for demonstration purposes. The repository of SBeA is at https://github.com/YNCris/SBeA_release/blob/main/README_SBeA_mapper.md. 

4. Neuroethological data mapping

1. Prepare mapping datasets. Following the aforementioned preparations, seven datasets are available for neuroethological mapping, including neuronal activities, subject poses, object poses, body distances, subject behavior motifs, object behavior motifs, and social behavior motifs. Manually organize these files as follows:
   Data path # The path to save all of the data
   ----\A-B-C-D-E-neu.mat # Neuronal activities
   ----\A-B-C-D-E-pose-tpm.mat # Subject poses
   ----\A-B-C-D-E-pose-free.mat # Object poses
   ----\A-B-C-D-E-rel_dist.mat # Body distances
   ----\A-B-C-D-E-mov-tpm.mat # Subject behavior motifs
   ----\A-B-C-D-E-mov-free.mat # Object behavior motifs
   ----\A-B-C-D-E-sbea.mat # Social behavior motifs
   The definitions of fields A, B, C, D, and E are the same as 1.5.7. The demo data for reproduction is at https://doi.org/10.6084/m9.figshare.29606795.v1. 
2. Create the embeddings of the mapping using CEBRA.
   NOTE: Given that CEBRA integrates both hypothesis-driven and discovery-driven approaches to construct embeddings, it is well-suited for uncovering neural representations associated with various auxiliary variables (Figure 5A).
   1. To ensure a systematic comparison of embeddings across different modalities, standardize the parameters of the CEBRA models. The model architecture employs the 10 offsets, with a batch size of 1024, a learning rate of 0.0001, and a temperature parameter of 1.
   2. Set the output dimension to 3, and conduct training for a maximum of 15,000 iterations. Use cosine distance as the similarity metric, while the conditional variable is defined as the time delta with a time offset of 10. Use the neuronal activity data as input, and auxiliary variables as labels to generate joint embeddings.
3. For the self-embedding of neuronal activities, apply the same CEBRA parameters, but only use neuronal activity data as input.

Results

The study of natural behavior presents greater complexity compared to trial-based experiments. First, in natural conditions, both neural activity and behavior lack a fixed baseline. These activities are recurrent, meaning that they are influenced by previous states, and thus, aligning the onset of specific behaviors for comparison of neural activity fails to disentangle the effects of prior neuroethological states. Second, neural encoding in natural behavior primarily occurs at the population level^4,5. The variability observed in single neurons is substantial enough to be considered as noise. To validate this, this part conducted a correlation analysis between neural activity and natural behavioral poses (Figure 2F, Figure 3F, Figure 4F). The resulting correlation coefficient matrices revealed no neuron-specific correspondence with the pose traces. Specifically, the correlation coefficients between neural signals and subject poses, object poses, or inter-body distances all fell within the range of -0.3 to +0.3, commonly regarded as weak correlations¹⁵ (Figure 2G, Figure 3G, Figure 4G). These findings indicate that, under naturalistic conditions, pose-related information is not encoded in a neuron-specific manner.

Given these factors, this framework offers an objective approach to capture and map neuroethological data at the neural population level. mTPM imaging ensures that the variability of individual neurons is preserved to the greatest extent possible. Additionally, the use of deep learning-based pose estimation by ADPT and unsupervised behavior decomposition methods, such as BeA and SBeA, generates rich auxiliary variables, enabling CEBRA to effectively interpret the variability within neural populations.

These examples demonstrate that the joint CEBRA embeddings are present across all auxiliary variables, including subject poses, object poses, body distances, subject behavior motifs, object behavior motifs, and social behavior motifs (Figure 5A). In order to verify the consistency of behavioral motifs and neural embeddings across sessions or subjects, Procrustes analysis³⁵ is used across three mouse pairs (Figure 5B). Given that CEBRA embeddings are distributed on a unit sphere, only the rotation parameter in Procrustes analysis was enabled. Since the natural-behavior CEBRA embeddings lack a clear baseline, this part first performed label-guided alignment sampling on the embeddings to align, ensuring consistent anchor points before applying Procrustes analysis. Visually, these CEBRA embeddings exhibit a degree of intrinsic consistency, with body distance and social motifs showing the highest alignment. It fits the quantification of the RMSE before and after Procrustes alignment (Figure 5C). Then, the embedding decoding accuracy is compared for poses (Figure 5D) and motifs (Figure 5E). While their representations differ, each is decodable with high accuracy. Although the decoding RMSE of body distance is significantly higher than the subject and object poses, it is no more than the tracking accuracy of ADPT⁷.

To explore the origins of these hypothesis-driven embeddings, a self-organized embedding of neural activity was generated through CEBRA (Figure 5A, right column). The shape of the neural embedding is more intricate than the other joint embeddings, incorporating patterns from various joint embeddings. Additionally, the similarities between neural embeddings and joint embeddings were compared using Procrustes transformation, and then their cosine similarities were compared (Figure 5F). The cosine similarity is derived per minute between the aligned embeddings across corresponding time points.

The S1-subject pose joint embedding was selected as the baseline for similarity comparisons based on the well-established role of the S1 in encoding self-organized somatosensory inputs³⁶. This embedding serves as a biologically meaningful reference point for evaluating how other variables-such as object-related motifs-are represented within the same neural space. Such comparisons enable us to assess the relative strength of encoding for different behavioral dimensions with respect to a self-related somatosensory baseline.

When comparing the cosine similarity of neural embeddings with the S1-subject poses joint embedding as the baseline, this study finds that the joint embeddings for object motifs are significantly lower. This suggests that, during the 15 min period of freely social interaction in this example, the S1 neural activities of the subject mouse primarily encode both its behavior and the ongoing social interactions. While this analysis serves as a demonstrative case, the same methodological framework can be readily applied to more granular investigations, for instance, by comparing embedding structures across distinct temporal epochs to uncover dynamic shifts in neural encoding.

Figure 1: Procedure for collecting neuroethological data. (A) The integration of devices. (B) The operation of data recording. (C) Neural signal extraction, 2D pose estimation, and 3D body trajectory reconstruction after recording. Please click here to view a larger version of this figure.

Figure 2: Preprocessed data of mouse 1 for further analysis. (A) Neuronal activities. (B) Subject poses. (C) Object poses. (D) Body distance. (E) Behavior motifs. From top to bottom are subject, object, and social behavior motifs. (F) The correlation coefficient matrices between neural activity and poses. Left: the correlation coefficients between neural activity and subject poses. Center: the correlation coefficients between neural activity and object poses. Right: the correlation coefficients between neural activity and body distance. The correlation coefficients are between each neuron trace and each pose dimension. (G) The distributions of correlation coefficients of F. The neuron indices are sorted according to the correlation coefficients between neural activity and subject poses. Abbreviations: N & S = neural activity and subject poses, N & O = neural activity and object poses, N & B = neural activity and body distances, CC = correlation coefficients. Please click here to view a larger version of this figure.

Figure 3: Preprocessed data of mouse 2 for further analysis. (A) Neuronal activities. (B) Subject poses. (C) Object poses. (D) Body distance. (E) Behavior motifs. From top to bottom are subject, object, and social behavior motifs. (F) The correlation coefficient matrices between neural activity and poses. Left: the correlation coefficients between neural activity and subject poses. Center: the correlation coefficients between neural activity and object poses. Right: the correlation coefficients between neural activity and body distance. The correlation coefficients are between each neuron trace and each pose dimension. (G) The distributions of correlation coefficients of F. The neuron indices are sorted according to the correlation coefficients between neural activity and subject poses. Abbreviations: N & S = neural activity and subject poses, N & O = neural activity and object poses, N & B = neural activity and body distances, CC = correlation coefficients. Please click here to view a larger version of this figure.

Figure 4: Preprocessed data of mouse 3 for further analysis. (A) Neuronal activities. (B) Subject poses. (C) Object poses. (D) Body distance. (E) Behavior motifs. From top to bottom are subject, object, and social behavior motifs. (F) The correlation coefficient matrices between neural activity and poses. Left: the correlation coefficients between neural activity and subject poses. Center: the correlation coefficients between neural activity and object poses. Right: the correlation coefficients between neural activity and body distance. The correlation coefficients are between each neuron trace and each pose dimension. (G) The distributions of correlation coefficients of F. The neuron indices are sorted according to the correlation coefficients between neural activity and subject poses. Abbreviations: N & S = neural activity and subject poses, N & O = neural activity and object poses, N & B = neural activity and body distances, CC = correlation coefficients. Please click here to view a larger version of this figure.

Figure 5: Analysis of CEBRA embeddings of neuroethological data. (A) CEBRA embeddings. From left to right are the joint embedding of S1 neural activity and subject poses, the joint embedding of S1 neural activity and object poses, the joint embedding of S1 neural activity and body distances between two animals, the joint embedding of S1 neural activity and subject behavior motifs, the joint embedding of S1 neural activity and object behavior motifs, the joint embedding of S1 neural activity and social behavior motifs, and the neural embedding of S1. (B) The Procrustes analysis aligns the embeddings above. The gray circles represent the mouse pair 1, serving as the reference embedding. The green plus signs represent mouse pair 2, and the orange crosses represent mouse pair 3, both aligned to mouse pair 1. (C) The root mean square error (RMSE) before (left) and after (right) Procrustes alignment (Paired t-test, n=3, mean ± SEM). (D) The RMSE of pose reconstruction from CEBRA embeddings (one-way ANOVA followed by Tukey's multiple comparisons test, n=3, mean ± SEM). (E) The accuracy of motif reconstruction from CEBRA embeddings (one-way ANOVA followed by Tukey's multiple comparisons test, n=3, mean ± SEM). (F) The cosine similarities between joint embeddings and neural embedding of S1 (one-way ANOVA followed by Dunnett's multiple comparisons test, n=45, mean ± SEM). *p < 0.05, **p < 0.01, ***p < 0.001. Please click here to view a larger version of this figure.

No.	Observed problem	Likely cause	Possible solutions
1	No behavior timestamps	(1)   Faulty SMA or BNC cables	(1)   Replace SMA and BNC cables
		(2)   Missing USB-to-TTL driver	(2)   Install the Prolific PL2303 USB driver
		(3)   Incorrect COM port selection	(3)   Verify the COM port number in Device Manager and update it in both the mTPM software and behavior camera script.
2	No visible fluorescence during mTPM mounting	(1)   Lack of viral expression	(1)   Use a different mouse
		(2)   Incorrect field of view	(2)   Readjust the field of view
		(3)   Insufficient laser power	(3)   Gradually increase laser power
		(4)   Dried Carbomer gel	(4)   Reapply fresh Carbomer gel
3	mTPM imaging shows a completely white screen	(1)   Light leakage	(1)   Rewrap the aluminum foil for proper shielding
		(2)   Insufficient laser power	(2)   Gradually increase laser power
		(3)   Detached fiber from the mTPM head	(3)   Reinsert the fiber into the mTPM and tighten the fixing screw
4	Dropped frames in behavioral video	(1)   Low ambient lighting	(1)   Increase background illumination
		(2)   Incorrect USB port	(2)   Use at least USB 3.0 ports
		(3)   Insufficient computer performance	(3)   Use a machine with Intel i7-9700K or higher, dual-channel RAM, and SSD storage.
5	No locomotion in mTPM-mounted mice	(1)   Repetitive use of the same mouse	(1)   Avoid reusing mice within 3 days
		(2)   Excessive use of aluminum foil	(2)   Use minimal foil necessary for light shielding
		(3)   Insufficient number or volume of helium balloons	(3)   Adjust the number and inflation of balloons to support the mTPM fiber while allowing natural posture and movement of mice.
6	Inaccurate 2D pose estimation	(1)   Insufficient number of manually labeled frames	(1)   Annotate at least 200 frames incrementally
		(2)   Undertrained ADPT model	(2)   Increase training epochs in the config.yaml file of ADPT
7	Abnormal 3D pose reconstruction	(1)   Improper camera calibration	(1)   Improve calibration contrast and tilt angle
		(2)   Inaccurate 2D pose input	(2)   Increase the number of checkboard frames captured
			(3)   Resolve the 2D pose issues first (see Problem 6)
8	Misalignment between neural and behavioral data	(1)   Incorrect sequence of software initialization	(1)   Always start mTPM recording before behavior camera
		(2)   Dropped behavior frames	(2)   Address frame drop issues (see Problem 4)
			(3)   Ensure sufficient disk space is available
9	Memory overflow during BeA/SBeA processing	(1)   Excessive recording duration	(1)   Split recordings into shorter segments (5–60 minutes), then run BeA/SBeA
		(2)   Limited system RAM	(2)   Increase the temporal reduction factor (e.g., from 5 to 10) in BeA
			(3)   Upgrade RAM to at least 64 GB
10	CEBRA fails to run on GPU	(1)   Mismatch between CUDA and GPU driver	(1)   Do not directly follow the tutorial to install CUDA 11.3
		(2)   Incompatible PyTorch version	(2)   Check your GPU model and driver version (nvidia-smi)
			(3)   Install the correct CUDA and PyTorch versions accordingly and then install CEBRA via pip

Table 1: Troubleshooting list. The following is a list of 10 previously encountered non-trivial problems and possible solutions.

Discussion

This neuroethological recording and decoding framework is built upon commercially available devices, ensuring that most troubleshooting issues can be addressed by the respective companies. Even so, this study provides a list of frequently encountered issues to facilitate reference and streamline troubleshooting (Table 1). This accessibility makes the framework more user-friendly for newcomers. Additionally, the framework is highly flexible, with synchronization between neural and behavioral recordings relying on standard TTL signals. As a result, it is straightforward to integrate other physiological recording devices into the framework if needed. The subsequent analysis procedures are also sufficiently general to support fully customized neural and behavioral recording systems.

The cost associated with this framework, which is based on commercialized devices, is relatively high (~500,000 USD), thereby imposing an additional financial burden on the lab. While recent open-source tools such as MINI2P¹³ and Anipose³⁷ can help reduce material costs, this experience suggests that the overall expenses will remain similar when accounting for the human resource costs involved in debugging. Another limitation of this framework lies in the interpretability of CEBRA embeddings. As a method based on artificial neural networks, it is inherently challenging to interpret. While this example provides a simple approach to explain the embeddings, further methods will need to be developed on a case-by-case basis for different projects. One potential solution for further interpretation of CEBRA embeddings is the application of dynamic systems³⁸. Additionally, natural behavior can be segmented into distinct phases, such as interactions when the two mice are either distant or nearby. Different scientific questions may require the development of customized data analysis workflows.

While the current mTPM + 3D camera system is deployed in an open-field arena, its application is not limited to this specific behavioral context. The main constraints arise from the physical tethering of the imaging system, which limits the extent of animal mobility, and the field of view of the 3D camera, which constrains the trackable volume. These factors may be addressed in future iterations by incorporating wireless imaging modules³⁹ or trap-camera arrays⁴⁰ to enable more complex and naturalistic behavioral paradigms. Notably, both the mTPM system and the 3D camera setup are capable of continuous 24 h data acquisition^10,41, making the entire pipeline well-suited for long-timescale behavioral and neural recording studies.

This study adopts a fully data-driven approach to investigate the neural encoding of spontaneous behavior, and therefore intentionally refrains from assigning predefined semantic labels to clustered behavioral motifs. This decision is rooted in the aim to preserve the generalizability of the neural-behavior mapping framework, allowing it to operate independently of experimenter-imposed behavioral categories. Readers interested in the biological interpretability and supervised classification of behavioral motifs may refer to previous works^1,10, as well as a recent study⁴², which systematically compared unsupervised motif clustering with manually labeled behaviors using the same underlying Behavior Atlas framework. These studies also provide extensive visualizations, including 3D pose sequences, trajectories, and motif-level embeddings, available through public repositories. Together, these resources offer complementary insights into the semantic structure of behavior while supporting the flexible, generalizable neural decoding approach adopted here.

This data processing pipeline was designed with modularity and flexibility in mind, allowing for adaptation to diverse experimental settings and user preferences. Each major component of the pipeline, ranging from 3D pose estimation, unsupervised behavioral motif clustering, neural signal preprocessing, to joint neuroethological embedding, is implemented as an independent module with clearly defined input and output interfaces. This architecture enables users to substitute alternative tools or algorithms at each stage (e.g., different pose estimation frameworks^6,22, behavior clustering algorithms^43,44, or neural decoders^45,46) without disrupting the overall workflow. While these components are designed to be interoperable, this study has not exhaustively tested all possible combinations of alternative methods, and users may need to perform additional tuning to ensure compatibility in their specific applications. Such modularity facilitates both reproducibility and extensibility, and allows the framework to be tailored to species, recording modalities, or behavioral paradigms beyond those demonstrated here. To support broader community use, this study provides a schematic overview and a summary table (Figure 1, Table of Materials).

The parameter settings used for SBeA and CEBRA in this pipeline are based on a combination of default values and empirical tuning specific to this experimental context, freely moving mice under natural social interaction. These parameters have been validated to reproduce all results presented in this study without requiring further adjustment. While users may wish to fine-tune certain parameters to accommodate different recording setups or behavioral tasks, such modifications are not necessary for replicating this pipeline. For users working in other contexts, consulting the original documentation and literature for SBeA and CEBRA is recommended, where parameter ranges and task-specific guidance are provided. This implementation serves as a robust reference configuration that can be directly applied or adapted as needed.

The primary advancement of this framework lies in its application to freely moving animals. Previous studies conducted with head-fixed animals can be adapted to freely moving conditions within this framework. For instance, tasks such as the Go/No-Go Task⁴⁷ and Two-Alternative Forced Choice⁴⁸ can be modified and integrated into this framework based on natural behavior paradigms. This approach eliminates artifacts caused by head restriction, enabling the study of the relationship between task and natural behavioral states. This framework provides animals with greater autonomy in decision-making. It also supports the studies of the spinal cord in naturalistic contexts, combining the mTPM spinal cord recording method⁴⁹. Additionally, it facilitates the study of free group behavior, a phenomenon that is not feasible in head-fixed setups. The data analysis workflow enables the interpretation of neural population activity across multiple variables, using embeddings to unravel the complexity of the brain function behind neural populations.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
3D behavior recording system	BayONE Scientific	BA-3D-Mouse	Integrated synchronization module
Balloon	AliExpress	URL: https://tinyurl.com/3uex669s	Any balloon that is light enough to fly when helium-filled. The balloons are spherical foil balloons, approximately 45 cm in diameter, and feature self-sealing valves. The URL provides an example of the balloons.
Carbomer eye gel	Vidisic	Carbomer 980-based lubricating eye gel	10g
Cotton twine	AliExpress	URL: https://tinyurl.com/ywu7u754	Thick and light, 1-2 mm diameter. The URL provides an example of the cotton twine.
Cranial drill	RWD	78001	0.8, 1.4, and 2.1 mm drill
Custom camera module configurable	Intel	RealSense D435	/
High-performance acrylic structural adhesive	HUITIAN	1320	490ml
Mouse for imaging	TRANSCEND VIVOSCOPE	URL: https://en.tv-scope.com/	The male mouse with C57BL/6J background (10 weeks old) was housed in 1 mouse per cage under a 12 h light-dark cycle at 22–25 °C with 40%–70% humidity and was allowed to access water and food ad libitum. AAV9-CaMKII-GCaMP6s viruses were injected into its primary somatosensory cortex (AP, −0.60 mm; ML, −2.40 mm; DV, 2.00 mm). In our study, the mice were prepared by TRANSCEND VIVOSCOPE as part of their professional animal preparation service. This service includes virus injection, cranial window implantation, and baseplate installation specifically tailored for their miniature two-photon microscopy system.
Mouse for interaction	BayONE LAC	URL: https://lac.bayonesci.com/	The male mice with C57BL/6J background (10 weeks old) were housed in 5 mice per cage under a 12 h light-dark cycle at 22–25 °C with 40–70% humidity and were allowed to access water and food ad libitum. All husbandry and experimental procedures were approved by the Animal Care and Use Committee at the Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences.
mTPM neural recording system	TRANSCEND VIVOSCOPE	SUPERNOVA-600	The SUPERNOVA-600 is a fully integrated miniature two-photon imaging system for freely moving rodents, including all essential optical and recording components but excluding external stimulation devices. It should contain the integrated synchronization module.