Name: Decoding Natural Behavior from Neuroethological Embedding (Video) | JoVE Journal
Uploaded: 2025-10-03T13:40:00.000+00:00
Duration: 8 min
Description: Learn precise neural decoding in freely moving mice using 3D behavior devices and mTPM imaging. This method reveals how neurodynamics encode natural behavior and advance brain intelligence research.

ai_agent_IP

ai_agent

maintenance

instantTrialDurationMinutes

isBiopharma

enableNewMyJoveSectionDesign

educationFeatures

seoMetaApi

ai_copilot_institution_blacklist

ai_chat_institution_blacklist

freeAccessWindow

ssoRoleCapture

scheduledMaintenanceBanner

joveLabsFeatures

disable_autoplay_for_bots

extended_trial_duration

5 minutes

This work was supported by the Strategic Priority Research Program of the Chinese Academy of Sciences (grant No. XDB1010101 to P.W.), STI2030-Major Projects (grant no. 2021ZD0203900 to P.W.), National Natural Science Foundation of China (grant no. 32222036 to P.W.), National Natural Science Foundation of China (grant no. T2394530 to P.W.), and Shenzhen Science and Technology Program (grant no. KJZD20230923115114028 to P.W.). The authors would also like to thank the Nanjing Brain Observatory (NBO) and the PKU-Nanjing Joint Institute of Translational Medicine (Nanjing 211800, China) for their support and assistance with the use of the two-photon microscope.

<ol><li>Han, Y., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Multi-animal+3d+social+pose+estimation%2C+identification+and+behaviour+embedding+with+a+few-shot+learning+framework%5BTitle%5D)+AND+%22Nat+Machine+Intellig%22%5BJournal%5D)">Multi-animal 3d social pose estimation, identification and behaviour embedding with a few-shot learning framework.</a> Nat Machine Intellig. 6 (1), 48-61 (2024).</li>
<li>Zong, W., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Miniature+two-photon+microscopy+for+enlarged+field-of-view%2C+multi-plane+and+long-term+brain+imaging%5BTitle%5D)+AND+%22Nat+Methods%22%5BJournal%5D)">Miniature two-photon microscopy for enlarged field-of-view, multi-plane and long-term brain imaging.</a> Nat Methods. 18 (1), 46-49 (2021).</li>
<li>Schneider, S., Lee, J. H., Mathis, M. W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Learnable+latent+embeddings+for+joint+behavioural+and+neural+analysis%5BTitle%5D)+AND+%22Nature%22%5BJournal%5D)">Learnable latent embeddings for joint behavioural and neural analysis.</a> Nature. 617 (7960), 360-368 (2023).</li>
<li>Lanzarini, F., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Neuroethology+of+natural+actions+in+freely+moving+monkeys%5BTitle%5D)+AND+%22Science%22%5BJournal%5D)">Neuroethology of natural actions in freely moving monkeys.</a> Science. 387 (6730), 214-220 (2025).</li>
<li>Testard, C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Neural+signatures+of+natural+behaviour+in+socializing+macaques%5BTitle%5D)+AND+%22Nature%22%5BJournal%5D)">Neural signatures of natural behaviour in socializing macaques.</a> Nature. 628 (8007), 381-390 (2024).</li>
<li>Mathis, A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Deeplabcut%3A+Markerless+pose+estimation+of+user-defined+body+parts+with+deep+learning%5BTitle%5D)+AND+%22Nat+Neurosci%22%5BJournal%5D)">Deeplabcut: Markerless pose estimation of user-defined body parts with deep learning.</a> Nat Neurosci. 21 (9), 1281-1289 (2018).</li>
<li>Tang, G., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Anti-drift+pose+tracker+%28adpt%29%2C+a+transformer-based+network+for+robust+animal+pose+estimation+cross-species%5BTitle%5D)+AND+%22Elife%22%5BJournal%5D)">Anti-drift pose tracker (adpt), a transformer-based network for robust animal pose estimation cross-species.</a> Elife. 13, RP95709(2025).</li>
<li>Panadeiro, V., Rodriguez, A., Henry, J., Wlodkowic, D., Andersson, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+review+of+28+free+animal-tracking+software+applications%3A+Current+features+and+limitations%5BTitle%5D)+AND+%22Lab+Anim+%28NY%29%22%5BJournal%5D)">A review of 28 free animal-tracking software applications: Current features and limitations.</a> Lab Anim (NY). 50 (9), 246-254 (2021).</li>
<li>Jhuang, H., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Automated+home-cage+behavioural+phenotyping+of+mice%5BTitle%5D)+AND+%22Nat+Commun%22%5BJournal%5D)">Automated home-cage behavioural phenotyping of mice.</a> Nat Commun. 1, 68(2010).</li>
<li>Huang, K., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+hierarchical+3d-motion+learning+framework+for+animal+spontaneous+behavior+mapping%5BTitle%5D)+AND+%22Nat+Commun%22%5BJournal%5D)">A hierarchical 3d-motion learning framework for animal spontaneous behavior mapping.</a> Nat Commun. 12 (1), 2784(2021).</li>
<li>Juavinett, A. L., Bekheet, G., Churchland, A. K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Chronically+implanted+neuropixels+probes+enable+high-yield+recordings+in+freely+moving+mice%5BTitle%5D)+AND+%22Elife%22%5BJournal%5D)">Chronically implanted neuropixels probes enable high-yield recordings in freely moving mice.</a> Elife. 8, e47188(2019).</li>
<li>Chen, S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Miniature+fluorescence+microscopy+for+imaging+brain+activity+in+freely-behaving+animals%5BTitle%5D)+AND+%22Neurosci+Bull%22%5BJournal%5D)">Miniature fluorescence microscopy for imaging brain activity in freely-behaving animals.</a> Neurosci Bull. 36 (10), 1182-1190 (2020).</li>
<li>Zong, W., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Large-scale+two-photon+calcium+imaging+in+freely+moving+mice%5BTitle%5D)+AND+%22Cell%22%5BJournal%5D)">Large-scale two-photon calcium imaging in freely moving mice.</a> Cell. 185 (7), 1240-1256.e30 (2022).</li>
<li>Zong, W., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Fast+high-resolution+miniature+two-photon+microscopy+for+brain+imaging+in+freely+behaving+mice%5BTitle%5D)+AND+%22Nat+Methods%22%5BJournal%5D)">Fast high-resolution miniature two-photon microscopy for brain imaging in freely behaving mice.</a> Nat Methods. 14 (7), 713-719 (2017).</li>
<li>Han, Y., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Mousevenue3d%3A+A+markerless+three-dimension+behavioral+tracking+system+for+matching+two-photon+brain+imaging+in+free-moving+mice%5BTitle%5D)+AND+%22Neurosci+Bull%22%5BJournal%5D)">Mousevenue3d: A markerless three-dimension behavioral tracking system for matching two-photon brain imaging in free-moving mice.</a> Neurosci Bull. 38 (3), 303-317 (2022).</li>
<li>Jazayeri, M., Ostojic, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Interpreting+neural+computations+by+examining+intrinsic+and+embedding+dimensionality+of+neural+activity%5BTitle%5D)+AND+%22Curr+Opin+Neurobiol%22%5BJournal%5D)">Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity.</a> Curr Opin Neurobiol. 70, 113-120 (2021).</li>
<li>Churchland, M. M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Neural+population+dynamics+during+reaching%5BTitle%5D)+AND+%22Nature%22%5BJournal%5D)">Neural population dynamics during reaching.</a> Nature. 487 (7405), 51-56 (2012).</li>
<li>Van Der Maaten, L., Hinton, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Visualizing+data+using+t-sne%5BTitle%5D)+AND+%22J+Machine+Learning+Res%22%5BJournal%5D)">Visualizing data using t-sne.</a> J Machine Learning Res. 9, 2579-2605 (2008).</li>
<li>Mcinnes, L., Healy, J., Melville, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Umap%3A+Uniform+manifold+approximation+and+projection+for+dimension+reduction%5BTitle%5D)+AND+%22arXiv%22%5BJournal%5D)">Umap: Uniform manifold approximation and projection for dimension reduction.</a> arXiv. , (2018).</li>
<li>Klibaite, U., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Mapping+the+landscape+of+social+behavior%5BTitle%5D)+AND+%22Cell%22%5BJournal%5D)">Mapping the landscape of social behavior.</a> Cell. 188 (8), 2249-2266.e23 (2025).</li>
<li>Zhou, D., Wei, X. 34th Conference on Neural Information Processing Systems (NeurIPS), , (2020).</li>
<li>Pereira, T. D., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Sleap%3A+A+deep+learning+system+for+multi-animal+pose+tracking%5BTitle%5D)+AND+%22Nat+Methods%22%5BJournal%5D)">Sleap: A deep learning system for multi-animal pose tracking.</a> Nat Methods. 19 (4), 486-495 (2022).</li>
<li>Von Ziegler, L., Sturman, O., Bohacek, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Big+behavior%3A+Challenges+and+opportunities+in+a+new+era+of+deep+behavior+profiling%5BTitle%5D)+AND+%22Neuropsychopharmacology%22%5BJournal%5D)">Big behavior: Challenges and opportunities in a new era of deep behavior profiling.</a> Neuropsychopharmacology. 46 (1), 33-44 (2021).</li>
<li>Pereira, T. D., Shaevitz, J. W., Murthy, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Quantifying+behavior+to+understand+the+brain%5BTitle%5D)+AND+%22Nat+Neurosci%22%5BJournal%5D)">Quantifying behavior to understand the brain.</a> Nat Neurosci. 23 (12), 1537-1549 (2020).</li>
<li>Segalin, C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((The+mouse+action+recognition+system+%28mars%29+software+pipeline+for+automated+analysis+of+social+behaviors+in+mice%5BTitle%5D)+AND+%22Elife%22%5BJournal%5D)">The mouse action recognition system (mars) software pipeline for automated analysis of social behaviors in mice.</a> Elife. 10, e63720(2021).</li>
<li>Gris, K. V., Coutu, J. P., Gris, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Supervised+and+unsupervised+learning+technology+in+the+study+of+rodent+behavior%5BTitle%5D)+AND+%22Front+Behav+Neurosci%22%5BJournal%5D)">Supervised and unsupervised learning technology in the study of rodent behavior.</a> Front Behav Neurosci. 11, 141(2017).</li>
<li>Liu, N., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Objective+and+comprehensive+re-evaluation+of+anxiety-like+behaviors+in+mice+using+the+behavior+atlas%5BTitle%5D)+AND+%22Biochem+Biophys+Res+Commun%22%5BJournal%5D)">Objective and comprehensive re-evaluation of anxiety-like behaviors in mice using the behavior atlas.</a> Biochem Biophys Res Commun. 559, 1-7 (2021).</li>
<li>Bando, Y., Wenzel, M., Yuste, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Simultaneous+two-photon+imaging+of+action+potentials+and+subthreshold+inputs+in+vivo%5BTitle%5D)+AND+%22Nat+Commun%22%5BJournal%5D)">Simultaneous two-photon imaging of action potentials and subthreshold inputs in vivo.</a> Nat Commun. 12 (1), 7229(2021).</li>
<li>Zhang, Z. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+flexible+new+technique+for+camera+calibration%5BTitle%5D)+AND+%22IEEE+Transactions+on+pattern+analysis+and+machine+intelligence%22%5BJournal%5D)">A flexible new technique for camera calibration.</a> IEEE Transactions on pattern analysis and machine intelligence. 22 (11), 1330-1334 (2002).</li>
<li>Pachitariu, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Suite2p%3A+Beyond+10%2C000+neurons+with+standard+two-photon+microscopy%5BTitle%5D)+AND+%22bioRxiv%22%5BJournal%5D)">Suite2p: Beyond 10,000 neurons with standard two-photon microscopy.</a> bioRxiv. , (2017).</li>
<li>Han, Y., Huang, K., Chen, K., Wang, L., Wei, P. IEEE 11th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), , 306-310 (2021).</li>
<li>Sheshadri, S., Dann, B., Hueser, T., Scherberger, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((3d+reconstruction+toolbox+for+behavior+tracked+with+multiple+cameras%5BTitle%5D)+AND+%22J+Open+Source+Soft%22%5BJournal%5D)">3d reconstruction toolbox for behavior tracked with multiple cameras.</a> J Open Source Soft. 5 (45), 1849(2020).</li>
<li>Zhong, H., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((S-ketamine+exposure+in+early+postnatal+period+induces+social+deficit+mediated+by+excessive+microglial+synaptic+pruning%5BTitle%5D)+AND+%22Mol+Psychiatry%22%5BJournal%5D)">S-ketamine exposure in early postnatal period induces social deficit mediated by excessive microglial synaptic pruning.</a> Mol Psychiatry. 30 (8), 3615-3631 (2025).</li>
<li>Datta, S. R., Anderson, D. J., Branson, K., Perona, P., Leifer, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Computational+neuroethology%3A+A+call+to+action%5BTitle%5D)+AND+%22Neuron%22%5BJournal%5D)">Computational neuroethology: A call to action.</a> Neuron. 104 (1), 11-24 (2019).</li>
<li>Schönemann, P. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+generalized+solution+of+the+orthogonal+procrustes+problem%5BTitle%5D)+AND+%22Psychometrika%22%5BJournal%5D)">A generalized solution of the orthogonal procrustes problem.</a> Psychometrika. 31 (1), 1-10 (1966).</li>
<li>Feldman, D. E., Brecht, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Map+plasticity+in+somatosensory+cortex%5BTitle%5D)+AND+%22Science%22%5BJournal%5D)">Map plasticity in somatosensory cortex.</a> Science. 310 (5749), 810-815 (2005).</li>
<li>Karashchuk, P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Anipose%3A+A+toolkit+for+robust+markerless+3d+pose+estimation%5BTitle%5D)+AND+%22Cell+Rep%22%5BJournal%5D)">Anipose: A toolkit for robust markerless 3d pose estimation.</a> Cell Rep. 36 (13), 109730(2021).</li>
<li>Laiz, R. G., Schmidt, T., Schneider, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Self-supervised+contrastive+learning+performs+non-linear+system+identification%5BTitle%5D)+AND+%22arXiv%22%5BJournal%5D)">Self-supervised contrastive learning performs non-linear system identification.</a> arXiv. , (2024).</li>
<li>Liberti, W. A., Perkins, L. N., Leman, D. P., Gardner, T. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((An+open+source%2C+wireless+capable+miniature+microscope+system%5BTitle%5D)+AND+%22J+Neural+Eng%22%5BJournal%5D)">An open source, wireless capable miniature microscope system.</a> J Neural Eng. 14 (4), 045001(2017).</li>
<li>Tuia, D., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Perspectives+in+machine+learning+for+wildlife+conservation%5BTitle%5D)+AND+%22Nat+Commun%22%5BJournal%5D)">Perspectives in machine learning for wildlife conservation.</a> Nat Commun. 13 (1), 792(2022).</li>
<li>Han, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((24-hour+simultaneous+MPFC%27s+miniature+2-photon+imaging%2C+EEG-EMG%2C+and+video+recording+during+natural+behaviors%5BTitle%5D)+AND+%22Sci+Data%22%5BJournal%5D)">24-hour simultaneous MPFC's miniature 2-photon imaging, EEG-EMG, and video recording during natural behaviors.</a> Sci Data. 12 (1), 1226(2025).</li>
<li>Ye, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Hierarchical+behavioral+analysis+framework+as+a+platform+for+standardized+quantitative+identification+of+behaviors%5BTitle%5D)+AND+%22Cell+Rep%22%5BJournal%5D)">Hierarchical behavioral analysis framework as a platform for standardized quantitative identification of behaviors.</a> Cell Rep. 44 (2), 115239(2025).</li>
<li>Weinreb, C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Keypoint-moseq%3A+Parsing+behavior+by+linking+point+tracking+to+pose+dynamics%5BTitle%5D)+AND+%22Nat+Methods%22%5BJournal%5D)">Keypoint-moseq: Parsing behavior by linking point tracking to pose dynamics.</a> Nat Methods. 21 (7), 1329-1339 (2024).</li>
<li>Jia, Y., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((self-supervised+features+extraction+of+animal+behaviors%5BTitle%5D)+AND+%22Elife%22%5BJournal%5D)">self-supervised features extraction of animal behaviors.</a> Elife. 11, e76218(2022).</li>
<li>Gosztolai, A., Peach, R. L., Arnaudon, A., Barahona, M., Vandergheynst, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Marble%3A+Interpretable+representations+of+neural+population+dynamics+using+geometric+deep+learning%5BTitle%5D)+AND+%22Nat+Methods%22%5BJournal%5D)">Marble: Interpretable representations of neural population dynamics using geometric deep learning.</a> Nat Methods. 22 (3), 612-620 (2025).</li>
<li>Stringer, C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Rastermap%3A+A+discovery+method+for+neural+population+recordings%5BTitle%5D)+AND+%22Nat+Neurosci%22%5BJournal%5D)">Rastermap: A discovery method for neural population recordings.</a> Nat Neurosci. 28 (1), 201-212 (2025).</li>
<li>Vassilev, P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Custom-built+operant+conditioning+setup+for+calcium+imaging+and+cognitive+testing+in+freely+moving+mice%5BTitle%5D)+AND+%22eNeuro%22%5BJournal%5D)">Custom-built operant conditioning setup for calcium imaging and cognitive testing in freely moving mice.</a> eNeuro. 9 (1), (2022).</li>
<li>Verma Rodriguez, A. K., Ramírez-Jarquin, J. O., Rossi-Pool, R., Tecuapetla, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Basal+ganglia+output+%28entopeduncular+nucleus%29+coding+of+contextual+kinematics+and+reward+in+the+freely+moving+mouse%5BTitle%5D)+AND+%22Elife%22%5BJournal%5D)">Basal ganglia output (entopeduncular nucleus) coding of contextual kinematics and reward in the freely moving mouse.</a> Elife. 13, RP98159(2025).</li>
<li>Ju, F., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Long-term+two-photon+imaging+of+spinal+cord+in+freely+behaving+mice%5BTitle%5D)+AND+%22bioRxiv%22%5BJournal%5D)">Long-term two-photon imaging of spinal cord in freely behaving mice.</a> bioRxiv. , (2022).</li>
</ol>

The authors declare that they have no conflicts of interest.

This neuroethological recording and decoding framework is built upon commercially available devices, ensuring that most troubleshooting issues can be addressed by the respective companies. Even so, this study provides a list of frequently encountered issues to facilitate reference and streamline troubleshooting (Table 1). This accessibility makes the framework more user-friendly for newcomers. Additionally, the framework is highly flexible, with synchronization between neural and behavioral recordings relying on standard TTL signals. As a result, it is straightforward to integrate other physiological recording devices into the framework if needed. The subsequent analysis procedures are also sufficiently general to support fully customized neural and behavioral recording systems.
The cost associated with this framework, which is based on commercialized devices, is relatively high (~500,000 USD), thereby imposing an additional financial burden on the lab. While recent open-source tools such as MINI2P13 and Anipose37 can help reduce material costs, this experience suggests that the overall expenses will remain similar when accounting for the human resource costs involved in debugging. Another limitation of this framework lies in the interpretability of CEBRA embeddings. As a method based on artificial neural networks, it is inherently challenging to interpret. While this example provides a simple approach to explain the embeddings, further methods will need to be developed on a case-by-case basis for different projects. One potential solution for further interpretation of CEBRA embeddings is the application of dynamic systems38. Additionally, natural behavior can be segmented into distinct phases, such as interactions when the two mice are either distant or nearby. Different scientific questions may require the development of customized data analysis workflows.
While the current mTPM + 3D camera system is deployed in an open-field arena, its application is not limited to this specific behavioral context. The main constraints arise from the physical tethering of the imaging system, which limits the extent of animal mobility, and the field of view of the 3D camera, which constrains the trackable volume. These factors may be addressed in future iterations by incorporating wireless imaging modules39 or trap-camera arrays40 to enable more complex and naturalistic behavioral paradigms. Notably, both the mTPM system and the 3D camera setup are capable of continuous 24 h data acquisition10,41, making the entire pipeline well-suited for long-timescale behavioral and neural recording studies.
This study adopts a fully data-driven approach to investigate the neural encoding of spontaneous behavior, and therefore intentionally refrains from assigning predefined semantic labels to clustered behavioral motifs. This decision is rooted in the aim to preserve the generalizability of the neural-behavior mapping framework, allowing it to operate independently of experimenter-imposed behavioral categories. Readers interested in the biological interpretability and supervised classification of behavioral motifs may refer to previous works1,10, as well as a recent study42, which systematically compared unsupervised motif clustering with manually labeled behaviors using the same underlying Behavior Atlas framework. These studies also provide extensive visualizations, including 3D pose sequences, trajectories, and motif-level embeddings, available through public repositories. Together, these resources offer complementary insights into the semantic structure of behavior while supporting the flexible, generalizable neural decoding approach adopted here.
This data processing pipeline was designed with modularity and flexibility in mind, allowing for adaptation to diverse experimental settings and user preferences. Each major component of the pipeline, ranging from 3D pose estimation, unsupervised behavioral motif clustering, neural signal preprocessing, to joint neuroethological embedding, is implemented as an independent module with clearly defined input and output interfaces. This architecture enables users to substitute alternative tools or algorithms at each stage (e.g., different pose estimation frameworks6,22, behavior clustering algorithms43,44, or neural decoders45,46) without disrupting the overall workflow. While these components are designed to be interoperable, this study has not exhaustively tested all possible combinations of alternative methods, and users may need to perform additional tuning to ensure compatibility in their specific applications. Such modularity facilitates both reproducibility and extensibility, and allows the framework to be tailored to species, recording modalities, or behavioral paradigms beyond those demonstrated here. To support broader community use, this study provides a schematic overview and a summary table (Figure 1, Table of Materials).
The parameter settings used for SBeA and CEBRA in this pipeline are based on a combination of default values and empirical tuning specific to this experimental context, freely moving mice under natural social interaction. These parameters have been validated to reproduce all results presented in this study without requiring further adjustment. While users may wish to fine-tune certain parameters to accommodate different recording setups or behavioral tasks, such modifications are not necessary for replicating this pipeline. For users working in other contexts, consulting the original documentation and literature for SBeA and CEBRA is recommended, where parameter ranges and task-specific guidance are provided. This implementation serves as a robust reference configuration that can be directly applied or adapted as needed.
The primary advancement of this framework lies in its application to freely moving animals. Previous studies conducted with head-fixed animals can be adapted to freely moving conditions within this framework. For instance, tasks such as the Go/No-Go Task47 and Two-Alternative Forced Choice48 can be modified and integrated into this framework based on natural behavior paradigms. This approach eliminates artifacts caused by head restriction, enabling the study of the relationship between task and natural behavioral states. This framework provides animals with greater autonomy in decision-making. It also supports the studies of the spinal cord in naturalistic contexts, combining the mTPM spinal cord recording method49. Additionally, it facilitates the study of free group behavior, a phenomenon that is not feasible in head-fixed setups. The data analysis workflow enables the interpretation of neural population activity across multiple variables, using embeddings to unravel the complexity of the brain function behind neural populations.

This framework is designed to capture and decode behavioral and neuroimaging data from freely moving animals in naturalistic experimental settings. It comprises three key components: deep learning-based pose estimation and behavior classification methods, SBeA1, miniature fluorescence imaging techniques mTPM2, and a contrastive learning-based neuroethological embedding algorithm, CEBRA3. Recent studies have highlighted the complexity of neuroethological processes in freely moving animals, which surpasses that observed in head-fixed experimental paradigms4,5. However, technical limitations and variability have hindered the widespread application of these approaches to broader investigations of natural behavior. This protocol presents a stable and integrated framework that ensures the accessibility of behavioral and neural data collected in naturalistic contexts for a wide range of research laboratories.
Given that animals move freely in natural environments, this framework incorporates deep learning-based pose estimation to achieve precise tracking of postures6,7. Traditional image processing-based tracking methods are insufficient for capturing fine-scale movements, such as limb and paw dynamics, compared to deep learning-based approaches8. The diverse and complex behaviors exhibited by freely moving animals pose challenges for supervised behavior classification methods9, as predefined behavioral categories often fail to encompass the full range of natural behavioral phenotypes10. Consequently, unsupervised learning-based classification methods are better suited for analyzing behavior in naturalistic settings1. They can comprehensively decompose continuous behavior into discrete subsecond motifs according to their intrinsic structural similarities, and then their consistent definitions are given through data-driven clusters.
Brain imaging in freely moving animals requires capturing the extensive variability of single-neuron activity4,5. Electrophysiological recordings in freely moving animals are limited in their ability to detect neurons with predominantly subthreshold activity11. Additionally, single-photon microscopy suffers from low resolution and contrast, making it difficult to maintain consistent neuron identities across imaging sessions12. mTPM offers superior resolution and contrast compared to single-photon microscopy, making it a more effective tool for investigating the neural coding of natural behaviors2,13,14,15.
Establishing a robust mapping between behavior and neural data necessitates methods capable of revealing their shared informational structure16. Conventional dimensionality reduction techniques, such as Principal Component Analysis (PCA)17, t-Distributed Stochastic Neighbor Embedding (t-SNE)18, and Uniform Manifold Approximation and Projection (UMAP)19, cannot effectively embed behavioral and neural data within a common feature space. In contrast, deep learning-based embedding approaches, such as CEBRA, enable the integration of multiple data modalities in both supervised and self-supervised frameworks, generating high-quality latent representations3. While various alternative methods have emerged in recent years20,21,22, this proposed framework prioritizes practical applications by incorporating well-established methods that are either commercially available or supported by comprehensive tutorials.
Compared to recent studies4,5, this framework offers three key advancements. First, it eliminates human bias in behavior classification. Previous studies relied on manual behavior labeling, which is labor-intensive and prone to inconsistency, particularly as annotators experience fatigue23,24,25. In contrast, this framework employs unsupervised behavior classification, which preserves the natural structure of behavioral patterns by objectively decomposing and clustering behavior motifs before assigning definitions26,27. Second, the use of mTPM enables the capture of more intricate neuronal dynamics at the single-neuron level. This methodological advantage expands the applicability of this framework to decoding complex natural behaviors from diverse neural populations, including those involved in subthreshold coding28. Third, this framework integrates behavioral and neural data into a unified representational space, rather than employing UMAP to embed each modality separately or using support vector machines to impose a rigid mapping between neural activity and behavior while disregarding their intrinsic dynamics4,5. This joint embedding approach ensures a more comprehensive and biologically meaningful representation of the relationship between behavior and brain activity.
This framework is well-suited for research projects that involve the recording and decoding of behavioral and neural data from freely moving animals in naturalistic experimental conditions. While the current implementation is optimized for mouse studies, adapting it to other animal models may require additional development. As the hardware components utilized in this framework are commercially available, on the one hand, the overall cost may be relatively high. On the other hand, this commercial availability significantly reduces the time spent on troubleshooting logistical issues and ensures the acquisition of stable and reliable results in an efficient manner.
This protocol is designed to be reproducible and accessible to neuroscience laboratories equipped for small-animal imaging and behavior tracking. The complete system integrates a commercially available mTPM device with a multi-angle behavioral acquisition setup. Typical neural recordings are acquired at 4.84 Hz with 512 &#215; 512 pixel resolution, and behavioral data are captured at 30 frames per second. Data synchronization is achieved through TTL pulse alignment during preprocessing. Training and decoding can be performed on a standard workstation with a GPU (e.g., NVIDIA RTX 3090 or equivalent), and the full pipeline requires approximately 100 GB of storage per experiment. While the current implementation is optimized for freely moving mice, the modular design of the workflow allows adaptation to other species by adjusting tracking calibration and imaging parameters based on the animal&#39;s size and mobility. These practical details support the protocol&#39;s adaptability and reproducibility across a range of experimental settings.

<table><tbody><tr><td>3D behavior recording system</td><td>BayONE Scientific</td><td>BA-3D-Mouse</td><td>Integrated synchronization module</td></tr><tr><td>Balloon</td><td>AliExpress</td><td>URL: https://tinyurl.com/3uex669s</td><td>Any balloon that is light enough to fly when helium-filled. The balloons are spherical foil balloons, approximately 45 cm in diameter, and feature self-sealing valves. The URL provides an example of the balloons.&amp;nbsp;</td></tr><tr><td>Carbomer eye gel</td><td>Vidisic</td><td>Carbomer 980-based lubricating eye gel</td><td>10g</td></tr><tr><td>Cotton twine</td><td>AliExpress</td><td>URL: https://tinyurl.com/ywu7u754</td><td>Thick and light, 1-2 mm diameter. The URL provides an example of the cotton twine.</td></tr><tr><td>Cranial drill</td><td>RWD</td><td>78001</td><td>0.8, 1.4, and 2.1 mm drill</td></tr><tr><td>Custom camera module configurable</td><td>Intel</td><td>RealSense D435</td><td>/</td></tr><tr><td>High-performance acrylic structural adhesive</td><td>HUITIAN</td><td>1320</td><td>490ml</td></tr><tr><td>Mouse for imaging</td><td>TRANSCEND VIVOSCOPE</td><td>URL: https://en.tv-scope.com/</td><td>The male mouse with C57BL/6J background (10 weeks old) was housed in 1 mouse per cage under a 12&amp;thinsp;h light-dark cycle at 22&amp;ndash;25&amp;thinsp;&amp;deg;C with 40%&amp;ndash;70% humidity and was allowed to access water and food ad libitum. AAV9-CaMKII-GCaMP6s viruses were injected into its primary somatosensory cortex (AP, &amp;minus;0.60 mm; ML, &amp;minus;2.40 mm; DV, 2.00 mm). In our study, the mice were prepared by TRANSCEND VIVOSCOPE as part of their professional animal preparation service. This service includes virus injection, cranial window implantation, and baseplate installation specifically tailored for their miniature two-photon microscopy system.</td></tr><tr><td>Mouse for interaction</td><td>BayONE LAC</td><td>URL: https://lac.bayonesci.com/</td><td>The male mice with C57BL/6J background (10 weeks old) were housed in 5 mice per cage under a 12&amp;thinsp;h light-dark cycle at 22&amp;ndash;25&amp;thinsp;&amp;deg;C with 40&amp;ndash;70% humidity and were allowed to access water and food ad libitum. All husbandry and experimental procedures were approved by the Animal Care and Use Committee at the Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences.</td></tr><tr><td>mTPM neural recording system</td><td>TRANSCEND VIVOSCOPE</td><td>SUPERNOVA-600</td><td>The SUPERNOVA-600 is a fully integrated miniature two-photon imaging system for freely moving rodents, including all essential optical and recording components but excluding external stimulation devices. It should contain the integrated synchronization module.&amp;nbsp;</td></tr></tbody></table>

decoding natural behavior from neuroethological embedding

Decoding Natural Behavior from Neuroethological Embedding

Animals engage with their natural environment through rich and dynamic brain activity. Understanding how neural population dynamics encode naturalistic behavior remains a fundamental challenge in systems neuroscience. Recent advances in deep learning-based behavior analysis and miniature fluorescence imaging have opened new avenues for investigating how the brain encodes natural behavior. Here, this study presents an integrated experimental and computational framework that combines the Social Behavior Atlas (SBeA), miniature Two-Photon Microscopy (mTPM), and Consistent EmBeddings of high-dimensional Recordings using Auxiliary variables (CEBRA) to decode complex behaviors from brain dynamics. This study uses naturalistic social interactions between freely moving mice as a model system, enabling high-resolution behavioral annotation alongside simultaneous neural imaging. This framework includes precise behavioral pose estimation, synchronized dual-mouse tracking, neural embedding alignment, and decoding of behavioral features directly from neural principal components. This study demonstrates that this approach achieves a decoding precision of 3. &#177; 1.5 pixels for posture and 89 &#177; 6% accuracy for motif decoding across animals, highlighting its robustness and generalizability. This method provides a powerful tool for exploring how brain activity reflects structured behavioral states, and it lays the groundwork for future studies of naturalistic neural coding principles.

The study of natural behavior presents greater complexity compared to trial-based experiments. First, in natural conditions, both neural activity and behavior lack a fixed baseline. These activities are recurrent, meaning that they are influenced by previous states, and thus, aligning the onset of specific behaviors for comparison of neural activity fails to disentangle the effects of prior neuroethological states. Second, neural encoding in natural behavior primarily occurs at the population level4,5. The variability observed in single neurons is substantial enough to be considered as noise. To validate this, this part conducted a correlation analysis between neural activity and natural behavioral poses (Figure 2F, Figure 3F, Figure 4F). The resulting correlation coefficient matrices revealed no neuron-specific correspondence with the pose traces. Specifically, the correlation coefficients between neural signals and subject poses, object poses, or inter-body distances all fell within the range of -0.3 to +0.3, commonly regarded as weak correlations15 (Figure 2G, Figure 3G, Figure 4G). These findings indicate that, under naturalistic conditions, pose-related information is not encoded in a neuron-specific manner.
Given these factors, this framework offers an objective approach to capture and map neuroethological data at the neural population level. mTPM imaging ensures that the variability of individual neurons is preserved to the greatest extent possible. Additionally, the use of deep learning-based pose estimation by ADPT and unsupervised behavior decomposition methods, such as BeA and SBeA, generates rich auxiliary variables, enabling CEBRA to effectively interpret the variability within neural populations.
These examples demonstrate that the joint CEBRA embeddings are present across all auxiliary variables, including subject poses, object poses, body distances, subject behavior motifs, object behavior motifs, and social behavior motifs (Figure 5A). In order to verify the consistency of behavioral motifs and neural embeddings across sessions or subjects, Procrustes analysis35 is used across three mouse pairs (Figure 5B). Given that CEBRA embeddings are distributed on a unit sphere, only the rotation parameter in Procrustes analysis was enabled. Since the natural-behavior CEBRA embeddings lack a clear baseline, this part first performed label-guided alignment sampling on the embeddings to align, ensuring consistent anchor points before applying Procrustes analysis. Visually, these CEBRA embeddings exhibit a degree of intrinsic consistency, with body distance and social motifs showing the highest alignment. It fits the quantification of the RMSE before and after Procrustes alignment (Figure 5C). Then, the embedding decoding accuracy is compared for poses (Figure 5D) and motifs (Figure 5E). While their representations differ, each is decodable with high accuracy. Although the decoding RMSE of body distance is significantly higher than the subject and object poses, it is no more than the tracking accuracy of ADPT7.
To explore the origins of these hypothesis-driven embeddings, a self-organized embedding of neural activity was generated through CEBRA (Figure 5A, right column). The shape of the neural embedding is more intricate than the other joint embeddings, incorporating patterns from various joint embeddings. Additionally, the similarities between neural embeddings and joint embeddings were compared using Procrustes transformation, and then their cosine similarities were compared (Figure 5F). The cosine similarity is derived per minute between the aligned embeddings across corresponding time points.
The S1-subject pose joint embedding was selected as the baseline for similarity comparisons based on the well-established role of the S1 in encoding self-organized somatosensory inputs36. This embedding serves as a biologically meaningful reference point for evaluating how other variables-such as object-related motifs-are represented within the same neural space. Such comparisons enable us to assess the relative strength of encoding for different behavioral dimensions with respect to a self-related somatosensory baseline.
When comparing the cosine similarity of neural embeddings with the S1-subject poses joint embedding as the baseline, this study finds that the joint embeddings for object motifs are significantly lower. This suggests that, during the 15 min period of freely social interaction in this example, the S1 neural activities of the subject mouse primarily encode both its behavior and the ongoing social interactions. While this analysis serves as a demonstrative case, the same methodological framework can be readily applied to more granular investigations, for instance, by comparing embedding structures across distinct temporal epochs to uncover dynamic shifts in neural encoding.
<img alt="Supernova-600 setup for mTPM behavior sync; mouse tracking; signal extraction, 3D reconstruction diagrams." class="xfigimg" src="/files/ftp_upload/68668/68668fig1.jpg" data-altupdatedat="1762509930000" data-alt="Figure 1"> 
Figure 1: Procedure for collecting neuroethological data. (A) The integration of devices. (B) The operation of data recording. (C) Neural signal extraction, 2D pose estimation, and 3D body trajectory reconstruction after recording. <a href="https://www.jove.com/files/ftp_upload/68668/68668fig1large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Neural activity correlation graphs; diagrams of subject and object poses, body distance, motif patterns." class="xfigimg" src="/files/ftp_upload/68668/68668fig2.jpg" data-altupdatedat="1762509930000" data-alt="Figure 2"> 
Figure 2: Preprocessed data of mouse 1 for further analysis. (A) Neuronal activities. (B) Subject poses. (C) Object poses. (D) Body distance. (E) Behavior motifs. From top to bottom are subject, object, and social behavior motifs. (F) The correlation coefficient matrices between neural activity and poses. Left: the correlation coefficients between neural activity and subject poses. Center: the correlation coefficients between neural activity and object poses. Right: the correlation coefficients between neural activity and body distance. The correlation coefficients are between each neuron trace and each pose dimension. (G) The distributions of correlation coefficients of F. The neuron indices are sorted according to the correlation coefficients between neural activity and subject poses. Abbreviations: N &amp; S = neural activity and subject poses, N &amp; O = neural activity and object poses, N &amp; B = neural activity and body distances, CC = correlation coefficients. <a href="https://www.jove.com/files/ftp_upload/68668/68668fig2large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Neural activity time-series analysis; diagram of subject, object poses, and body distances in correlation." class="xfigimg" src="/files/ftp_upload/68668/68668fig3.jpg" data-altupdatedat="1762509930000" data-alt="Figure 3"> 
Figure 3: Preprocessed data of mouse 2 for further analysis. (A) Neuronal activities. (B) Subject poses. (C) Object poses. (D) Body distance. (E) Behavior motifs. From top to bottom are subject, object, and social behavior motifs. (F) The correlation coefficient matrices between neural activity and poses. Left: the correlation coefficients between neural activity and subject poses. Center: the correlation coefficients between neural activity and object poses. Right: the correlation coefficients between neural activity and body distance. The correlation coefficients are between each neuron trace and each pose dimension. (G) The distributions of correlation coefficients of F. The neuron indices are sorted according to the correlation coefficients between neural activity and subject poses. Abbreviations: N &amp; S = neural activity and subject poses, N &amp; O = neural activity and object poses, N &amp; B = neural activity and body distances, CC = correlation coefficients. <a href="https://www.jove.com/files/ftp_upload/68668/68668fig3large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Neural activity correlation chart; motifs analysis; subject and object poses; experiment results." class="xfigimg" src="/files/ftp_upload/68668/68668fig4.jpg" data-altupdatedat="1762509930000" data-alt="Figure 4"> 
Figure 4: Preprocessed data of mouse 3 for further analysis. (A) Neuronal activities. (B) Subject poses. (C) Object poses. (D) Body distance. (E) Behavior motifs. From top to bottom are subject, object, and social behavior motifs. (F) The correlation coefficient matrices between neural activity and poses. Left: the correlation coefficients between neural activity and subject poses. Center: the correlation coefficients between neural activity and object poses. Right: the correlation coefficients between neural activity and body distance. The correlation coefficients are between each neuron trace and each pose dimension. (G) The distributions of correlation coefficients of F. The neuron indices are sorted according to the correlation coefficients between neural activity and subject poses. Abbreviations: N &amp; S = neural activity and subject poses, N &amp; O = neural activity and object poses, N &amp; B = neural activity and body distances, CC = correlation coefficients. <a href="https://www.jove.com/files/ftp_upload/68668/68668fig4large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Mouse behavior data analysis; graph of 3D positions, BeA categories, and statistical results." class="xfigimg" src="/files/ftp_upload/68668/68668fig5.jpg" data-altupdatedat="1762509930000" data-alt="Figure 5"> 
Figure 5: Analysis of CEBRA embeddings of neuroethological data. (A) CEBRA embeddings. From left to right are the joint embedding of S1 neural activity and subject poses, the joint embedding of S1 neural activity and object poses, the joint embedding of S1 neural activity and body distances between two animals, the joint embedding of S1 neural activity and subject behavior motifs, the joint embedding of S1 neural activity and object behavior motifs, the joint embedding of S1 neural activity and social behavior motifs, and the neural embedding of S1. (B) The Procrustes analysis aligns the embeddings above. The gray circles represent the mouse pair 1, serving as the reference embedding. The green plus signs represent mouse pair 2, and the orange crosses represent mouse pair 3, both aligned to mouse pair 1. (C) The root mean square error (RMSE) before (left) and after (right) Procrustes alignment (Paired t-test, n=3, mean ± SEM). (D) The RMSE of pose reconstruction from CEBRA embeddings (one-way ANOVA followed by Tukey's multiple comparisons test, n=3, mean ± SEM). (E) The accuracy of motif reconstruction from CEBRA embeddings (one-way ANOVA followed by Tukey's multiple comparisons test, n=3, mean ± SEM). (F) The cosine similarities between joint embeddings and neural embedding of S1 (one-way ANOVA followed by Dunnett's multiple comparisons test, n=45, mean ± SEM). *p &lt; 0.05, **p &lt; 0.01, ***p &lt; 0.001. <a href="https://www.jove.com/files/ftp_upload/68668/68668fig5large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>No.</td>
			<td>Observed problem</td>
			<td>Likely cause</td>
			<td>Possible solutions</td>
		</tr>
		<tr>
			<td rowspan="3">1</td>
			<td rowspan="3">No behavior timestamps</td>
			<td>(1)   Faulty SMA or BNC cables</td>
			<td>(1)   Replace SMA and BNC cables</td>
		</tr>
		<tr>
			<td>(2)   Missing USB-to-TTL driver</td>
			<td>(2)   Install the Prolific PL2303 USB driver</td>
		</tr>
		<tr>
			<td>(3)   Incorrect COM port selection</td>
			<td>(3)   Verify the COM port number in Device Manager and update it in both the mTPM software and behavior camera script.</td>
		</tr>
		<tr>
			<td rowspan="4">2</td>
			<td rowspan="4">No visible fluorescence during mTPM mounting</td>
			<td>(1)   Lack of viral expression</td>
			<td>(1)   Use a different mouse</td>
		</tr>
		<tr>
			<td>(2)   Incorrect field of view</td>
			<td>(2)   Readjust the field of view</td>
		</tr>
		<tr>
			<td>(3)   Insufficient laser power</td>
			<td>(3)   Gradually increase laser power</td>
		</tr>
		<tr>
			<td>(4)   Dried Carbomer gel</td>
			<td>(4)   Reapply fresh Carbomer gel</td>
		</tr>
		<tr>
			<td rowspan="3">3</td>
			<td rowspan="3">mTPM imaging shows a completely white screen</td>
			<td>(1)   Light leakage</td>
			<td>(1)   Rewrap the aluminum foil for proper shielding</td>
		</tr>
		<tr>
			<td>(2)   Insufficient laser power</td>
			<td>(2)   Gradually increase laser power</td>
		</tr>
		<tr>
			<td>(3)   Detached fiber from the mTPM head</td>
			<td>(3)   Reinsert the fiber into the mTPM and tighten the fixing screw</td>
		</tr>
		<tr>
			<td rowspan="3">4</td>
			<td rowspan="3">Dropped frames in behavioral video</td>
			<td>(1)   Low ambient lighting</td>
			<td>(1)   Increase background illumination</td>
		</tr>
		<tr>
			<td>(2)   Incorrect USB port</td>
			<td>(2)   Use at least USB 3.0 ports</td>
		</tr>
		<tr>
			<td>(3)   Insufficient computer performance</td>
			<td>(3)   Use a machine with Intel i7-9700K or higher, dual-channel RAM, and SSD storage.</td>
		</tr>
		<tr>
			<td rowspan="3">5</td>
			<td rowspan="3">No locomotion in mTPM-mounted mice</td>
			<td>(1)   Repetitive use of the same mouse</td>
			<td>(1)   Avoid reusing mice within 3 days</td>
		</tr>
		<tr>
			<td>(2)   Excessive use of aluminum foil</td>
			<td>(2)   Use minimal foil necessary for light shielding</td>
		</tr>
		<tr>
			<td>(3)   Insufficient number or volume of helium balloons</td>
			<td>(3)   Adjust the number and inflation of balloons to support the mTPM fiber while allowing natural posture and movement of mice.</td>
		</tr>
		<tr>
			<td rowspan="2">6</td>
			<td rowspan="2">Inaccurate 2D pose estimation</td>
			<td>(1)   Insufficient number of manually labeled frames</td>
			<td>(1)   Annotate at least 200 frames incrementally</td>
		</tr>
		<tr>
			<td>(2)   Undertrained ADPT model</td>
			<td>(2)   Increase training epochs in the config.yaml file of ADPT</td>
		</tr>
		<tr>
			<td rowspan="3">7</td>
			<td rowspan="3">Abnormal 3D pose reconstruction</td>
			<td>(1)   Improper camera calibration</td>
			<td>(1)   Improve calibration contrast and tilt angle</td>
		</tr>
		<tr>
			<td>(2)   Inaccurate 2D pose input</td>
			<td>(2)   Increase the number of checkboard frames captured</td>
		</tr>
		<tr>
			<td></td>
			<td>(3)   Resolve the 2D pose issues first (see Problem 6)</td>
		</tr>
		<tr>
			<td rowspan="3">8</td>
			<td rowspan="3">Misalignment between neural and behavioral data</td>
			<td>(1)   Incorrect sequence of software initialization</td>
			<td>(1)   Always start mTPM recording before behavior camera</td>
		</tr>
		<tr>
			<td>(2)   Dropped behavior frames</td>
			<td>(2)   Address frame drop issues (see Problem 4)</td>
		</tr>
		<tr>
			<td></td>
			<td>(3)   Ensure sufficient disk space is available</td>
		</tr>
		<tr>
			<td rowspan="3">9</td>
			<td rowspan="3">Memory overflow during BeA/SBeA processing</td>
			<td>(1)   Excessive recording duration</td>
			<td>(1)   Split recordings into shorter segments (5–60 minutes), then run BeA/SBeA </td>
		</tr>
		<tr>
			<td>(2)   Limited system RAM</td>
			<td>(2)   Increase the temporal reduction factor (e.g., from 5 to 10) in BeA</td>
		</tr>
		<tr>
			<td></td>
			<td>(3)   Upgrade RAM to at least 64 GB</td>
		</tr>
		<tr>
			<td rowspan="3">10</td>
			<td rowspan="3">CEBRA fails to run on GPU</td>
			<td>(1)   Mismatch between CUDA and GPU driver</td>
			<td>(1)   Do not directly follow the tutorial to install CUDA 11.3</td>
		</tr>
		<tr>
			<td>(2)   Incompatible PyTorch version</td>
			<td>(2)   Check your GPU model and driver version (nvidia-smi)</td>
		</tr>
		<tr>
			<td></td>
			<td>(3)   Install the correct CUDA and PyTorch versions accordingly and then install CEBRA via pip</td>
		</tr>
	</tbody>
</table>
Table 1: Troubleshooting list. The following is a list of 10 previously encountered non-trivial problems and possible solutions.

This protocol gives an integrated framework based on advanced computational neuroethological methods to understand brain coding in naturalistic contexts.

Animals engage with their natural environment through rich and dynamic brain activity. Understanding how neural population dynamics encode naturalistic ...

The scope of my research is to understand how neurodynamics encode natural behavior and how the brain controls complex actions that supports survival in natural environments. Traditional head-fixed paradigms limit our understanding of natural behavior. Our protocol updates this paradigm by embodying precise neural behavior decoding in freely moving animals toward natural brain intelligence.We will focus on collecting rich uncontrolled data to build digital life models using holistic approaches to understand intelligence in complex living systems. To begin, connect the universal serial bus cable of the synchronization module of the three-dimensional behavior device to the workstation of the same device. Then, connect the synchronization module of the mTPM device to its controller using one SMA cable.Connect the TTL output port of the synchronization module of the three-dimensional behavior device to the TTL input port of the synchronization module of the mTPM device using one SMA BNC conversion cable. To begin calibration, adjust the shooting angle of all four cameras so that they cover the entire base of the open field and extend their field of view at least 20 centimeters above the farthest boundary to capture mouse rearing behavior. Then, place the calibration module at the center of the shooting area.Switch off all the lights and run the camera calibration software. Now, fix the mouse restrainer to the micro-manipulator of the mTPM. Using the metal plate, secure the head of the mouse to the restrainer.Switch off all the lights. Then, fix the mTPM to its holder and switch on the imaging system to locate the fluorescent signal. Add one drop of Carbomer eye gel to the top of the cranial window.Move the mouse using the motion platform so that the cranial window is aligned directly beneath the objective of the mTPM. Move the micro-manipulator vertically to locate the imaging plane. Then, move the micro-manipulator in plane to center the imaging plane.Then, fix the upper base to the mTPM. Apply adhesive to glue the lower base to the upper base and secure it to the cranial window. To ensure structural stability, fill the gap between the two bases and the metal plate bracket attached to the mouse's head using a high-performance acrylic structural adhesive.Then, assess the bond stability by gently probing the base with tweezers. After that, add one drop of Carbomer eye gel into the base chamber. Observe the neuronal fluorescence through the mTPM.If the fluorescence is not clearly visible, remove the adhesive using a cranial drill to detach the base. Then, repeat the procedure until clear fluorescence is achieved. Then, secure aluminum foil with tape between the fiber of the mTPM and the cranial window.Switch on the room light and test the clarity of the frames captured by the mTPM. To put the mouse in an open field, inflate at least 10 helium balloons and tie each one separately with cotton twine. Then, detach the metal plate from the mouse restrainer.Hold the mouse gently by its tail using one hand. With the other hand, support the optical fiber of the mTPM. Carefully place the mouse into the open field.Suspend the helium balloons by attaching the cotton twine to the fiber. Adjust the number of balloons so that the mouse can move and explore the open field without restriction. Close the door of the mTPM enclosure to reduce external disturbances.Start the mTPM recording software and the synchronization software. Set the file paths and recording parameters according to the platform establishment procedure. Start the recording of the mTPM through the recording software.Check the synchronization software to verify that time markers are accurately recorded for each two-photon frame. Evaluate whether the contrast of the two-photon images remain stable during recording. Also confirm that the mouse's movements do not disrupt the stability of the imaging frames.Now, start the customized camera synchronization script to initiate behavior recording. Set the file path and parameters according to the platform establishment procedure. Then, start the behavior recording using the customized synchronization script.Confirm the presence of a time marker in the synchronization software for every 30 frames of behavior video. Check that all four video streams from the cameras are synchronized correctly. Verify that the video capture parameters of the three-dimensional behavioral tracking system are set properly.Once the behavioral recording stops automatically, manually switch off both the mTPM recording and synchronization software to conclude the trial. Correlation coefficient matrices showed no distinct neuron-specific patterns for subject poses, object poses, or body distances, indicating weak correspondence between neural signals and behavioral metrics. All neuron behavior correlation coefficients fell between 0.3 and 0.3, confirming weak associations under naturalistic conditions.Zebra-derived neural embeddings form intricate patterns, incorporating components from multiple joint embeddings. Zebra embeddings demonstrated consistent alignment of behavioral and neural variables across three mouse pairs, particularly for body distance and social motifs. The decoding error for body distance embeddings was significantly higher than subject and object poses, but remained within expected tracking error limits.Joint embeddings of neural activity with various behavioral variables revealed high decoding accuracy across subject poses, object poses, and motifs. Cosine similarity analysis using the S1-subject pose embedding as a reference showed lower alignment for object-related motifs, suggesting primary encoding of self and social behavior.

decoding-natural-behavior-from-neuroethological-embedding

Research

JoVE Journal

Neuroscience

Non-invasive measurements of neural activity patterns in freely behaving animals are obtained by combining neurophysiological recordings with high speed videography.

Recordings of Neural Circuit Activation in Freely Behaving Animals

The comparative species approach allows behavioral neuroscientists to explore various neurobiological factors associated with specific behaviors viewed as characteristic of a specific animal model.  Taking advantage of naturally occurring differences in behavior between closely related species, this technique doesn’t require invasive techniques to manipulate the expression of the behavior.

Using a Comparative Species Approach to Investigate the Neurobiology of Paternal Responses

In this paper, an experimental framework to perform closed-loop experiments is presented, in which information processing (i.e., coding and decoding) and learning of neuronal assemblies are studied during the continuous interaction with a robotic body.

Closed-loop Neuro-robotic Experiments to Test Computational Properties of Neuronal Networks

This protocol presents a novel methodology for the neural decoding of intent from freely-behaving infants during unscripted social interaction with an actor. Neural activity is acquired using non-invasive high-density active scalp electroencephalography (EEG). Kinematic data is collected with inertial measurement units and supplemented with synchronized video recording.

A Novel Experimental and Analytical Approach to the Multimodal Neural Decoding of Intent During Social Interaction in Freely-behaving Human Infants

Here, we present methods for high-throughput study of a series of the Mexican cavefish behaviors and vital staining of a mechanosensory system. These methods use free-software and custom-made scripts, providing a practical and cost-effective method for the studies of behaviors.

Behavioral Tracking and Neuromast Imaging of Mexican Cavefish

The purpose of this protocol is to utilize pre-built convolutional neural nets to automate behavior tracking and perform detailed behavior analysis. Behavior tracking can be applied to any video data or sequences of images and is generalizable to track any user-defined object.

A Step-by-Step Implementation of DeepBehavior, Deep Learning Toolbox for Automated Behavior Analysis

This study presents a naturalistic experimental setup that allows researchers to present real-time action stimuli, obtain response time and mouse tracking data while participants respond after each stimulus display, and change actors between experimental conditions with a unique system including a special transparent organic light-emitting diode (OLED) screen and light manipulation.

A Naturalistic Setup for Presenting Real People and Live Actions in Experimental Psychology and Cognitive Neuroscience Studies

Presented here is a video analysis pipeline that overcomes challenges in behavioral monitoring within MRI environments, allowing for the detection of uninstructed behavioral responses to external cues. This analysis will facilitate a more comprehensive understanding of evoked internal state changes and brain-wide activity.

Profiling Maternal Behavior Responses During Whole-Brain Imaging

Electrophysiological responses of olfactory sensory neurons to odorants can be measured in insects using single sensillum recordings. In this video article we will demonstrate how to perform single sensillum recordings in the antennae of the vinegar fly (Drosophila melanogaster) and the maxillary palps of the malaria mosquito (Anopheles gambiae).

Single Sensillum Recordings in the Insects Drosophila melanogaster and Anopheles gambiae

A way to study the integration of newborn dentate granule cells in adult animals is described. This technique uses an engineered retrovirus to label newborn neurons, followed by electrophysiological recordings to determine in vivo functional integration.

Studying the Integration of Adult-born Neurons

Learn precise neural decoding in freely moving mice using 3D behavior devices and mTPM imaging. This method reveals how neurodynamics encode natural behavior and advance brain intelligence research.

Decoding Natural Behavior from Neuroethological Embedding

Explore More Videos