A Step-by-Step Implementation of DeepBehavior, Deep Learning Toolbox for Automated Behavior Analysis

Summary

The purpose of this protocol is to utilize pre-built convolutional neural nets to automate behavior tracking and perform detailed behavior analysis. Behavior tracking can be applied to any video data or sequences of images and is generalizable to track any user-defined object.

Abstract

Understanding behavior is the first step to truly understanding neural mechanisms in the brain that drive it. Traditional behavioral analysis methods often do not capture the richness inherent to the natural behavior. Here, we provide detailed step-by-step instructions with visualizations of our recent methodology, DeepBehavior. The DeepBehavior toolbox uses deep learning frameworks built with convolutional neural networks to rapidly process and analyze behavioral videos. This protocol demonstrates three different frameworks for single object detection, multiple object detection, and three-dimensional (3D) human joint pose tracking. These frameworks return cartesian coordinates of the object of interest for each frame of the behavior video. Data collected from the DeepBehavior toolbox contain much more detail than traditional behavior analysis methods and provides detailed insights to the behavior dynamics. DeepBehavior quantifies behavior tasks in a robust, automated, and precise way. Following the identification of behavior, post-processing code is provided to extract information and visualizations from the behavioral videos.

Introduction

A detailed analysis of behavior is key to understanding the brain and behavior relationships. There have been many exciting advances in methodologies for recording and manipulating neuronal populations with high temporal resolution, however, behavior analysis methods have not developed at the same rate and are limited to indirect measurements and a reductionist approach¹. Recently, deep learning based methods have been developed to perform automated and detailed behavior analysis^2,3,4,5. This protocol provides a step-by-step implementation guide for the DeepBehavior toolbox.

Traditional behavioral analysis methods often include manually labeling data by multiple evaluators, leading to variance in how experimenters define a behavior⁶. Manual labeling of the data requires time and resources that increase disproportionately to the amount of data collected. Moreover, manually labelled data reduce the behavior outcomes into categorical measurements which do not capture the richness of the behavior, and will be more subjective. Thus, the current traditional methods may be limited in capturing the details in the natural behaviors.

The DeepBehavior toolbox presents a precise, detailed, highly temporal, and automated solution using deep learning for behavioral analysis. Deep learning has quickly become accessible to all with open-source tools and packages. Convolutional neural networks (CNNs) are proven to be highly effective in object recognition and tracking tasks^7,8. Using modern day CNNs and high-performance graphics-processing-units (GPUs), large image and video datasets can be processed quickly with high precision^7,9,10,11. In DeepBehavior, there are three different convolutional neural net architectures, TensorBox, YOLOv3, and OpenPose².

The first framework, Tensorbox, is a versatile framework that incorporates many different CNN architectures for object detection¹². TensorBox is best suited for detecting only one object class per image. The resulting outputs are bounding boxes of the object of interest (Figure 1) and the cartesian coordinates of the bounding box.

The second CNN framework is YOLOv3, which stands for "You Only Look Once"¹³. YOLOv3 is advantageous when there are multiple objects of interest that must be tracked separately. The output of this network includes the bounding box with the associated object label class as well as the bounding box cartesian coordinates of the object in the video frame (Figure 2).

The previous two frameworks are advantageous for generalized behavioral data collected from standard laboratory experiments in animal subjects. The last CNN framework is OpenPose^14,15,16 which is used for human joint pose estimation. OpenPose detects human body, hand, facial, and foot key points on images. The outputs of the framework are labeled images of the human subject as well as the coordinates of all the 25 key points in the body and 21 key points of each hand (Figure 3).

This detailed step-by-step guide for implementation of our recently developed open-source DeepBehavior toolbox employs state-of-the-art convolutional neural nets to track animal behavior (e.g. movement of a paw) or human behavior (e.g. reaching tasks). By tracking the behavior, useful kinematics can be derived from the behavior such as position, velocity, and acceleration. The protocol explains the installation of each CNN architecture, demonstrates how to create training datasets, how to train the networks, how to process new videos on the trained network, how to extract the data from the network on the new videos, and how to post-process the output data to make it useful for further analysis.

Protocol

1. GPU and Python Setup

1. GPU Software
   When the computer is first setup for deep learning applications, GPU-appropriate software and drivers should be installed which can be found on the GPU's respective website. (see the Table of Materials for those used in this study).
2. Python 2.7 Installation
   Open a command line prompt on your machine.
   Command line: sudo apt-get install python-pip python-dev python-virtualenv

2. TENSORBOX

1. Tensorbox Setup
   1. Create Virtual Environment for Tensorbox
      Command line: cd ~
      Command line: virtualenv --system-site-packages ~/tensorflow
      NOTE: '~/tensorflow' is the name of the environment and is arbitrary
   2. Activate environment
      Command line: source ~/tensorflow/bin/activate
2. Tensorbox Installation
   We will be using GitHub to clone TensorBox from http://github.com/aarac/TensorBox and install it on our machine as well as installing additional dependencies.
   Command line: cd ~
   Command line: git clone http://github.com/aarac/TensorBox
   Command line: cd TensorBox
   Command line: pip install -r requirements.txt
3. Label Data
   1. Create a folder of images of behavior
      Open source tools such as ffmpeg are useful to accomplish converting videos to individual frames We recommend labeling at least 600 images from a wide-distribution of behavior frames for training. Put these images in a folder.
   2. Launch labeling graphical user interface
      Command line: python make_json.py <path to image folder> labels.json
      To label an image, click the top left corner of the object of interest (i.e. paw) first and then click the bottom right corner of the object of interest (Figure 4). Inspect that the bounding box captures the entire object of interest. Press 'undo' to re-label the same image or press 'next' to move onto the next frame.
4. Train TensorBox
   1. Link training images to network hyperparameters file
      Within the tensorbox folder, open the following folder in a text editor:
      /TensorBox/hypes/overfeat_rezoom.json. Navigate to the attribute under data named train_idl and replace the file path from ./data/brainwash/train_boxes.json to the labels.json filepath. Save the changes to file.
   2. Begin training script
      Command line: cd ~/TensorBox
      Command line: python train.py --hypes hypes/overfeat_rezoom.json --gpu 0 --logdir output
      The network will then begin training for 600,000 iterations. In the output folder, the resulting trained weights of the convolutional neural network will be generated.
5. Predict on New Images
   For image labeling:
   Command line: cd ~/TensorBox
   Command line: python label_images.py --folder <path to image folder> --weights output/overfeat_rezoom_<timestamp>/save.ckpt-600000 --hypes /hypes/overfeat_rezoom.json --gpu 0
   To get coordinates of bounding boxes:
   Command line: cd ~/TensorBox
   Command line: python predict_images_to_json.py --folder <path to image folder> --weights
   output/overfeat_rezoom_<timestamp>/save.ckpt-600000 --hypes
   /hypes/overfeat_rezoom.json --gpu 0
6. MATLAB Post-Processing for TensorBox
   Additional MATLAB code has been provided to extract kinematics and visualizations of the coordinates using the resulting JSON coordinate file from the model
   Run the "Process_files_3Dreaching_mouse.m" script for 3D kinematic analysis of single food pellet reaching task.

3. YOLOv3

1. Install YOLOv3
   Command line: cd ~
   Command line: git clone cd darknet
   For GPU usage, open 'Makefile' and change the following lines: GPU=1; CUDNN=1.
   Command line: make
2. Labeling Training Data using Yolo_mark
   Command line: cd ~
   Command line: git clone cd ~/Yolo_Mark
   Command line: cmake .
   Command line: make
   Place the training images in ~/Yolo_mark/data/obj folder
   Command line: chmod +x ./linux_mark.sh
   Command line: ./linux_mark.sh
   Label the images one by one in the graphical user interface (Figure 5). The recommended amount of images is approximately 200.
3. Training YOLOv3
   1. Setup configuration file
      Command line: cd ~/Yolo_mark
      Command line: scp -r ./data ~/darknet
      Command line: cd ~/darknet/cfg
      Command line: cp yolov3.cfg yolo-obj.cfg
   2. Modify the configuration file
      Open the yolo-obj.cfg folder and modify the following lines: batch=64, subdivision=8, classes=(# of class to detect), and for each convolutional layer before a yolo layer change the filter=(classes+5)x3. Details on these changes can be found at https://github.com/aarac/darknet/blob/master/README.md
   3. Download network weights
      Download the network weights from https://www.dropbox.com/s/613n2hwm5ztbtuf/darknet53.conv.74?dl=0
      Place the downloaded weight file into ~/darknet/build/darknet/x64
   4. Run training algorithm
      Command line: cd ~/darknet
      Command line: ./darknet detector train data/obj.data cfg/yolo-obj.cfg darknet53.conv.74
   5. YOLOv3 Evaluation
      After the training is complete based on a set number of iterations (ITERATIONNUMBER), you can view them by
      Command line: ./darknet detector test data/obj.data cfg/yolo-obj.cfg backup/yolo-obj_ITERATIONNUMBER.weights <IMAGE>.jpg
4. Predict on new videos and get coordinates
   This command can be run to obtain the coordinates of the labels in the new video:
   Command line: ./darknet detector demo data/obj.data cfg/yolo-obj.cfg backup/yolo-obj_ITERATIONNUMBER.weights VIDEO.avi -ext_output <VIDEO.avi> FILENAME.txt
5. YOLOv3 PostProcessing in MATLAB
   Take the FILENAME.txt file to MATLAB, and run the "Process_socialtest_mini.m" script for two mice social interaction test. See results in Figure 2

4. OpenPose

OpenPose is ideal to track multiple body parts in a human subject. The setup and installation processes are very similar to the previous two frameworks. However, there is no training step as the network is already trained on human data.

1. OpenPose Installation
   Navigate to https://github.com/aarac/openpose and follow the installation instructions.
2. Process Video
   ./build/examples/openpose/openpose.bin --video VIDEONAME.avi --net_resolution "1312x736" --scale_number 4 --scale_gap 0.25 --hand --hand_scale_number 6 --hand_scale_range 0.4 --write_json JSONFOLDERNAME --write_video RESULTINGVIDEONAME.avi
   Here the --net_resolution, --scale_number, --scale_gap, --hand_scale_number and --hand_scale_range handles can be omitted if a high precision detection is not needed (this would decrease the processing time).
3. OpenPose Post-Processing
   In MATLAB folder, please use 'process_files_human3D.m' script to run the code after adding the appropriate folder containing json files from cameras 1 and 2, as well as the calibration file. This will create a "cell" file with all the 3D poses of the joints. It will also make a movie of the 3D skeletal view. For camera calibration, please follow the instructions at this link: http://www.vision.caltech.edu/bouguetj/calib_doc/

Results

When the protocol is followed, the data for each network architecture should be similar to the following. For TensorBox, it outputs a bounding box around the object of interest. In our example, we used videos from a food pellet reaching task, and labeled the right paws to track their movement. As seen in Figure 1, the right paw can be detected in different positions in both the front view and side view cameras. After post-processing with camera calibration, 3...

Discussion

Here, we provide a step-by-step guide for implementation of DeepBehavior, our recently developed deep learning based toolbox for animal and human behavior imaging data analysis². We provide detailed explanations for each step for installation of the frameworks for each network architecture, and provide links for installation of the open-source requirements to be able to run these frameworks. We demonstrate how to install them, how to create training data, how to train the network, and how to proce...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
CUDA v8.0.61	NVIDIA	n/a	GPU Software
MATLAB R2016b	Mathworks	n/a	Matlab
Python 2.7	Python	n/a	Python Version
Quadro P6000	NVIDIA	n/a	GPU Processor
Ubuntu v16.04	Ubuntu	n/a	Operating System