Automated Interactive Video Playback for Studies of Animal Communication

1. Setting up hardware and Viewer system.

The system consists of a testing tank flanked by two monitors. A video camera connected to the BIOBSERVE Viewer system records the movement of the subject in real time. Motion data is passed to the interactive video playback (IVP) program on a separate server, which determines the motion of the animated stimulus on screen.

1. Place two CRT monitors at opposite ends of an 80-l aquarium filled with clean water; match monitor output.
2. Place camera overhead to capture full view of aquarium; connect to BIOBSERVE video card.
3. Configure the Viewer system to receive tracking information from the camera.
4. Enable plug-in to Viewer that sends the coordinates of the animal's snout, body, and tail to animation server in real time via a specified network IP address.
5. Turn on animation server.

2. Calibration.

1. On the animation server, open InteractiveDisplaySetup.txt. Enter screen width and screen height in pixels under "Monitor Info". Under "ProgramType" enter "calibration". Save changes to file.
2. Double-click on IVP program icon to start program.
3. Enter in the extents of the tank dimensions as set up in the Viewer system in step 2.3. This will ensure correspondence between Viewer data and interactive output from IVP.
4. Since the ends of fish tanks are smaller than the monitors, the software allows the user to use the keyboard (a and s for width, z and x for height, and arrow keys for position) to move the viewing area (viewport) of the stimuli to align with the ends of the testing tank. Viewed on the animation server, the pink viewport is echoed on one testing monitor and the blue viewport on the other (figure 1).
5. Use z and x on the keyboard to scale the display of the male fish on both viewports until it matches the desired stimulus size.
6. Move the boxes displayed so that they align to the edges of the test tank. The user uses the Q or W keys to move the boxes toward or away from the center of the fish.
7. Enter experiment information when prompted by the program. The user has the option of selecting among preloaded models (in our demonstration, courting male Xiphophorus birchmanni, X. malinche). The user also specifies the behavior of each fish in each stage (non-interactive or interactive), the side and stage in which each stimulus will appear, and the desired size (standard length) of each stimulus.
8. The next question asks if the user wants the fish movements to be mirrored, i.e. that the movement of the model on either side of the aquarium will be exactly the same. This is only used when fish are both in animation mode. The final questions concern the movement of the dorsal fin. The user can select default dorsal; this would make the dorsal only raise during a lateral courtship display. If the dorsal is not set to default, the user can decide if the model raises its dorsal fin according to the distance of the subject from the monitor.

3.Starting IVP for mate-choice test.

1. Open InteractiveDisplaySetup.txt and set "ProgramType" to liveTesting and set 'experimentName' to the same file name that contains the Experiment Data.
2. Gently place subject fish in aquarium and wait 10 min.
3. Start the Viewer 2.0 and the IVP.

4. Mate-choice trial.

The overall experimental sequence follows previous studies using non-interactive video stimuli^1-4. Females are presented with two different stimuli on monitors at opposite ends of a testing aquarium. Female behavior is tracked in real-time by the Biobserve Viewer system. The response assay is association time, the amount of time that a female spends within 10 cm of one monitor or the other. Association time is automatically computed by the Viewer.

1. The video stimuli are displayed to subjects in a trial consisting of two, 20-minute stages. A stage consists of four 5-minute segments:
   1. The first segment acclimatizes the female to the test tank for five minutes by displaying a monochromatic screen on both monitors.
   2. In the second segment, two different video stimuli are displayed to the female, one on the left monitor and the other on the right for five minutes.
   3. Immediately following the display of the video stimuli, a monochromatic screen is again displayed on both monitors for five minutes.
   4. In the fourth and final segment, stimuli from the second segment are presented, but the position of each stimulus is switched. This provides a within-female control for side bias.
2. The second stage repeats the same pattern, with a different set of simultaneously-presented stimuli.
3. Additional trials are run by returning to step 2 and systematically varying side and order of presentation.

5. Representative Results.

We evaluated the effectiveness of interactive playback by comparing female response to interactive and non-interactive animations of courting males. The non-interactive stimulus performed a courtship stimulus onscreen, independent of female behavior, as in previous studies^1-4.

The interactive stimulus tracked the horizontal position of the female across the screen. The position of either the simulated fish or the subject fish was operationally defined as the midpoint of the line connecting the centroid and the snout. The three rules guiding male behavior were as follows, where the X direction refers to the length of the tank, and the Z direction refers to the width of the tank (figure 2):

Rule 1: The simulated male always follows the subject across the screen, tracking her in the Z direction.

Rule 2: The simulated male's dorsal fin is only raised when it is performing a lateral courtship display.

Rule 3: The simulated male swordtail fish will only perform a lateral courtship display for 50% of the total time it is being displayed to the female. The lateral courtship display is triggered by the male fish being within one-fourth male body length of the female fish in the Z direction. The lateral courtship display is independent of how close the female is to the monitor in the X direction.

The interactive stimulus closely tracked the female's position in real time (figure 3).

Previous work using non-interactive stimuli^4,5 showed that female X. birchmanni strongly preferred visual signals of their own species . The non-interactive stimuli created by the IVP were equally effective at eliciting conspecific mate preferences (t = 1.923, N = 9, p = .046). When females were tested on simulated interactive conspecific and heterospecific males immediately before or afterwards in the same trial, however, they failed to show a preference (figure 4).

Figure 1. Monitor setup for calibration.

Figure 2. Schematic view of tank and monitor setup, indicating axes used to describe position.

Figure 3. Horizontal position of animated exemplar and representative female subject over time.

Figure 4. Association time (s) of females with interactive and non-interactive conspecific and heterospecific male stimuli. Each plot is created from a five-minute record of the positions output by the prototype. In these plots, time progresses in the vertical Y direction from 0 to 5 minutes.

Figure 5. Representative association time data and tracking by two interactive stimuli. Two five-minute segments for two females are shown.

Previous methods for interactive video playback in animal behavior have relied on a human operator to provide responses to behavioral cues from subjects. With IVP, we created a program that applied rules-based interactivity in response to real-time, automated data on subject behavior. We briefly outline the steps involved in creating the program below.

The first step was to create digital male exemplars of X. birchmanni and X. malinche. We took an approach similar to previous studies⁶. We created 3D meshes that are modeled on textures based on photographs of real X. birchmanni and X. malinche. To capture the realistic textures of the real fish, the same photographs used to model the fish shapes were used as textures for the fish. A planar map applied to their UV coordinates aligned the UV map with the photograph texture. Second, the digital fish mesh must deform like a real fish. To accomplish this, a virtual skeleton was created for body and fins and "skinned" to the mesh. The skinning process enables the mesh to be deformed when the joints are rotated.

Second, we added motion to the digital fish. Six key movements that a male swordtail fish makes were animated. Three of the movements were used to represent the different speeds at which a fish would swim. The other three movements were the fish remaining still, turning, or exhibiting a lateral courtship display. Since males can raise or lower their dorsal fin in accordance to whether male or female receivers are present³, we decoupled the movement of the dorsal fin from that of the lateral courtship display. The dorsal fin was keyed so that it could be raised or lowered at any point during the cycle. A total of twenty-four animation cycles were used. Each cycle started and ended with the fish in the same posture so that the animation cycles could easily blend together. All of the twenty-four animation cycles were created by rotoscoping^7,8 the desired motion from overhead video of a live, courting male X. birchmanni.

Third, we enabled interactivity. We used the Biobserve Viewer system to track in real time the position of the snout, body, and tail of the female swordtail and transmit that information in real time to the IVP program. This was done separately for each courting male on each monitor. The male animation followed the subject fish's position. We modeled following using Reynolds arrived steering behavior^9,10, which allowed the male to follow the female and decelerate as it approaches the female.

To calculate the position of the male swordtail fish at each time step, the system was supplied with the current position of the female, which allowed the program to calculate the forces that drive the male. First, the target-offset vector was calculated by subtracting the position of the male fish from the position of the female fish. Second the distance from the male fish to the female fish was determined by taking the magnitude of the target-offset vector. Third, the desired speed of the male fish was determined by dividing the distance by a constant deceleration value. This allowed the male fish to slow down as it approached the female fish. Last, the desired acceleration was calculated by subtracting the male's current velocity from the desired velocity.

Since animations are rendered as discrete frames of video at 60 Hz, calculations were made for each discrete time step, at intervals of 0.016 seconds. Maximum velocity was set to a value of 10 cm/s for these experiments. If the magnitude of the new velocity was greater than the maximum velocity, the velocity was set to the maximum.

For this particular simulation, the interactive male fish raised its dorsal fin 50% of the time, and only during courtship interactions. The lateral courtship display behavior was triggered whenever the male stimulus was within 0.25 body lengths of the female swordtail fish in the Z dimension.

We were surprised that interactivity abolished the female preference for conspecifics, despite the fact that the non-interactive animations elicited a strong preference and given that females spent the majority of time associating with the interactive stimuli. One possibility is that closely following the female may override visual cues used to evaluate males, such as the sword and dorsal fin. Alternatively, females may be less likely to lose interest in a courting male, and therefore less likely to sample both individuals (figure 5).

Nevertheless, our results show that the operating principle of video-game technology namely, software-driven, rules-based agents responding to user input can be successfully applied to interactive playback in studies of animal behavior. This type of rules-based interactive playback should prove useful for studies of shoaling and collective movement^11,12. In particular, the ability to manipulate the rules that a virtual exemplar uses for shoaling should give us insight into the processes that animals use to make shoaling decisions.