Name: An Experimental Platform to Study the Closed-loop Performance of Brain-machine Interfaces (Video) | JoVE Journal
Uploaded: 2011-03-10T12:00:00.000+00:00
Duration: 10 min 51 s
Description: Learn how to create a closed-loop brain-machine interface using the H1 neuron in the fly brain to control and stabilize a mobile robot. This method enables real-time neuronal control of robotic systems.

ai_agent_IP

ai_agent

maintenance

instantTrialDurationMinutes

isBiopharma

enableNewMyJoveSectionDesign

educationFeatures

seoMetaApi

ai_copilot_institution_blacklist

ai_chat_institution_blacklist

freeAccessWindow

ssoRoleCapture

scheduledMaintenanceBanner

joveLabsFeatures

disable_autoplay_for_bots

extended_trial_duration

5 minutes

K. Peterson was supported by a PhD studentship from the Department of Bioengineering and funding from the US Air Force Research Labs.
N. Ejaz was supported by a PhD studentship from the Higher Education Commission Pakistan and funding from the US Air Force Research Labs.

1. Fly Preparation
<ol>
 <li>The first step in setting up the experiments involves preparing the fly so that no involuntary motion corrupts the stability of the neuronal recordings and that the head of the fly is correctly oriented with the visual stimulation equipment. To begin preparing the fly, cool it on ice and then use blunted cocktail sticks to hold the wings down, and fix the back of the fly to a piece of double-sided-tape on a microscope slide.</li>
 <li>Next, use an electro-cauterizing needle to apply bees wax to attach the wings to the slide and also to block the action of the flight motor. This step requires quick and accurate handling so that the fly does not warm up during the procedure.</li>
 <li>Now under the microscope, hold each leg with forceps and use a pair of small scissors to cut them off at the joints closest to the body. Repeat this for the proboscis. To prevent the fly from drying out, the holes must be sealed with wax.</li>
 <li>Next, cut one of the wings off and then turn the fly on its side. Remove any remaining pieces of wing, while leaving the calyptra covering the halteres, and seal the hole with wax. Repeat this procedure for the other wing.</li>
 <li>To stimulate a target neuron in a defined way, the fly's head has to be properly aligned with the computer monitors. To do this, you will need a customized holder that has a broad space for the fly's body and an appendage on one end with a notch cut where the fly's neck will be placed.</li>
 <li>Place the fly onto the holder with its neck in the notch, pressing it down while gluing the abdomen in place. Now place the fly holder in a stand so that you can see the front of the fly's head through the microscope.</li>
 <li>Viewing the fly with red light, an optical phenomenon called the pseudo-pupil can be seen in each eye. The pseudo-pupil provides a reference frame which can be used to align the fly's head with the stimulus (Franceschini 1975). If the pseudo-pupil assumes a certain shape, as shown by the image insert below, then the orientation of the fly's head is perfectly defined. <img src="/files/ftp_upload/1677/1677fig0.jpg" alt="Cellular structure analysis, microscopy image, alignment comparison, detailed morphology study." data-altupdatedat="1788235205000" data-alt="pseudo-pupil"> </li>
 <li>Use a micromanipulator to correctly orient the fly's head, and then use wax to glue it to the holder.</li>
 <li>Next, Press the thorax down flat and wax it to the holder. This allows the rear head capsule to be opened so that electrodes can be inserted into the fly brain.</li>
 <li>Use a micro scalpel or a fine injection needle to carefully cut a window into the cuticle of the right head capsule. Be careful not to cut the neural tissue right underneath the cuticle. Once the piece of cuticle is removed, add a few drops of Ringer solution.</li>
 <li>Use forceps to remove any floating hairs, fat deposits, or muscle tissue that may cover the lobula plate. The lobula plate can be identified by a characteristic branching pattern of silvery trachea that covers its posterior surface. </li>
 <li>Cut a small hole into the cuticle of the left rear head capsule for positioning a reference electrode. With the fly prepared, lets see how to position the recording electrode.</li>
</ol>
2. Positioning the Recording Electrode
<ol>
 <li>With the fly prepared, let’s proceed to locating and recording signals from the H1 neuron. The recording electrode must be placed in close proximity to the H1 neuron. The H1 neuron mainly responds to horizontal back-to-front motion presented to its receptive field (Krapp et al. 2001).</li>
 <li>To position the recording electrode, use the trachea as a visual landmark. Initially, place the electrode between the uppermost trachea.</li>
 <li>It helps to use an audio amplifier to convert the recorded electrical potentials into acoustic signals. Each individual spike is turned into a characteristic clicking sound. The closer the electrode gets to an individual neuron, the clearer the clicking sound becomes.</li>
 <li>To identify the H1 neuron by means of its motion preference, stimulate it with motion in the horizontal direction. With the recording electrode in place, lets move on to visual stimulation and recordings.</li>
</ol>
3. Visual Stimulation and Recordings
<ol>
 <li>The closed-loop experiments are setup such that such that stimulation of the H1 neuron results in the robot compensating for the movement of the turn-table. To begin, place a fly in front of two CRT computer monitors. Because the fly visual system is 10 times faster than the human’s, the monitors must display 200 frames per second. The monitors and other electrical equipment must be electromagnetically shielded to minimize external noise in the measured neuronal signal.</li>
 <li>Position the centres of the monitors at +/- 45 degrees relative to the fly's orientation. As seen from the fly's eye equator, each monitor subtends an angle of +/-25 degrees in the horizontal, and +/-19 degrees in the vertical plane.</li>
 <li>Synchronized input to the computer monitors is provided by two video cameras mounted on a small, two-wheeled ASURO robot that has been modified for the experiment.</li>
 <li>Position the robot on a turn-table within a cylindrical area whose walls are lined with a pattern of vertically oriented, black and white stripes. By rotating the turn-table in the horizontal plane, the movements of the robot are limited to only one degree of freedom.</li>
 <li>Initially both the turn-table and robot are at rest. When the turn-table starts moving, its rotation carries the robot, in the same direction and the video cameras record the relative motion between the robot and the striped pattern of the arena.</li>
 <li>The battery-powered video cameras on the robot are mounted at an orientation of +/- 45 degrees. They capture 200 images per second to match the frame rate of the computer monitors in front of the fly.</li>
 <li>Log the images presented to the computer monitors at 200 frames per second at a resolution of 640 x 480 (gray-scale).</li>
 <li>While the fly is watching the movements of the striped pattern, record the band-passed filtered (for example, between 300 and 2 kHz) electrical signals with a Digital Acquisition board using a sampling rate of at least 10 kHz.</li>
 <li>A threshold is applied to the band-passed filtered electrical signals to separate the spikes from the background activity. A causal, half-gaussian filter is convolved with the spikes to obtain a smooth spiking activity estimate for the H1 cell. </li>
 <li>To close the loop of the brain machine interface, a control algorithm is used to convert the spike rate of the H1 cell to a robot speed which is fed back via a Bluetooth interface to control the two DC motors driving the wheels of the robot.</li>
 <li> Pure sine waves are chosen as velocity profiles for the turn-table. The sine waves have a DC-offset such that the turn-table only rotates in the direction which stimulates the H1 neuron along its preferred direction. Stimulation of the H1 neuron results in the robot compensating for the movement of the turn-table. <img src="/files/ftp_upload/1677/1677fig1.jpg" alt="Grating pattern diagram with robot turntable; charts show spiking response and compensation speed." data-altupdatedat="1788235205000" data-alt="Figure 1"> Figure 1: Closed loop setup. In our setup, spiking activity of the left H1 cell is used to control the motion of a robot mounted on a turntable. Visual image motion generated as a result of relative motion between the robot and the turn-table is captured via high-speed cameras and displayed on two CRT monitors in front of the fly. H1 spiking activity from the left hemisphere is used to estimate the real-time spike rate which then uses a control law to calculate a compensation speed for the robot. The robot counter-rotation stabilizes the visual image motion observed by the fly during closed-loop control. </li>
</ol>
4. Representative Outcome and Results
<ol>
 <li>When set up correctly, visual stabilization is achieved when the counter-rotation of the robot matches the rotation of the turn-table, resulting in little or no pattern movement on the computer monitors. The overall performance of the system depends on the control algorithm being used to close the loop.</li>
 <li>The first algorithm we test is a proportional controller (figure 2) where the updated robot speed is proportional to the difference in angular velocities between the robot, ωr, and the turn-table, ωp. Different values for the static gain, Kp, and input frequencies for the turn-table signal, ωp, are chosen to test the performance of the controller. <img src="/files/ftp_upload/1677/1677fig2.jpg" alt="Proportional control system diagram for robot speed regulation using angular velocity feedback." data-altupdatedat="1788235205000" data-alt="Figure 2"> Figure 2: Proportional controller. (a) Relative motion between the robot and the turn-table in the preferred direction stimulates the H1 cell to give the spike rate, F. This spike rate is converted into a speed error, E, and a proportional controller is used to estimate the updated robot speed, vr(t+1). The spike rate, F, is converted to the speed error, E, based on whether it is less than or greater than the spontaneous spike rate, Fspont. The spike rate to speed error conversion is done by projecting F over a cosine (interval [π,0]) to account for spiking rate threshold nonlinearities. The constants 70 and 150 are used to match the 8-bit input speed of
the robot to the minimum and maximum angular velocities of the turn-table. (b) Block diagram showing closed-loop system using a proportional controller. The input to the system is a sinosuidal modulation of the turn-table angular velocity, ωp(t), and the corresponding robot response ωr(t+1) is recorded. </li>
 <li>Sample traces for ωp and ωr are shown here for Kp=1 and an input frequency of 0.6 Hz for ωp (see figure 3). The robot (in green) follows the turn-table (in blue) with a lag and a smaller peak-amplitude. The horizontal component of the pattern motion that stimulates the H1 cell is shown below (in red). <img src="/files/ftp_upload/1677/1677fig3.jpg" alt="Platform vs. robot angular velocities graph; optic flow comparison; ωp−ωr equation; velocity analysis." data-altupdatedat="1788235205000" data-alt="Figure 3"> Figure 3: Closed loop responses. (a) Angular velocities for the turn-table (blue), ωp, and robot (green), ωr, at input frequency = 0.6 Hz. (b) Horizontal optic flow is shown as calculated from logged images (red). Pyramidal Lucas Kanade method (3 pyramid levels) is used to compute the optic flow field between successive image frames. Horizontal angular velocity is calculated by summing the projections of individual vectors in the flow fields onto the horizontal unit vector i. Horizontal back-to-front motion that excites the H1 cell is referred to as PD (preferred direction) while horizontal front-to-back motion that inhibits the H1 cell is referred to as ND (null direction). </li>
 <li>Input frequencies for the turn-table signal, ωp, are chosen between 0.03-3 Hz and the corresponding robot signal, ωr, is recorded. Both signals are transformed into the frequency domain by a fast Fourier transform (see figure 4) and the amplitude and phase values are calculated at the input frequency. <img src="/files/ftp_upload/1677/1677fig4.jpg" alt="Angular velocity vs frequency FFT graph; turn-table and robot analysis, time domain to frequency domain." data-altupdatedat="1788235205000" data-alt="Figure 4"> Figure 4: Frequency response. Angular velocity signals for the turn-table, ωp, and robot, ωr, are transformed into the frequency domain using the fast Fourier transform (FFT) method to calculate amplitude and phase components at the input frequency. Phase components of FFT are not shown in the figure. </li>
 <li>The Bode magnitude plot for the proportional controller with Kp=1 shows the response of the system over the tested input frequencies (see figure 5-a). The performance of the controller generally decreases with increasing frequencies. The slightly increased gain at 1 Hz is as a result of oscillations in the robot signal due to using only one H1-cell whose dynamic (output) range mainly covers horizontal back-to-front motion. <img src="/files/ftp_upload/1677/1677fig5.jpg" alt="Frequency response graphs; gain (dB) vs. frequency (Hz), phase (degrees) vs. frequency, data analysis." data-altupdatedat="1788235205000" data-alt="Figure 5"> Figure 5: Proportional controller performance. The Bode magnitude and phase plots for the proportional controller (averaged over 8 flies), static gain Kp = 1.0 (a) the bode magnitude plot roughly follows a low pass filter characteristic. The slightly increased gain at 1 Hz is as a result of oscillations in the robot signal, ωr, due to using only one H1-cell whose dynamic (output) range mainly covers horizontal back to front motion. The number of oscillations in the robot signal, ωr, decrease with increasing input frequencies leading to slightly increased gain at these frequencies. (b) the Bode phase plot is less than 180° for input frequencies ≤ 1 Hz and approaches instability at 3 Hz. Beyond a certain input frequency, the controller becomes instable due to the kinematics of the robot. This instability only occurs beyond the known optimal response range of the fly visual system (Warzecha et al. 1999). </li>
 <li>The Bode phase plot (see figure 5-b) shows a controller phase lag less than Π for input frequencies &lt; 0.6 Hz. This shows that the controller is stable for frequencies &lt; 0.6 Hz and unstable for input frequencies ≥ 1 Hz.</li>
 <li>The performance of the proportional controller with a static Kp (in blue) was compared with an adaptive controller (in red), where the value for Kp is updated every 50 ms based on the peak spike rate, Fmax, calculated over time interval [t-500ms - t] (see figure 6). As a result of the large integration time window, the proportional controller performs better than the adaptive controller for the parameter range tested (see figure 7-a). The initial time integration window of 500 ms was chosen for technical reasons related to the robot platform we are using. The adaptive controller had a similar phase characteristic as the proportional controller (see figure 7-b). 
<img src="/files/ftp_upload/1677/1677fig6.jpg" alt="Adaptive parameter estimation graph, dynamic gain estimation formula, spike rate analysis." data-altupdatedat="1788235205000" data-alt="Figure 6"> Figure 6: Adaptive controller gain. The proportional controller uses a static gain, Kp, whereas the adaptive controller estimates the gain continuously during closed loop control. The dynamic gain, Kp, is inversely proportional to the maximum spike rate, Fmax, over the interval t-500ms ≤ τ ≤ t. Figure shows three instances where Fmax is estimated over time. Based on estimated values of Fmax, Kp is highest during time window 2 (green) and lowest during time window 3 (orange). 
<img src="/files/ftp_upload/1677/1677fig7.jpg" alt="Frequency response analysis, gain and phase diagrams, comparing proportional vs adaptive control." data-altupdatedat="1788235205000" data-alt="Figure 7"> Figure 7: Proportional vs. adaptive controller. Bode magnitude and phase plots for the proportional (Kp = 1) and adaptive controller (a) the adaptive controller updates the value of Kp every 50ms based on the peak spike rate estimate Fmax over the last 500ms. The proportional controller gain plot (blue) is higher than that of the adaptive controller (red) indicating that it is performing better at all input frequencies. (b) the Bode phase plots for both controllers are similar with a significant difference at f = 0.3 Hz. Both controllers approach instability at 3 Hz. Significantly different gain and phase values are indicated by an asterix (Wilcoxon rank sum method, p = 0.05). </li>
 <li>The grating pattern around the turntable was removed and the lab environment was used as an approximation of naturalistic visual input for the fly H1-cell. On average, the Bode magnitude plot (see figure 8-a) for the naturalistic visual input (in blue) showed slightly higher gains than the one with grating visual input (in red) probably because the wider range of spatial frequencies in naturalistic visual images is exploited. The Bode phase plot characteristics for grating vs naturalistic visual inputs were similar (see figure 8-b). <img src="/files/ftp_upload/1677/1677fig8.jpg" alt="Frequency response graph showing gain and phase in dB vs Hz for lab and striped environments." data-altupdatedat="1788235205000" data-alt="Figure 8"> Figure 8: Stripped pattern vs. lab environment. The Bode magnitude and phase plots for the proportional controller when presented with stripped patterns (red) vs lab environment (blue) visual images under closed loop (a) the Bode magnitude plot for when lab environment images are used is marginally higher than when stripped pattern is used (except at f = 0.1 Hz) indicating better performance under such stimulus. (b) the Bode phase plot under both visual conditions follow the same pattern, with both approaching instability at 3 Hz. Significantly different gain and phase values are indicated by an asterix (Wilcoxon rank sum method, p = 0.05). </li>
</ol>

<ol><li>Chiappe, E. M., Seelig, J. D., Reiser, M. B., Jayaraman, V. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Walking+modulates+speed+sensitivity+in+Drosophila+motion+vision%5BTitle%5D)+AND+%22Curr.+Biol%22%5BJournal%5D)">Walking modulates speed sensitivity in Drosophila motion vision.</a> Curr. Biol. 20, 1470-1475 (2010).</li>
<li>Franceschini, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Sampling+of+the+visual+environment+by+the+compound+eye+of+the+fly%3A+fundamentals+and+applications%5BTitle%5D)+AND+%22Photoreceptor+optics%22%5BJournal%5D)">Sampling of the visual environment by the compound eye of the fly: fundamentals and applications.</a> Photoreceptor optics. Snyder, A. W., Menzel, R. , Springer. Berlin Heidelberg New York. 98-125 (1975).</li>
<li>Karmeier, K., Tabor, R., Egelhaaf, M., Krapp, H. G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Early+visual+experience+and+the+receptive-field+organization+of+optic+flow+processing+interneurons+in+the+fly+motion+pathway%5BTitle%5D)+AND+%22Vis.+Neurosci%22%5BJournal%5D)">Early visual experience and the receptive-field organization of optic flow processing interneurons in the fly motion pathway.</a> Vis. Neurosci. 18, 1-8 (2001).</li>
<li>Krapp, H. G., Hengstenberg, B., Hengstenberg, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Dendritic+structure+and+receptive-field+organization+of+optic+flow+processing+interneurons+in+the+fly%5BTitle%5D)+AND+%22Jour.+of+Neurophys%22%5BJournal%5D)">Dendritic structure and receptive-field organization of optic flow processing interneurons in the fly.</a> Jour. of Neurophys. 79, 1902-1917 (1998).</li>
<li>Krapp, H. G., Hengstenberg, R., Egelhaaf, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Binocular+contributions+to+optic+flow+processing+in+the+fly+visual+system%5BTitle%5D)+AND+%22Jour.+of+Neurophys%22%5BJournal%5D)">Binocular contributions to optic flow processing in the fly visual system.</a> Jour. of Neurophys. 85, 724-734 (2001).</li>
<li>Longden, K. D., Krapp, H. G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Octopaminergic+modulation+of+temporal+frequency+coding+in+an+identified+optic-flow+processing+interneuron%5BTitle%5D)+AND+%22Frontiers+in+Systems+Neuroscience%22%5BJournal%5D)">Octopaminergic modulation of temporal frequency coding in an identified optic-flow processing interneuron.</a> Frontiers in Systems Neuroscience. 4, 153-153 (2010).</li>
<li>Bender, J. A., Dickinson, M. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+comparison+of+visual+and+haltere-mediated+feedback+in+the+control+of+body+saccades+in+Drosophila+melanogaster%5BTitle%5D)+AND+%22J.+Exp.+Bio%22%5BJournal%5D)">A comparison of visual and haltere-mediated feedback in the control of body saccades in Drosophila melanogaster.</a> J. Exp. Bio. 209, 4597-4606 (2006).</li>
<li>Longden, K. D., Krapp, H. G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((State-dependent+performance+of+optic+flow-processing+interneurons%5BTitle%5D)+AND+%22J.+Neurophysiol%22%5BJournal%5D)">State-dependent performance of optic flow-processing interneurons.</a> J. Neurophysiol. 102, 3606-3618 (2009).</li>
<li>Maimon, G., Straw, A. D., Dickinson, M. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Active+flight+increases+the+gain+of+visual+motion+processing+in%5BTitle%5D)+AND+%22%22%5BJournal%5D)">Active flight increases the gain of visual motion processing in.</a> 13, 393-399 (2010).</li>
<li>Petrovitz, R., Dahmen, H., Egelhaaf, M., Krapp, H. G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Arrangement+of+optical+axes+and+spatial+resolution+in+the+compound+eye+of+the+female+blowfly+Calliphora%5BTitle%5D)+AND+%22J+Comp+Physiol+A%22%5BJournal%5D)">Arrangement of optical axes and spatial resolution in the compound eye of the female blowfly Calliphora.</a> J Comp Physiol A. 186, 737-746 (2000).</li>
<li>Warzecha, A. K., Horstmann, W., Egelhaaf, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Temperature-dependence+of+neuronal+performance+in+the+motion+pathway+of+the+blowfly+Calliphora+Erythrocephala%5BTitle%5D)+AND+%22J+Exp+Biology%22%5BJournal%5D)">Temperature-dependence of neuronal performance in the motion pathway of the blowfly Calliphora Erythrocephala.</a> J Exp Biology. 202, 3161-3170 (1999).</li>
</ol>

<ol>
 <li>Fly dissection needs to be carried out carefully making sure that we correctly orient the fly with respect to the computer monitors.</li>
 <li>Separating the spikes from the H1 neuron from the spikes from all other neurons to get a good signal to noise ratio which can then be used to control the robot reliably.</li>
 <li>Care should be taken to prevent any neural tissue from drying out during the course of the experiment.</li>
 <li>The cameras are connected to computers using an Ethernet cable. Care should be taken so that they are not over-wound during the experiment as that would affect the rotation of the robot. </li>
 <li>Currently we are using the spike activity of one H1 to setup a stabilization task in only one direction of yaw rotation. We can add a second electrode to get signals from both left and right H1's so that we can study stabilization control algorithms in both directions of yaw rotation.</li>
 <li>We can remove the arena containing the vertically oriented black and white stripes and use the lab environment as the visual stimulus for the fly. This will allow us to study closed-loop performance with naturalistic images.</li>
 <li>The robot can be removed from the turn-table and allowed to move around the lab environment while in closed-loop control. This will allow us to investigate control algorithms involved in collision avoidance. </li>
 <li>The wires connecting the cameras to the computers can be removed by implementing a wireless transmission system, giving us a completely unrestrained robotic setup.</li>
 <li>The performance measures of the different control algorithms will give us an understanding of how different strategies are able to cope with non-stationary and variable neuronal signals. This knowledge can then be applied to different clinical and non-clinical brain-machine interfaces.</li>
 <li>This experimental setup is the first step towards recording neuronal signals from behaving animals. Our goal is to place the fly on the robot and use its neural activity for closed-loop control. In such a setup, we would be able to record neural activity from the fly while it is receives multisensory stimulation as a result of the motion of the robot.</li>
<li>Record only signals from one cell. In the configuration we should you, the most suitable cell would be the H1-cell. Isolating the responses of the H1 neuron from other neurons with similar receptive fields, e.g. H2, is important to maintain a good signal to noise ratio for the neuronal recordings. The responses of the H1 and H2 neurons can be discriminated by noting that the H2 has characteristically lower spontaneous and mean spiking rate than the H1 neuron. The neurons can also be discriminated on the basis of their morphology (Krapp et al. 2001).</li>
<li>Our system allows us to compare the closed-loop gain achieved by our fly-robot interface on an image stabilization task, and to compare its performance with closed-loop optomotor gains observed in previous experiments on flies (Bender &amp; Dickinson 2006, Warzecha et al. 1996, Heisenberg &amp; Wolf, 1990). Additionally, it allows us to investigate state-dependant changes in visual information processing under closed-loop conditions (Chiappe et al. 2010, Maimon et al. 2010, Longden &amp; Krapp 2009, Longden &amp; Krapp 2010). </li>
</ol>

<table><tbody><tr><td>RosetteSep&amp;#174; human T cell enrichment cocktail</td><td>Stem Cell Technologies</td><td>15061</td><td></td></tr><tr><td>RosetteSep&amp;#174; density medium</td><td>Stem Cell Technologies</td><td>15705</td><td></td></tr><tr><td>RPMI 1640 medium w/glutamine/HEPES</td><td>Fisher Scientific</td><td>SH3025501</td><td></td></tr><tr><td>Fetal calf serum</td><td>Omega Scientific</td><td>FB-01</td><td></td></tr><tr><td>GlutaMAX&amp;#8482;-I</td><td>Invitrogen</td><td>35050</td><td></td></tr><tr><td>RPMI 1640 vitamin solution</td><td>Sigma-Aldrich</td><td>7256</td><td></td></tr><tr><td>RPMI 1640 amino acids solution</td><td>Sigma-Aldrich</td><td>R7131</td><td></td></tr><tr><td>Sodium pyruvate</td><td>Sigma-Aldrich</td><td>S8636</td><td></td></tr><tr><td>&amp;#946;-Mercapt&amp;#8211;thanol</td><td>Sigma-Aldrich</td><td>M7522</td><td></td></tr><tr><td>BAPTA</td><td>Sigma-Aldrich</td><td>A4926</td><td></td></tr><tr><td>Poly-l-lysine hydrobromide</td><td>Sigma-Aldrich</td><td>P2636</td><td></td></tr><tr><td>Thapsigargin</td><td>Calbiochem</td><td>586005</td><td></td></tr><tr><td>Sylgard&amp;#174; 184 silicon elastomer kit</td><td>Dow Corning</td><td>3097358-1004</td><td></td></tr><tr><td>HIPEC&amp;#174; R6101 semiconductor protective coating</td><td>Dow Corning</td><td></td><td></td></tr><tr><td>EPC 10 patch clamp amplifier</td><td>HEKA Instruments</td><td></td><td></td></tr><tr><td>Motorized micromanipulator </td><td>Sutter Instrument Co.</td><td>MP-285</td><td></td></tr><tr><td>Olympus 1X71 inverted microscope with 40X oil immersion objective</td><td>Olympus Corporation</td><td>1X71</td><td></td></tr><tr><td>Patch pipette puller</td><td>Sutter Instrument Co.</td><td>P-97</td><td></td></tr><tr><td>Borosilicate tubing with filament (O.D.: 1.5 mm and I.D.: 1.10 mm)</td><td>Sutter Instrument Co.</td><td>BF150-110-7.5</td><td></td></tr><tr><td>Microforge</td><td>Narishige International</td><td>MF-830</td><td></td></tr><tr><td>Silicon O-rings </td><td>McMaster-Carr</td><td>111 S70</td><td></td></tr><tr><td>Microscope cover glass</td><td>Fisher Scientific</td><td>12-545-102 25 mm 25CIR-1</td><td></td></tr><tr><td>Pulse software</td><td>HEKA Instruments</td><td></td><td></td></tr><tr><td>Origin scientific graphing and analysis software</td><td>OriginLab</td><td></td><td></td></tr><tr><td colspan="4">&lt;ol&gt;&lt;li&gt;1)High-Speed CRT Displays (LG Studioworks 221U)&lt;/li&gt;&lt;li&gt;High-Speed Cameras (Prosilica, GC640)&lt;/li&gt;&lt;li&gt;Robot Platform (ASURO with custom modifications)&lt;/li&gt;&lt;li&gt;Serial Bluetooth Modem (Bluesmirf)&lt;/li&gt;&lt;li&gt;Micro Step Drive (Applied Motion Systems, ST5-SI)&lt;/li&gt;&lt;li&gt;Stepper Motor (Sanyo Denki, 103H6704-0140)&lt;/li&gt;&lt;li&gt;Tungsten electrodes (fh-co.com - Item code - UEW SHG SE 3P1M)&lt;/li&gt;&lt;li&gt;Extracellular Amplifier (NPI EXT 10-2F)&lt;/li&gt;&lt;li&gt;Motorized Micromanipulator (Scientifica, PS-700-Z)&lt;/li&gt;&lt;li&gt;Stereo Microscope (Leica, MZ95)&lt;/li&gt;&lt;/ol&gt;</td></tr></tbody></table>

an experimental platform to study the closed-loop performance of brain-machine interfaces

An Experimental Platform to Study the Closed-loop Performance of Brain-machine Interfaces

The non-stationary nature and variability of neuronal signals is a fundamental problem in brain-machine interfacing. We developed a brain-machine interface to assess the robustness of different control-laws applied to a closed-loop image stabilization task. Taking advantage of the well-characterized fly visuomotor pathway we record the electrical activity from an identified, motion-sensitive neuron, H1, to control the yaw rotation of a two-wheeled robot. The robot is equipped with 2 high-speed video cameras providing visual motion input to a fly placed in front of 2 CRT computer monitors. The activity of the H1 neuron indicates the direction and relative speed of the robot's rotation. The neural activity is filtered and fed back into the steering system of the robot by means of proportional and proportional/adaptive control. Our goal is to test and optimize the performance of various control laws under closed-loop conditions for a broader application also in other brain machine interfaces.

We use a closed-loop fly-machine interface to investigate general principles in neuronal control.

The non-stationary nature and variability of neuronal signals is a fundamental problem in brain-machine interfacing. We developed a brain-machine ...

We use the activity of a visual inter neuron in the fly brain, the H one cell, to control the motors of a mobile robot. The robot is placed on a constantly rotating turntable and the cell's activity is used to stabilize the robot relative to the environment against external movements. Images of pattern motion recorded from the robot are sampled and sent to two computer CRT monitors, which are positioned in front of the fly.The signals of the H one cell measured in spikes per second indicate the speed of the pattern motion. Different control laws are then applied to convert the recorded spikes per second into a control signal for the robot's motors. Hi, I'm Navita Josh from Holger Crops Lab in the department of Bioengineering Imperial College London.I'm Chris Peterson, also from Holger Cups Lab. Hi, and I am Holger Cup. Today we'll show you a procedure for creating a brain machine interface between individual cells in the fly visual system and a robot.We use this procedure to test the performance of different control strategies using neuronal signals to control robotic systems under closed lip conditions. So let's get started. To begin preparing the fly, cool it on ice, and then use blunted cocktail sticks to hold the wings down and fix the back of the fly to a piece of double-sided tape on a microscope Slide next, use B wax to attach the wings to the slide and also to block the action of the flight motor.This step requires quick and accurate handling so that the fly does not warm up during the procedure. Now, under the microscope, hold each leg with forceps and use a pair of small scissors to cut them off at the joints closest to the body. Repeat this for the proboscis.To prevent the fly from drying out, the holes must be sealed with wax. Next, cut one of the wings off, and then turn the fly on its side. Remove any remaining pieces of wing while leaving the katra, covering the Hal tears and seal the hole with wax.Repeat this procedure for the other wing. To stimulate a target neuron in a defined way, the fly's head has to be properly aligned with the computer monitors. To do this, you will need a customized holder that has a broad space for the fly's body and an appendage on one end with a notch cut where the fly's neck will be placed.Place the fly onto the holder with its neck in the notch, pressing it down while gluing the abdomen in place. Now, place the fly holder in a stand so that you can see the front of the fly's head through the microscope. Viewing the fly with red light and optical phenomenon called the pseudo pupil can be seen in each eye.If the pseudo pupil assumes a certain shape, then the orientation of the fly's head is perfectly defined. Use a micro manipulator to correctly orient the fly's head and then use wax to glue it to the holder. Next, press the thorax down flat and wax it to the holder.This allows the rear head capsule to be opened so that electrodes can be inserted into the fly Brain use a micro scalpel or a fine injection needle to carefully cut a window into the cuticle of the right head capsule. Be careful not to cut the neural tissue right underneath the cuticle. Once the piece of cuticle is removed, add a few drops of ringer solution.Use forceps to remove any floating hairs, fat deposits, or muscle tissue that may cover the LOA plate. The LOA plate can be identified by a characteristic branching pattern of silvery trachea that covers its posterior surface. Cut a small hole into the cuticle of the left rear head capsule for positioning a reference electrode with the flying prepared.Let's see how to position the recording electrode. The recording electrode must be placed in close proximity to the H one neuron. The H one neuron mainly responds to horizontal back to front motion presented to its receptive field.To position the recording electrode, use the trachea as a visual landmark. Initially place the electrode between the uppermost trachea. It helps to use an audio amplifier to convert the recorded electrical potentials into acoustic signals.Each individual spike is turned into a characteristic clicking sound. The closer the electrode gets to an individual neuron, the clearer the clicking sound becomes. To identify the H one neuron by means of its motion preference, stimulate it with motion in the horizontal direction.With the recording electrode in place, let's move on to visual stimulation and recordings. To begin place a fly in front of two CRT computer monitors. Because the fly visual system is 10 times faster than the humans, the monitors must display 200 frames per second position.The centers of the monitors at plus or minus 45 degrees relative to the flies orientation. As seen from the flies eye equator, each monitor sub tens an angle of plus or minus 25 degrees in the horizontal and plus or minus 19 degrees in the vertical plane. Input to the computer monitors is provided by two video cameras mounted on a small two-wheeled SRO robot that has been modified for the experiment.Position the robot on a turntable within a cylindrical area whose walls are lined with a pattern of vertically oriented black and white stripes. By rotating the turntable in the horizontal plane, the movements of the robot are limited to only one degree of freedom. Initially, both the turntable and robot are at rest.When the turntable starts moving its rotation carries the robot in the same direction, and the video cameras record the relative motion between the robot and the striped pattern of the arena. The battery powered video cameras on the robot are mounted at an orientation of plus or minus 45 degrees. They capture 200 images per second to match the frame rate of the computer monitors in front of the fly log.The images presented to the computer monitors at 200 frames per second at a resolution of six 40 by four 80 gray scale. While the fly is watching the movements of the striped pattern record, the band passed filtered. For example, between 302 kilohertz electrical signals with a digital acquisition board using a sampling rate of at least 10 kilohertz, A threshold is applied to the band passed filtered electrical signals to separate the spikes from the background activity.A causal half Gaussian filter is convolved with the spikes to obtain a smooth spiking activity estimate for the H one cell to close the loop of the brain machine interface. A control algorithm is used to convert the spiking rate of the H one cell to a robot speed, which is fed back via a Bluetooth interface to control the two DC motors driving the wheels of the robot. Pure sign waves are chosen as velocity profiles for the turntable.The sign waves have a DC offset such that the turntable only rotates in the direction which stimulates the H one neuron along its preferred direction. The whole control system is set up such that stimulation of the H one neuron results in the robot compensating for the movement of the turntable when set up correctly. Visual stabilization is achieved when the counter rotation of the robot matches the rotation of the turntable, resulting in little or no pattern movement on the computer monitors.The overall performance of the system depends on the control algorithm being used to close the loop. The first algorithm we test is a proportional controller where the updated robot speed is proportional to the difference in angular velocities between the robot omega R and the turntable Omega P.Different values for the static gain. KP and input frequencies for the turntable signal omega P are chosen to test the performance of the controller.Sample traces for omega P and omega R are shown here for KP equals one and an input frequency of 0.6 hertz for omega P, the robot in green follows the turntable in blue with a lag and a smaller peak amplitude. The horizontal component of the pattern motion that stimulates the H one cell is shown on the right in red input frequencies for the turntable signal. Omega P are chosen between 0.03 and three hertz, and the corresponding robot signal omega R is recorded.Both signals are transformed into the frequency domain by a fast four-year transform, and the amplitude and phase values are calculated at the input frequency. The BO d magnitude plot for the proportional controller with KP equals one shows the response of the system over the tested input frequencies. The performance of the controller generally decreases with increasing frequencies.The slightly increased gain at one hertz is as a result of oscillations in the robot signal due to using only one H one cell whose dynamic output range mainly covers horizontal back to front motion. The Bodhi phase plot shows a controller phase lag less than PI for input frequencies less than 0.6 hertz. This shows that the controller is stable for frequencies less than 0.6 hertz and unstable for input frequencies greater than or equal to one hertz.The performance of the proportional controller with a static KP was compared with an adaptive controller where the value for KP is updated every 50 milliseconds. Based on the peak spike rate. F max calculated over time interval T minus 500 milliseconds to T.As a result of the large integration time window, the proportional controller performs better than the adaptive controller for the parameter range tested, the adaptive controller had a similar phase characteristic as the proportional controller, the grading pattern around the turntable was removed and the lab environment was used as an approximation of naturalistic visual input for the fly H one cell.On average, the Bodhi magnitude plot for the naturalistic visual input showed slightly higher gains than the one with grading visual input, probably because the wider range of spatial frequencies in naturalistic visual images is exploited. The Bodhi phase plot characteristics for grading versus naturalistic visual inputs were similar. We've just shown you how to create a brain machine interface between a cell and the five visual system and a robot.There are a few critical steps during this procedure. First, avoid deep cuts during opening the head capsule to prevent from injuring the brain. Second, position the electrode carefully so that it records from only one cell we record from the H one cell.Third, keep the brain moist at all times and prevent it from drying out. So that's it. Thanks for watching and good luck with the experiments.

an-experimental-platform-to-study-closed-loop-performance-brain

Research

JoVE Journal

Neuroscience

We present a method for collecting electrocorticographic signals for research purposes from humans who are undergoing invasive epilepsy monitoring.  We show how to use the BCI2000 software platform for data collection, signal processing and stimulus presentation.  Specifically, we demonstrate SIGFRIED, a BCI2000-based tool for real-time functional brain mapping.

Recording Human Electrocorticographic (ECoG) Signals for Neuroscientific Research and Real-time Functional Cortical Mapping

Development of an effective brain-machine-interface (BMI) system for restoration and rehabilitation of bipedal locomotion requires accurate decoding of user's intent.  Here we present a novel experimental protocol and data collection technique for simultaneous non-invasive acquisition of neural activity, muscle activity, and whole-body kinematics during various locomotion tasks and conditions.

Simultaneous Scalp Electroencephalography (EEG), Electromyography (EMG), and Whole-body Segmental Inertial Recording for Multi-modal Neural Decoding

In this paper, an experimental framework to perform closed-loop experiments is presented, in which information processing (i.e., coding and decoding) and learning of neuronal assemblies are studied during the continuous interaction with a robotic body.

Closed-loop Neuro-robotic Experiments to Test Computational Properties of Neuronal Networks

Monitoring brain activity during upright motor tasks is of great value when investigating the neural source of movement disorders. Here, we demonstrate a protocol that combines functional near infrared spectroscopy with continuous monitoring of muscle and kinematic activity during 4 types of motor tasks.

Functional Near Infrared Spectroscopy of the Sensory and Motor Brain Regions with Simultaneous Kinematic and EMG Monitoring During Motor Tasks

With this experiment, one might be able to detect consciousness in people with disorders of consciousness. Furthermore, the approach can create a simple communication channel that enables people to give simple YES/NO answers to questions.

Assessment and Communication for People with Disorders of Consciousness

We present protocols and methods of analyses to build co-adaptive interfaces that stream, parameterize, analyze, and modify human body and heart signals in close-loop. This setup interfaces signals derived from the peripheral and central nervous systems of the person with external sensory inputs to help track biophysical change.

Real-Time Proxy-Control of Re-Parameterized Peripheral Signals using a Close-Loop Interface

This paper discusses how to build a brain-computer interface by relying on consumer-grade equipment and steady-state visually evoked potentials. For this, a single-channel electroencephalograph exploiting dry electrodes was integrated with augmented reality glasses for stimuli presentation and output data visualization. The final system was non-invasive, wearable, and portable.

A Single-Channel and Non-Invasive Wearable Brain-Computer Interface for Industry and Healthcare

The purpose of this study is to provide an important reference for the standard clinical operation of motor imagery brain-computer interface (MI-BCI) for upper limb motor dysfunction after stroke.

Motor Imagery Brain-Computer Interface in Rehabilitation of Upper Limb Motor Dysfunction After Stroke

This study introduces a brain-computer interface (BCI) system for stroke patients, which combines electroencephalography and electrooculography signals to control an upper limb robotic hand, enhancing daily activities. The evaluation used the Berlin Bimanual Test for Stroke (BeBiTS).

Brain-Computer Interface-controlled Upper Limb Robotic System for Enhancing Daily Activities in Stroke Patients

This study presents a standardized and reproducible protocol for implementing the Spatial-Temporal-Frequency EEG analysis tool (STFEEG) for motor imagery EEG decoding, incorporating configurable spatial-temporal-frequency segmentation, Common Spatial Patterns (CSP)-based feature extraction, multiple classification algorithms, and visualization capabilities.

STFEEG-Tool: A Spatial-Temporal-Frequency EEG Analysis Tool for Motor Imagery Brain-Computer Interfaces

Learn how to create a closed-loop brain-machine interface using the H1 neuron in the fly brain to control and stabilize a mobile robot. This method enables real-time neuronal control of robotic systems.

An Experimental Platform to Study the Closed-loop Performance of Brain-machine Interfaces

Explore More Videos