Name: Capturing Dynamic Finger Gesturing with High-resolution Surface Electromyography and Computer Vision (Video) | JoVE Journal
Uploaded: 2025-03-28T14:05:00.000+00:00
Duration: 8 min 15 s
Description: Learn to capture dynamic finger gestures using synchronized high-resolution surface EMG and computer vision. This protocol enables accurate muscle-to-gesture mapping for prosthetic rehabilitation and human-computer interaction research.

ai_agent_IP

ai_agent

maintenance

instantTrialDurationMinutes

isBiopharma

enableNewMyJoveSectionDesign

educationFeatures

seoMetaApi

ai_copilot_institution_blacklist

ai_chat_institution_blacklist

freeAccessWindow

ssoRoleCapture

scheduledMaintenanceBanner

joveLabsFeatures

disable_autoplay_for_bots

extended_trial_duration

5 minutes

This project was partially funded with a grant from the ERC (OuterRetina) and ISF. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. We thank David Buzaglo, Cheni Hermon, Liron Ben Ari and Adi Ben Ari for their assistance with designing the original version of the protocol.

Healthy participants (n = 18, aged 18-32 years, both males and females) were recruited for this study, which was approved by the Tel Aviv University Ethics Review Board (Approval No. 0004877-3). The protocol adheres to the board's guidelines for research involving human participants. Informed consent was obtained from all participants in accordance with institutional requirements.
1. Experimenter briefing
<ol>
	<li>Ask participants to perform a series of 14 distinct finger gestures (see Figure 1) and repeat each gesture 7x in a random sequence. Ask them to maintain each gesture firmly for 5 s, followed by a 3 s rest period. The total duration for each session is 13:04 min.</li>
	<li>A large image of the gesture displayed on a computer screen is accompanied by a countdown timer to indicate gesture performance. During the rest period, ask the participant to look at the small image of the upcoming gesture shown, along with a timer indicating the remaining rest time. Two distinct beep sounds signal the start and end of each gesture, helping participants prepare for the next gesture.</li>
	<li>Ask each participant to execute the procedure in four different positions, similar to previously presented22: 
	Position 1: Participant standing. Hand down, straight, and relaxed. 
	Position 2: Participant sitting in the armchair. Hand extended forward at 90°, palm relaxed (a support device may be used). 
	Position 3: Hand folded upwards (with an elbow resting on the armchair), palm relaxed. 
	Position 4: Participant chooses one of the previous positions and may move the hand freely within the camera's detection range, monitored in real-time on a PC screen (see step 1.4 for more details).</li>
	<li>For each session, make the participant wear an electromyography device on the arm and position a hand-tracking camera towards them. Ask the participants to ensure that their palms always face the camera. The hand-tracking software is displayed on a separate screen so that both the participant and the conductor can verify that the hand is correctly recognized.</li>
	<li>For each position, adjust the hand-tracking camera's position and angle to ensure accurate hand recognition. Additionally, assess the quality of the signals from the electrodes using the spectrogram script.</li>
</ol>
<img alt="Dynamic finger gesturing; soft electrode array; real-time EMG, CV data; gesture recognition diagram." class="xfigimg" src="/files/ftp_upload/67766/67766fig01.jpg" data-altupdatedat="1788304762000" data-alt="Figure 1"> 
Figure 1: Schematic representation of the data collection process. The subject is equipped with a soft electrode array placed on the forearm (3), which captures high-resolution surface electromyography (sEMG) signals during gesture performance. The subject performs 14 different finger gestures presented in random order on a computer display (4). The EMG data is streamed wirelessly to a personal computer (PC) from the data acquisition unit (DAU; 1). Simultaneously, hand kinematic data (HKD) representing finger joint angles is captured using a hand-tracking camera (2). <a href="https://www.jove.com/files/ftp_upload/67766/67766fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
2. Setting up the data acquisition units
<ol>
	<li>Open Github repository at https://github.com/NeuroEngLabTAU/Fingers_Gestures_Recognition.git and follow the detailed instructions in the Installation section. Locate the primary Python file data_collection.py in the folder finger_pose_estimation/data_acquisition. Use this to run the experiment, use the script spectrogram.py to assess EMG signal quality before the experiment begins, and the script data_analysis.py for signal filtering and segmentation.</li>
	<li>Ensure that the EMG Data Acquisition Unit (DAU) is fully charged before each session and turn it on.</li>
	<li>Connect the DAU to the PC through Bluetooth using the dedicated application. Set the Bluetooth communication rate to 500 samples per second (S/s).</li>
	<li>Install and open the hand-tracking camera's software on the PC. Connect the hand-tracking camera to the PC using a cable.</li>
	<li>Use one screen to always display hand-tracking camera software. This way, the conductor and the participant will be able to ensure that the camera recognizes the hand correctly during the experiment.</li>
</ol>
3. Participant preparation
<ol>
	<li>Introduction and consent
	<ol>
		<li>Briefly explain the study's relevance and the experimental procedure to the participant. Obtain informed consent following institutional guidelines for research involving humans.</li>
	</ol>
	</li>
	<li>Electrode placement
	<ol>
		<li>Instruct the participant to flex their right hand by forming a strong fist. While the participant flexes, palpate the forearm by gently pressing along the muscle to identify the spot where muscle activation is most prominent. This location is easily identifiable by feeling the area where the muscle bulges during contraction.</li>
		<li>Optional: Prepare the identified skin area by cleaning with an alcohol fiber-less cloth, prep gel, or water and soap. Allow the area to air dry. Avoid excessive cleaning with alcohol, as it may dry the skin. This step is optional; see the discussion section.</li>
		<li>Peel off the white protective layer from the EMG electrode array and carefully attach the electrodes to the identified forearm area as determined in step 3.2.1. (see Figure 1). Ensure the adhesive tape is closer to the palm. Secure the electrode array to the skin by gently tapping.</li>
		<li>Once the electrode array is attached to the skin, peel off the transparent support layer.</li>
		<li>Insert the electrode array connector card into the DAU's connector socket. Attach the DAU to the adhesive tape next to the electrodes.</li>
		<li>Run custom Python spectrogram script Spectrogram.py to verify real-time signal quality. A window will appear displaying raw data (on the left) and frequency domain (on the right) for all electrodes (see Supplementary Figure 1 for reference).
		<ol>
			<li>Verify that all electrodes are detected and function properly and that the signal is clean from excessive noise and 50 Hz noise.</li>
			<li>If needed, reduce 50 Hz noise by moving away from electronic devices that may cause interference and unplugging unnecessary devices from the power. Allow time for the signal to stabilize.</li>
			<li>Verify EMG signal capture: instruct participant to place an elbow on the armchair and move fingers, then relax. Ensure that a clear EMG signal is displayed followed by static baseline noise.</li>
			<li>Close the script once signal verification is complete.</li>
		</ol>
		</li>
	</ol>
	</li>
	<li>Gesture and hand position review
	<ol>
		<li>Open Images folder by clicking on Finger_pose_estimation &gt; Data_acquisition. Review the gesture images with the participants.</li>
		<li>Ensure they understand each movement and can perform them accurately. Explain the four hand positions clearly to the participant.</li>
		<li>Instruct the participant on how to hold the hand before each session, ensuring proper posture and positioning.</li>
	</ol>
	</li>
	<li>Participant and camera positioning
	<ol>
		<li>For hand position 1, instruct the participant to stand straight about 1 m away from the table. Instruct the participant to hold the right hand down, straight and relaxed, with the palm facing the hand-tracking camera. Fix the hand-tracking camera on the table with a selfie stick and direct it to face the participant's hand.</li>
		<li>For hand position 2, instruct the participant to sit comfortably in an armchair positioned 40-70 cm from the monitors. Instruct the participant to extend the right hand forward at 90° with a relaxed palm facing the hand-tracking camera. Use a support device, if needed, to hold the hand stable. Place the hand-tracking camera on the table facing up. 
		NOTE: As the participant is requested to remain in a fixed posture, it is important to find a comfortable position they can maintain throughout the session.</li>
		<li>For hand position 3, instruct the participant to sit as described in step 3.4.2. Instruct the participant to fold the hand upwards while resting the elbow on the armchair. The palm should be relaxed, and the participant should face the hand-tracking camera. Fix the hand-tracking camera on the table facing the participant's hand (use a selfie stick if necessary). Ensure the participant's position is optimal for both viewing the screens and being within the camera's field of view.</li>
		<li>Continuously monitor the screen displaying hand-tracking data to ensure the camera detects the hand and fingers throughout the experiment. Optional: verify EMG signal quality (step 3.2.6.) in each hand position before starting the experiment.</li>
	</ol>
	</li>
</ol>
4. Data collection
<ol>
	<li>Running the experiment
	<ol>
		<li>Open Python and load data_collection.py. Verify the parameters num_repetition, gesture_duration, rest_duration are set as desired.
		<ol>
			<li>num_repetition: Define the number of times each gesture image is shown. For this experiment, set it to 7, meaning each image is shown 7 times. gesture_duration: Specify the duration (in s) for which the participant performs the hand gesture. For this experiment, set it to 5 s, determining how long each gesture image is displayed. Rest_duration: Specify the duration (in s) for which the participant relaxes their palm between gestures. For this experiment, set it to 3 s.</li>
		</ol>
		</li>
		<li>Adjust the hand-tracking camera position and angle to the participant's hand position.</li>
		<li>Run the data_collection.py script. A window will appear to enter the participant's details (serial number, age, sex, session number, and hand position). Complete this information and press OK to start the experiment automatically.</li>
	</ol>
	</li>
	<li>Data collection
	<ol>
		<li>For each session, record EMG and hand-tracking data which are automatically saved. Repeat the experiment 4x for each participant, once per hand position.</li>
	</ol>
	</li>
</ol>
5. End of experiment and post-experiment data handling
<ol>
	<li>As the experiment is completed, the data are automatically saved. Ensure data is saved in a folder labeled with the participant's serial number. Each session is stored in a subfolder named S# (e.g., S1), with four subfolders for each hand position P# (P1, P2, P3, and P4). The folder size for a single session is approximately 160 MB.</li>
	<li>If a participant completes multiple sessions, ensure all data is saved in the corresponding session folder (e.g., S1, S2).</li>
	<li>Data files 
	Ensure that each hand-position folder (P#) contains the following files: EMG data saved in an EDF file, named as follows: fpe_pos{position number}_{subject number}_S{session number}_rep0_BT; hand-tracking data saved in a CSV file, named fpe_pos{position number}_{subject number}_S{session number}_rep0_BT_full; and a log file, log.txt, containing metadata about the session.</li>
	<li>Data processing 
	​NOTE: A user may choose how to proceed with signal analysis and which tools to use. Here, we provide a script for performing signal filtering and data segmentation in Python. When using Python, ensure all dependencies (e.g., Numpy, Pandas, SciPy, MNE, Sklearn) are installed.
	<ol>
		<li>Open Python, load data_analysis.py and run the script.</li>
		<li>Request will appear in the console to provide necessary parameters for data processing: path to EMG file, path to hand kinematic data, path where the processed data will be saved, sampling rate in Hz, window duration in ms, and stride interval in ms.</li>
		<li>Following that step the script will perform the data processing.</li>
		<li>EMG signal filtering: Run the script as above. The script first filters the sEMG signal by applying a 4th-order Butterworth high-pass filter with a 20 Hz cutoff to remove non-EMG signals, then a notch filter to remove 50 Hz and 100 Hz harmonics. Additionally, script applies normalization of the EMG signal.</li>
		<li>EMG, HKD data, and instructed gesture segmentation: Run the script as above. The script applies segmentation, utilizing a rolling window technique defined by the specified window duration and stride interval. In this experiment, set them to 512 and 2 ms, respectively. The script then transforms the sEMG channel organization into a 4 x 4 spatial grid configuration while maintaining the electrode array layout. Finally, the script generates a dictionary containing metadata as a pickle file.</li>
		<li>Data cleaning and validation steps
		<ol>
			<li>Identify and exclude segments containing artifacts, noise, or inconsistent gesture labels from the dataset.</li>
			<li>Ensure segment completeness and temporal continuity across windows to maintain data reliability.</li>
			<li>Cross-check gesture data against the HKD for consistency. Remove windows displaying gesture patterns that deviate from HKD session standards.</li>
			<li>Detect and discard outlier segments that fail to conform to the expected kinematic patterns for the session.</li>
			<li>Perform further data analysis using advanced algorithms. These are not provided in the current protocol.</li>
		</ol>
		</li>
	</ol>
	</li>
</ol>

<ol><li>Fang, B., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Simultaneous+sEMG+recognition+of+gestures+and+force+levels+for+interaction+with+prosthetic+hand%5BTitle%5D)+AND+%22IEEE+Trans+Neural+Sys+Rehabilitation+Eng%22%5BJournal%5D)">Simultaneous sEMG recognition of gestures and force levels for interaction with prosthetic hand.</a> IEEE Trans Neural Sys Rehabilitation Eng. 30, 2426-2436 (2022).</li>
<li>Yadav, D., Veer, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Recent+trends+and+challenges+of+surface+electromyography+in+prosthetic+applications%5BTitle%5D)+AND+%22Biomed+Eng+Lett%22%5BJournal%5D)">Recent trends and challenges of surface electromyography in prosthetic applications.</a> Biomed Eng Lett. 13 (3), 353-373 (2023).</li>
<li>EMG based classification of basic hand movements based on time-frequency features. Sapsanis, C., Georgoulas, G., Tzes, A. 21st Mediterranean Conf Control and Automation, , 716-722 (2013).</li>
<li>sEMG signal based hand gesture recognition by using selective subbands coefficients and machine learning. Qaisar, S. M., Lopez, A., Dallet, D., Ferrero, F. J. 2022 IEEE Int Instrument Measurement Technol Conf, , 1-6 (2022).</li>
<li>Zhang, X., Zhou, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((High-density+myoelectric+pattern+recognition+toward+improved+stroke+rehabilitation%5BTitle%5D)+AND+%22IEEE+Trans+Biomed+Eng%22%5BJournal%5D)">High-density myoelectric pattern recognition toward improved stroke rehabilitation.</a> IEEE Trans Biomed Eng. 59 (6), 1649-1657 (2012).</li>
<li>Guo, K., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Empowering+hand+rehabilitation+with+AI-powered+gesture+recognition%3A+a+study+of+an+sEMG-based+system%5BTitle%5D)+AND+%22Bioengineering%22%5BJournal%5D)">Empowering hand rehabilitation with AI-powered gesture recognition: a study of an sEMG-based system.</a> Bioengineering. 10 (5), 557(2023).</li>
<li>Sun, T., Hu, Q., Libby, J., Atashzar, S. F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Deep+heterogeneous+dilation+of+LSTM+for+transient-phase+gesture+prediction+through+high-density+electromyography%3A+towards+application+in+neurorobotics%5BTitle%5D)+AND+%22IEEE+Robot+Autom+Lett%22%5BJournal%5D)">Deep heterogeneous dilation of LSTM for transient-phase gesture prediction through high-density electromyography: towards application in neurorobotics.</a> IEEE Robot Autom Lett. 7 (2), 2851-2858 (2022).</li>
<li>Atzori, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Characterization+of+a+benchmark+database+for+myoelectric+movement+classification%5BTitle%5D)+AND+%22IEEE+Trans+Neural+Sys+Rehabilitat+Eng%22%5BJournal%5D)">Characterization of a benchmark database for myoelectric movement classification.</a> IEEE Trans Neural Sys Rehabilitat Eng. 23 (1), 73-83 (2015).</li>
<li>Advancing muscle-computer interfaces with high-density electromyography. Amma, C., Krings, T., Schultz, T. Proc 33rd Ann ACM Conf Human Factors Computing Sys, , 929-938 (2015).</li>
<li>Geng, W., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Gesture+recognition+by+instantaneous+surface+EMG+images%5BTitle%5D)+AND+%22Sci+Rep%22%5BJournal%5D)">Gesture recognition by instantaneous surface EMG images.</a> Sci Rep. 6 (1), 36571(2016).</li>
<li>Wei, W., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+multi-stream+convolutional+neural+network+for+sEMG-based+gesture+recognition+in+muscle-computer+interface%5BTitle%5D)+AND+%22Pattern+Recognit+Lett%22%5BJournal%5D)">A multi-stream convolutional neural network for sEMG-based gesture recognition in muscle-computer interface.</a> Pattern Recognit Lett. 119, 131-138 (2019).</li>
<li>Padhy, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+tensor-based+approach+using+multilinear+SVD+for+hand+gesture+recognition+from+sEMG+signals%5BTitle%5D)+AND+%22IEEE+Sens+J%22%5BJournal%5D)">A tensor-based approach using multilinear SVD for hand gesture recognition from sEMG signals.</a> IEEE Sens J. 21 (5), 6634-6642 (2021).</li>
<li>Moin, A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+wearable+biosensing+system+with+in-sensor+adaptive+machine+learning+for+hand+gesture+recognition%5BTitle%5D)+AND+%22Nat+Electron%22%5BJournal%5D)">A wearable biosensing system with in-sensor adaptive machine learning for hand gesture recognition.</a> Nat Electron. 4 (1), 54-63 (2021).</li>
<li>Côté-Allard, U., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Deep+learning+for+electromyographic+hand+gesture+signal+classification+using+transfer+learning%5BTitle%5D)+AND+%22IEEE+Trans+Neural+Sys+Rehabilita+Eng%22%5BJournal%5D)">Deep learning for electromyographic hand gesture signal classification using transfer learning.</a> IEEE Trans Neural Sys Rehabilita Eng. 27 (4), 760-771 (2019).</li>
<li>NeuroPose: 3D hand pose tracking using EMG wearables. Liu, Y., Zhang, S., Gowda, M. Web Conference 2021 Proc World Wide Web Conf, , 1471-1482 (2021).</li>
<li>Dere, M. D., Lee, B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+novel+approach+to+surface+EMG-based+gesture+classification+using+a+vision+transformer+integrated+with+convolutive+blind+source+separation%5BTitle%5D)+AND+%22IEEE+J+Biomed+Health+Inform%22%5BJournal%5D)">A novel approach to surface EMG-based gesture classification using a vision transformer integrated with convolutive blind source separation.</a> IEEE J Biomed Health Inform. 28 (1), 181-192 (2024).</li>
<li>Chen, X., Li, Y., Hu, R., Zhang, X., Chen, X. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Hand+gesture+recognition+based+on+surface+electromyography+using+convolutional+neural+network+with+transfer+learning+method%5BTitle%5D)+AND+%22IEEE+J+Biomed+Health+Inform%22%5BJournal%5D)">Hand gesture recognition based on surface electromyography using convolutional neural network with transfer learning method.</a> IEEE J Biomed Health Inform. 25 (4), 1292-1304 (2021).</li>
<li>Lee, K. H., Min, J. Y., Byun, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Electromyogram-based+classification+of+hand+and+finger+gestures+using+artificial+neural+networks%5BTitle%5D)+AND+%22Sensors%22%5BJournal%5D)">Electromyogram-based classification of hand and finger gestures using artificial neural networks.</a> Sensors. 22 (1), 225(2022).</li>
<li>Zhou, X., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+novel+muscle-computer+interface+for+hand+gesture+recognition+using+depth+vision%5BTitle%5D)+AND+%22J+Ambient+Intell+Humaniz+Comput%22%5BJournal%5D)">A novel muscle-computer interface for hand gesture recognition using depth vision.</a> J Ambient Intell Humaniz Comput. 11 (11), 5569-5580 (2020).</li>
<li>Zhang, Z., Yang, K., Qian, J., Zhang, L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Real-time+surface+EMG+pattern+recognition+for+hand+gestures+based+on+an+artificial+neural+network%5BTitle%5D)+AND+%22Sensors%22%5BJournal%5D)">Real-time surface EMG pattern recognition for hand gestures based on an artificial neural network.</a> Sensors. 19 (14), 3170(2019).</li>
<li>Nieuwoudt, L., Fisher, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Investigation+of+real-time+control+of+finger+movements+utilizing+surface+EMG+signals%5BTitle%5D)+AND+%22IEEE+Sens+J%22%5BJournal%5D)">Investigation of real-time control of finger movements utilizing surface EMG signals.</a> IEEE Sens J. 23 (18), 21989-21997 (2023).</li>
<li>Ben-Ari, L., Ben-Ari, A., Hermon, C., Hanein, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Finger+gesture+recognition+with+smart+skin+technology+and+deep+learning%5BTitle%5D)+AND+%22Flexible+Printed+Electron%22%5BJournal%5D)">Finger gesture recognition with smart skin technology and deep learning.</a> Flexible Printed Electron. 8 (2), 25012(2023).</li>
<li>Yang, C., Xie, L. Gesture recognition method based on computer vision and surface electromyography: implementing intention recognition of the healthy side in the hand assessment process. 2024 4th Int Conf Neural Network Info Comm Eng, , 663-668 (2024).</li>
<li>Lin, J., Wu, Y., Huang, T. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Modeling+the+constraints+of+human+hand+motion%5BTitle%5D)+AND+%22Proc+Workshop+Human+Motion%22%5BJournal%5D)">Modeling the constraints of human hand motion.</a> Proc Workshop Human Motion. , 121-126 (2000).</li>
<li>Arché-Núñez, A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Bio-potential+noise+of+dry+printed+electrodes%3A+physiology+versus+the+skin-electrode+impedance%5BTitle%5D)+AND+%22Physiol+Meas%22%5BJournal%5D)">Bio-potential noise of dry printed electrodes: physiology versus the skin-electrode impedance.</a> Physiol Meas. 44 (9), 95006(2023).</li>
<li>Gracia-Ibáñez, V., Vergara, M., Sancho-Bru, J. L., Mora, M. C., Piqueras, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Functional+range+of+motion+of+the+hand+joints+in+activities+of+the+International+Classification+of+Functioning%2C+Disability+and+Health%5BTitle%5D)+AND+%22J+Hand+Ther%22%5BJournal%5D)">Functional range of motion of the hand joints in activities of the International Classification of Functioning, Disability and Health.</a> J Hand Ther. 30 (3), 337-347 (2017).</li>
<li>Exploring arm posture and temporal variability in myoelectric hand gesture recognition. Milosevic, B., Farella, F., Benatti, S. 2018 7th IEEE Int Conf Biomed Robotics Biomech, , 1032-1037 (2018).</li>
<li>Gijsberts, A., Atzori, M., Castellini, C., Müller, H., Caputo, B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Movement+error+rate+for+evaluation+of+machine+learning+methods+for+sEMG-based+hand+movement+classification%5BTitle%5D)+AND+%22IEEE+Trans+Neural+Sys+Rehabilitation+Eng%22%5BJournal%5D)">Movement error rate for evaluation of machine learning methods for sEMG-based hand movement classification.</a> IEEE Trans Neural Sys Rehabilitation Eng. 22 (4), 735-744 (2014).</li>
<li>Armitano-Lago, C., Willoughby, D., Kiefer, A. W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+SWOT+analysis+of+portable+and+low-cost+markerless+motion+capture+systems+to+assess+lower-limb+musculoskeletal+kinematics+in+sport%5BTitle%5D)+AND+%22Front+Sports+Act+Living%22%5BJournal%5D)">A SWOT analysis of portable and low-cost markerless motion capture systems to assess lower-limb musculoskeletal kinematics in sport.</a> Front Sports Act Living. 3, 809898(2022).</li>
</ol>

Yael Hanein declares a financial interest in X-trodes Ltd, which commercialized the screen-printed electrode technology used in this paper. The&#160;other authors have no other relevant financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

The protocol presented in this study outlines critical steps, modifications, and troubleshooting strategies aimed at enhancing hand gesture recognition through the combination of sEMG signals and HKD. It addresses key limitations and compares this approach to existing alternatives, highlighting its potential applications in various research domains. One of the most important aspects of the protocol is ensuring the correct positioning and alignment of the hand-tracking camera. Accurate gesture capture is highly dependent on the angle and distance of the camera relative to the participant&#39;s hand. Even slight deviations in camera positioning can lead to tracking inaccuracies, reducing the fidelity of the gesture data. This alignment must be carefully adjusted for each participant and hand position to ensure consistent and reliable data collection. Additionally, it is crucial that participants are well-acquainted with the protocol to prevent junk data - where gestures are either incorrectly executed or misaligned with the experimental flow. Ensuring that participants are comfortable and familiar with the gestures and the experimental setup can minimize data noise and improve the quality of the recordings.
A common challenge in this type of study is noise contamination in both sEMG and HKD. sEMG signals are particularly sensitive to factors such as muscle fatigue, motion artifacts, and environmental noise like electromagnetic interference. Pre-processing techniques, such as band-pass filtering, are essential for reducing noise and improving signal clarity. Proper electrode placement and instructing participants to maintain relaxed muscles during rest phases can further mitigate motion artifacts. Despite these precautions, some variability in sEMG signals is inevitable due to individual differences in anatomy, hand strength, and muscle activation patterns. This variability can be addressed through flexible algorithms capable of normalizing these differences across subjects and conditions.
A key factor in achieving high-quality sEMG signals is initial signal verification. Traditional protocols using gel electrodes require skin preparation, such as exfoliating or cleaning with alcohol, to improve signal clarity. However, in a previous study we showed that with dry electrodes, skin preparation may not significantly impact signal quality25. In this protocol, skin cleaning is optional and thus simplifies the process. Another skin-related issue affecting signal quality is excessive and thick arm hair. In such cases, we suggest either shaving the area or excluding the subject from the study.
One of the critical challenges in using sEMG for gesture recognition is its sensitivity to hand positioning. Even when performing the same gesture, variations in hand orientation can lead to different EMG signal patterns. To address this issue, machine learning models that can accommodate variability in hand positions are essential22. These models must be trained with data from multiple hand postures to improve robustness and generalizability. Synchronization of visual and sEMG data is another important consideration. Consistent timing of gestures is critical to avoid discrepancies between the gesture execution and the data recording. This protocol uses visual countdowns and auditory cues to help ensure accurate timing and recalibration steps are employed when necessary to correct any misalignment during data collection.
Despite its strengths, this protocol has several limitations. One major constraint is the limited field of view of the hand-tracking camera, which requires the participant&#39;s hands to remain within the camera&#39;s detection range. This restricts the analysis to a small set of movements. For outside the lab experiments a more complex video imaging will be required or the use of smart gloves. Participant fatigue also poses a challenge during longer sessions, potentially affecting gesture accuracy and muscle activation, which can degrade the quality of the sEMG data. To mitigate these effects, it may be necessary to limit the session length or introduce breaks to minimize fatigue. Additionally, powerline interference can introduce noise into the sEMG signals, particularly when the participants are close to the PC for data capture. A wireless version of the system could reduce such interference by allowing participants to be farther from the computer.
A significant methodological limitation of EMG-based finger gesture detection stems from the high inter-subject variability in sEMG signals, which requires the development of custom models for each participant. This subject-specific approach, while more accurate, limits the protocol&#39;s scalability and requires additional calibration and training time for each new user. EMG and HKD data streams show minor temporal synchronization differences due to dual process recording. These timing discrepancies have a minimal impact on the static gesture analysis since the maintained poses are temporally stable. The sustained nature of static gestures provides adequate time for both EMG and kinematic features to stabilize, unlike dynamic gestures, which require more precise synchronization.
A key advantage of this method is its flexibility in capturing gestures. Unlike other systems that require rigid setups and strict gesture parameters, this protocol accommodates dynamic and flexible hand positions19. This flexibility is especially useful in studies aimed at analyzing a broad range of motions, making it more adaptable to real-world applications. Furthermore, this protocol is cost-effective compared to more advanced motion capture and sEMG systems, which often involve complex setups29. By integrating a hand-tracking camera with semi-automated sEMG algorithms, this method provides a viable alternative for gesture recognition studies without compromising data quality. Additionally, the system&#39;s potential for real-time data processing opens possibilities for immediate feedback in applications such as neuroprosthetics and rehabilitation, where real-time responsiveness is essential. This protocol has significant implications for several fields, particularly neuroprosthetics. Accurate prediction of hand gestures from sEMG signals is crucial for controlling prosthetic limbs, and the flexibility in hand positioning offered by this method makes it an ideal candidate for real-time prosthetic devices. In rehabilitation, this protocol could be employed to monitor and enhance motor recovery in patients with hand or finger impairments. By analyzing muscle activation patterns during gesture performance, this system could be used to tailor rehabilitation exercises to individual needs, offering a personalized approach to motor recovery. For human-computer interaction (HCI), this method enables more natural gesture-based control systems, improving the intuitiveness and efficacy of user interfaces. Lastly, the protocol could be applied to ergonomic studies to assess how different hand positions and gestures influence muscle activity and fatigue, potentially leading to advancements in workplace design and user ergonomics.
To ensure consistent contraction strength across participants, future studies could implement a glove with force-sensitive resistors to measure force directly. This would allow for standardized effort across subjects, improving the reliability of EMG data. Additionally, integrating this force measurement as a label in joint kinematics would provide a more detailed representation of the muscle&#39;s internal state, potentially enriching the analysis of muscle function and movement patterns. This approach would not only enhance data consistency but also offer deeper insights into the relationship between muscle contraction and joint motion.
In conclusion, this protocol provides a novel and flexible approach to hand gesture recognition with broad applications across neuroprosthetics, rehabilitation, HCI, and ergonomics. Although the system has limitations, its flexibility, cost-effectiveness, and potential for real-time use represent substantial advancements over existing methods. These strengths make it a promising tool for further development and innovation in gesture recognition technologies.

Hand gesturing is essential in human communication, making the recognition of finger gestures a crucial area of research across fields such as human-computer interaction, advanced prosthetics1,2,3,4, and rehabilitation technologies5,6. As a result, finger gesture recognition has garnered significant attention for its potential to enhance intuitive control systems and assistive devices. Surface electromyography (sEMG) combined with deep learning algorithms is emerging as a highly promising approach for capturing and interpreting these gestures due to its ability to detect the electrical activity of muscles associated with hand movements7,8,9,10,11,12,13,14,15.
However, despite these advances, current approaches face limitations in real-world applications. Most existing systems require complex, cumbersome recording setups with numerous electrodes5,7,9,16,17 and precise positioning3,18, which are often difficult to implement outside of controlled environments. Additionally, these systems tend to focus on static hand positions13,18,19,20,21, limiting their ability to interpret dynamic, fluid gestures that occur in daily activities. The protocol aims to address these limitations by supporting dynamic gesture recognition in more natural conditions. Such methodology would enable more practical and user-friendly applications in areas like prosthetics and rehabilitation, where real-time, natural gesture interpretation is essential.
To address these challenges, developing more accurate and adaptable algorithms requires datasets that reflect natural, everyday conditions3,4. Such datasets must capture a wide range of dynamic movements, various hand positions, and large volumes of data to ensure model robustness. Furthermore, the variability between training and test datasets is crucial, allowing models to generalize across different hand postures, muscle activation patterns, and motions. Incorporating such diversity into the data will enable algorithms to perform gesture recognition more accurately in everyday, real-world applications22.
Overcoming these challenges will be essential for the future development of more practical and widely applicable gesture recognition systems. The study and protocol described here stem&#160;from the need to have a portable, user-friendly setup that can capture dynamic hand movements in natural settings. Comprehensive datasets and advanced algorithms are critical to fully unlocking the potential of sEMG and deep learning in human-computer interfaces, neuroprosthetics, and rehabilitation technologies. We expect this protocol to contribute to the field by facilitating comprehensive data collection to further enable the development of algorithm models that generalize across diverse hand positions.
A significant challenge in gesture recognition lies in the sensitivity of sEMG signals to hand positioning. While many studies focus on fixed-hand positions for gesture prediction, real-world applications demand models capable of recognizing finger movements across various hand postures. Recent approaches have addressed this by incorporating computer vision as a ground truth reference, enhancing the accuracy and flexibility of these models15,19. Additionally, hybrid models that integrate sEMG signals with visual data offer further improvements in recognition accuracy across diverse scenarios23.
In this protocol, we present a synchronized approach to data collection that enhances dynamic gesture recognition by incorporating both EMG and hand-tracking data in real-world-like conditions. Unlike traditional methods that restrict gesture performance to static positions, this protocol includes gestures performed across four distinct positions: hand down, hand up, hand straight, and hand moving. The hand-tracking camera tracks hand movements within a three-dimensional interactive zone, identifying distinct hand elements and capturing dynamic movements with high resolution. A soft electrode array of 16 electrodes placed on the forearm to record muscle activity offers stable and wireless recordings without impeding participant mobility. The synchronized data from these two sources provides a comprehensive foundation for developing advanced gesture recognition algorithms capable of operating in real-world conditions. The approach specifically addresses the limitations of current setups by facilitating free movement and stable signal recording in realistic scenarios. This advancement supports gesture recognition technologies for applications in prosthetics, rehabilitation, and interactive technologies, where intuitive control and flexibility are essential.

<table><tbody><tr><td>Adjustable Selfie Stick</td><td></td><td></td><td>Used to position and angle the hand-tracking camera in the desired orientation for optimal data capture during the experiment.</td></tr><tr><td>Alcohol pad</td><td></td><td></td><td>To clean the area for electrode placement.</td></tr><tr><td>Data acquisition unit (DAU)</td><td>X-trodes Ltd.</td><td>&amp;nbsp;XTR-BT V1.3</td><td>Realtime compatible DAU 2.0.17 or 4.0.1</td></tr><tr><td>Finger Gestures Recognition library</td><td></td><td></td><td>https://github.com/NeuroEngLabTAU/Fingers_Gestures_Recognition.git</td></tr><tr><td>Leap Motion Controller 2</td><td>Ultraleap</td><td>129-SP4-00005-03</td><td>Hand-tracking camera</td></tr><tr><td>Long Type-C to Type-C cable</td><td></td><td></td><td>Connection of the hand-tracking camera to the PC.</td></tr><tr><td>PC Monitors</td><td></td><td></td><td>One for guidelines, one for viewing the hand-tracking camera data&amp;nbsp;</td></tr><tr><td>Personal Computer (PC)</td><td>Windows</td><td></td><td>Windows 10+; Processors: Inteli7 processor. BT receiver.</td></tr><tr><td>Python code</td><td></td><td></td><td>A script enabling seamless data streaming and recording up to 500 S/s when DAU is connected to PC via Bluetooth</td></tr><tr><td>Ultraleap Camera Python API</td><td>Ultraleap</td><td></td><td>Python API and instructions from Ultraleap&amp;rsquo;s GitHub repository (https://github.com/ultraleap/leapc-python-bindings) used to collect data from the Ultraleap unit during the experiment</td></tr><tr><td>Ultraleap Hyperion</td><td>Ultraleap</td><td></td><td>Tracking software</td></tr><tr><td>XTR EXG16</td><td>X-trodes Ltd.</td><td>XTELC0003405RM</td><td>16-channel wearable dry electrode array patch for EMG, ECG monitoring</td></tr><tr><td>X-trodes PC App</td><td>X-trodes Ltd.</td><td>1.1.35.0</td><td>An application for connecting X-trodes DAU to PC via BT</td></tr></tbody></table>

capturing dynamic finger gesturing with high-resolution surface electromyography and computer vision

Capturing Dynamic Finger Gesturing with High-resolution Surface Electromyography and Computer Vision

Finger gestures are a critical element in human communication, and as such, finger gesture recognition is widely studied as a human-computer interface for state-of-the-art prosthetics and optimized rehabilitation. Surface electromyography (sEMG), in conjunction with deep learning methods, is considered a promising method in this domain. However, current methods often rely on cumbersome recording setups and the identification of static hand positions, limiting their effectiveness in real-world applications. The protocol we report here presents an advanced approach combining a wearable surface EMG and finger&#160;tracking system to capture comprehensive data during dynamic hand movements. The method records muscle activity from soft printed electrode arrays (16 electrodes) placed on the forearm as subjects perform gestures in different hand positions and during movement. Visual instructions prompt subjects to perform specific gestures while EMG and finger positions are recorded. The integration of synchronized EMG recordings and finger tracking data enables comprehensive analysis of muscle activity patterns and corresponding gestures. The reported approach demonstrates the potential of combining EMG and visual tracking technologies as an important resource for developing intuitive and responsive gesture recognition systems with applications in prosthetics, rehabilitation, and interactive technologies. This protocol aims to guide researchers and practitioners, fostering further innovation and application of gesture recognition in dynamic and real-world scenarios.

The dataset consists of two time-synchronized components: a 16-channel EMG dataset and data from a hand-tracking camera system. The 16-channel EMG data captures muscle activity by recording electrical signals from different muscles over time. The hand-tracking system provides 16 channels of data corresponding to key points on a skeletal model of the hand. While the model has 21 points, excluding the wrist, this number was reduced to 16 due to motion constraints24. The EMG and visual data were collected by running two separate processes on the same computer during recording to establish synchrony. A timestamp was used to mark the start of each process, allowing data analysis code to align muscle activity and hand movement data at the end of the recording. Timestamp annotations were saved automatically in both EDF and CSV files, marking the exact time when specific finger gestures were instructed and facilitating the alignment during data analysis. The filtered EMG signal (20 Hz 4th-order Butterworth high pass filter) is characterized by a low baseline (grey-shaded areas), which typically falls within the range of 3-9 µV25. This baseline is observed when the subject's hand is stationary and the muscles are at rest. However, if muscle tone is present even in the rest position, a distinct EMG signal can be detected. Mechanical artifacts caused by movement usually manifest in the 10-20 Hz range and should be filtered out accordingly. Significantly elevated baseline values may indicate 50 Hz line interference and should be avoided during the experimental setup stage. In cases where moderate 50 Hz noise persists; a notch filter is applied. Sharp movement artifacts, which are more difficult to remove, often appear as pronounced high-amplitude spikes in the signal (see asterisk in Figure 2A). The amplitude of the EMG signal across the 16-electrode array varies, reflecting the spatial distribution of the muscle activity over the region measured. This variance provides valuable insight into the heterogeneity of muscle contraction during hand gestures.
The hand-tracking camera provides direct information of finger angles (hand kinematic data, HKD), which are expected to correlate closely with the recorded EMG signals. During gestures, finger angles in normal range26, depending on the specific gesture. When the visual path between the hand-tracking camera and the hand is unobstructed, the resulting signal is stable and accurate, as demonstrated in Figure 2. However, in instances where visual contact is lost or when the system experiences technical limitations, the HKD may become erratic, displaying jumps between incorrect values. Such outlier data should be minimized during data collection and discarded in the final analysis to maintain the integrity of the results.
The HKD is intuitive and provides a direct comparison with the actual gestures performed. It exhibits low variability between subjects and across different hand positions. In contrast, the EMG data tends to vary significantly between individuals due to anatomical differences such as hand size and muscle development27. Additionally, variability may be observed between dominant and non-dominant hands. This subject-specific variability can be addressed during offline analysis.
In Figure 2, it is evident that both the EMG and HKD are offset relative to the instructed gesture trigger. This discrepancy arises due to response time and the natural movement execution28. In regression tasks, such variability could contribute to the richness of the data, while in classification tasks, it can be managed using a generalized likelihood ratio approach, as applied in similar scenarios28.
<img alt="EMG measurement; time-series comparison of hand kinematics and muscle activity; experimental graph." class="xfigimg" src="/files/ftp_upload/67766/67766fig02.jpg" data-altupdatedat="1788304762000" data-alt="Figure 2"> 
Figure 2: Representative sEMG and HKD during finger abduction. Surface electromyography (sEMG) signals and hand kinematic data (HKD) recorded during dynamic finger abduction and rest performed during hand position 1 (hand down, straight, and relaxed) by a single participant. (A) Filtered EMG signals from 16 channels as a function of time. Asterisk (*) denotes a mechanical artifact detected in the EMG recording of Channel 5. (B) HKD, showing the joint angles as a function of time. Joint angles are measured at various joints: trapeziometacarpal (TMC), metacarpophalangeal (MCP), and proximal interphalangeal (PIP). The phases of the experiment (rest and abduction) are indicated along the x-axis. <a href="https://www.jove.com/files/ftp_upload/67766/67766fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
These representative results demonstrated the utility of the synchronized EMG and HKD data in capturing hand gestures. The alignment of EMG signals with corresponding HKD allows mapping muscle activity to specific finger movements. When constructing a predictive model, researchers can use HKD as ground truth, iteratively verifying and refining EMG-based gesture predictions. This approach highlights the practical applicability of the protocol and suggests the need for further research in more natural settings.
Supplementary Figure 1: Spectrogram windows displayed during the signal verification step. The left panels show raw EMG data, while the right panels show the detected frequency domains. (A) Example of a very noisy EMG signal with strong 50 Hz and 100 Hz interference. (B) Example of the same EMG signal recording after moving the participant further away from electrical devices, resulting in a clean EMG signal with minimal interference. <a href="https://www.jove.com/files/ftp_upload/67766/S1_R1.jpg" target="_blank">Please click here to download this File.</a>

The paper presents a comprehensive protocol for simultaneously recording hand-electromyography (EMG) and visual finger tracking during natural finger gesturing. The visual data is designed to serve as the ground truth for the development of accurate EMG-based computational models for finger gesture recognition.

Finger gestures are a critical element in human communication, and as such, finger gesture recognition is widely studied as a human-computer interface for ...

capturing-dynamic-finger-gesturing-with-high-resolution-surface

Research

JoVE Journal

Bioengineering

Source:Hila Man1, Paul F. Funk2, Ieva Vebraite-Adereth3, Christoph Anders4, Orlando Guntinas-Lichius2, Yael Hanein1,3,51Sagol School of Neuroscience, Tel Aviv University, 2Department of Otorhinolaryngology, Jena University Hospital, Friedrich-Schiller-University Jena, 3School of Electrical Engineering, Tel Aviv University, 4Division Motor Research, Pathophysiology and Biomechanics, Department of Trauma, Hand and Reconstructive Surgery, Jena University Hospital, Friedrich-Schiller-University...

Wireless Surface Electromyography Recording of Facial Muscle Activity During Expressions

Monitoring brain activity during upright motor tasks is of great value when investigating the neural source of movement disorders. Here, we demonstrate a protocol that combines functional near infrared spectroscopy with continuous monitoring of muscle and kinematic activity during 4 types of motor tasks.

Functional Near Infrared Spectroscopy of the Sensory and Motor Brain Regions with Simultaneous Kinematic and EMG Monitoring During Motor Tasks

A novel low-cost human-machine interface for interactive post-stroke balance rehabilitation system is presented in this article. The system integrates off-the-shelf low-cost sensors towards volitionally driven electrotherapy paradigm. The proof-of-concept software interface is demonstrated on healthy volunteers.

A Human-machine-interface Integrating Low-cost Sensors with a Neuromuscular Electrical Stimulation System for Post-stroke Balance Rehabilitation

Transcranial magnetic stimulation, electromyography, and 3D motion capture are commonly used non-invasive techniques for investigating neuromuscular function in humans. In this paper, we describe a protocol that synchronously samples data generated by all three of these tools along with the unique addition of virtual reality stimulus presentation and feedback.

Multifunctional Setup for Studying Human Motor Control Using Transcranial Magnetic Stimulation, Electromyography, Motion Capture, and Virtual Reality

Presented here is a behavioral paradigm that elicits robust fast visuomotor responses on human upper limb muscles during visually guided reaches.

An Emerging Target Paradigm to Evoke Fast Visuomotor Responses on Human Upper Limb Muscles

When we grasp an object, multiple regions of the fingers and hand typically make contact with the object's surface. Reconstructing such contact regions is challenging. Here, we present a method for approximately estimating the contact regions by combining marker-based motion capture with existing deep learning-based hand mesh reconstruction.

Estimation of Contact Regions Between Hands and Objects During Human Multi-Digit Grasping

Magnetoencephalography (MEG) and high-density electroencephalography (HD-EEG) are rarely recorded simultaneously, although they yield confirmatory and complementary information. Here, we illustrate the experimental setup for recording simultaneous MEG and HD-EEG and the methodology for analyzing these data aiming to localize epileptogenic and eloquent brain areas in children with drug-resistant epilepsy.

Electromagnetic Source Imaging in Presurgical Evaluation of Children with Drug-Resistant Epilepsy

In many biological and clinical situations it is advantageous to study cellular processes as they evolve in their native microenvironment.  Here we describe the assembly and use of a low-cost fiber-optic microscope which can provide real time imaging in cell culture, animal studies, and clinical patient studies.

High-resolution Fiber-optic Microendoscopy for in situ Cellular Imaging

Non-destructive volume visualization can be achieved only by tomographic techniques, of which the most efficient is the x-ray micro computerized tomography ( CT).

High Resolution 3D Imaging of Ex-Vivo Biological Samples by Micro CT

We describe here the operation of a microfluidic device that allows continuous and high-resolution microscopic imaging of single budding yeast cells during their complete replicative and/or chronological lifespan.

Continuous High-resolution Microscopic Observation of Replicative Aging in Budding Yeast

Learn to capture dynamic finger gestures using synchronized high-resolution surface EMG and computer vision. This protocol enables accurate muscle-to-gesture mapping for prosthetic rehabilitation and human-computer interaction research.

Capturing Dynamic Finger Gesturing with High-resolution Surface Electromyography and Computer Vision

Explore More Videos