Dynamic Digital Biomarkers of Motor and Cognitive Function in Parkinson's Disease

Summary

This protocol offers a digitization of portions of traditional clinical tasks commonly used to measure cognition and motor control in Parkinson’s disease. Clinical tasks are digitized while biophysical rhythms are co-registered from different functional levels of the nervous systems, ranging from voluntary, spontaneous, automatic to autonomic.

Abstract

As Parkinson’s disease (PD) is a heterogeneous disorder, personalized medicine is truly required to optimize care. In their current form, standard scores from paper and pencil symptom- measures traditionally used to track disease progression are too coarse (discrete) to capture the granularity of the clinical phenomena under consideration, in the face of tremendous symptom diversity. For this reason, sensors, wearables, and mobile devices are increasingly incorporated into PD research and routine care. These digital measures, while more precise, yield data that are less standardized and interpretable than traditional measures, and consequently, the two types of data remain largely siloed. Both of these issues present barriers to the broad clinical application of the field’s most precise assessment tools. This protocol addresses both problems. Using traditional tasks to measure cognition and motor control, we test the participant, while co-registering biophysical signals unobtrusively using wearables. We then integrate the scores from traditional paper-and-pencil methods with the digital data that we continuously register. We offer a new standardized data type and unifying statistical platform that enables the dynamic tracking of change in the person’s stochastic signatures under different conditions that probe different functional levels of neuromotor control, ranging from voluntary to autonomic. The protocol and standardized statistical framework offer dynamic digital biomarkers of physical and cognitive function in PD that correspond to validated clinical scales, while significantly improving their precision. 

Introduction

Precision medicine (PM) (Figure 1) has emerged as a powerful platform to develop personalized targeted treatments. In the field of cancer research, this model has been very successful and its tenets are bound to revolutionize the medical field in the near future¹. PM combines multiple layers of knowledge, ranging from patients’ self-reports to genomics. Integrating information across all these layers results in a personalized assessment that enables interpretation of the data and more precise treatment recommendations aimed at considering all aspects of the person’s life.

There are several challenges when trying to adapt the PM platform to neuropsychiatric and neurological disorders of the nervous systems^2,3, and these challenges have recently been voiced⁴. Among those are the disparity in the data that is acquired, namely discrete scores from clinical pencil-and-paper methods guided by observation, and continuous biophysical data physically acquired from the nervous systems output (e.g., using biosensors). The data from clinical scores tend to assume a one-size-fits all static model that enforces a single (theoretical) probability distribution function (PDF). This a priori assumption is imposed on the data without proper empirical validation, because normative data has not been acquired and characterized in the first place. As such, there is no proper similarity-metric-based criteria describing the neurotypical maturational states of the human nervous systems, as the healthy person ages and the probability spaces used to cast these parameter variations shift at some rate. Without normative data and proper similarity metrics, it is not possible to measure departures from typical states as they dynamically change across the person’s life. It is also not possible to predict the sensory consequences of the upcoming changes.

Figure 1: Precision medicine platform: Filling the gap between behaviors and genomics to enable targeted treatment development in personalized medicine translated to neurological and neuropsychiatric disorders of the nervous systems. The precision medicine platform for the development of personalized targeted treatments can be translated to diagnose and treat neurological and neuropsychiatric disorders of the nervous systems. However, in the knowledge network, the layer of behavioral analyses needs a paradigm shift to integrate new emerging digital outcomes from biophysical data with more traditional clinical criteria. A challenge ahead is to provide statistically sound methods and new intuitive visualization tools for such integration, while encouraging use of digital outcome measures by clinicians, patients and caregivers. This figure has been modified from Hawgood et al.¹ with permission from the American Association for the Advancement of Science. Please click here to view a larger version of this figure.

The current “grand averaging” approach smooths out as noise the individual’s stochastic fluctuations in the data, i.e., the signal variability that manifests as the person naturally ages, as the disorder progresses, and as the person’s nervous systems receive and respond to treatments. The lack of normative data (i.e., assessing large cross sectional and longitudinal portions of the healthy population) prevents us from understanding the neurotypical dynamics of healthy aging. As such, it becomes a challenge to know how to more generally anticipate the consequences of a given pathology, as that pathology begins to systemically manifest in the individual. Predictive approaches are critical to design regenerative therapies and/or neuroprotective therapies that slow down the degenerative process. Parkinson’s disease is a good example of pathologies whereby the manifestations of the disorder are preceded by many other measurable symptoms. We know today that the visible motor disorders were preceded by less visible sensory issues such as diminished olfactory function^5,6, changes in speech patterns, rapid eye movement (REM) sleep⁷, and other non-motor symptoms related to the functioning of the enteric nervous systems⁸. By the time the disorder manifests, there is already high dopaminergic depletion in the system; yet non-motor symptoms could have forecasted some of the visible motor impairments, by which the disorder is currently primarily assessed.

There is a need to change the current analytical models and consider the importance of properly characterizing empirical data across all levels of the nervous systems, whereby biorhythmic motions manifest and can be dynamically harnessed in the form of time series co-registered with a multitude of sensors. Motion data in its more general sense ought not to be limited to movements and the disorders they broadcast. Digital data from all biorhythms of the nervous systems (inclusive of non-movement waveforms) offer the forecasting potential that we may we need to help prevent or to slow down rapid neurodegeneration. Yet, as we augment our repertoire of data types, we should avoid the inherent assumption of parametric linear models for statistical inference and interpretation currently in use to analyze such data. It will be critical to evaluate the adequacy of such linear models for the types of highly non-linear problems that we study in nervous systems pathologies subject to stochastic shifts and dynamical changes. Caveats in current assumption-analysis pipeline loops are present in both data types being harnessed: those from the discrete clinical scores and those from the continuous digital biophysical waveforms. While they remain disconnected, it will be important to design new frameworks that enable proper integration of both types of data in ways that align digital outcomes with clinical criteria to facilitate the use of emerging digital technologies by patients, caregivers and clinicians.

To overcome some of these challenges, we have recently adapted the PM platform in Figure 1 to provide precision phenotyping for neurological and neuropsychiatric conditions³. To that end, we have designed a new way to gather, analyze and interpret behavioral data in tandem with traditional clinical scoring tests that ascertain complex relations between cognitive and motor phenomena. More precisely, we have digitized the pencil-and-paper methods. The data from such methods alone are much too coarse to capture important information escaping the naked eye. But their use in combination with digital data from biophysical sensors offers a new avenue to connect new emerging digital technologies with clinical criteria amenable to encourage clinicians to adopt them in the near future.

Here, we introduce the use of digital data within the context of clinical assessments. Namely, as the person performs the clinical task, e.g., drawing a clock in the Montreal Cognitive Assessment (MoCA) test, the biorhythms output by the nervous systems are co-registered across different functional layers. These include electroencephalography (EEG), electrocardiography (ECG or EKG), voice patterns and kinematics from the body, and the kinematic output from the hand-held pen that the person uses to draw the clock on a digitized tablet. We also collect video data from the face as the person draws, to perform sentiment analyses predictive of emotional states. These data are then analyzed through the optics of a new statistical platform for individualized behavioral analysis (SPIBA) and interpreted according to the clinical criteria underlying such tests. More specifically, the discrete scores are used to median-rank the cohort of patients and in this way, stratify the group based on that clinical criteria. We can then examine the continuous biophysical data of the groups thus identified, in search for digitally-driven stochastic criteria that fundamentally separates one subset of patients from another, across more than one parametric dimension. Moreover, by examining the continuous biophysical data in its own right, according to the inherent fluctuations of each person within the cohort and blinded from the clinical criteria, we can search for self-emerging clusters within the cohort, and compare the extent to which such clusters map onto those that the subtypes informed by the clinical criteria revealed.

This approach offers a new way to identify parameters within the wealth of biophysical digital data, that most effectively capture the differences across subtypes and renders those differences as potentially good candidates to stratify patients with Parkinson’s disease (PWP) in the blind, i.e., across a random draw from the general population. The relevance of this method is twofold. We can truly personalize the treatments, while properly integrating disparate data types from biosensors and clinical criteria; i.e., continuous digital biophysical data in the form of time series, and discrete clinical scores from traditional tests.

Although this is a general approach, applicable to all disorders of the nervous systems, we frame the work within the context of PWP and offer new ways to make statistical inferences about the continuous digital data co-registered during the performance of such clinical tests considering the discrete clinical scoring system. As such, the work enables clinical interpretation of the digital results amenable to use in clinical settings. Lastly, we provide recommendations to begin designing new ways to visualize such individual outcomes to embed in new apps for ease of use in the home and clinical settings by the patients, the caregivers and the clinical personnel alike.

Protocol

All methods described here have been approved by the Rutgers University Institutional Review Board.

1. Participants and acquisition system set up

1. Obtain informed consent from participants.
   NOTE: Participants should either be diagnosed with Parkinson’s disorder, or should not be diagnosed with any neurological disorder to serve as healthy control participants. Healthy participants should be gender and age matched to the patient participants. All participants should have the mobility to continuously walk for 5 minutes.
2. Measure the participant’s body dimension (body height, foot length, arm span, ankle height, hip height, hip width, knee height, shoulder width, shoulder height; Figure 2A) to later create his/her body avatar in the motion capture system.
   NOTE: This information is used in the motion capture system to accurately record positional data of the participant’s body.
3. Set up the motion capture system, including the 17 wireless motion-tracking sensors and motion-tracking software.
   1. Place sensors on the following body parts: head, sternum, pelvis, right shoulder, right upper arm, right lower arm, right hand, left shoulder, left upper arm, left lower arm, left hand, right upper leg, right lower leg, right foot, left upper leg, left lower leg, left foot (Figure 2B). Secure these sensors with strap bands to allow for unobtrusive movement.
   2. Once all sensors are placed at proper places, calibrate the participant’s position to create his/her avatar.
      NOTE: Details on calibration method can be found in Roetenberg et al.⁹.
4. Set up EEG device and EEG recording software.
   1. Position 31 sensors across the scalp and the recording device on the back of the participant’s head. Place channel sensors on the following locations: P7, P4, Cz, Pz, P3, P8, O1, O2, T8, F8, C4, F4, Fp2, Fz, C3, F3, Fp1, T7, F7, PO4, FC6, FC2, AF4, CP6, CP2, CP1, CP5, FC1, FC5, AF3, PO3 (Figure 2C).
   2. Attach the remaining channel sensor (Oz) to a connector to measure the heart signal, by positioning that connected sensor on the participant’s left-side stomach (Figure 2C).
   3. Attach two reference channel sensors behind the participant’s left ear, and then insert electrode gel into the sensors on the EEG cap using a syringe (Figure 2C).
   4. Once completed, start streaming the electrical activities on the recording software for a few minutes until it stabilizes.
      NOTE: Figure 2D shows sample traces from the EEG signals harnessed from the central nervous system (CNS) and autonomic nervous system (ANS).
5. Set up microphone to capture the participant’s voice. Place the microphone in front of the participant and connect it to the computer where lab stream layer (LSL) will be running (see LSL below) (Figure 2E).

Figure 2: Set up to digitize traditional clinical tests, while integrating multiple waveforms from unobtrusive wireless wearable biosensors. (A) Set up for the drawing tasks: actual patient wearing unobtrusive wearable sensors and avatar rendered in real time from the kinematics being collected. (B) Sensor locations of a set of small and lightweight motion-tracking sensors (60 Hz), from which motion data are co-registered synchronously across the body. (C) EEG map and reference location. (D) Sample EEG waveforms from the 31 leads and heart signal extracted from the Oz lead. (E) Sample waveform from the participant’s voice during a counting task. Please click here to view a larger version of this figure.

6. Set up the LSL system to synchronize the streams of EEG, motion, and audio signals, along with mouse click timestamps (Figure 3A).
   1. Open the Lab Recorder app.
   2. Open LSL Apps for Mouse and in-house built Xsens synchronizing app and link them to the Lab Recorder app by checking the boxes Mouse Buttons and Tracker Kinematics in the Record from Streams section.
      NOTE: The Mouse app will be used to timestamp events during the experiment.
   3. Link EEG, motion, and audio streams to the Lab Recorder app by checking the boxes AudioCaptureWin, LSL-EEG, Position in the Record from Streams section.
      NOTE: The LSL system enables a unified collection of measurement time series in research experiments that handles time-synchronization, along with networking and real-time access, and centralized collection, viewing and disk recording of the data. In the context of this protocol, the LSL system synchronously streams from the CNS, peripheral nervous system (PNS) and ANS, as the person naturally performs traditional clinical tasks.
7. Set up recording for pen movement, including pen tablet and movement analysis software (Figure 3B).
   1. Place the drawing tablet and tablet pen in front of the participant.
   2. Connect the tablet to the computer from which movement analysis software will be recording.
   3. Place a white paper on the tablet and secure with a tape.
      NOTE: Prepare in advance the papers to show a box to indicate space where the participant would be drawing in.
8. Start recording.
   1. Press record in LSL, motion capture software, and EEG recording software.
   2. At the start and end of each task, timestamp with mouse clicks by pressing on the timestamping button on the motion capture software.
      NOTE: This way, timestamps will be recorded in both LSL stream data and in the motion capture data. This will ensure that there is a back-up timestamp, in case one of the timestamping functions fails during recording.

Figure 3: Signal digitization and synchronization through a common central processing unit’s Lab Stream Layer (LSL) system. (A) LSL system synchronously allow co-registration of motion (PNS), EEG (CNS), ECG (ANS), voice and time stamping through mouse clicks. (B) Digitized tablet and pen recording kinematics (position) of the pen tip, during drawing tasks from standardized cognitive tests. Please click here to view a larger version of this figure.

2. Experimental procedure

1. Conduct task 1 — Benson Complex Figure Copy (Immediate)¹⁰ (1 min).
   1. Instruct the participant to copy the Benson figure on the paper, and to remember the design because the participant will be asked to draw it again from memory later.
      NOTE: This test is designed to assess the participant’s visuo-constructional and visual memory function (Figure 4A).
2. Conduct tasks 2 and 3 — Trail Making Test Part A (3 min).
   1. Instruct the participant to draw a line between circles that are numbered in ascending order.
      NOTE: There are two tasks in this section, where the first (task 2) is to complete a sample test (consisting of 8 numbers), and the next (task 3) is to complete the actual test (consisting of 25 numbers). This test was a component of the Army Individual Test Battery¹¹, and assesses processing speed and executive function and depends on visuomotor and perceptual-scanning skills (Figure 4B).
3. Conduct tasks 4 and 5 — Trail Making Test Part B (5 min).
   1. Instruct the participant to draw a line between circles that contain either numbers or letters, in ascending order, while alternating between numbers and letters. For example, the sequence would be: A to 1; 1 to B; B to 2; 2 to C.
      NOTE: There are two tasks in this section, where the first (task 4) is to complete a sample test (consisting of 4 numbers and 4 letters), and the next (task 5) is to complete the actual test (consisting of 13 numbers and 12 letters). Trail Making Test Part B is similar to Trail Making Test Part A, but requires more cognitive flexibility as the participant shifts from number to letter sets. (Figure 4B).
4. Conduct task 6 — Clock Drawing (1 min).
   1. Instruct the participant to draw an analog clock with numbers 1 through 12 and set the time to 10 past 11.
      NOTE: This test is part of the MoCA¹² and assesses the participant’s visuo-constructional skills. (Figure 4C).
5. Conduct task 7 — Benson Complex Figure Copy (Delayed)¹⁰ (1 min).
   1. Instruct the participant to draw the Benson complex figure from memory on a blank piece of paper.
      NOTE: This test assesses the participant’s visuo-constructional and visual memory function. (Figure 4A).
6. Conduct tasks 8 and 9 — Number Span Test (Forward and Backward)¹³ (10 min).
   1. Instruct the participant to repeat the numbers that the experimenter reads out loud.
   2. For task 8 (forward), instruct the participant to repeat the numbers in the same order, and for task 9 (backward), to repeat them in the reverse order. For both tests, there are two trials per sequence length ranging from 3 to 9 digits for the forward, and 2 to 8 digits for the backward test. Continue testing until two number strings at the same length are failed.
      NOTE: Both tests measure the capacity to briefly hold information, but the backward test (task 9) also measures the ability to manipulate the numbers and reverse the sequence. As such, in the context of the present protocol, this task measures a memory and a cognitive component in relation to the voice biorhythms output.

Figure 4: Towards clinically informed digital biomarkers and scoring cards: Digitization of traditional clinical tests to enable integration of clinical criteria and digital biophysical data. Sample traces from standardized cognitive tests. (A) Benson complex figure provided to participant to reproduce immediately (center), or after a 10-minute delay from memory (right). (B) Trail A task involving numbers to be connected by a line in an ascending order (top), and Trail B task involving alternating between letters and numbers (bottom). (C) Drawing a clock with instructions to set time to 10 past 11. For all sample traces, gray lines represent the trajectory of pen lifts during the drawing task, and blue lines represent the actual pen drawings. For analysis, we examine both types of trajectories. Please click here to view a larger version of this figure.

7. Conduct tasks 10, 11, and 12 — Pointing (10 min).
   1. Position a target in front of the participant to point and touch.
   2. For task 10 (control), instruct the participant to point at the target 40 times at a self-paced manner with the dominant hand.
   3. For task 11 (metronome), instruct the participant to point at the target 40 times at a self-paced manner, while setting the metronome beat at 35 bpm in the background, but do not instruct anything about the metronome beat.
   4. For task 12 (paced pointing), instruct the participant to point at the target 40 times at the pace following the metronome beat set at 35 bpm.
      NOTE: The pointing task allows for separating the participant’s motion by deliberate (forward movement, from the time the hand is resting until the hand reaches the target) and spontaneous (backward movement, from the time the hand is touching the target until the hand reaches back to its resting position) segments, thereby analyzing the biophysical signals when the participant is exerting voluntary control (deliberate motion), and when the participant is exerting little control (spontaneous motion) (Figure 5A). By introducing the metronome beats and requiring the participant to pace the pointing at the metronome beat, comparison can be made of the biophysical signals when the participant is moving at a self-paced manner, and when the participant is actively controlling the movement pace to an external beat.
8. Conduct tasks 13, 14, and 15 — Walking (25 min).
   1. Instruct the participant to walk naturally for 5 minutes under three different conditions.
   2. For task 13 (control), instruct the participant to walk naturally around the room.
   3. For task 14 (metronome), instruct the participant to walk naturally around the room while setting the metronome to beat in the background for 12 bpm.
   4. For task 15 (paced breathing), instruct the participant to walk naturally around the room while he/she paces the breathing rate to the metronome beat set at 12 bpm.
      NOTE: By introducing the metronome beat, comparison can be made of the biophysical signals while the participant is using the entire body to move and breathe at a self-paced manner, and when the participant is actively controlling the breathing pace (which usually occurs automatically) while moving the entire body (Figure 5B).

Figure 5: Three-dimensional (3D) trajectories and their speed-amplitude time series output during voluntary motions of the upper body. Goal-directed pointing behavior to probe volitional control using a version of the kinetic tremor task of the Movement Disorder Society – Unified Parkinson’s Disease Rating Scale (MDS-UPDRS). (A) Participant and avatar in sitting position performing the upper body, goal-directed pointing task (top); 3D-positional trajectories of forward (to target) and backward (to resting position) movement (bottom left); and the corresponding linear speed profiles showing the time series of fluctuations in speed amplitude (m/s) and inter-peak interval timings (ms), derived from the linear velocity vector flow (bottom right). (B) Participant and avatar in the walking task(top); 3D-positional trajectories of different body parts (bottom left) and corresponding speed profiles (bottom right). Please click here to view a larger version of this figure.

9. Conduct tasks 16 and 17 — Face Video (10 min).
   1. Instruct the participant to sit comfortably, and set up a camera in front of the participant.
   2. For task 16 (control), instruct the participant to stare at a space without any stimuli for 5 minutes.
   3. For task 17 (smile), instruct the participant to watch a funny video for 5 minutes.
      NOTE: The set up takes approximately 30 minutes, and the entire protocol takes approximately 60 minutes, and an additional 10 minutes for PWP.

Results

Parameters of interest

There are many motion parameters that we can extract from the trajectories of biophysical digital signals generated by the person’s nervous systems. Here we focus on the electroencephalogram EEG waveforms (representative of the central nervous systems, CNS output), bodily movements (representative of the peripheral nervous systems, the PNS output), and the heart signals (representative of the autonomic nervous systems, ANS output).

For the CNS- and ANS- related signals, we use the fluctuations in peak amplitude of the EEG and ECG waveforms (µV) converted to the unitless (standardized) micro-movement spikes (MMS) (see below). For the PNS-related signals, we use the trajectories of the center of mass (COM) and their speed profiles’ time series (m/s), to derive the corresponding unitless MMS. Once we gather the MMS, we can integrate them into a weighted-undirected graph, based on a set of pairwise signal analytics across sensors and levels of nervous systems function. This step enables us to use network connectivity analyses on the combined signals. We then produce interpretable graphs^14,15 depicting changes in self-emerging network topology. In particular, when we compare these graphs across the three pointing and/or walking tasks, we can observe how the biophysical signals respond to an external rhythm in a passive manner (i.e., when the metronome beats spontaneously entrain the biorhythm) and in an active manner (i.e., when the participant deliberately attempts to pace the hand pointing or the walking motions to the metronome beat). We can also study patterns of information transmission across the network nodes representing the CNS, the PNS and the ANS levels of function.

Stochastic Analyses on the MMS from multi-functional layers of biophysical signals

The biophysical signals harnessed from the wearable grid of sensors across the body, give rise to time series of peaks and valleys, which vary in amplitude and timing. The MMS of these biophysical signals¹⁶ are the fluctuations in amplitude and timing of the peaks, whereby the amplitudes are normalized to a unitless value ranging in the real [0,1] interval, thus allowing integration and comparison across signals from different functional layers of the nervous systems: CNS, PNS and ANS. These disparate layers of functionality require different levels of neuromotor control and possess different amplitude ranges across individuals. They also have different inter-peak-interval timings. While the MMS normalization conserves the timing of the original peaks, it also captures the variations in the amplitude. Such a normalization is obtained by dividing each local peak amplitude by the sum of the peak amplitude and the average of the signals sampled within the two neighboring local minima surrounding the peak:

These continuous spikes in the [0,1] real numbers interval preserve the information on the timing and amplitude fluctuations, while enabling us to treat the time series as a random process. We then adopt a Gamma process under the general rubric of Poisson random processes, commonly used in the field of computational neuroscience to analyze binary spikes.

These analytical methods have been explained elsewhere in detail^3,14,17,18 and see Figure 6 for explanation of the analytical pipeline and proposed visualization to aid clinical interpretation. Here, we use the Gamma parameter plane spanned by the shape and scale parameters of the Gamma PDFs that we empirically estimate from the MMS waveforms. We also plot the points of the corresponding Gamma moments on a four-dimensional graph, whereby the mean, variance and skewness span three of the dimensions and the kurtosis is represented in the size of the marker that we use to denote the person’s stochastic signatures.

For each task, the MMS peak data derived from the biophysical time series are gathered and fitted to a Gamma PDF using maximum likelihood estimation (MLE) with 95% confidence intervals for each Gamma-parameter: the shape denoting the distribution’s shape, and the scale denoting the dispersion (noise to signal ratio). As such, we estimate with high confidence, the best continuous family of PDFs that captures the biorhythms fluctuations of the person’s nervous systems across different levels of functionality that these tasks demand. These functional layers span from high-level abstract cognitive skills and memory abilities, to voluntary neuromotor control and spontaneous consequential motions derived from the goal-directed tasks. We also examine automatic motions and the system’s capacity for physical entrainment with the metronome’s beats per minute.

Figure 6: Statistical analytical pipeline for development of dynamic digital biomarkers and applications to future application (APP) development. (A) Center of mass (COM) 3D positional trajectory while the person walks around. (B) Speed fluctuations of the COM with peaks in amplitude highlighted with a red dot. (C) Standardized MMS from the fluctuations in COM-speed peaks (red dots) in the [0,1] real-value interval. (D) Frequency histograms of the MMS peaks (red dots in MMS). (E) Probability density functions (PDF) fitting the frequency histogram evolving in time from the memory less, most random, exponential distribution (red) to some transitional skewed distribution (blue), to the Gaussian (predictive) with low dispersion (GOAL, green circle). This ideal distribution (green) appears in young athletes and sets the target for predictive, high signal-to-noise ratio cases. (F) Maximum likelihood estimation (MLE) used to fit the best PDF (with 95% confidence) to the empirical data. Resulting parameter values localize on the Gamma parameter plane the evolving stochastic signatures (Gamma process) from the COM speed fluctuations: “log shape” represents the shape of the distribution ranging from exponential to skewed to symmetric (ideal Gaussian); “log scale” is the noise-to-signal ratio (dispersion) implying the type of kinesthetic feedback the brain (most likely) gets^22,23. Colors represent stochastic states as they dynamically evolve over time. (G) Probability distance (Wasserstein metric distance⁷) from the ideal GOAL of low noise-to-signal ratio (low dispersion) and high predictability (symmetric distribution) found in neurotypicals; away from poor feedback (random noise) found in more advanced PD, deafferented patients^24,25,26,27, schizophrenia²⁸ and autistic individuals^3,18,22,29. (H) Simplified visualization to represent these stochastic states evolving in time is based on the power law relationship between the shape and scale parameters. Such visuals for future app development can provide real-time, easy to understand, clinical feedback to PWP and the healthcare team in order to improve precision in assessment and treatment planning. Please click here to view a larger version of this figure.

Results from Different Modes of Data

For the purpose of this paper, we examined three PWP and three healthy participants’ data with the demographics shown in Table 1. The three PWP were chosen from 10 PWP we recorded, to represent a case with mild PD (unified Parkinson's disease rating scale [UPDRS] score 16), a moderate case (UPDRS score 25), and a severe case (UPDRS score 44). Two healthy participants were chosen from 15 healthy individuals we recorded, as they most closely matched the PWP in age and gender; one healthy participant was chosen from the younger age group to represent an ideal healthy reference for comparison.

Participant	Disorder	Sex	Age	UPDRSa
1	PWP	F	64	44
2	PWP	M	65	25
3	PWP	M	64	16
4	none	M	26	n/a
5	none	F	65	n/a
6	none	M	67	n/a
aMaximum score 108.

Table 1: Demographics of the participants.

In the cognitive and memory test (pen movement), the positional trajectories of the pen motions were recorded, and the linear velocity was extracted to derive the time series of the speed amplitude. Then the MMS from the fluctuations in speed amplitude were derived for each drawing task. The patients were grouped according to the Movement Disorder Society−UPDRS (MDS-UPDRS) median ranked scores, with highest PD severity indicated by the highest ranked score above the median value of the cohort. Three representative participants from each cluster that the median ranking of the scores determined are used to display the results in relation to three representative controls. One control is young (26-year-old male) representing an ideal state of neuromotor control during youth. The other two are healthy aging controls; one 65-year-old female and one 67-year-old male. Figure 7 depicts the trajectories of COM and Figure 8 depicts the corresponding Gamma processes from the MMS derived from its trajectory speed profiles.

Figure 7: Sample trajectories of the COM summarizing the trajectories of 17 body locations during the performance of select cognitive drawing tests with actual digitized traces. Sample performance during the Benson complex figure (tasks 1 and 7) and Trail making tests (tasks 2−5). (A) Benson complex figure used in this protocol. (B) Trail test with numbers and letters whereby the aim is to connect them in a prescribed order by drawing a line along an orderly path (task 4 — Trail B). (C) Sample pen traces of the Benson complex figure and COM 3D trajectory from 65-year-old female control (blue) and PWP with MDS-UPDRS score of 44 (red). Left side shows the results when the participant immediately copied the figure (task 1), and right side is when the participant recalled the figure after 10 minutes of delay (task 7). In both cases the continuous drawing along with pen lifts are captured to show variations from hesitation, etc. (D) Performance of the control vs. PWP in the Trail A (task 3) and Trail B (task 5) tasks. Notice the changes in the COM trajectories and actual drawings. Please click here to view a larger version of this figure.

Figure 8 shows the results from the best MLE fitted Gamma-PDF with 95% confidence for the healthy participants and patient participants. In each drawing task, the patients stratified apart from the controls. Further, they stratified within their group and differentiated according to the MDS-UPDRS median-ranked scores. Each patient is represented as the moments of the empirically estimated PDF on the top panels, while the bottom panels depict the PDF curves for each participant. Across panels, the reader can appreciate the family of PDFs spanned by each person across the tasks. Contrast this approach with the one-size-fits all (assumed) parametric models. Figure 8D shows the continuous drawing of the clock figure (task 6) as captured by the tip of the pen (including pen lifts).

Figure 8: Micro-movement spikes (MMS) stratify the cohorts and build interpretable personalized dynamic digital biomarkers for cognitive tasks. The moment-by-moment fluctuations (the unitless MMS) derived from the 3D trajectories of the COM during cognitive testing uniquely localize each participant along a stochastic map. The COM summarizes 17 position-trajectories across the body co-registering motions while the person performs cognitive tasks and draws on a digital tablet. (A) The Gamma moments were empirically estimated during the Trail B test (task 5) connecting letters and numbers (mean x-axis; standard deviation y-axis; skewness z-axis and kurtosis marker size) from max-likelihood estimation with 95% confidence on the top panel. Each marker represents the person’s unique location in probability space. Each point denotes a unique separable PDF shown on the bottom panel, thus stratifying the UPDRS median ranked scores of the PWP (legend). (B) The task of copying the complex Benson figure (task 1). (C) Clock drawing task (task 6). (D) the actual clock drawings captured by the digitized pen showing the continuous trace (inclusive of pen lifting). All kinematics from the pen and full body in motion are synchronously registered with EEG-EKG (not shown) to empirically derive the multi-layered (cognitive, voluntary, spontaneous, automatic, autonomic) stochastic signatures. This personalized approach (Gamma process) contrasts with the ‘one-size-fits-all-model’ that assumes a theoretical PDF, and through grand averages, smooths out as ‘noise’ important fluctuations in amplitude and timing of the waveforms. The micro-movement spikes (MMS) and Gamma process approach stratify the cohorts and build interpretable personalized digital biomarkers for cognitive tasks. Please click here to view a larger version of this figure.

Next we present results from bodily movements (voluntary pointing vs. automatic walking). The drawing motions do not require the same level of goal-directness as the task of pointing to a target (i.e., tasks 10−12). To be able to assess the levels of volitional control, we next use the task of pointing to a spatial target. As before, we use the trajectories from the COM summarizing the kinematic motions of 17 sensor positions (Figure 9). We first take the speed amplitude time series and then derived the MMS from the moment-by-moment fluctuations in amplitude. Left panel in Figure 10 shows results from the stochastic analyses during the pointing task at baseline (task 10). Middle panel in Figure 10 shows the results from the pointing task when a metronome is set at 35 bpm, without instructing the participant of its presence (task 11). Right panel in Figure 10 shows the results from the case when the participants are instructed to pace their pointing movements to the metronome’s rhythm (task 12).

Figure 9: Three-dimensional trajectories of the COM during the pointing tasks in three different contexts, regular pointing to take a baseline measurement (task 10), pointing while a metronome beats in the background at 35 bpm without alerting the participant of the presence of the metronome (task 11), and pointing in the presence of the same metronome beat but after instructing the participant to pace the motions to the metronome’s rhythm (task 12). (A) Performance of the control participant. (B) Performance of the PWP in the group with the lowest severity rating according to the median ranking of the MDS-UPDRS scores of the entire cohort. (C) PWP in the mid severity-range group. (D) PWP in the highest severity group. Note the degradation of the COM trajectory with the increase in the MDS-UPDRS scores. Please click here to view a larger version of this figure.

The results from the Gamma process are shown in Figure 10 whereby it is possible to distinguish each PWP sub-type and track the change in stochastic signatures from context to context.

The pointing task revealed the sensitivity of these analyses to contextual situations. For the same pointing task, changes in the metronome condition elicited different stochastic signatures between conditions. Particularly, we can observe the change in COM biorhythms when they spontaneously (without instructions) entrained with the metronome beat against the condition whereby the participant was instructed to deliberately pace the pointing motions to the metronome’s beat. This task demonstrated that at baseline of upper body motions, the levels of voluntary control differ across PWP according to different MSD-UPDRS scores. Specifically, the lower the score, the lower the noise to signal ratio (the scale parameter value) on the Gamma parameter plane is (Figure 10A), and the more symmetric the PDF shape value is. This orderly relation between median-ranked UPDRS scores and digital data was altered with the presence of the metronome and further differentiated between the spontaneous (uninstructed) and the deliberate (instructed) pointing conditions.

Figure 10: Dynamic digital assessment of three specific pointing tasks. Gamma process output from the MMS derived from the fluctuations in the COM speed time series differentiates within and between PWP and controls groups during the three pointing tasks (tasks 10−12). (A) The Gamma parameter plane shows the differences between PWP and controls. (B) For each pointing condition, the Gamma moments empirically estimated from the Gamma process, distinguish between PWP and controls; and within each group, the stochastic signatures stratify the participants into different points. Each task context changes the location of the point on the map. (C) The family of PDFs also distinguish each participant, each group and reveals the statistical differentiation across task contexts for goal-directed pointing. Please click here to view a larger version of this figure.

We then asked if these influences of the different conditions on the voluntary pointing behavior would extend to automatic walking motions. To that end, we performed the same protocol as above, i.e., using the metronome, while the participant walked in the room. The metronome’s beat was set in this case to 12 bpm. Figure 11 shows the COM trajectories for the controls and the PWP median-ranked by the MDS-UPDRS scores. The results from the stochastic analysis of the walking task are depicted in Figure 12.

Figure 11: Dynamic digital assessment of three specific walking tasks. Walking task to ascertain noise-to-signal ratio of the fluctuations in speed amplitude, derived from the 3D trajectories of the COM from 17 locations across the body. (A) 3D trajectories of the COM in the control participant, while the person paces back and forth during natural walking (task 13); walking in the presence of a metronome without instruction, to measure spontaneous entrainment to the metronome beats (task 14); and walking while deliberately pacing the breathing rate to the metronome’s beats, as instructed (task 15). (B) PWP with lower ranked UPDRS score. (C) PWP with higher UPDRS score degrades the 3D COM trajectory. (D) PWP with the highest ranked UPDRS score shows very perturbed COM trajectories. Please click here to view a larger version of this figure.

Figure 12: Capacity for spontaneous and instructed physical entrainment while walking. MDS-UPDRS-informed digital biomarker of walking task. (A) Stratification of PWP during natural walking in line with median-ranked scores. (B) Metronome spontaneously shifts stochastic signatures. (C) Instructed paced walking to the metronome’s beat again changes signatures. (D-F) Gamma log-log parameter plane localizes groups along different PDFs shapes and scale (noise-to-signal ratio) with noisier and more random fluctuations in PWP. (G-I) Empirically estimated PDFs represented in panels D-F above span a family that shifts with context in a manner that is unique to each person. Please click here to view a larger version of this figure.

Since all cognitive and memory tasks can be performed while the computer’s webcam records the face, it is possible to use OpenPose, an open source machine learning software that is openly available to researchers³⁵, and extract the facial information, which can be used to infer information related to sentiment or emotional content. Often in PD the facial expressions decrease as the dopamine depletion may eventually result in low muscle tone. Here the Gamma Process can also be used to ascertain areas of the face that are most active during a given task, or to probe emotional content by ascertaining area-transitions across emotions. Figure 13 shows an example of such analyses using video from the face of a participant during tasks 16 and 17. The 70 points used to capture micro-motions of the face are placed in correspondence with the known trigeminal nerves areas V1 (29), V2 (14), V3 (27)⁸ (Figure 13A) to assess, for example, in this case, which area changes maximally when transitioning from a neutral face to a smile. This type of analysis can be systematically used to probe other non-motor aspects of PD, including depression and social communication in general.

To compensate for uncertain camera zoom, natural human motion, and actual face size, we normalize the face in the following way: Assuming the camera is stationary, we map each face to a “unit face” with x̄’, ȳ’ = 0 and unit variance. For each frame in the video, we normalize each point x’ = x- x̄, y’ = y- ȳ, and scale each coordinate by the variance of the overall mask for the given frame, to achieve unit variance for each mask. We then treat each point in the time series of faces as deviations from the previous mask, as we assume that the face does not plastically deform during the recording. The result is a 70-channel timeseries of position coordinates (Figure 13B). The fluctuations in speed amplitude extracted from the positional and velocity flows giving raise to these time series are converted to MMS and input to the Gamma process, thus revealing PDFs and their shifts with emotional transitions (Figure 13C). For example, transitioning from a neutral expression to a smile seems imperceptible in Figure 13B, yet the stochastic shifts reveal the zone V2 as the most sensitive, maximally changing the PDF.

Figure 13: Sentiment analyses from video data captured using OpenPose. (A) Facial areas according to the trigeminal nerve, which carries general somatic afferent fibers (GSA). These fibers innervate the skin of the face via ophthalmic (V1), maxillary (V2) and mandibular (V3) divisions used here to study the transitions across facial expressions (neutral vs. smile). (B) Using a few minutes of video captured with commercially available video cameras, it is possible to extract face information with OpenPose and render the 70 points across the face according to the V1, V2, V3 areas (color codes as in panel A). The MMS from these time series are then input to a Gamma process and the scale and shape Gamma parameters empirically estimated for each condition. (C) The analyses reveal the area V2 is in this case maximally affected by transitioning from neutral to smile, for this particular person, as the change in PDF is the largest. Please click here to view a larger version of this figure.

Figure 14: Integrating digital biophysical signals from multiple layers of the nervous systems using weighted un-directed graphs and information theoretical methods. Network connectivity of pair-wise mutual information between all EEG, motion (magnetometer), and EKG signals. (A) Representative healthy participant’s connectivity measure during the three walking tasks - task 13 control (left), task 14 spontaneous metronome placement (middle), and task 15 instructed paced breathing (right). Each node represents a single sensor’s signal; the color of the line represents the level of MI, where brighter color indicates higher connectivity; the node color represents the average MI of that sensor’s signal with all other sensors. The color scale is set to be the same across all tasks and all participants and is arbitrarily set to have the brightest color for the maximum MI value across all tasks and participants, and to have the darkest color for the minimum MI value across all tasks and participants. The healthy participant’s connectivity shows the strongest connection across the brain and body nodes. (B) PWP with UPDRS 16 connectivity measure, with the same schematic layout as in panel A, shows less density in its connectivity than the healthy participant’s network. (C) PWP with UPDRS 44 connectivity measure, with the same schematic layout as in panel A, shows the sparsest connectivity pattern across the brain and body. Please click here to view a larger version of this figure.

We next ascertain the amount of information transmitted by these biorhythms. To that end, we use the information theoretical approach developed by Shannon¹⁹, using mutual information (MI) between each pair of sensors’ signals (i.e., EEG sensor, motion sensor, ECG sensor). To that end, we obtained the MMS amplitudes from each waveform type.

MI between two sensors assesses the level of uncertainty reduced in one sensor’s signal by introducing the information of the other sensor’s signal; when MI is high, this implies that the two signals are highly connected, and when it is low, this implies that the two signals are mostly independent. Specifically, MI is computed as:

where is the normalized histogram value of the distribution of values for signal X, and is the normalized histogram value of the joint distribution of values for signal X and Y. The sampling bins were set to increment in 0.05, ranging from 0.5 to the maximum amplitude value. Details on the derivation of this formula can be found in many literature on information theory with application on clinical analyses^20,21.

Overall, the healthy participant exhibited a more connected network during the three walking tasks, while the patient participants had a sparser connection, as shown in Figure 13. Not only information transmission across the network, whereby mutual information was derived from self-sensed biorhythms, is much lower in PWP; more importantly, there are fundamental differences in patterns of MI transmission between conditions, that vary from patient to patient. Here we can further utilize connectivity metrics from network analyses to summarize topological features of these evolving networks and as such, provide additional indexes of physical entrainment capacity, reflecting as well the communication levels between brain, body and heart when physical entrainment related to the metronome occurs vs. when it does not.

Discussion

This work introduces a new protocol that enables integration of traditional clinical tests with digital data from biophysical signals output by the nervous systems as the person performs such tests. We introduce the use of SPIBA and the MMS as a unifying platform to combine disparate types of data such as discrete scores from pencil-and-paper observational methods and continuous digital data from biophysical sensors. The methods are illustrated using a cohort of PWP and age- and sex- matched controls, with an additional young control as the ideal healthy reference for comparison. We show that traditional clinical tests (e.g., those that may be part of the MoCA and the MDS-UPDRS) can be used to median rank the cohort and automatically extract, from the inherent variability of the group’s scores, information that stratifies data in the digital realm according to clinically defined levels of severity. Such levels align well with levels of clinical MDS-UPDRS and the cognitive/memory test performance. At another layer of implementation then, we examine the biorhythms of the nervous systems harnessed from the CNS, the PNS and the ANS layers, thus characterizing different levels of autonomy and control. We provide sample data and stochastic signatures derived from such data, examined through the optics of the clinical criteria. Under such approach, we can differentiate the patients from the healthy controls; and through digital biorhythms, differentiate within the PWP, on their clinically-defined levels of severity.

By aligning the biophysical digital data with the clinical criteria this way, we provide an interpretable set of criteria that can more dynamically track individualized shifts in outcomes. We coin these new outcome measures dynamic digital biomarkers, because they are based on digital data, yet they provide interpretable outcomes according to well established and validated clinical criteria. They are derived from the time series of nervous system processes, and they capture the dynamic nature of such. In particular, we are able to use both motor and non-motor criteria. As such, we can begin to quantify non-motor aspects of PD that are now known to precede the deterioration of motor symptoms that conventionally have defined the disorder thus far. For example, facial analyses such as those presented here could be used to examine spontaneous facial micro-gestures during REM sleep to build a repertoire of those which could forecast the deterioration of motor activities. Likewise, we could use these methods to examine levels of pain during daily activities and assess their potential correlates with activities during REM sleep time. This is important, because both REM sleep and pain dysregulation are known to precede motor issues later defining the level of severity of PD^5,6,7. Using our digitization of the various traditional clinical tasks, other non-motor activities embedded in the clinical tasks used to probe cognitive and memory abilities were characterized here, and indexes of such activities mapped to motor symptoms. These methods are amenable to connect basic research and clinical practices in Parkinson’s disease. They can also be extended to other disorders of the nervous systems.

Besides motor output from the face, during REM sleep, we can examine facial gestures during natural social situations within the frame of self-sensing or kinesthetic reafference to gauge the levels of reafferent feedback the brain of the patient most likely gets. Here, despite the very subtle differences in micro-gestures across the facial areas corresponding to the trigeminal afferent regions V1, V2, V3, it was possible to pinpoint in the representative participant, which region of the face maximally shifted the stochastic signatures when transitioning between neutral and naturally smiling states. This suggests that using the SPIBA and the MMS, we will be able to evaluate other non-motor(sensory) aspects of PD related to difficulties with sensory input from kinesthetic-touch channels. These have been found to be problematic in PD, even during early stages of the disorder²⁴. Because sensory and motor go hand in hand, this information could help us forecast more obvious motor issues surfacing later in the progression of this disorder^1,7. We posit that these kinesthetic channels along trigeminal areas of the face may also help us dissociate different types of pain dysregulation, including those related to sleep alterations^5,6.

The current methods provide a new way to examine the biophysical signals obtained from the central, peripheral and autonomic nervous systems in tandem, under different conditions requiring different cognitive skills and different levels of autonomy and neuromotor control. Using the SPIBA framework, where stochastic analyses and pair-wise network analyses are applied on the standardized MMS data, it is also possible to objectively characterize cognitive activities. The fifteen tasks that were used in this experiment require different types of cognitive skills (e.g., visuo-constructional skills, visual memory, perceptual-scanning skills) and different levels of cognitive control (e.g., deliberately pace the pointing speed, pace the breathing rate as instructed). For that reason, the stochasticity and connectivity patterns of CNS-PNS-ANS information transmission of biophysical signals exhibited during these tasks, can be used to characterize different levels of cognitive loads and their impact on the motor output.

While we underscore the advantages of our new analytical methods and protocols, we also point out caveats and practical limitations that ought to be considered when adopting our recording platform for synchronous data gathering. This is because in this set up, there are multiple recording software types streaming on a single computer for synchronization purposes, requiring the computational power of the computer to be high, or else one could incur data loss, computer freezing and/or excess noise. In the current design, two streaming software (EEG, and motion capture) and LSL were run on a single computer. As such, we had to be mindful of processing overload and possible computer freeze. This was one of the reasons why we used one of the EEG channels to extract the ECG signals. Computers with higher memory capacity and faster processors may be able to handle a separate ECG software simultaneously streaming with the EEG and sensor grid of kinematics. These issues are practical in nature and independent of the analytical methods (SPIBA) and standard data types (MMS) that we offer. Yet, we believe it is important to alert the end user of the need to assess computational power before designing the protocol for data co-registration from multiple streams.

Another caveat we point out is that, the 15 tasks illustrated in the protocol are a subset of what can be used to develop dynamic digital biomarkers. For the purpose of this paper, we limited to a few tasks due to space constraint, and chose the ones that involved different levels of control and bodily motion, and indeed we can add other tasks not included in this paper. Our goal is to derive a smaller subset of tasks that would require less time and effort. In fact, from our lab, pointing tasks (tasks 10−12) are a set of tasks that we found to be an effective and efficient way to characterize the stochastic signatures of biorhythms varied by differing levels of voluntary control and neurological disorders, including PD^24,30.

The representative results shown in this study are a small subset of what can be done with the MMS datasets derived from biosensors waveforms and cameras’ motion caption, using SPIBA methods. For illustrative purposes, we examined the MMS in amplitude and focused on the fluctuations in the linear speed amplitude derived from the person’s COM. The COM is a summary signal from all 17 body sensors grid that we co-registered. However, we could extend the analyses to other rotational parameters, and to other kinetic variables (e.g., forces and pressure) that generate time series of fluctuating parameters (e.g., as we did with the face data.) Also, due to space constraint, we only illustrated the analysis of EEG data based on its scalp amplitude information, but we can also apply these analytics to data derived from the source space³¹. For all modes of data, we could also examine the stochasticity of the times between peaks (instead of peak amplitude), which also generate time series. Other time series of parameters can be derived from such waveforms, and their MMS can be used to ascertain cohesiveness and connectivity from the network that was constructed^32,33,34. Furthermore, these analyses can also be extended to the frequency domain³⁴. In addition to the mutual information network analysis, we could have focused on other topological features of the network to differentiate PWP and controls and to stratify PWP. For the purpose of this paper, we focus on the usefulness of these analytics as a tool, but through this type of characterization, we will gain knowledge to provide clinically-informed interpretations of the digital data that these analytical tools provide.

The current methods described in this study serve to introduce some of the many possible ways that SPIBA and the MMS can be applied to clinical and digital data integration. We offer this unifying platform, standardized data type and experimental protocol in the hopes to finally inform digital data of clinical criteria, and likewise add more precision from the digital data to the traditional pencil-and-paper methods. Such improvement will 1) enable more accurate tracking of symptom change in response to treatment, 2) enhance understanding of natural PD progression over time, and 3) facilitate stratification of PD symptom presentation (which may dictate unique clinical recommendations for each subgroup). As such, we hope to apply these methods to further research in PD, but also see usefulness in clinical application as well. Using commercial-grade devices such as mobile phones, biophysical data can be obtained to perform the analytics that we illustrated in this paper. Currently, there are efforts in collecting such digital data on a larger scale such as the mPower app study from University of Rochester (https://parkinsonmpower.org) and Kaggle. Indeed, using these open-access data repositories, we were able to stratify PD and normal aging individuals from accelerometer data obtained from mobile phones, and to automatically classify activities that are embedded in the clinical tests presented here³⁵.

As a next step, we aim to collect more data from a wider range of PWP population and their matched control participants and record them at different time points to be able to perform both a cross-sectional and longitudinal analysis using our methods. We foresee such collected digital data would offer much more than the sum of their parts, and truly realize the tenets of precision medicine in neurology and psychiatry.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Enobio 32	NE Neuroelectrics	NE006WF	wearable, wireless electrophysiology sensor system for the recording of EEG.
Inking Pen	Wacom	KP1302	tablet pen
Intuos Pro	Wacom	PTH451	pen tablet
Lab Stream Layer System	n/a	n/a	open source software to synchronize different devices
Microphone	Zaffiro	B07BDFP6XC	computer microphone
MovAlyzeR	Neuroscript	Version 6.1.0.0.	pen movement caption software
MTw Awinda wireless motion tracker	Xsens	MTw Awinda	motion capture system
MVN Analyze	Xsens	Version 2019	motion-tracking software
NIC 2.0	NE Neuroelectrics	NE001SW2	Neuroelectrics Instrument Controller (NIC) EEG streaming software
OpenPose	n/a	n/a	open source machine learning software to extract facial information