A Machine Learning Approach to Design an Efficient Selective Screening of Mild Cognitive Impairment

Summary

This methodology produces decision trees that target population groups more prone to suffering from mild cognitive impairment and are useful for cost-effective selective screening of the disease.

Abstract

Mild cognitive impairment (MCI) is the first sign of dementia among elderly populations and its early detection is crucial in our aging societies. Common MCI tests are time-consuming such that indiscriminate massive screening would not be cost-effective. Here, we describe a protocol that uses machine learning techniques to rapidly select candidates for further screening via a question-based MCI test. This minimizes the number of resources required for screening because only patients who are potentially MCI positive are tested further.

This methodology was applied in an initial MCI research study that formed the starting point for the design of a selective screening decision tree. The initial study collected many demographic and lifestyle variables as well as details about patient medications. The Short Portable Mental Status Questionnaire (SPMSQ) and the Mini-Mental State Examination (MMSE) were used to detect possible cases of MCI. Finally, we used this method to design an efficient process for classifying individuals at risk of MCI. This work also provides insights into lifestyle-related factors associated with MCI that could be leveraged in the prevention and early detection of MCI among elderly populations.

Introduction

Population aging is increasing the prevalence of chronic and degenerative diseases, especially degenerative dementias, which are expected to affect more than 131 million people worldwide by 2050¹. Among all the degenerative dementias, Alzheimer's disease (AD) is the most common with an overall prevalence in Europe of 6.88%². Due to the ever-declining independence of AD patients, this group should start receiving support as soon as AD starts to manifest. Therefore, the early detection of prodromal signs of AD, such as mild cognitive impairment (MCI), is essential.

MCI is defined as an intermediate cognitive decline stage corresponding to normal aging and severe deterioration due to dementia³. According to estimates by Petersen et al.⁴, the prevalence of MCI is 8.4% among people aged 65-69 years and reaches 25.2% for those aged over 80 years. MCI results in individuals experiencing more difficulties than expected in the execution of low-level cognitive skills, especially those related to memory and language, but does not interfere with the activities of daily living.

Screening is not synonymous with diagnosis; the diagnosis of MCI will always be a clinical task whereas screening methods can only inform us that a patient has a higher probability of suffering from this pathology and that there is a well-founded suspicion of MCI that should be confirmed clinically. Hence, primary healthcare workers (doctors, pharmacists, nurses, etc.) could benefit from the availability of simple screening methods (brief cognitive tests) that can be applied in minutes. Ideally, these would objectively identify patients with a high probability of suffering an MCI so that they can then be clinically tested by general or specialized physicians.

Given that the early detection of MCI is becoming an essential task within the context of public health, this work aimed to identify which characteristics are useful in the targeted identification of MCI in screening tests of elderly populations. These groups would then be more thoroughly tested for MCI in tests administered by primary health care providers. This methodology provides a decision tree with the appropriate algorithms for identifying the population groups to target.

Among these characteristics, age is one of the most consistent factors associated with the development of this pathology. Other relevant characteristics are related to demographics or lifestyle⁵. Among the latter, some studies have identified the duration of daytime or nighttime sleep as a risk factor that can lead to the diagnosis of MCI^5,6,7,8,9. The prolonged consumption of medications such as benzodiazepines, consumed by an estimated 20%-25% of older adults^10,11, can also influence sleep hours and the development of MCI^12,13. Indeed, prolonged treatments for chronic diseases may be important features useful in the pre-selection of individuals with a high risk of suffering from MCI.

Here, we developed data-based models that use automatic learning algorithms, a decision tree, and a predictive tool to increase the efficiency of the methodology for detecting MCI by discriminating which characteristics play an important role in the early detection of MCI. The resultant decision tree presented here was produced using a specific cohort of Spanish patients using community pharmacies. However, this method would also be useful among other populations with different characteristics.

This work was completed in collaboration with primary healthcare and specialized medical doctors. Community pharmacies were ideal for testing this algorithm because they are close to patients, have long opening hours, and are frequently visited and consulted. Degenerative dementias are complex conditions which are not always well understood by primary health care providers¹⁴. Therefore, becoming involved in the process will raise awareness of people suffering from MCI and dementias.

Protocol

The methodology applied in this study has been previously published⁵ in work carried out at the University CEU Cardenal Herrera together with community pharmacies in the region of Valencia (Spain) associated with the Spanish Society of Family and Community Pharmacy (SEFAC). This current study was reviewed and approved by the Research Ethics Committee at the Universidad CEU Cardenal Herrera (approval no. CEI11/001) in March 2011. All individuals involved in the study gave their written informed consent to participation in accordance with the Declaration of Helsinki.

1. Selection of factors associated with mild cognitive impairment

1. Search for terms related to MCI for use in screening Cochrane Systematic Reviews (e.g., cognitive impairment, dementia, risk factors, etc.).
2. Search for terms for which there is some evidence of a relationship with cognitive deterioration or dementia published in the PubMed database; these include demographic factors (sex, age, education level, and economic status), social factors (cognitive and social activities), chronic pathologies (cholesterol, depression, hypertension, diabetes, and obesity), and lifestyle behaviors (alcohol consumption, smoking habit, diet, physical activity, and sleep hours).
3. Calculate the odds ratio for qualitative variables or Cohen's d effect size for quantitative variables¹⁵. Select the variables with larger effect sizes for cognitive deterioration or dementia for use in elaborating a questionnaire.

2. Design of the questionnaires

1. Design a questionnaire to collect information about the selected variables, following the guidelines provided by Nardi¹⁶. For instance, the variables used in Climent et al.⁵ were demographic (age, weight, and height [measured with standardized procedures using calibrated scales and stadiometers], sex, education level, and employment type), lifestyle (physical exercise, reading, time spent sleeping overnight and during the day, puzzles, games, TV consumption time, and tobacco and alcohol consumption), and chronic pathologies (hypertension, hyperlipidemia, and diabetes). In addition, record the presence or absence of depression, which is frequently associated with cognitive deterioration.
2. Design a pharmacotherapy follow-up sheet to report all the drugs consumed by the participants at the time of the interview, as in Climent et al.⁵, which used Dader's method¹⁷ to design this sheet.

3. Selection of tests for MCI screening

1. Determine all the tests used to screen for MCI that could be administered by primary healthcare workers (e.g., pharmacists). Reject any tests that must be administered by a specialist. Some of the tests that fulfill these conditions are the Short Portable Mental State Questionnaire (SPMSQ)¹⁸, Mini Mental State Examination (MMSE)¹⁹, Memory Impairment Screen (MIS)²⁰, Picture Memory Impairment Screen (PMIS)²¹, Montreal Cognitive Assessment (MoCA)²², Saint Louis University Mental Status (SLUMS)²³, and Quick Mild Cognitive Impairment (Qmci)²⁴. An exhaustive review of each MCI test is available in Cullen et al.²⁵.
2. Search for a good estimation of the test sensitivities and specificities in the scientific literature.
3. Estimate the time required to administer these tests to healthy individuals.
4. Consider the basic patient characteristics required for completion of these tests. For example, a minimum education level may be necessary because many MCI tests are not suitable for illiterate participants. A set of MCI screening tests is usually applied to increase sensitivity; however, the minimum number of tests must be quickly administered by pharmacists if the final selective screening is intended for a large population. Climent et al.⁵ assessed MCI using the MMSE and SPMSQ tests, with the latter being suitable for the large number of individuals who lived through the Spanish civil war who are illiterate.
   1. Use a variant of the SPMSQ by Pfeiffer¹⁸ was validated in Spanish by Martínez de la Iglesia²⁶. This test has a maximum score of 10 and the cut-off point for establishing cognitive impairment is 3 or more errors (4 or more for illiterate individuals). This test takes between 8 and 10 minutes to complete.
   2. Use a NORMACODERM version of the MMSE validated for Spanish speakers by Blesa²⁷ by adapting the original version by Folstein¹⁹. This screening test has a maximum score of 30 and is corrected according to the patients' years of schooling and ages. Participants who score less than or equal to 24 are considered as MCI cases. The MMSE is a measure of general cognitive function and includes orientation to time and place, written and spoken language, attention span, calculation, and memory. It was administered to all the participants in this study because it is a very short test which takes only around 5 minutes to complete.

4. Subject recruitment

1. Find pharmacists willing to recruit non-institutionalized people to form the study population. The mentioned study by Climent et al.⁵ included people aged 65 years or more who went regularly to the pharmacy and who agreed to participate in this study. Exclude patients with any difficulty in performing these evaluation tests (e.g., because of blindness, deafness, etc.) or who were already being treated for dementia.
2. Provide the participating pharmacists with informed consent forms, which must be completed by every individual taking part in the study. This consent form specifies the title of the research, the objectives of the project, a comprehensible explanation of all the procedures that the participant would take part in, the absence of specific risks, the confidentiality of all the collected data, and the right to withdraw from the study for any reason at any time.
3. Train the pharmacists to administer structured personal interviews to the participants, which should last approximately half an hour per person. Collect data for 1 year and send all the forms to the researchers responsible for data protection in the study. Subsequently follow-up with the patients for 3 months.
4. Instruct the pharmacists how to identify a probable MCI case using MCI tests. Based on Climent et al.⁵ we used SPMSQ scores of 4 or more points (for illiterate participants) or 3 or more points for the other participants and scores of 24 points or less were used in the corrected MMSE test.
5. Instruct pharmacists how to refer MCI cases to a medical specialist (a neurologist) for their clinical diagnosis-the last step in the flow chart used in this research study (Figure 1).

Figure 1: Flowchart of the research study and the proposed selective screening. The left side represents the initial study whose data were analyzed with machine-learning techniques to propose the selective screening for early detection of MCI shown in the right panel. This figure was modified from Climent³⁴. Please click here to view a larger version of this figure.

5. Pharmacist researcher training

1. Contact specialists to organize sessions for training the participating pharmacists in basic knowledge related to cognitive impairment and in managing its screening tools, for instance, the SPMSQ and MMSE.
2. Ensure that the participating pharmacists are aware of the procedures, data collection protocol, and all the possible issues related to data protection. Inform them that the project was approved by a Research Ethics Committee and of the importance of the consent form according to the Declaration of Helsinki.
   NOTE: To perform the study described by Climent et al.⁵, workshops were held at the Official College of Pharmacists and the Cardenal-Herrera CEU University (UCH-CEU), and covered the following: MCI and dementia; diagnostic approaches to MCI and management of the SPMQP and MMSE (taught by the Neurology Service at La Plana Hospital in Castellón); project presentation and explanation of the methodology by senior community pharmacist researchers; and health education and cognitive training by researchers from the Department of Pharmacy at the UCH-CEU University.

6. Study design

1. Calculate a sample size to assess the feasibility of the project. Because this was an observational study, a larger sample will produce more effective tools. There are two ways to determine sample size: one is based on the estimation of the prevalence and the other is more precise, taking into account effect sizes.
   1. Calculate an accurate estimation of the prevalence of the condition in the population
      
      
      
      where α is the significance level, p₀ is the initial estimation and error is the maximum error expected with a 100(1 - α)% confidence.
   2. According to the effect sizes found in the literature for each factor, use tools like the pwr package in R to estimate how much power is required to detect differences^15,28.
      NOTE: For instance, in our study⁵ we designed the first proposal with an error of 3% at 95% confidence and an initial estimation of the prevalence of MCI at 15% in the population aged 65 years or older, resulting in an estimated required sample size of 541 individuals.

7. Interdisciplinary communication network, pharmacists, primary healthcare physicians, and specialists

1. Design letters to communicate information about the project to the healthcare centers involved.
2. Explain to participating pharmacists how to inform their assigned physicians about the results of the screenings through a letter to the primary healthcare center.
3. Send written communications to the medical coordinators of the healthcare centers related to the participating pharmacies and to the Neurology Services of the hospitals to which they are assigned.
4. Contact participating neurologists to find out each patient's definitive diagnosis obtained via specific tests undertaken by specialized healthcare providers. Before this, primary healthcare providers should carry out the following protocol, as summarized by the clinical guidelines (Figure 2).

Figure 2: Protocol for primary healthcare action. An example of primary healthcare actions that should be considered for early MCI detection before the patient is referred for a medical diagnosis by specialists. Please click here to view a larger version of this figure.

8. Statistical analysis and preprocessing

NOTE: Before applying machine-learning techniques a preparatory step is required to transform the original data into a new data set according to the final study objective and the procedures to be applied. For this transformation, several things should be considered, including the characteristics of the algorithms. This is because some of them are sensitive to a lack of variability or sharing of information across columns, although the algorithms used to generate decision trees are particularly robust against these problems. This initial phase aims to categorize qualitative variables and gather values with enough cases for each variable. For efficient screening it is important to choose variables whose acquisition is proven to be easy and accurate. Participants are selected by a short interview in which the algorithms used were constrained to a white-box model, making it easy to check the criteria used to decide if the individual should take the test. We suggest using the rpart²⁹ package in R software for these algorithms, and implementing recursive partitioning.

1. Collect all the forms from the participating pharmacies and convert them into a table in which every column is a variable and every participating individual is a row.
2. Assign an identification number to each participant. Save the identification number and contact information in a different document so that it is not used by the machine-learning algorithm.
3. Generate variables to classify whether each drug the patient takes corresponds to second or third ATC³⁰ (Anatomical Therapeutic Chemical) level codes, according to the active principal ingredients on the pharmacotherapy follow-up sheet.
4. Perform an initial descriptive analysis.
   1. For every ordinal variable, choose an adequate contrast for the variable. For categorical variables, select the value considered as the baseline.
   2. For categorical variables, calculate a univariate logistic regression with a response variable for screening for MCI. Analyze the outcome of the regression with a contingency table, the p-value, sample odds ratio, and the 95% confidence interval of the odds ratio.
   3. For quantitative variables, calculate the mean, standard deviation, coefficient of logistic regression, and the 95% confidence interval of their coefficients.
5. Reject variables with missing (unavailable) values, considering these variables difficult to accurately collect.
6. Select only variables for which there is at least one statistically significant category (α < 0.01) according to the logistic regression analysis. The outcome of this step produces a reduced data set compared to the initial one.

9. Algorithms to create a decision tree

NOTE: Machine-learning algorithms must be properly parameterized to predict which individuals are likely to have a positive MCI test result. One of the main problems while screening for a condition is that the original data is expected to be imbalanced (i.e., few positive cases compared to the negative ones). To get models with balanced data we used a technique called down-sampling, or random sampling, to equalize the frequency with that of the lowest frequency class³¹. Efficient screening also requires reducing the number of false negatives as much as possible (i.e., increasing the sensitivity of the selection of participants suffering from MCI). One of the techniques used to achieve a greater sensitivity is the introduction of penalties in the calculation of Gini's impurity index (i.e., the index used by the algorithm to select the best split for the decision tree)³².

1. Generate a training and test data set with 80% and 20% of the whole data set, respectively using the createDataPartition function in the caret library³³.
2. Apply the algorithms used to generate decision trees to the training data set. Use the train again function in the caret library³³. The following steps are different parameters of the function; for instance, the tree used in this paper was generated with rpart²⁹ (method = "rpart"), but other algorithms are available.
   1. Select the 'down sampling' sampling method and introduce the sampling = "down" parameter into the caret.
   2. Set the prior probabilities for both classes.
   3. Provide a loss matrix with the Gini's impurity index penalties applied in order to focus on the increasing sensitivity.
   4. For every parameter in the algorithm, choose an appropriate grid of values.
   5. Use a cross-validation estimation of the receiver operating curve (ROC) values to select the best models within the parameter grid.
3. Calculate a confusion matrix and the area under the ROC curve (AUC) for the test set prediction to assess the true performance of the model.

Representative Results

The participating pharmacies gathered data from 728 users and collected demographic variables in addition to the drugs prescribed to the participants. A univariate logistic regression was performed for all the variables³⁴; the error bar graphs shown in Figure 3 and Figure 4 are convenient graphical representations of the confidence interval of the odds ratio (for qualitative variables) and the confidence interval of the coefficient of the logistic regression (for quantitative variables). Variables with p-values exceeding 0.01 (sex, age, education level, reading habit, time spent sleeping, depression, and memory complaints) were selected and used to generate a white-box model based on a decision tree. This decision tree was generated using a training data set comprising 583 individuals as an input and was validated with a test set of a cohort of 145 participants.

After using the caret³³ library in R, the resultant tree assigned a probability of suffering MCI to each individual depending on their final node in the tree (depicted in Figure 5) as well as their answers to a few questions. To evaluate the forecasting capability of these probabilities, a ROC analysis of the test set was performed (Figure 6); its AUC was 0.763 and its 95% confidence interval was (0.6624, 0.8632). In addition to the probabilities, the tree shown in Figure 5 also used very simple questions about how long the person sleeps and how often they read, to recommend (with a sensitivity of 0.76 and specificity of 0.70) whether patients should take the MCI tests.

Using this decision tree and short interview to select users at risk of MCI we were able to significantly reduce the number of patients requiring MCI tests (administration is quite time-consuming). This reduction can be estimated by using data in the test set and interpreting the confusion matrix of the observed and predicted classes shown in Table 1. In this work, 55 out of 145 participants in the test set were identified by the decision tree for further MCI testing, (representing a reduction of 62% of users taking the tests) while also selecting most of the individuals (19 out of 25) who were positive for MCI.

Figure 3: Example of the variables selected during preprocessing. A 99% confidence interval of the odds ratio was calculated and is represented as an error bar. The base value for the logistic regression is indicated below the name of the variable at the top of every panel. For every value of the variable, an error bar represents the confidence interval of the odds ratio of taking that value versus taking the base value. Because the variables used to generate the tree were selected, the confidence intervals do not include the value 0 for some values as these showed significant differences. The scale of the vertical axis is logarithmic to help in comparisons across groups. Please click here to view a larger version of this figure.

Figure 4: Example of non-selected variables during preprocessing. A 99% Confidence Interval of the odds ratio was calculated and is represented with an error bar. The base value for the logistic regression is indicated below the name of the variable at the top of every panel. For every value of the variable, an error bar represents the confidence interval of the odds ratio of taking that value versus taking the base value. In contrast with the previous figure, all the confidence intervals of the selected variables include the value 0, since no significant differences were found to be included to generate the tree. The scale of the vertical axis is logarithmic to help comparison across groups. Please click here to view a larger version of this figure.

Figure 5: Proposed partition tree for selection of pharmacy users. The following tree shows the selection algorithm for MCI tests for individuals aged over 65 years. The text at the top of the box corresponds to the recommendation of taking the MCI screening tests, the two numbers below are the estimated probability of a negative or positive MCI testing outcome, respectively. The value at the bottom of the box is the percentage of individuals with these characteristics in the training set. The warmer the color of the box, the more likely the MCI tests was positive. The top node corresponds to the question about whether the participant has a memory complaint. If the individual does not have a memory complaint, the tree leads to the left branch and the ensuing questions ask about the individual's sex; patients with a memory complaint are asked about the amount of time they sleep per day. This figure was modified from Climent³⁴. Please click here to view a larger version of this figure.

Figure 6: Receiver operating curves for the partition tree and sensitivity and specificity of the final decision in the test set. The graph represents the ROC curve of the probabilities assigned by the partition tree algorithm in the test set. The red surface corresponds to the AUC and the blue point on the curve shows the sensitivity and specificity of the final recommendation made by the tree. Please click here to view a larger version of this figure.

		Reference
		No	Yes
Prediction	No	84	6
	Yes	36	19

Table 1: Confusion matrix. Confusion matrix of the predicted and observed values in the test set which were used to validate the proposed model.

Discussion

After searching for terms associated with MCI in Cochrane studies in the PubMed database, a specific questionnaire was created for this study that used the most evident variables with a proven association with MCI. Demographic, lifestyle, and social factors, as well as the patient's pharmacotherapy and some relevant pathologies were also recorded. Additionally, the SPMSQ and MMSE MCI tests were also selected. Importantly, the SPMSQ was not affected by participants' level of schooling. Pharmacists were trained to administer this study and communication with primary and specialized care was assured via letters informing them of this work. Only specialized healthcare providers could definitively make a diagnosis if MCI was suspected as a result of these tests.

In conclusion, in this study we screened for MCI among a population with a low prevalence of the condition (17%). We designed a set of selection criteria for use with machine-learning techniques, which increased the percentage of MCI positives up to more than 30% among the selected users. Consequently, these tools help increase the screening efficiency and substantially reduce the cost of mass screening among the population group selected by the decision tree.

A limitation of this method is that the decision tree may become invalid in this specific cohort as the population changes and thus, will likely require periodic updates. For instance, many individuals in this population were illiterate, but the number of illiterate individuals aged over 65 years will decrease in the future. These demographic changes will affect the variables related to reading and will require future recalibration of the decision tree.

Remarkably, this data-driven model provided information about the most important variables (from among hundreds) in the construction of a concise yet informative and efficient model. Constructing a decision tree provides insight into the best variables to focus on and is both a cost-effective way to help select people for whom further MCI testing is recommended and furthers our knowledge of these populations in this context.

To increase the future percentage detection rate of MCI, we will require new cost-effective techniques that can assure increased effectiveness. This protocol is time-consuming and is difficult for pharmacists to integrate into their daily work. Thus, other tests such as the MoCA²² or SLUMS²³ (both with adequate sensitivity and specificity) could be considered for fast the detection of MCI in the future.

A systematic evaluation of the trade-off between specificity and test duration should improve the effectiveness of the set of MCI tests used for screening. Moreover, relevant quantitative variables included in the study should have a wide range so that an efficient cut-off can be selected for them; a narrow range would exclude a large portion of the population from early detection. For instance, the age variable (which is always considered an important criteria in MCI diagnoses) was not considered relevant in this decision tree because the recruitment criteria (age over 65 years) was too conservative; inclusion of younger individuals in a future study would allow the optimal age for starting MCI screening to be calculated.

Materials

Name	Company	Catalog Number	Comments
caret	Max Kuhn		R package
rpart	Terry Therneau, Beth Atkinson, Brian Ripley		R package
SPMSQ in Spanish	Farmaceuticoscomunitarios.org		http://farmaceuticoscomunitarios.org/anexos/vol11_n1/ANEXO1.pdf
SPMSQ in English	geriatrics.stanford.edu		https://geriatrics.stanford.edu/culturemed/overview/assessment/assessment_toolkit/spmsq.html
MMSE in Spanish	Farmaceuticoscomunitarios.org		http://farmaceuticoscomunitarios.org/anexos/vol11_n1/ANEXO2.pdf
MMSE in English	oxfordmedicaleducation.com		http://www.oxfordmedicaleducation.com/geriatrics/mini-mental-state-examination-mmse/