Meta-analysis of Voxel-Based Neuroimaging Studies using Seed-based d Mapping with Permutation of Subject Images (SDM-PSI)

Summary

We detail how to conduct a meta-analysis of voxel-based neuroimaging studies using Seed-based d Mapping with Permutation of Subject Images (SDM-PSI).

Abstract

Most methods for conducting meta-analysis of voxel-based neuroimaging studies do not assess whether effects are not null, but whether there is a convergence of peaks of statistical significance, and reduce the assessment of the evidence to a binary classification exclusively based on p-values (i.e., voxels can only be “statistically significant” or “non-statistically significant”). Here, we detail how to conduct a meta-analysis using Seed-based d Mapping with Permutation of Subject Images (SDM-PSI), a novel method that uses a standard permutation test to assess whether effects are not null. We also show how to grade the strength of the evidence according to a set of criteria that considers a range of statistical significance levels (from more liberal to more conservative), the amount of data or the detection of potential biases (e.g., small-study effect and excess of significance). To exemplify the procedure, we detail the conduction of a meta-analysis of voxel-based morphometry studies in obsessive-compulsive disorder, and we provide all the data already extracted from the manuscripts to allow the reader to replicate the meta-analysis easily. SDM-PSI can also be used for meta-analyses of functional magnetic resonance imaging, diffusion tensor imaging, position emission tomography and surface-based morphometry studies.

Introduction

Since the introduction of magnetic resonance imaging, the neuroimaging community has published thousands of studies of the neural substrates of psychological functions and neuropsychiatric disorders. To summarize these findings, several methods have been developed^1,2,3,4,5,6. Original voxel-based neuroimaging studies report the coordinates of the peaks of statistical significance (e.g., in a comparison of gray matter volume between patients and controls), and meta-analytic methods commonly assess whether there is convergence of peaks in certain brain regions.

However, we have previously shown that these tests for convergence of peaks rely on delicate assumptions that might influence the patterns of meta-analysis results and their statistical significance⁷. Specifically, these tests assume that voxels are independent and that they have the same probability of a "false" peak, while in real gray matter, voxels correlate with their neighbors and the probability that a voxel has a "false" peak depends on its tissue composition. In addition, they also encompass paradoxes such as that the statistical power increases in the presence of few true effects, and decreases when there are multiple true effects.

To overcome these problems, we developed a method that imputes the brain maps of statistical effects for each study and then conducts a standard random-effects meta-analysis to formally test whether the effects are different from zero. This method is called "Seed-based d Mapping with Permutation of Subject Images" (SDM-PSI)⁸ and its main features include:

• Accounting for both increases and decreases of the outcome of interest (e.g., activation and deactivation) so that contradictory findings cancel each other⁴;
• Use of effect size estimates with random-effects modeling, which increases reliability and performance⁹;
• Potential simultaneous inclusion of available 3D statistical images (i.e., maps of t-test values)¹⁰;
• Subject-based permutation tests identical to those of FSL "randomize" tool¹¹;
• Use of threshold-free cluster enhancement (TFCE) statistics¹².

We have detailed and fully validated the SDM methods elsewhere^4,8,10,13,14.

In addition, we suggest not relying on a binary classification of the voxels based on the level of statistical significance (significant vs. not significant) but, conversely, assessing the strength of the evidence using a set of criteria²². The binary statistical significance reductionism leads to poor control of the false positive and false negative rates¹⁵, whereas the criteria use ranges of statistical significance levels and take into account the amount of data or potential biases. The SDM-PSI software returns the necessary elements to conduct such a classification⁸ and thus they can be employed to afford a more granular classification of the strength of the evidence.

Here we show how to conduct a meta-analysis of voxel-based neuroimaging studies using SDM-PSI. To exemplify the protocol, we use data from a published meta-analysis of voxel-based morphometry studies that investigated gray matter abnormalities in patients with obsessive-compulsive disorder (OCD)⁴. However, we will not use the methods employed in that early meta-analysis, but the aforementioned state-of-the-art procedures. The reader can download the software and these data from our website (http://www.sdmproject.com/) to replicate the analysis.

All researchers who aim to conduct a meta-analysis of voxel-based neuroimaging studies can follow this protocol. The method can be used with functional magnetic resonance imaging (fMRI, e.g., BOLD response to a stimulus)¹⁶, voxel-based morphometry (VBM, e.g., gray matter volume)¹⁷, diffusion tensor imaging (DTI, e.g., fractional anisotropy)¹⁸, position emission tomography (PET, e.g., receptor occupancy)¹⁹ and surface-based morphometry (SBM, e.g. cortical thickness) studies/datasets.

Protocol

1. Installation of SDM-PSI

1. Go to https://www.sdmproject.com/software/ to download the version of SDM-PSI for the operating system of the computer as a ZIP file.
2. Decompress the ZIP file. To avoid problems, decompress it within a local folder without blank spaces in its path.
3. Click the file SdmPsiGui to execute the graphical interface of SDM-PSI, and close the About splash window that will automatically open.
4. If SdmPsiGui does not find all required paths, it will automatically offer to display the preferences window. Press Yes.
   1. If MRICron is not installed in the computer go to https://www.nitrc.org/frs/?group_id=152 to download the version for the operating system of the computer as a ZIP file, and decompress the ZIP file.
   2. In the Brain viewer tab of the preferences window, ensure that Brain viewer is set to MRICron, and click the folder icon next to Brain viewer executable to find the MRICron executable.
   3. Ensure that all remaining paths in the different tabs have blue marks, which indicate that the paths are correct.
5. In case that SdmPsiGui has not automatically displayed the preferences window, go to the Tools menu and click Preferences.
6. In the Multithreading tab, specify the number of concurrent threads to use in the calculations. Some SDM-PSI calculations take a very long time (from hours to days) and consume a large amount of RAM memory (from hundreds of megabytes to gigabytes). The use of multiple threads (parallel processing) substantially decreases the time but increases the memory used.
7. Close the preferences window and SdmPsiGui.

2. Meta-analysis plan

1. Specify a precise question. For instance, "do patients with OCD have gray matter volume regional abnormalities?"
2. Write clear inclusion criteria that allow a systematic inclusion of the studies. For instance, "all studies that performed whole-brain voxel-based comparison of gray matter volume between individuals with OCD and healthy controls".
3. Write clear exclusion criteria that allow a systematic exclusion of those studies that cannot or should not be included for specific reasons. For instance, "studies with less than 10 patients, duplicated datasets, or studies from which the required information cannot be retrieved".
4. Write down the data to extract from each study. The following list includes the recommended data (some of them are not strictly required, but their absence would impoverish the meta-analysis):
   • An identification of the study.
   • The sample sizes.
   • The statistical significance level, which is the t-value, z-value or p-value used in the study to determine which voxels were statistically significant.
   • The software and stereotactic space. See Table 1 for the list of software packages and stereotactic spaces understood by SDM-PSI.
   • The coordinates and height of the peaks. The height of a peak is its t-value or z-value, but a p-value is also useful.
   • Variables that will be used to describe the samples or to conduct subgroup analyses or meta-regressions.
5. To increase the quality of the review, consider following the "Ten simple rules for neuroimaging meta-analysis"²⁰ and the PRISMA checklist²¹.
6. To increase the transparency of the review, consider registering the protocol beforehand on a publicly available database such as PROSPERO (https://www.crd.york.ac.uk/PROSPERO/).

3. Exhaustive search

1. Select a set of keywords that allow finding any study that might meet the inclusion criteria. For instance, the keywords might be "obsessive–compulsive disorder" plus "morphometry", "voxel-based" or "voxelwise".
2. Conduct the search on databases such as PubMed and Web of Science:
   1. Go to database website, e.g. https://www.ncbi.nlm.nih.gov/pubmed/ for PubMed.
   2. Type the search query. In the example meta-analysis, the query might be "obsessive–compulsive disorder" AND ("morphometry" OR "voxel-based" OR "voxelwise"). In this query, the operator "AND" means that the studies must have all keywords, the operator "OR" means that the studies must have at least one of the keywords, and the parentheses indicate the order of these logical operations. Therefore, the retrieved studies must have the keyword "obsessive–compulsive disorder" and at least one of the keywords "morphometry", "voxel-based" or "voxelwise". Note that other strategies are possible.
3. Apply the inclusion/exclusion criteria. For instance, from the results provided by the databases, select only the articles that analyze differences in gray matter volume between patients with obsessive-compulsive disorder and controls performing whole-brain voxel-based morphometry studies, and discard studies including less than 10 patients and studies that re-analyzed previously published data.
4. To increase the exhaustiveness of the search, consider conducting a manual search over the referenced works in the selected studies.
5. To maximize the inclusion of studies and avoid uncertainty in the collection of data, consider contacting the corresponding authors to ask for any missing or unclear data.
6. Record the number of studies retrieved and the number of studies excluded for each reason. Consider creating a PRISMA flow diagram²¹ with these numbers.

4. Collection of the data

1. For each included study, read the manuscript to find the specific data to extract.
2. Save the data from the studies systematically, e.g., typing the data in pre-formatted spreadsheet files. To minimize typing errors, consider copying and pasting the numbers of interest and double-checking the saved data.
3. When the statistical significance level is unclear, consider following these recommendations:
   1. If the manuscript reports peaks obtained using two whole-brain statistical significance levels, e.g. p-value < 0.001 without correction for multiple comparisons (from now on, "uncorrected threshold") and familywise error rate (FWER) < 0.05 (from now on "corrected threshold"), select the uncorrected threshold, and include all peaks obtained using this uncorrected threshold. The reason to prefer the uncorrected threshold is that studies usually obtain more peaks applying an uncorrected threshold, and SDM estimates the maps more accurately if it has information from more peaks.
   2. If the manuscript reports peaks obtained using an uncorrected threshold for increases and a corrected threshold for decreases (or vice versa), select the uncorrected threshold but include only the peaks obtained using the corrected threshold. This is a conservative approximation because it might discard some peaks obtained using the uncorrected threshold. An example of this situation might be when a manuscript states something such as "we detected FWER-corrected larger gray matter volumes in some regions, while we did not detect smaller gray matter volume in any region even using uncorrected p-value < 0.001".
   3. If the authors applied cluster-based statistics, use the cluster-forming height threshold. This is a conservative approximation because some voxels might have had t-values higher than the threshold, but the authors discarded them because their clusters were not large enough.
   4. If the manuscript does not specify a threshold, use a slightly smaller value than the t-value of the smallest peak. The reason to use this t-value is that if the authors had applied this statistical significance threshold without requiring a minimum size for the clusters, they would have found the same peaks.
4. When recording the peak information do the following:
   1. Exclude peaks obtained using a statistical significance threshold that is more liberal than the threshold selected for the rest of the brain. An example of this situation is when the authors applied more liberal thresholds or small volume corrections to some a priori brain regions.
   2. Convert z-values and p-values into t-values. Click the button Convert peaks in the SDM-PSI software to convert them easily. Alternatively, convert them in the same spreadsheet file (e.g., "=T.INV(1-0.001,34)" for p-value = 0.001 and 34 degrees of freedom; the degrees of freedom are the sum of the sample sizes minus the number of parameters, which in a two-sample comparison are two plus the number of covariates used in the original comparison).
   3. Use positive t-values for peaks of increase (e.g., activation) and negative t-values for peaks of decrease (e.g., deactivation). See Table 2 for guidance on how to decide the sign of the t-values.
      NOTE: We obtained information from studies "Heuvel" and "Soriano-Mas" after personal communication.

5. Introduction of data into SDM-PSI

1. Open SdmPsiGui and close the About splash window (avoid having any key pressed while closing it).
2. Click the Change meta-analysis button at the upper-left part of the graphical interface to select a directory for the meta-analysis (any new empty directory of choice will do).
3. Click the button SDM table editor to input general information from the studies, including their identification (column "study"), their sample sizes (columns "n1" and "n2"), the t-value that they used as statistical thresholds (column "t_thr"), and other potential variables to conduct subgroup analyses or meta-regressions.
4. Within the selected directory, create a text file for each study with the coordinates and t-value of each peak:
   1. Open a text editor to create a text file named as [the identification of study] + "." + [the software] + "_" + [the stereotactic space] + ".txt". For instance, for the study "Carmona" that was conducted with SPM and reports the coordinates in MNI space, the name of the text file must be "Carmona.spm_mni.txt". If the study has no peaks, the software and stereotactic space can be replaced by "no_peaks".
   2. Write the coordinates and t-value of each peak in a different row. For instance, the first rows of the text file "Carmona.spm_mni.txt" should be:
      40,39,21,-5.14
      53,27,21,-3.77
      56,23,20,-3.63

6. Pre-processing

1. Click in the Preprocessing button at the left menu bar, select the modality of the studies at the list box labeled "Modality" and press OK. In the example meta-analysis, the modality is "VBM – gray matter".
2. Wait (some minutes) while SDM-PSI calculates the maps of the lower and upper bounds of potential effect sizes. SdmPsiGui will show four progress bars that display the status of the execution and the expected remaining time for the current process. During the calculations, the color of the circle next to "Processing status" will be yellow, and will change to green if the execution ends successfully, or to red if it fails.

7. Main analysis

1. Click the Mean button at the left menu bar and press OK.
2. Wait (some minutes) while SDM-PSI conducts the multiple imputation and meta-analysis (Figure 1).
3. Click the Threshold button in the left menu toolbox, select the uncorrected p-values of the main analysis ("MyTest_uncorrp" by default) and press OK. SDM-PSI will automatically open both MRICron to visualize the results and a webpage with a detailed report of them.
4. Press the FWE correction button at the left menu toolbox, select the main analysis in the list box ("MyTest" by default) and press OK.
5. Wait (some hours or even days) while SDM-PSI conducts the permutation test.
6. Click the Threshold button in the left menu toolbox, select the TFCE-correction of the main analysis ("MyTest_corrp_tfce" by default) and press OK. SDM-PSI will automatically open both MRICron to visualize the results and a webpage with a detailed report of them.

8. Heterogeneity, publication bias and grading

1. Click the Extract button on the left menu toolbox, select a peak from the main analysis ("MyTest_z_p_0.05000_10_neg_peak1" by default) and press OK. SDM-PSI will automatically open a webpage with statistics of this peak. Write down the heterogeneity I² statistic.
2. Click the Bias Test button on the left menu toolbox, select a peak from the main analysis ("MyTest_z_p_0.05000_10_neg_peak1" by default) and press OK. SDM-PSI will automatically open a webpage with a funnel plot and the results of a test for small-study effect and a test for excess significance. The former tests whether there is asymmetry in the funnel plot (i.e., larger effect size in small studies), which could indicate that small studies are only published if they find large effect sizes or other sources of bias. The latter tests whether the number of studies with statistically significant results is larger than expected, which could indicate that studies are only published if they find statistically significant results or other sources of bias.
3. Press the Evidence grading button from the top toolbox, select the main analysis ("MyTest" by default) from the list box and press OK. After some seconds, SDM-PSI will automatically open MRICron to show the classes of evidence.

Representative Results

As shown in the map opened in MRICron when thresholding of the main analysis (Figure 2, step 7.6), patients with OCD had statistically significantly smaller gray matter volume in dorsal anterior cingulate/medial frontal cortex. The accompanying webpage details that the cluster is moderately small (143 voxels) and mainly located at Brodmann area 32, and that the peak of the cluster is at MNI [2, 32, 32], has a z-value of -4.97 and a FWER-corrected p-value of 0.01.

In the webpages obtained in steps 8.1 and 8.2, the low I² statistic (1.5%) indicates very small heterogeneity, the funnel plot does not show asymmetries (Figure 3), and both the test for small-study effect and the test for excess of significance are negative. However, the evidence of smaller gray matter volume in dorsal anterior cingulate cortex is moderately weak, as shown in the map open in MRICron when grading evidence, especially due to limited amount of data.

When the main analysis was thresholded using a less stringent statistical significance level (step 7.3), patients also showed statistically significantly larger gray matter volume in the striatum and superior parietal gyrus (uncorrected p-values = 0.00006 and 0.0002 respectively), but the evidence of these abnormalities should be considered weaker.

Figure 1: Main window of the SDP-PSI graphical user interface during a mean execution. Please click here to view a larger version of this figure.

Figure 2: Regions of statistically significantly smaller gray matter volume in patients with obsessive-compulsive disorder as compared to matched healthy controls.
The cluster of statistical significance covers 143 voxels, has its peak at MNI [2,32,32], and includes mostly dorsal anterior cingulate/medial frontal cortex, Brodmann area 32. Please click here to view a larger version of this figure.

Figure 3: Funnel plot for the peak of the cluster of statistically significantly smaller gray matter volume in dorsal anterior cingulate cortex. Please click here to view a larger version of this figure.

Software packages	Coding in SDM
Statistical Parametric Mapping (SPM)	spm
FMRIB Software Library (FSL)	fsl
Other packages	other
Stereotactic space	Coding in SDM
Montreal Neurological Institute (MNI)	mni
Raw Talairach	tal
MNI converted to Talairach using Brett transform	brett

Table 1: List of software packages and stereotactic spaces understood by SDM-PSI.

	t-values must be positive when:	t-values must be negative when:
One-sample fMRI studies	task > baseline (activation)	task < baseline (deactivation)
Two-sample fMRI studies	patients > controls in task > baseline (hyper-activation)	patients < controls in task > baseline (hypo-activation)
	patients < controls in task < baseline (failure of deactivation)	patients > controls in task < baseline (hyper-deactivation)
Two-sample VBM / FA studies	patients > controls (larger volume / FA)	patients < controls (smaller volume / FA)

Table 2: Sign of the t-values of the peaks.

Discussion

As introduced earlier, most voxel-based meta-analytic methods use a test for convergence of peaks that has some limitations, and then conduct a binary classification of the evidence exclusively based on p-values.

In this protocol, we detailed how to conduct a voxel-based meta-analysis using SDM-PSI, which has a number of positive features including a standard permutation test to assess the statistical significance of the effects. In addition, we show how the strength of the evidence can be graded using a set of criteria that go beyond a binary classification that solely relies on one statistical significance level.

To facilitate the replication of the example meta-analysis, we provide the data already extracted from the manuscripts from a previous meta-analysis. Interestingly, in the manuscript of that meta-analysis, the evidence “seemed” stronger than the evidence that we found with the updated methods. We therefore suggest that unsystematic evaluations of the evidence in previous voxel-based meta-analyses are taken with caution.

We hope that following this protocol, neuroimaging meta-analyses provide a richer and more granulate description of the evidence of neuroimaging findings.

Materials

Name	Company	Catalog Number	Comments
Computer			32- or 64-bit Windows, 64-bit Mac OSX, or 64-bit Linux