ORIGINAL: Neuroskeptic
THURSDAY, 14 JUNE 2012
Update - A number of articles linking to this post are wrongly stating that FreeSurfer is medical software used to diagnose diseases or measure the size of brain tumors. It's not. It is purely for research purposes as the software license states, "The Software has been designed for research purposes only and has not been reviewed or approved by the Food and Drug Administration or by any other agency. CLINICAL APPLICATIONS ARE NEITHER RECOMMENDED NOR ADVISED."
Last month, neuroscientists were warned about potential biases in SPM8, a popular software tool for analysis of fMRI data.
Now a paper highlights another software pitfall: The Effects of FreeSurfer Version, Workstation Type, and Macintosh Operating System Version on Anatomical Volume and Cortical Thickness Measurements
FreeSurfer is one of the major image analysis packages and amongst other things, you can use it to measure the size of different parts of the brain.
German Dutch researchers Ed Gronenschild and colleagues took a set of 30 brains and got FreeSurfer to estimate the size and thickness of various structures. Then they did the same thing, on the exact same brains, with a different version of the software.
They found substantial differences in regional volumes, depending upon the version of FreeSurfer used. Running the same version of the software on a Mac vs a PC also created differences, and even the version of Mac OS had an impact.
How much of a difference it made varied by brain location. The differences were 5-15% with version changes. For Mac vs PC and Mac OS updates it was less bad, 2-5% mostly, but in the worst regions - the parahippocampal and entorhinal cortex - it was still almost 15% different. Why those regions are so variable is unclear.
The paper goes into lots more detail, but the lesson for researchers is extremely simple: don't cross the streams of data-analysis. Set up your analysis stream and then use it on all of your data. Same hardware, same software, same settings.
Imagine you're doing a study comparing brain structure in two groups. Halfway through analyzing your data, you upgrade your MacOS. All of the brains you analyze after that will be, say, 5% "bigger". That'll certainly make your data much noisier, and if you happen to analyze most of Group A before Group B, it'll give you a false positive finding.
Sometimes you just can't avoid changes in hardware or software - IT techs have a habit of upgrading things without asking - but in these cases, you should run the same data under the old and the new regime to see if it's making a difference.
Finally, it would be wrong to blame FreeSurfer for this. I'd be surprised if they were any worse than the other software packages. Mixing and matching versions is something that the FreeSurfer developers specifically warn against. This paper shows why.
Gronenschild EH, Habets P, Jacobs HI, Mengelers R, Rozendaal N, van Os J, and Marcelis M (2012). The Effects of FreeSurfer Version, Workstation Type, and Macintosh Operating System Version on Anatomical Volume and Cortical Thickness Measurements. PloS one, 7 (6) PMID: 22675527
