Brain scanning studies are usually too small to find reliable results

Most studies that have used MRI machines to find links between the brain’s structure or function and complex mental traits had an average of 23 participants, but thousands are needed to find reliable results

By Clare Wilson

Brain scanning studies that use magnetic resonance imaging (MRI) machines are often said to show links between the brain’s structure or patterns of activity and complex traits such as depression, autism and certain aspects of personality. But for years, there have been suspicions that some of the headline-grabbing results aren’t trustworthy. Now, that has been confirmed.

Almost all such research so far has had too few participants to find reliable results, according to a study by Scott Marek at Washington University in St Louis and his colleagues. His group found that such studies need to look at the brains of thousands of people; they usually include an average of about 23 individuals.

Marek and his team analysed results from three of the largest ongoing neuroimaging studies to date, including the UK Biobank study, which had scanned nearly 36,000 participants at the time. They looked at links between brain structure or functioning and two relatively well-studied traits: cognitive ability and, in children, scores on a checklist for “psychopathology”, a combination of several behavioural measures.

By doing multiple analyses with varying numbers of people, they showed that when small sample sizes were used, they could find apparent correlations between these traits and the brain’s structure or function. But analyses of larger groups showed that these effects were either exaggerated or completely spurious. In some cases, different small samples could reach opposite conclusions, simply because people’s brains are so variable that random chance can sway the results one way or another.

Although Marek’s team only looked at two traits, it is likely that its conclusion that these kinds of brain scanning studies need thousands of participants is true more broadly, says Kevin Mitchell at Trinity College Dublin in Ireland.

There is no reason to think that complex intellectual traits like people’s appetite for risk-taking or their political leanings will necessarily have any manifestation in brain tissues that would be visible on a scan, says Mitchell.

Results from previous small brain scanning studies of this type should now be viewed with suspicion. And unless it contains thousands of participants, the same goes for ongoing research.

The new analysis may lead to the MRI field having a kind of existential crisis, says Cassandra Gould van Praag at the University of Oxford. “There’s a bit of The Emperors’ New Clothes about this.”

There are parallels here with research into the genetics of mental health. In the 1990s, there were similar eye-catching claims that researchers had found the genes “for” things like depression and aggression, based on small studies with tens or hundreds of participants.

Then, larger studies that sequenced all the DNA of several hundred thousand people at a time showed that common conditions like depression are actually influenced by hundreds of gene variants that each have tiny effects.

None of the results from small studies on the genetics behind mental health were borne out. They probably found false results because of a failure to use good research practices, like preregistering the study’s methods before collecting and analysing the data, as well as checking the results in a second group of people before making any claims, says Mitchell.

The field of brain scanning needs a similar overhaul, with researchers committing to proper research practices before they get funding in future. Existing small studies could still be salvaged if multiple research teams band together to produce one large study, says Marek.

Journal reference: Nature, DOI: 10.1038/s41586-022-04492-9

Sign up to our free Health Check newsletter for a round-up of all the health and fitness news you need to know, every Saturday