Today’s Brain Teaser Question: What product….
- Was inspired by its co-creator’s experience with the Alzheimer’s disease of both of his grandmothers?[1]
- Has attracted $70 million dollars in venture capital investments?[2]
- Has 50 million subscribers?[3]
- Had, as of February 2013, achieved $23.6 million in revenue?[4]
- Claims to be serious training for your brain that can improve memory, attention, and other cognitive abilities through the “science of neuroplasticity”?
- But, according to a consensus statement signed by seventy neuroscientists and memory researchers, lacks any “compelling evidence” that the product offers consumers “a scientifically grounded avenue to reduce or reverse cognitive decline”?[5]
Answer: (You guessed it.) Lumosity. It is clear that much of that $70 million in investments has gone into advertising because almost everyone has heard of Lumosity and understands its basic claims. Unfortunately, far fewer people know what the research really tells us about Lumosity.
The Brain Training Promise
As the population ages, concerns about Alzheimer’s disease and dementia have created a demand for anything that might stave off the course of mental decline. The idea that simple video games on your computer or smartphone could keep you mentally sharp is very appealing. Furthermore, brain training programs have a kind of simple plausibility. They sound scientific, and the analogy to physical exercise makes intuitive sense.
In addition, Attention Deficit Hyperactivity Disorder (ADHD) continues to be a concern for parents of school-aged children, and memory and attention training programs designed for children with ADHD have emerged to meet the need.
Lumosity appears to be the market leader in this brain training stampede, but it is by no means the only brain game in town. Here are just a few of the other products in this lively marketplace.
Lumosity has kept its claims relatively modest, but its competitor brainHQ, a division of Posit Science, Inc., is less restrained. The brainHQ website claims this collection of online games can “improve your tennis game” and help you “hear better in crowded places.” Most importantly they assert that brainHQ “turns back the clock on your memory loss by about 10 years.”[6]
CogniFit appears to be very similar to Lumosity and brainHQ but with a less elaborate website and advertising campaign.
Developed by Swedish researcher Torkel Klingberg and now owned by the publishing giant Pearson, Cogmed has a very different marketing approach. Lumosity and brainHQ sell subscriptions to their online games directly to the end user, but Cogmed is marketed through a network of professional providers. This approach makes sense because, although the program is for all ages, Cogmed makes a special effort to pitch its product to parents of children with ADHD. The Cogmed website features a video testimonial by Dr. Edward Hallowell, psychiatrist and author of the popular book Driven to Distraction.
Jungle Memory is a subscription memory program specifically for children, marketed directly to parents. It claims to be effective with kids who suffer from ADHD, dyslexia, autism spectrum disorders, motor dyspraxia, and learning difficulties. Like all the other brain training products, Jungle Memory comes complete with many testimonials and claims to be based on “cutting-edge science.”
The Pseudoscience Behind Brain Training
If you have been following this industry at all, you may have heard that the brain training claims are not supported by evidence. In April 2013, New Yorker writer Gareth Cook published a scathing review of CogMed called “Brain Games are Bogus,” and last autumn, the consensus statement mentioned above, a collaborative project of the Stanford Center on Longevity and the Max Planck Institute on Human Development, received some media coverage.[7]
But the pushback on these programs has not received the kind of attention it deserves, and media coverage written by people who have swallowed the industry public relations line is still common. For example, “The Silver Economy: Brain training fired up by hard evidence,” an article published last autumn in the Financial Times appeared after the consensus statement was released and made reference to the consensus statement, but still presented a favorable image of the brain training industry.
As is often the case, the problem lies in the inability of journalists[8] and the general public to determine what constitutes “hard evidence.” Lumosity tends to keep their claims relatively vague. They point to ongoing research at their own Lumos Labs, but much of the research is either not peer reviewed or not relevant to the question of whether Lumosity actually works. For example, a recent Lumos Labs publication in the peer-reviewed journal Frontiers in Human Neuroscience simply outlines the research potential of the enormous dataset Lumosity has collected from its subscribers.[9]
In contrast to Lumosity, brainHQ’s claims are whoopers. For example, the brainHQ website says, “We know it cuts your risk of crashing your car in half,” but the research cited in support of this claim is not a study of brain training at all. It is a study of car crashes among older drivers that finds a relationship between neurological disease and car accidents.[10]
So, no, you don’t know that brainHQ cuts the risk of car crashes. Or if you do, you are using the word “know” in a much different way than I do.
The IMPACT Study
On the surface, the study that supplies the foundation of brainHQ’s most important claim to “turn back the clock on your memory loss,” as well as other claims that the brain game system fights depression and “improves health-related quality of life measures for five years,” sounds pretty impressive. The “IMPACT study” was conducted in part at the Mayo Clinic and published in the Journal of the American Geriatric Society in 2009. However the conflict of interest statement accompanying the article indicates that the study was funded by brainHQ’s parent company, Posit Science, Inc. and that five of the seven authors had some kind of financial relationship with Posit Science.
Perhaps most troubling, the study methodology did not eliminate the possibility of experimenter bias because the person who installed the software for the participants and introduced them to its use was not blind to the study design.[11] Even so, the results just made it over the bar for statistical significance (p = .02) and the size of the observed effect—the practical or clinical significance of the study—was less than a quarter of a standard deviation (d = 0.23) or what is generally considered a weak effect.[12] Not the most impressive of outcomes.
The ACTIVE Study
A more recent study that impressed the Financial Times writer as being “hard evidence,” is the “ACTIVE study” published in 2014 in the same Journal of the American Geriatric Society. It was a large-scale multi-site clinical trial supported by grants from the U.S. National Institute on Aging and the National Institute of Nursing Research. In this case, the two lead investigators had financial interests in products used in the research. However, the most damning feature of the study was that almost all the positive effects that were observed were changes in self-report measures. In other words, participants believed they had improved and reported belief to the investigators. In contrast, the objective measures—actual tests of functioning—produced much more mixed results. Importantly, the objective measures of problem solving and memory functioning showed no significant effect.[13]
The Portal 2 Study—Just for Fun
To make matters worse, a 2014 Florida State University study published in Computers & Education, suggests that you would be better off playing the popular first-person video game Portal 2 than playing Lumosity’s brain games. In this case, the researchers recruited college students and randomly assigned them to play either eight hours of Portal 2 or eight hours of Lumosity in a laboratory setting over a series of sessions. Before and after training, both groups were given a battery of objective tests of problem solving, spatial ability, and persistence.
Surprise!
The group who played Portal 2—which makes no claims about any cognitive benefits—scored significantly better than the Lumosity group on all three measures. Furthermore, the effect sizes were double those of the IMPACT and ACTIVE studies. The Lumosity group showed no significant difference pre-versus-post on any measure.[14]
Like any study, this one has strengths and weaknesses. The experimenters statistically controlled for the participant’s level of enjoyment of the game, and, as a result, the outcome cannot be dismissed solely on the assumption that Portal 2 is more fun to play than Lumosity. But the sample was relatively small (N = 77) and made up entirely of college students. Furthermore, because the post-test measures were completed shortly after the last game session, it is impossible to know how long the effects would last.
So What’s The Harm?
The purveyors of the “science of neuroplasticity” have clearly oversold their goods. But how bad can it be? The games are harmless and sometimes even fun. Because the typical brain training business model is to sell modestly priced subscriptions (approximately $150-$240 per year) to a large client base, the financial loss for any particular individual is not great. Although at this point, there is little evidence that playing computer games (except perhaps Portal 2!) produces cognitive and memory gains that transfer to everyday life, perhaps someday more effective programs will be developed. So what’s the harm?
I can think of two ways this brain training episode causes harm:
Opportunity Costs
The most important harm for the individual user is what economists call opportunity costs. Making one choice eliminates the opportunity to do something else. As the authors of the consensus statement make clear, for people who want to preserve their cognitive capacity, sitting at a computer for several hours a week may not be the best choice:
If an hour spent doing solo software drills is an hour not spent hiking, learning Italian, making a new recipe, or playing with your grandchildren, it may not be worth it. But if it replaces time spent in a sedentary state, like watching television, the choice may make more sense for you.[15]
The message is clear. Based on the available evidence, playing brain games on the computer is probably not the best way to maintain cognitive health. It is better to get up and do something more active with your life. The authors of the consensus statement put it this way:
…the promise of a magic bullet detracts from the best evidence to date, which is that cognitive health in old age reflects the long-term effects of healthy, engaged lifestyles.
The Misuse of Science
In recent years, science has been battered by a number of self-inflicted wounds—fraud and sloppy or absent peer review. But pseudoscience, the unearned use of the trappings of science to bolster a claim (in this case, a product claim) has been with us a long time. As soon as science achieved a level of stature, people began hijacking scientific language for their own purposes.
Today, neuroscience has created a great boon for the purveyors of pseudoscientific products, because, as sciences go, it has a lot of trappings. Neuroscience is concerned with our most mysterious organ, the brain, and it makes use of elaborate equipment, technical language, and colorful images. (To learn more about the misuses of neuroscience, see Brainwashed: The Seductive Appeal of Mindless Neuroscience by Sally Satel and Scott O. Lilienfeld.)
To be sure, neuroscience is a wonderful and exciting field, but like many shiny new things, its benefits have been overstated. In this case, neuroscience has become the indentured servant of an industry whose products may be all but useless. The brain training programs that are currently available are potentially quite lucrative for their makers but a waste of time and money for most users.
To protect the integrity of the science, it is important for science-minded people to call out pseudoscience when they see it, and it is hard to think of a more dramatic case of contemporary pseudoscience than the brain training industry. Because brain training programs are not drugs, they do not fall under Food and Drug Administration standards of proof of effectiveness. Brain training companies are subject to the truth in advertising standards of the Federal Trade Commission, but somehow, they seem to have avoided prosecution while spinning tales of improved tennis games, better memory, and safer driving. So it is up to individual scientists, journalists, and science advocates to speak up about the brain training scam. No one else is going to do it.
After that, you can go back to playing Portal 2.
[1] http://blog.lumosity.com/the-lumosity-story-video/
[2] http://techcrunch.com/2012/08/22/lumosity-raises-31-5m-from-discovery-communications-for-brain-fitness-games/
[3] http://www.lumosity.com/press/releases/lumosity-reaches-50-million-members-and-1-1-billion-game-plays
[4] http://www.forbes.com/companies/lumosity/
[5] http://longevity3.stanford.edu/blog/2014/10/15/the-consensus-on-the-brain-training-industry-from-the-scientific-community-2/
[6] http://blog.brainhq.com/2010/10/27/giving-health-advice-for-older-people-dont-forget-the-brain/
[7] http://chronicle.com/article/Brain-Training-Companies-Get/149555/, http://www.scientificamerican.com/article/brain-training-doesn-t-make-you-smarter/
[8] As an aside: I have come to believe that all science journalists should have a PhD in a science. Figuring out the scientific claims of both scientists and non-scientists requires some fairly serious training. Interviewing scientists is not a sufficient methodology for getting to the bottom of some questions.
[9] Sternberg, D. A., Ballard, K., Hardy, J. L., Katz, B., Doraiswamy, P. M., and Scanlon, M. (2013). The largest human cognitive performance dataset reveals insights into the effects of lifestyle factors and aging. Frontiers in Human Neuroscience, 7, 292. doi:10.3389/fnhum.2013.00292
[10] http://www.ncbi.nlm.nih.gov/pubmed/10719769
[11] Smith, G. E., Housen, P., Yaffe, K., Ruff, R., Kennison, R. F., Mahncke, H. W., & Zelinski, E. M. (2009). A Cognitive Training Program Based on Principles of Brain Plasticity: Results from the Improvement in Memory with Plasticity-based Adaptive Cognitive Training (IMPACT) Study. Journal of the American Geriatrics Society, 57(4), 594–603. doi:10.1111/j.1532-5415.2008.02167.x
[12] https://en.wikipedia.org/wiki/Effect_size#Cohen.27s_d
[13] http://onlinelibrary.wiley.com/doi/10.1111/jgs.12607/abstract
[14] Shute, V. J., Ventura, M., & Ke, F. (2015). The power of play: The effects of Portal 2 and Lumosity on cognitive and noncognitive skills. Computers & Education. doi:10.1016/j.compedu.2014.08.013
[15] http://longevity3.stanford.edu/blog/2014/10/15/the-consensus-on-the-brain-training-industry-from-the-scientific-community-2/