AAAS Kavli Lecture: Richard Harris on the Challenges of Replication in Science

With increasing evidence that many studies in biomedicine and social science do not stand the test of time, journalists are facing new challenges in reporting and interpreting the results of such research, NPR science correspondent Richard Harris told a University of Texas audience on Nov. 2.

Speaking in the second in a series of AAAS Kavli Science Journalism Award lectures this fall, Harris described the so-called “reproducibility crisis” in science, in which intriguing findings in pre-clinical cancer research and other areas often cannot be replicated, even by the researchers who did the original studies.

Harris discussed ways the science can go astray, steps researchers can take to correct flaws in their research design and practice, and precautions journalists should keep in mind as they sort through the claims and counter-claims made in the vast number of research papers published each year.

Despite the attention the reproducibility issue has garnered among scientists, Harris argued against calling it a crisis. “There’s not something new that’s happened,” he said.  “What we are seeing is a realization that science is a lot less reliable” at the level of individual studies than most people realize. “We need to be much more cautious in interpreting any single result,” he said, “and the world of science also needs to think about ways to improve the reliability of these individual studies.”

Harris spoke to about 180 people at the Belo Center for New Media on the campus of the University of Texas at Austin. The AAAS Kavli lecture series brings winners of the distinguish journalism award to campuses for public talks and classroom workshops. The next lecture will be Friday, Nov. 10, when Paula S. Apsell, senior executive producer of PBS’s NOVA, visits the University of Oregon in Eugene.

Harris, who in recent years had been covering the environment and climate science for NPR, returned to the biomedical beat in 2014. He started by looking at the status of the research effort and some of the big questions in the field. He was struck by a paper by C. Glen Begley, head of cancer research at the drug company Amgen, and Lee M. Ellis, a surgical oncologist at the M.D. Anderson Cancer Center in Houston.

The pair looked at 53 promising papers on cancer research from academic laboratories and attempted to reproduce the findings. They even asked some of the authors of the original research to help them.  In the end, Begley and Ellis were only able to reproduce the findings of six of the 53 studies, or about 11 percent of them. Reproducibility is a touchstone of science. Without it, research findings remain necessarily tentative and science cannot advance.

Harris cited other examples of concern—a review of 100 studies in the field of psychology in which the findings in only about a third of the studies were reproducible; an effort by scientists at Bayer, another large drug company, that managed to reproduce the findings of only one-quarter of the studies under review; a just-published review of 25 historical candidate genes for schizophrenia which found no evidence that the candidate genes are more associated with the disease than other genes.

Just as important, the failure to replicate echoes through newspaper coverage as well. Harris noted a report earlier this year that tracked newspaper coverage of 156 initial scientific studies that were deemed quite exciting. The findings of only 76 of the 156 studies were confirmed by subsequent follow-up analyses.

The causes of such unpredictable results, Harris said, can include bad ingredients in the lab, including contaminated and misidentified cell lines; poor research design, including insufficient numbers of mice in animal studies; statistical error and overreach, including “HARKing” (hypothesizing after the results are known), a push beyond the limits of the data; and funding pressures, which can lead scientists to hype or exaggerate their results to remain competitive for additional grant money.

The solutions include better validation of tissue samples and cell lines. Harris said the National Institutes of Health (NIH) is now insisting that scientists verify that their cell lines are what they say they are. Transparency also helps, with scientists agreeing to more widely share their data and the computer codes they used to analyze it.

Better training also is important. “In biomedicine, very often graduate students and post-docs learn from the people who are running their labs,” Harris said, “and their mentors may or may not have a lot of time to do that.” In fact, they often must spend a lot of their time writing grant proposals. “The training in this preclinical research tends to be fairly thin,” Harris said. The NIH is funding development of training courses to give young researchers a better grounding in the fundamentals of biomedical research procedures.

As for journalists, Harris said, they need to better understand some of the pitfalls associated with science and the reproducibility of research findings. His recommendations:

Don’t Assume: “Even if it is peer reviewed, it doesn’t mean it’s correct,” Harris said.

Ask Questions:  Are there other studies like this?  Is it the first of 10 that found this result? Are there other findings that contradict this finding? Are the statistical findings barely significant?  What is the p-value, one measure of statistical robustness? “That is a red flag for me if the p-values are barely at that mark,” Harris said.   

Express uncertainty: “We need to do the best we can to convey to our audience that science is a gradual process, each piece is just one more piece of the puzzle,” Harris said. “We shouldn’t oversell any one particular finding … If the effect is small, you should say so.”

Follow up: “Look back from time to time on big stories and see if they are panning out,” Harris said.  Follow the chatter on blogs, Twitter and sites such as PubPeer, which allows scientists to engage anonymously in post-publication peer review of papers that may be making a splash.

While journalists can report on the underlying questions involving reproducibility of scientific research, they can’t solve them, Harris said. That is up to the scientific community. But journalists can “make sure that we’re not overselling things, that scientists aren’t overselling things,” Harris said, “and that we are keeping in perspective how science is actually built, piece by piece, small step by small step.” He added, “We need to find creative ways of telling dramatic stories about small steps.”

Harris deals with the reproducibility issue and more in his recently published book, “Rigor Mortis” (Basic Books, 2017).