by Craig Rowlands, PhD, DABT and Ruth A Roberts, PhD, ATS, FBTS, ERT, FRSB, FRCPath
The Society of Toxicology Contemporary Concepts in Toxicology (SOT CCT) Webinar on Reproducibility and Reliability of Biomedical Research was delivered on November 3, 2016, to a fascinated audience of more than 190 SOT members. The presenter, Glenn Begley, PhD, ERT, Chief Scientific Officer of Akriveia Therapeutics, has led efforts to bring attention to the issue via several highly cited articles and numerous presentations, including his well-attended presentation at the 2016 SOT Annual Meeting.
An increasing number of biomedical journals and funding agencies are raising significant concerns that much of what is published in the biomedical literature cannot be reproduced. Dr. Begley’s webinar presentation featured three highly cited case studies in an interactive format where participants were surveyed for their views in real-time throughout. Although these case studies were from high impact, well-cited papers in leading journals, Dr. Begley pointed out some fundamental issues with the data and the conclusions drawn.
At the start of the presentation, over half of the attendees believed that there was a problem with non-reproducibility; this increased to over 75% of attendees at the end of the webinar. Participants indicated they were most concerned by inappropriate interpretation of data but also by inappropriate processing of data such as the use of "% of control" rather than actual numbers. There also was concern around failure to repeat experiments.
Many attendee questions came in throughout and trended in several areas; how can journals better represent the large amounts of data in current computational approaches? How are data best presented when journals limit space? There also were several questions related to the need for journals to publish so called “negative results”— that is— results that support the null hypothesis. Most revealing was Dr. Begley’s response to a question about how to understand when statistically significant results are in fact biologically significant; he responded that while there is no hard rule, researchers need to understand the system they are working in and the models they employ. This should drive confidence in knowing when a result is biologically meaningful irrespective of statistical significance.