Blogs

During the 2025 SOT Annual Meeting and ToxExpo, the Roundtable Session “Is It Time to Right-Size Systematic Review?” posed a timely question to regulatory toxicologists.  Although federal workforce travel restrictions led to speaker substitutions, the panel still delivered a thought-provoking series of talks that elicited eager discussion from the audience.   

The Chair of the session, Heather Lynch, MPH, DABT (Integral Consulting), first gave an overview of our current challenges in systematic review, including (1) lack of standard approaches; (2) resource- and time-intensive protocols; and (3) the perception that systematic review “solves everything” but in reality, there is plenty of room for interpretation and the subjectivity of reviewers remains.   

Suzanne Hartigan, PhD (American Chemistry Council), brought a stakeholder perspective to the discussion, highlighted the emerging use of AI as a tool for systematic review, and emphasized the need for careful problem formulation up front. Dr. Hartigan proposed that the initial problem formulation would ideally yield a specific research question that clarifies the rationale behind the set PECO criteria.   

Seneca Fitch, BS (ToxStrategies), also stressed the importance of the problem formulation step, especially as it relates to streamlining the systematic review process, and discussed the idea of an iterative problem formulation based on results in initial screens.  Bonus discussant Kim Zaccaria, PhD, DABT (SRC, Inc.), gave an example of this situation in the discussion portion of the session: if initial results show that the chemical is used as a common positive control to model a certain disease state at high doses, then including and summarizing all such studies adds little value to the assessment if the goal is to identify a point-of-departure, which will by definition occur at relatively “low” doses. The speakers, therefore, suggested that revising PECO criteria based on initial screening results could streamline the process without affecting the outcome of the assessment. 

The utility and pitfalls of AI were also a salient topic of discussion. Ms. Fitch posed the question to the audience, “Does AI make systematic review less systematic?” with the thought experiment that systematic review using 100% human reviewers would lack efficiency, and 100% AI screening would lack accuracy and transparency. She argued that the ideal ratio of human-to-AI screening when resources are limited is somewhere in the middle, termed “human-in-the-loop” screening. 

Darrell McCant, MPH, DABT (Texas Commission on Environmental Quality; TCEQ) and Clemens Wittwehr, PhD, (European Commission Joint Research Centre) closed out the talks with discussions on current and future methods of systematic review in the regulatory space. Mr. McCant described TCEQ’s evolving approaches to systematic review, including updated guidelines that were updated in 2024 to include a Systematic Evidence Map Workflow, and emphasized that systematic review is not a solitary pursuit but a team sport. Dr. Wittwehr spoke on an OECD guidance document to harmonize the approach of using research data (as opposed to data from studies following standard guidelines) in regulatory assessments. Although not yet published, Dr. Wittwehr shared that this OECD guidance document will be forthcoming shortly. 

The discussion portion of this session showed the engagement of the audience in the topic at hand. The general sentiment on AI in systematic review was skeptical, with one audience member commenting on the “hallucinations of AI” and another noting the difficulty of identifying negative data with AI (a human reviewer can infer that some studies report only positive results in the text). Dr. Zaccaria argued that AI is best used with a very focused set of PECO criteria since including any and all health-relevant endpoints tends to confuse AI and therefore requires manual screening. Another question raised by the audience was the concept of “dueling systematic reviews” where conflicting conclusions are reached by different assessments. Panelists attributed this to (1) differences in the problem formulation questions leading to different answers; (2) lack of transparency in the PECO criteria and screening methods; and (3) the phrase “systematic review” being used quite loosely at times. The last point highlighted the need for education on systematic review in undergraduate- and graduate-level curricula. 

This blog reports on the Roundtable Session titled “Is It Time to Right-Size Systematic Review?” that was held during the 2025 SOT Annual Meeting and ToxExpo. An on-demand recording of this session is available for meeting registrants on the SOT Online Planner and SOT Event App. 

This blog was prepared by an SOT Reporter and represents the views of the author. SOT Reporters are SOT members who volunteer to write about sessions and events in which they participate during the SOT Annual Meeting and ToxExpo. SOT does not propose or endorse any position by posting this article. If you are interested in participating in the SOT Reporter program in the future, please email SOT Headquarters.