As adoption of medical AI rapidly expands, healthcare AI companies are moving from simply generating answers to proving why those answers should be trusted.
OpenEvidence, an AI-powered medical search engine, has developed medical AI co-pilot capabilities that assess and visualize the quality of published evidence that is cited and used to inform all answers to clinical questions. The company claims this feature, called EvidenceGrade, adds an important layer of context to help incorporate medical evidence into high-stakes medical decisions.
According to OpenEvidence, a well-known limitation of AI systems is that they ignore differences in the quality of multiple sources when summarizing them. This limitation can have significant implications for medical care due to the differences between evidence from randomized, blinded, placebo-controlled trials and small observational studies in foreign populations.
According to Daniel Nadler, Founder and CEO of OpenEvidence, EvidenceGrade surfaces, quantifies, grades and visualizes the quality of the evidence behind OpenEvidence answers from a safety perspective that allows physicians to apply that evidence in real-world clinical settings.
OpenEvidence’s new capabilities go beyond simply citing sources to scoring the underlying quality of the evidence behind those sources and exposing that assessment to clinicians in real time.
The company gave Fierce Healthcare a first look at this new feature.
EvidenceGrade feature screenshot
Evaluate the evidence behind answers with EvidenceGrade (OpenEvidence)
This signals an ongoing shift as healthcare AI companies increasingly compete on reliability as well as accuracy. As generative AI tools become more capable, the differentiator is no longer just answering clinical questions quickly.
EvidenceGrade was developed by OpenEvidence’s team of medical AI scientists led by physician-scientist Sam Finlayson, MD. and Travis Zak, MD. and lead machine learning scientist Dr. Evan Hernandez. said Dr. Eric Lehman, Nadler.
EvidenceGrade is built on the GRADE framework (Grading of Recommendations Assessment, Development, and Evaluation), a widely adopted standard for assessing the quality of evidence. This is the methodology behind Cochrane, a not-for-profit network of health researchers, experts and patients, the World Health Organization, and most major clinical guidelines.
“The team consulted with experts in evidence synthesis and considered approaches used by Cochrane and similar teams. These approaches were then adapted to the unique demands of real-time decision-making that OpenEvidence was designed to support,” Nadler told Fierce Healthcare.
Nadler explained how EvidenceGrade scores questions. Questions are first classified to determine whether they are suitable for scoring, and then all retrieved papers are scored for quality, authenticity, and relevance. The model is trained to weigh the strength of the study design, the consistency and precision across sources, and how directly the evidence applies to the question, and reflects how professional methodologists evaluate bodies of evidence, he said.
An “A” grade means the evidence is supported by the most achievable research design for the question. These include randomized controlled trials and rigorous systematic reviews, well-conducted prospective cohorts and registries, or current evidence-based guidelines with strong recommendations. A “B” grade is moderate evidence supported by an appropriate study design with significant limitations on one or more axes, such as a reasonable sample size, surrogate results, observational evidence applied to a therapeutic question, or moderate imprecision.
A “C” grade indicates limited evidence based on a weak design, such as a narrative review or expert opinion, or a strong design with significant limitations across multiple axes. Additionally, a “D” grade indicates minimal evidence based on case reports, case series, preclinical data, or mechanical extrapolation, providing a directional signal.
EvidenceGrade assigns a U when a grade cannot be assigned.
“Not all evidence is equally robust, and when clinicians act on answers, it is essential to understand how much weight the underlying evidence holds,” said Samuel Finlayson, MD, senior vice president of medical AI at OpenEvidence. “Expert research review teams do valuable work, but they cannot cover every question that arises in daily practice. We built EvidenceGrade to extend their methodology to as many questions as possible. We see it as a complement to traditional evidence synthesis, and we are releasing it as a starting point that can be improved through collaboration with the clinical community.”
Screenshot of the EvidenceGrade approach
How EvidenceGrade scores responses from gradable claims to final letter scoring. (open evidence)
OpenEvidence has developed an AI-powered medical search engine and physician-specific generative AI chatbot that summarizes and simplifies evidence-based medical information. As of July, 915,000 licensed U.S. physicians, including nurses, nurse practitioners, and physician assistants, were using OpenEvidence, and more than 690,000 licensed U.S. physicians were using OpenEvidence.
The company announced Thursday a collaboration with NewYork-Presbyterian and its affiliated medical schools, Columbia University Vagelos College of Physicians and Surgeons, and Weill Cornell Medical College to bring OpenEvidence to all hospitals and care settings.
The organizations say OpenEvidence will be available to clinical staff at NewYork-Presbyterian Hospital, Columbia University, and Weill Cornell University.
“Each patient is unique, and care should reflect that,” Umejay, chief information officer for NewYork-Presbyterian, said in a statement. “By making OpenEvidence available across our systems, clinicians will be able to use AI to seamlessly access and analyze medical research in real time, and use that information to provide informed and compassionate care to the communities we serve.”
The company is also collaborating with Mount Sinai, marking a new companywide integration with a major health system in the New York City area.
While the company gained traction in the market as a free tool for clinicians, OpenEvidence is now setting its sights on expanding its footprint with hospitals and health systems. We are also working with Sutter Health and Cedars-Sinai to integrate an AI-based medical search and decision support platform into the organizations’ electronic health record systems.
These health systems are currently using OpenEvidence’s ad-supported model. Nadler told Fierce Healthcare in May that an enterprise model was in the works.
“We are pioneering a standard, ad-supported AI enterprise model, similar to Anthropic’s business model, for large systems like Mount Sinai and Cedars-Sinai that need that option, and where there is an opportunity for enterprise-level customization and value-add beyond the core free OpenEvidence product for physicians,” he said.
