Paul Boyer, a psychotherapist at Kaiser Permanente in Oakland, California, has first-hand experience with the AI revolution. He’s a little overwhelmed.
The healthcare giant has unveiled a new note-taking software suite created by healthcare AI pioneer Abridge that aims to summarize patient visits at supersonic speed. For many clinicians, this technology alleviates one of life’s persistent headaches: administration and paperwork.
But the AI scribe caused another headache for Boyer and his colleagues. That means it’s not very useful. They end up revising notes written on the computer.
Mr. Boyer said Mr. Abridge is “not good at picking up on the clinical nuances and emotional tones” that can be important in the field of mental health. For example, for manic patients, how something is said is more important than what is said, and software has a hard time picking up on those cues, Boyer said.
Note-taking software is not the wave of the future. It’s the wave of the moment. It is being introduced at hospitals across the country. And researchers have discovered some benefits. A year after their introduction, doctors who used the products the most saved more than 30 minutes of work each day, according to a study of five hospitals published in April in the Journal of the American Medical Association.
Many doctors like the products that are in place. Several interview-based studies have found an overall positive response to scribes.
Nevertheless, as Boyer’s example illustrates, there are deep questions about the quality of the system. While Boyer and his colleagues spend time revising their notes, safety researchers worry that clinicians may not be diligent about catching mistakes. That may mean future doctors relying on incorrect information.
Abridge says it evaluates Scribe at every stage of deployment, including direct testing against previous versions of the software.
“After introducing the model, we are monitoring free text feedback from clinician users regarding clinician edits, star ratings, and note quality,” Davis Liang, the company’s director of applied sciences, told KFF Health News in a statement.
Scribe software with artificial intelligence is some of the AI-powered tools being introduced in the medical field. Clinicians and patient safety advocates argue that government regulations are not sufficiently structured to guard against the threat that new technology could harm patients by missing or obscuring important details of their conditions.
Raj Ratwani, a researcher who specializes in human factors, or how people interact with technology, at MedStar Health, a large hospital system based in Columbia, Maryland, said there are “no safeguards in place right now” to scrutinize writing software at the federal level.
Ratwani worries that protections for medical software will be further relaxed. Ratwani said the rules proposed by the Office of the National Coordinator for Health IT, the agency that regulates electronic medical records, which are central records of patient care, could weaken requirements to make medical records easy to understand, easy to use, and transparent about the use of AI. And incomprehensible records can confuse clinicians and lead to errors.
Since the Obama administration, the Department of Health and Human Services’ IT department has encouraged “user-centered design” testing, where developers try out products on doctors and nurses. Regulators also called for more transparency from companies in the fast-growing market for AI tools.
Both of these requirements are being repealed in a rule proposed by HHS Secretary Robert F. Kennedy Jr.’s Office of Health IT.
Physicians and other health care professionals refer to records for clinical information, such as notes summarizing a patient’s treatment history and a list of medications and treatments the patient has used. The doctor also enters treatment instructions.
A poorly designed or cluttered record system “can result in a list of medications that is so complex and confusing that orderers end up selecting the wrong medications,” Ratwani said.
Tim Huang, Abridge’s general counsel, said the company “broadly supports” the government’s rules as a “necessary modernization” that “keeps pace with the speed at which AI is evolving.”
Ryan Howells, a principal at Levitt Partners, which consults digital health companies, said the old rules “overburden” electronic health records systems. Mr. Levitt supports this proposal.
The administration argues that eliminating the requirement would encourage more innovation and competition. The electronic health records market is steadily consolidating, leaving hospitals and other clinicians with fewer vendors to choose from.
The 2022 study found that the top two vendors, Epic and Oracle Health, account for more than 70% of the hospital market. And Howells argued that too many rules burdens providers who want better record-keeping systems. Howells said federal regulation is “the single biggest impediment to true clinical innovation.”
Some critics say the Trump administration’s proposal to remove record-keeping requirements is too broad. This removes regulations intended to keep records safe. It also eliminates privacy protections for sensitive and protected health data and overhauls the standards governing the format in which data is transmitted. The rules could give clinicians “more choices in health IT to meet their needs through increased competition,” the government wrote in its proposal.
HHS’ health IT division declined to comment, noting that the proposal is still going through the regulatory process. Public comment ended in February.
But most concerning to some, even in hospitals and development departments, are proposals to clarify prerequisites for ensuring new products are tested on real users and that AI technology decisions are transparent to doctors and nurses.
“Historically, hospitals and health systems have been challenged by the black-box nature of certain AI tools and the way algorithms are developed,” said Jennifer Holloman of the American Hospital Association. And as more AI tools flood the market, transparency becomes even more important, the association said.
Complaints about the security of electronic medical records have been going on for years, even for a seemingly simple task. Ratwani likes the example of ordering medicine for a specific condition.
“A doctor is trying to order Tylenol, and the drug list is so confusing that there are 30 different types of Tylenol, all at different doses and for different purposes, but it could actually be designed to be simpler and make it easier for the doctor to actually choose the right type of Tylenol to order,” he said.
Actual user testing was aimed at simplifying physician record design. But Lee Burchell, vice president of policy and communications at EHR developer Altera Digital Health, said the administration is trying to end the requirement in a confusing manner.
Burchell’s interpretation of the rule refers to “enforcement discretion,” a principle that allows the government to choose not to enforce certain rules, but companies are still required to conduct testing (the heavy lifting part) but are not required to report the results to the federal government.
The administration is also ending a Biden-era initiative to create a transparency “model card” for AI. The concept was that with the click of a mouse, clinicians could explore data used to train AI tools that advise clinicians. But Trump administration regulators say few people took advantage of the year-old tool.
Still, hospitals and doctors are wary of removing it. The tool “provides information about how predictive or generative AI applications should be designed, developed, tested, evaluated, and used. These data are critical to fostering trust in AI tools and ensuring patient safety,” the AHA said in a comment letter to the HHS IT Office. The American College of Physicians issued a similar warning, saying, “Lack of clarity can undermine clinician trust, increase liability costs, and damage the patient-physician relationship.”
Even the developers aren’t completely sure about this idea. Burchell said her electronic health records industry group has “many different perspectives” on the issue. “Usually we tend to be a little more aligned on our responses.”
Still, Burchell’s group thought companies should be transparent about the data AI relies on to make decisions and how it comes up with recommendations.
Evidence regarding the effectiveness of AI tools is scant or contradictory.
A recent study comparing 11 AI scribes for potential pilot use at the Veterans Health Administration found that the software performed worse than humans across five simulated scenarios. “Although ambient AI scribes can produce thorough notes, the overall quality is still significantly below that of human-produced documents,” the authors note, noting that the omission of information is particularly concerning given its potential to impact follow-up care.
The vendors surveyed in the VA study were not identified for what the authors called “contractual reasons.”
It’s just one type of AI tool. A wave of tools is arriving, each requiring its own evaluation, not to mention the tools already installed.
Boyer said AI scribes are largely negligible for now. But he worries that management will design his work around the expected time savings and schedule more patients. This means more time needs to be spent on both patient care and software error correction.
KP spokesperson Vincent Staupe said the company does not require clinicians to use AI.
“As I’m revising that memo, I feel like this is a lot of work,” Boyer said. “This is definitely making the situation worse, taking away time that could otherwise be spent fixing AI tools.”
KFF Health News is a national newsroom that produces in-depth journalism on health issues and is one of KFF’s core operating programs and an independent source of health policy research, polling and journalism. Click here for details KFF.
