Doctors in Utah are challenging a prescription drug refill system that uses artificial intelligence, saying it risks patient safety.
In January, the state partnered with AI physician startup Doctronic to test an AI-powered system that would “automate routine guideline-based prescription renewals” for Utah residents. As Fierce Healthcare reported in January, this marked the first test of AI as an autonomous clinical decision maker under the regulatory moratorium paradigm.
Rather than having to wait days or weeks to make a doctor’s appointment to get regular medication refilled, Utahns can chat with an AI agent online and renew their prescriptions almost instantly.
According to a mitigation agreement (PDF) between the Utah Office of Artificial Intelligence Policy and Doctoronic, the AI system will allow 30, 60 or 90-day renewals for medications already prescribed by a licensed health care provider. The company limits its refills to 191 commonly prescribed drugs. The formulary was reviewed by an independent pharmacist and the state.
The Utah Medical Licensing Board said in a letter (PDF) to the Utah Department of Commerce that it learned of the agreement only after the system was up and running.
“Oversight of prescription refills is a task given to appropriately licensed physicians for important safety and clinical reasons. After each refill, the doctor must conduct a reassessment to safely adjust the dose, monitor for side effects, contraindications, new drug interactions, and ensure the drug continues to be effective.” “Clinical decision-making is required. Patients who continue to have their medications refilled without evaluation may remain on outdated or suboptimal treatments for months or even years. There is a reason why prescription refills require physician approval.”
The April 20 letter, signed by 11 of the medical board’s 14 physicians, said proceeding with an AI-powered pilot project without consulting the medical board “could put Utah residents at risk and remains a major concern of the board.”
“It is essential that medically-backed experts review all proposals before implementation to ensure that these programs do not compromise patient safety. We must not allow AI or other financial incentives to override this mandate, and that is exactly what happened here,” the medical board wrote.
The medical board is calling for an “immediate suspension” of the program pending further discussions.
In a statement to Fierce Healthcare, a Doctronic spokesperson said, “We are participating in the process as planned, with established safeguards, physician oversight of all prescriptions during the first phase of the program, and continued physician engagement throughout the program. We remain focused on demonstrating safe, evidence-based expanded access to daily care.”
Utah officials sought to address the medical board’s concerns in a response letter dated April 21, noting that companies requesting deregulatory agreements have submitted “thoroughly vetted and detailed implementation and safety plans.”
“For any agreement involving a medical application, OAIP consults with medical and public health experts in that specific field, as well as regulatory authorities, to ensure that the technology includes the necessary safety guardrails,” state officials said in a letter jointly written by OAIP and the Department of Professional Licensing (DOPL).
The pilot, including Doctoronic, “was rigorously reviewed by several medical experts prior to launch,” state officials claimed. “This evaluation process resulted in a number of substantive adjustments and guardrail recommendations, many of which were incorporated into the pilot. As we communicated in a recent email thread, we look forward to further involving the Board in the scrutiny and oversight process as we evaluate this pilot and consider future pilots.”
The pilot is currently in its first phase, and all AI-generated prescription renewals will be reviewed and approved by qualified human physicians before being sent to pharmacies, according to a letter from state officials.
“Any disagreement between the automated system and the reviewing clinician will be further reviewed and analyzed using established academic scoring procedures to assess risk,” the letter said.
State officials said those consistent safety standards must be demonstrated before the program moves into Phase 2, which requires clinician review immediately after a prescription is issued. “If the AI system is determined to be safe in Phases 1 and 2, the pilot will move to Phase 3, where the AI will be able to “semi-automatically update prescriptions (sic) and ensure continued compliance with standards, while providing ongoing clinical monitoring in the form of random sampling of AI output,” state officials said.
State officials argue that AI systems must conduct comprehensive medical evaluations at every step that “reflect human clinical decision-making, including validating medications, screening for new side effects, analyzing drug interactions, confirming allergic conditions, and evaluating ongoing effectiveness.”
It is “strictly prohibited” for AI systems to handle controlled substances, change treatment plans, or initiate new prescriptions. Patients should also be evaluated regularly by a doctor, either in person or via telemedicine.
“If a case deviates from established guidelines, exhibits clinical complexity, or contains conflicting information, AI should automatically escalate consultation to a physician. Patients and pharmacists also retain the permanent option to request human physician review,” state officials wrote.
The state plans to continue the pilot, and OAIP “reserves absolute authority to modify or terminate the pilot if safety standards are not met.”
The Department of Commerce and DOPL said they want to work with the Utah Medical Licensing Board, an advisory body to the department, to review the pilot’s progress, evaluate the system’s performance and evaluate future medical proposals.
“The biggest risk is maintaining the status quo,” Doctronic co-founders and co-CEOs Matt Pavel and Adam Oskowitz wrote in a recent blog post.
“The question is not whether AI is perfect. Neither AI nor human doctors are ever perfect. The question is whether it is safer than what we currently offer, which means patients don’t get timely updates and end up not taking their medications. 125,000 people die each year from noncompliance. This is not a statistic about AI. This is a statistic about the system we inherited,” Pavel and Oskowitz wrote.
“In Utah, we are still in Phase 1, where all renewals are reviewed by one of our physicians before they reach the pharmacy. By proving that we should, we earn the right to operate with complete autonomy. This is how we build trust,” the co-founders wrote.
Pavel and Oskowitz argue that the healthcare industry faces human capital issues due to a shortage of primary care physicians and increased demand for care, resulting in patients waiting weeks to see a clinician.
“The biggest risk in healthcare is continuing to rely on systems that cannot keep up with current or future demands. To expand access, AI will play a central role in managing day-to-day operations while doctors focus on complex decision-making, reshaping the way healthcare is delivered. “With careful implementation, measurable outcomes, and ongoing validation, this approach can improve access, reduce costs, and better align healthcare delivery to patient needs,” the Doctonic co-CEOs wrote.
