A new Pennsylvania-led randomized controlled trial found that an AI-powered chatbot was more likely to convince vaccine-hesitant parents to vaccinate their children against human papillomavirus (HPV), but only on standard public health protocol.
The findings raise questions about when, how, and to what extent AI will enhance public health communication. “Comparing chatbots to others is not really a fair test. The interesting question is whether chatbots are better than what public health agencies are already offering. In our study, that was not the case,” said Sharath Chandra Gunthuk, research associate professor of computer and information science (CIS) and lead author of the study.
explained in a new paper by JAMA network openThe trial, which involved about 1,300 participants in the US, UK and Canada, found that skeptical parents who interacted with the chatbot were more likely to say they intended to vaccinate their children than parents who did not receive the intervention.
But just a few minutes of reading the standard documentation provided online by government health agencies like the Centers for Disease Control and Prevention (CDC) about the benefits of the HPV vaccine had essentially the same effect.
It would have been easy to find positive results by comparing AI chatbots with no intervention or very weak control conditions. But we wanted to know whether chatbots would add value beyond what public health agencies already provide. ”
Neil Sehgal, PhD student at CIS and lead author of the study
Promises and challenges of AI in public health
As chatbots become more powerful and widely used, researchers around the world have begun investigating the extent to which AI can change people’s minds, and a growing wave of vaccine hesitancy has made the issue even more urgent.
Two years ago, a study found that AI-powered chatbots could reduce belief in conspiracy theories. Last year, a randomized controlled trial in China found that giving parents access to a chatbot made them more likely to give their daughters the HPV vaccine or schedule an appointment for it.
However, the design of these and similar studies often makes it difficult to assess the effectiveness of chatbots. For example, the Chinese study compared two weeks of access to a chatbot with no intervention, so it is unclear whether other persuasive content, such as standard public health materials, would have had a similar effect.
“What’s new here is the comparison with strong, realistic controls,” says Alison M. Buttenheim, professor of nursing and health policy in family and community health and co-author of the study.
Test your AI against stronger standards
In contrast, the Pennsylvania team set strict guidelines to make the chatbot as comparable as possible to existing public health materials and to evaluate its effectiveness over time. Participants assigned to the chatbot or document groups were exposed to each for a minimum of the same amount of time, 3 minutes, and all participants were assessed on their intention to vaccinate their children 15 and 45 days after the intervention.
Under these more demanding conditions, the chatbot’s innovative features (such as the ability to talk to parents in real time) provided essentially no additional benefit, even though parents spent more time engaging with the chatbot than with the document.
“While chatbot conversations can immediately drive intent, their benefits disappeared when compared to well-designed public health materials and did not persist over time,” said Lyle Unger, CIS professor and co-author of the study.
In fact, at day 45, participants assigned to read public health materials expressed higher intentions to vaccinate their children than participants who interacted with the chatbot or received no intervention.
“AI chatbots are promising, but they should not be considered better than existing tools simply because they are new and more interactive,” Guntuku adds. “A brief reading of a CDC webpage lasted at least as long as a chatbot conversation, and the effects actually lasted longer.”
The challenge of turning intentions into action
The researchers caution that the minimum exposure time for each intervention may overstate the actual strength of such written public health materials. “Would every parent who is hesitant about getting the HPV vaccine choose to spend three minutes on a CDC webpage?” Sehgal asks. “Maybe, maybe not. Chatbots are certainly more interactive.”
Ultimately, neither the written intervention nor the chatbot intervention increased the proportion of parents who said their child had actually received the HPV vaccine within 45 days of the study period, highlighting the challenge of addressing vaccine hesitancy.
Researchers say more participants may not have been able to vaccinate their children simply because the study was too short and the intervention focused too much on communication rather than on structural barriers to health care, such as time, money or access to a pediatrician.
“Vaccination is not just a matter of communication,” Buttenheim said. “Even if interventions change some parents’ minds, it takes a lot of time to turn intentions into actions.”
Towards evidence-based AI for public health
Next, the researchers want to test the use of AI in more complex scenarios. There, chatbots act not as one-time conversation partners, but as medical concierges, helping to schedule appointments, send reminders, and connect with clinicians. “Chatbots can become even more useful when combined with other features,” points out Unger.
The University of Pennsylvania team is also extending this research to global health settings, including a study of AI-powered vaccine communication in Nigeria. The goal is to understand how chatbot interventions can be adapted to local contexts, rather than simply transplanting them from studies conducted in the US, UK, and Canada. “For AI tools to benefit public health, they need to be evaluated in the communities where they are likely to be deployed,” Sehgal says. “This means not just translating chatbot prompts, but working with partners to understand concerns, language, trust, and access.”
Ultimately, the researchers hope their study will encourage a more evidence-based approach to the use of AI in public health. “AI tools need to be evaluated against realistic alternatives,” says Guntuku. “It’s time to shift the conversation from ‘Can AI persuade people?'” to answer more detailed questions such as “When will AI add meaningful value, for whom, and under what conditions?”
sauce:
University of Pennsylvania School of Engineering and Applied Science
Reference magazines:
Sehgal, NKR; Others. (2026) Large-scale language model chatbot conversations with public health materials and parental HPV vaccination intentions: A randomized clinical trial. JAMA net open. DOI: 10.1001/jamanetworkopen.2026.16822. https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2849959
