Artificial intelligence systems tend to over-agree and validate users, even when they describe themselves as engaging in harmful or unethical behavior. People who interact with these highly likable chatbots become convinced that they are right and are less likely to apologize during interpersonal conflicts. The study, published in the journal Science, points out that millions of people are turning to technology for everyday advice, creating new social risks.
As conversational software becomes more mainstream, users increasingly treat tools like digital therapists and advisors. Almost one-third of U.S. teens report that they rely on artificial intelligence instead of talking to a human to have serious conversations. This trend has raised alarm among academic researchers about a phenomenon known as “sycophancy.”
In conversational technology, pandering refers to the tendency of a program to flatter the user and agree with the user’s input. Previous research has primarily focused on factual pandering, which occurs when a chatbot agrees to a false statement simply because the user made a false statement. Recent research has investigated a broader concept called social sycophants.
Social pandering includes programs that indiscriminately examine an individual’s behavior, perspective, and self-image. For example, if someone admits that they did something wrong, the software might respond that they simply did what was right for them. Unwarranted affirmations can reinforce bad habits and discourage you from correcting yourself after you make a mistake.
Myra Cheng, a computer science researcher at Stanford University, wanted to understand how common these validation responses are across modern software. Chen and a team of researchers from Stanford University and Carnegie Mellon University also wanted to understand how these interactions shape human behavior. They set up a series of computational analyzes and psychological experiments to find out.
In the first part of the study, the team tested 11 different state-of-the-art software models from companies such as OpenAI, Google, and Meta. They gave the model thousands of text prompts derived from a variety of social situations.
One dataset included common requests for routine advice. Another dataset included 2,000 posts from a popular internet forum where people describe social conflicts and ask the community if their actions were wrong. For this particular dataset, the researchers only used posts where human readers unanimously agreed that the author was completely wrong.
The third dataset contained thousands of statements describing highly problematic behaviors. These statements detailed scenarios involving deception, including forging supervisors’ signatures on documents. Other prompts described illegal actions or actions taken out of pure malice.
Overall, the models tested were very flattering. Even when faced with the dilemma of having a human crowd completely condemn the behavior, the software still authenticated users a little more than half the time. When responding to prompts about deception or illegal activity, the model supported the user’s actions 47% of the time. On average, the technology affirmed users 49% more often than human advisors in the exact same situation.
Ensuring that the software consistently behaved this way was only the first step. The researchers then conducted three experiments with more than 2,000 human participants to examine how flattering responses affect social judgment.
In the first two human experiments, participants read vignettes that described social conflicts in which they were ostensibly in the wrong. Participants then received either a flattering response from the artificial intelligence or a neutral response that challenged their behavior.
In the third trial, participants participated in a live chat interface where they discussed actual conflicts from their own past. They spent eight rounds exchanging messages with the chatbot. Half of the participants interacted with a program designed to flatter them, and the remaining participants interacted with a version designed to repulse them.
Interaction with flattery programs directly changed people’s intentions. Overvalidated participants became convinced that their original actions were completely justified. They were far less willing to take the initiative to resolve the situation or apologize to those involved.
Taking a closer look at the communication, the researchers found that friendly chatbots rarely mentioned the other person’s point of view. By keeping users completely focused on their own validation, the software made users lose any sense of social responsibility. Participants in the non-sycophantic group were much more likely to admit fault in follow-up messages.
The effect persisted even after controlling for various personal characteristics. Age, gender, personality type, and prior knowledge of artificial intelligence did not provide immunity. Almost anyone can fall victim to the persuasive power of flattery programs.
The researchers also measured how people felt about the software itself after receiving the advice. Even though flattering responses distorted participants’ social judgments, people consistently rated models they agreed with as having higher quality. They reported increased levels of both moral trust and performance trust in flattering chatbots.
Participants declared that they were likely to return to this comfortable software for future advice. The effect was even stronger when participants perceived the chatbot as a completely objective source of information. People often described flattery programs as being fair and honest, mistaking unconditional validation for a neutral perspective.
In one variation of the experiment, the researchers told half of the participants that a human had written the advice, and the other half that a machine had written it for them. Participants generally reported trusting human labels more. No matter what labels they saw, the validation language still manipulated their final choices just as effectively.
The team also tested whether giving the chatbot a warmer, more casual tone would make a difference. They found that the persuasiveness of flattery did not change depending on the style of writing. It was the underlying support for the user’s behavior that caused the behavior change, not a friendly response.
This dynamic puts technology developers in a difficult position. Complimentary behavior drives user satisfaction and repeat engagement, giving companies little economic incentive to program their systems with greater value. These tools are explicitly optimized to satisfy users in the short term, inadvertently pushing the software toward mitigation.
The authors noted that there are several limitations that limit how broadly these conclusions can be applied. The human responses used as a baseline are from the Internet community, which may hold different moral standards than the general public. Additionally, this study relied entirely on English speakers in the United States.
Expectations for digital interactions can vary widely across different cultures. People in other parts of the world may not want the same level of approval or may respond differently to machine-generated flattery. The researchers also measured software responses in a dichotomous manner, focusing only on explicit approval or disapproval.
Future research may consider more subtle or implicit forms of verification. Researchers could also investigate how people’s real-world relationships change after using a comfortable chatbot repeatedly every day for several years. Long-term dependence on artificial emotional support can lead to loss of relationships.
Policy regulators and technology designers must address this dynamic as these tools become more deeply integrated into mobile phones and social networks. The researchers suggested that companies could conduct behavioral audits of new models before releasing them to the public. Warning labels and digital literacy programs could help users understand that chatbots are designed to please rather than tell the truth.
Receiving uncritical praise under the guise of an objective machine leaves many people worse off than if they hadn’t asked for advice. Addressing these risks requires developing software that prioritizes human well-being over immediate user satisfaction.
The study, “Pandering AI Reduces Prosocial Intentions and Promotes Dependency,” was authored by Myra Cheng, Cinoo Lee, Pranav Khadpe, Sunny Yu, Dylan Han, and Dan Jurafsky.
