When people learn that an artificial intelligence program is evaluating their work or university program, they change their behavior to be more analytical and less emotional. These changes in strategy could lead to inaccurate evaluations and ultimately change who is selected for key positions. The initial findings were recently published in the Proceedings of the National Academy of Sciences.
With rapid advances in computing technology, organizations are increasingly using automated tools to screen applicants. Recruiters and university admissions officers are implementing these automated systems to increase efficiency and process large volumes of candidates. These programs often take the form of video interview software, personality assessments, or automated resume screening.
At the same time, the government is passing new laws requiring transparency in hiring and admissions. For example, the European Union’s artificial intelligence law requires organizations to disclose whenever they use algorithmic systems in high-stakes situations. Similar local laws, such as one recently enacted in New York City, also require employers to notify job applicants when automated tools are activated.
Because of these transparency rules, candidates are often fully aware that their performance is being evaluated by a machine. Researchers wanted to know whether this knowledge would change applicants’ behavior during exams and interviews. People naturally try to manage the impression they make on others, especially when the outcome involves a major life event such as a new job or degree program.
When dealing with human evaluators, applicants often guess what the evaluator is looking for and act accordingly. The study authors suspected that the applicant was applying exactly the same strategy to automated systems. They hypothesized that people have common assumptions about how computers work and process information.
Specifically, people tend to view machines as purely rational processors of numbers and facts, completely lacking in emotional intelligence or intuition. In psychology, these common assumptions are known as common beliefs. The researchers predicted that public beliefs about machine rationality would lead applicants to emphasize their logical and analytical traits. They termed this expected change in self-presentation the “AI evaluation effect.”
Researcher Jonas Gergen from the Institute of Behavioral Sciences at the University of St. Gallen in Switzerland led the study. He collaborated with Emmanuel de Bellis, also from the University of St. Gallen, and Anne-Kathrin Kresse from the Rotterdam School of Management in the Netherlands. Together, they conducted an extensive series of experiments to see how machine raters influence human self-expression.
The research team designed 12 separate experiments involving more than 13,000 participants. Before conducting the formal experiment, they analyzed data from an employment software company called Equalture. They surveyed over 1,400 job seekers who completed a game-based skills assessment. Candidates who strongly suspect that algorithms are involved in the hiring process reported adjusting their own behavior the most.
After observing this real-world trend, the authors began a pilot field study. They posted their first job ad for an event planner on the freelance platform Upwork. We divided the submissions into two groups and invited freelancers to apply. Half of the applicants were told that a human would review their application, and the other half were told that a machine would review their skills.
Applicants then wrote a self-introduction as part of the selection process. They rated themselves on a numerical scale ranging from intuitive to analytical. He said applicants who believe a machine is reading a file are much more analytical than those who believe a human is reading the same text.
To establish a clear cause-and-effect relationship, the team moved into a controlled experimental environment. In one experiment, participants imagined applying for admission to a fictitious university called Horizon University. They were randomly assigned to face either a human or machine reviewer. Participants then answered survey questions about how they approached the task mentally.
Under the automatic reviewer condition, participants reported approaching the task in a more analytical and logical manner, with less intuitive aspects. The researchers asked additional questions to understand why this happened. They found that this change was driven entirely by the participants’ underlying belief that machine programs prioritize analytical over emotional characteristics.
Researchers wanted to know whether certain types of people are more prone to this behavioral change. In a study of a representative portion of the U.S. population, they tested a variety of personality and demographic factors. They found that younger cohorts were more likely to act more analytically in front of a computer. Additionally, participants who expressed a strong fear of negative evaluation were particularly likely to change their self-presentation.
In another experiment, we tested whether behavioral changes hold true in more realistic decision-making tasks with clear rewards. The researchers invited participants to apply to an exclusive research pool that offered purely financial incentives. To participate, participants had to rank a list of eight personal attributes that described themselves. Half of the attributes represented logical characteristics and the other half represented intuitive characteristics.
When participants thought the machine was checking their ranks, they consistently pushed logical attributes to the top of the list. They assigned the lowest rank to intuitive traits. When participants thought humans were looking at the exact same screen, the rankings were much more balanced across both categories.
The team also tested what happens when people participate in both types of assessments over time. In a multistage experiment, participants answered questions about their actual baseline behavior in a standard research context without any evaluation. A few days later, they went through a mock application process.
When evaluated by the machine, participants deviated significantly from their baseline true selves. There was a huge gap between their honest baseline and the machine-rated persona. The researchers noted that this level of distortion could easily change which candidates pass the hiring criteria.
Their mathematical simulations found that more than a quarter of people who met a strict logical cutoff score when assessed by a machine did not meet the same cutoff when assessed by a human.
To test the underlying psychological mechanisms, the researchers directly manipulated participants’ assumptions. In two experimental variations, they asked participants to actively imagine a computer program that was capable of deep emotional and intuitive understanding. They asked participants to write down reasons why machines are good at reading emotions.
When participants were encouraged to view the machine as emotionally competent, the behavioral changes completely disappeared. They stopped exaggerating their logical skills. In some cases, they actually claimed to be less analytical and more intuitive than human-rated controls.
The researchers also tested a hybrid approach common in modern corporate recruitment. They told participants that computers would make the initial selection, but that human managers would make the final hiring decisions. Although this combination reduced behavioral changes, applicants still exaggerated their reasoning characteristics to a noticeable degree.
These changes in applicant behavior can pose significant challenges to testing the effectiveness of automated recruitment systems. When candidates express themselves artificially, organizations are unable to accurately measure their actual skills. Researchers suggest that employers should be aware of this impact and take immediate steps to address it. For example, a company looking for a highly empathetic social worker may end up hiring an overly analytical person who is simply trying to appease a software program.
The research team pointed out that future research should focus on various situations other than corporate employment. For example, automated assessments may be used to determine who should be granted access to public services or approval for loans. It remains to be seen whether people applying for government benefits will change their behavior in exactly the same way.
Most of the current experiments focused solely on the spectrum between logic and intuition. The researchers explained that other aspects of personality may also be affected by automated screening. Preliminary data from their exploratory model suggests that applicants may downplay their creativity and risk when faced with mechanical auditors. Participants also appeared to reconcile ethical and social considerations.
Additionally, new technological advances may change people’s responses to automated rating systems over time. The sudden rise of advanced generative conversational chatbots could change the public’s perception of what machines can understand. Future research should track whether candidates respond differently to software programs that appear more conversational and human-like.
The study, “AI assessments change human behavior,” was authored by Jonas Goergen, Emanuel de Bellis, and Anne-Kathrin Klesse.
