Recent research published in Journal of Experimental Psychology: General It suggests that people consistently judge creative writing more harshly if they believe it was created by artificial intelligence. This bias seems incredibly difficult to overcome and is indicative of a deep-seated human preference for human-created art.
Generative artificial intelligence refers to computer programs that can generate new text, images, or music by predicting patterns from large amounts of data. Tools like ChatGPT and Claude allow you to write essays, poems, and stories that look like they were written by a real person. As these technologies become more common, scientists wanted to understand how people react to computer-generated art.
“We started this project in early 2023, shortly after the launch of ChatGPT. From our early interactions with the technology, it was clear that this tool was capable of creative production. We were very interested in whether and how humans would react to creative products produced by AI,” explained study author Manav Raj, assistant professor of business administration at the Wharton School at the University of Pennsylvania.
Previous research suggests that people may not be able to tell the difference between human and computer writing when placed in low light. However, researchers conducted this particular study to see what happens when they explicitly tell the audience that a machine wrote the text. They wanted to know whether this knowledge could change the way people enjoy art and dampen its negative reactions.
To explore these questions, the scientists conducted 16 separate experiments with a total of 27,491 participants. In the first group of five experiments, the researchers tested whether the actual content of the text changed how people responded to the label artificial intelligence. We had participants read poems and short stories generated by ChatGPT and rate them in terms of quality, creativity, and enjoyment.
Some participants were told that a machine wrote the text, while others were told that a human wrote it. The researchers varied writing styles, testing first- and third-person perspectives, poetry and prose, and different emotional tones. We also tested stories that contrasted human characters with aliens, animals, and robots.
Across all these variations and thousands of participants, readers consistently gave lower ratings to texts they thought were written by a machine. Changing story details did not consistently reduce this penalty. At this early stage, we have evidence that bias is largely independent of the specific content of the writing.
In the second phase of the study, the scientists conducted an experiment with 3,590 participants to see if the context of the assessment matters. They asked a group to judge a text as a work of art. They asked another group to make judgments based on objective qualities such as consistency and logicality.
This change in instructions did not alleviate negative reactions. Participants in both groups rated the text as less valuable even when they believed it came from a computer. This suggests that bias applies whether people are reading for pleasure or for practical evaluation.
The researchers then conducted five more experiments to see if changing people’s perceptions of computer programs would help. In these studies, participants were asked to read articles about machines’ impressive cognitive and emotional abilities before reading the generated stories. In some versions, scientists tried to humanize the software by giving it a name and gender.
None of these strategies could reliably reduce negativity bias. Even when the computer program was described as highly capable or endowed with human characteristics, participants still rated the text lower once they learned its origin. Across these diverse approaches, negative reactions were found to be significantly persistent.
“What was surprising to us was how long-lasting the effects were,” Raj told SciPost. “We really tried to ‘break out’ of that at various points and find situations where we could eliminate the AI disclosure discount. Despite our attempts to build on the existing literature on algorithmic evasion, we found this result to be very troubling. ”
In the fourth experiment, the scientists investigated whether just knowing that a computer had written the story made people feel ambivalent. Ambivalence refers to a complex emotion in which a person may feel both positive and negative qualities about the exact same thing at the same time. Researchers attempted to measure this specific emotional state by testing 423 and 1,280 participants in two studies, respectively.
They found that knowing about computer involvement did not evoke mixed emotions. It simply made participants’ judgments more negative overall. This exposure did not provoke a complex emotional response, but rather directly reduced ratings.
Finally, the researchers ran three experiments to test the concept of human involvement in the loop. They wanted to know whether framing the writing process as a human-machine collaboration would be viewed more favorably. They tested this with machine-generated stories and real award-winning short stories written by humans.
When it was explained that humans were using a computer program as a tool to write a story, participants still judged the work as harshly as if it had been written by a machine on its own.
Throughout the study, researchers collected data on a variety of underlying mechanisms, including humanness, effort, and emotional depth. They consistently found that perceived trustworthiness was the strongest factor explaining lower ratings. People simply view machine-generated text as less reliable than human-authored text, which explains the negative ratings.
“Our main finding, at least for now, is that humans have persistent negative reactions to knowing that creative goods (or at least creative writing) are being produced with the help of AI,” Raj said. “Everything that leverages AI is now a moving target, and this has been going on for a significant amount of research and about two years of data collection.”
Although these findings provide strong evidence of bias, there are some potential limitations that should be kept in mind. Participants were recruited from online platforms that tend to attract moderately tech-savvy individuals. This means the results may not be fully representative of the world’s population as a whole.
The observed biases can also manifest differently in visual art, music, and other physical products. As society becomes accustomed to this technology, it is very likely that attitudes will change. Future research could investigate whether this negative bias fades over time as machine-generated text becomes an everyday reality.
“One thing to note is that our study does not address the quality of creative products produced by AI at all,” Raj explained. “In all cases, we kept the sample of writing constant and only manipulated whether participants believed it was written by an AI. Therefore, the quality and nature of the creative product is an open question.”
“This last point is a question I would like to research in the future. We are using AI for creative purposes and innovation, but we don’t yet know what that means for the properties of creative goods (aside from some studies suggesting that in some circumstances it is difficult to distinguish between AI-generated creative goods and human-generated creative goods). I am very interested in pushing this area further.”
The study, “Artificial Intelligence Disclosure Penalties: Humans Persistently Disdain the Value of AI-Generated Creative Writing,” was authored by Manav Raj, Justin M. Berg, and Rob Seamans.
