When people debate highly controversial issues, they often misrepresent the other side’s arguments. A recent study tested whether individuals can accurately describe the beliefs of their political or social opponents and found that people are generally very good at understanding opinions with which they disagree. These results are cognitive science.
Measuring open-mindedness has historically been a challenge for psychologists. Traditionally, researchers have used self-report surveys that ask people to rate their intellectual habits. Participants may be asked to agree or disagree with statements about whether or not to listen to contradictory evidence. Other surveys ask people to say whether they think changing their mind is a sign of weakness.
Although these studies are easy to conduct, they are known to have methodological flaws. People often answer questions in ways that make them look good to researchers. This tendency is known in psychology as social desirability bias. Individuals may also lack the ability to accurately observe their own thought processes and thus may claim objectivity even when they routinely ignore contradictory evidence.
To get around the problem of self-reporting, researchers sought behavioral measures. They designed a task inspired by a classic concept in cognitive science known as the Turing test. The basic Turing Test, originally proposed by mathematician Alan Turing in the mid-20th century, assesses whether artificial intelligence can successfully imitate humans. The algorithm passes the test if the human judge cannot determine whether it is a machine or another human being talking to it.
An economist named Brian Caplan previously proposed adapting this concept to human discussion. In the ideological Turing test, you have to make enough of an argument to convince someone who actually holds the opposite point of view. The original idea suggests that if you can seamlessly pass as an opponent, you truly understand their worldview.
Psychologists Charlotte O. Brand, Daniel Brady and Tom Stafford from the University of Sheffield put this concept into practice. They wanted to see if open-mindedness could be reliably documented through experimental design without relying on introspection. The research team selected three topics that are polarizing in the UK as the basis for their study. These topics included veganism, the COVID-19 vaccine and the Brexit vote to leave the European Union.
It attracted 600 participants from all over the country. Each person had a strong opinion on at least one of these three issues. In the first phase of the experiment, participants provided arguments to support their beliefs. For example, vegans were asked to imagine having a conversation with a fellow vegan and write down three reasons why that person would follow the diet.
In the second phase, the researchers asked these same people to completely switch perspectives. Vegan participants had to imagine having a conversation with a meat eater and write down three reasons why that person would avoid a vegan diet. This process generated thousands of authentic and imitated arguments.
The researchers processed these submitted arguments and prepared them for the blind review process. They transformed all statements into first person to make them sound like authentic beliefs. For example, an argument originally written as “They might be afraid of needles” became “I’m afraid of needles.”
The researchers also removed text with sarcastic remarks. Derogatory comments such as “They think the vaccine will track them” have been changed to “I think the vaccine will track me.” This preparation made the discussion appear serious and did not reveal the author’s true identity.
A new group of 1,200 participants then acted as judges. These raters read the prepared arguments and indicated on a 7-point scale how strongly they agreed with each statement. The judges did not know whether the arguments were written by true supporters of the issue or by opponents posing as supporters.
The research team established a baseline threshold for passing the test. They calculated the average agreement score that true supporters gave to arguments written by their fellow supporters. Arguments written by opponents pass the test if they score above this baseline average.
This relative baseline explains differences in how much particular groups generally agree with their side. For example, vegans rated pro-vegan arguments highly across the board. Non-vegetarians tended to remain ambivalent regarding both perspectives. By establishing relative metrics, we were able to prevent misleading success rates across the data.
The results revealed a high baseline of mutual understanding among participants. In all three debates, participants were generally good at imitating opposing viewpoints. Success rates certainly vary depending on the problem in question.
About 54% of arguments intended to mimic the opposing side’s arguments over coronavirus vaccines passed the test. The debate on the Brexit vote achieved a 64% approval rate. The mock discussion on veganism reached the highest success criterion with a 71% pass rate.
The researchers observed no evidence that either side of a particular argument was inherently better at passing the test. Those who voted to leave the EU were as able to predict the Remain debate as Remainers were to predict the Leave debate. Being part of a minority group, such as vaccine refusers, did not predict better performance.
Passing the test provided predictive insight into how people view their ideological rivals. The Sheffield team checked their behavioral results against standard questions about their opponent’s perceptions. People who were more successful at imitating others were less likely to judge them negatively.
Specifically, successful participants were less likely to rate people with opposing views as ignorant or irrational. They were also less likely to perceive their rivals as immoral or unethical. Additionally, those who passed the test were more likely to acknowledge that their partner had a good argument for their beliefs.
The researchers recorded the amount of time participants self-reported researching topics of their choice and debating their opponents. They predicted that high engagement and frequent discussions would lead to improved test performance. This hypothesis could not be realized with the data collected.
Frequent discussions with self-reported opponents did not predict higher pass rates. In the specific case of veganism, those who reported spending the most time researching the topic were actually less likely to pass certain indicators of the test. High engagement with a topic does not automatically correlate with better perspective-taking. Reading about an issue for too long can isolate a person from the actual opinions on the other side.
Researchers note that open-mindedness may not be a stable personality trait that remains the same throughout life. They seem to react to specific situations and personal circumstances. A person may be very good at understanding others in one area, but have difficulty understanding others in another.
Future research could investigate how the ability to articulate reasons for opposition is related to broader cognitive abilities. Psychologists study a phenomenon known as theory of mind. This includes the ability to blame others for mental states or distinct beliefs. Investigating how natural differences in theory of mind predict argument performance may provide new insights into human social behavior.
Overall, this study provides the scientific community with a new behavioral tool. By moving away from research that relies on self-reflection, researchers can document exactly how we interact with different opinions. This design could ultimately help us assess whether different conflict resolution programs actually improve our ability to understand each other.
The study, “The Ideological Turing Test: Behavioral Measures of Open-mindedness and Perspective Taking,” was authored by Charlotte O. Brand, Daniel Brady, and Tom Stafford.
