Acquiescence Effect
Why "Yes" is literally overrated
The Acquiescence Effect (also known as Acquiescence bias, Yea-saying or Agreement effect) is an observed tendency for people to choose positive answers (such as Yes or Agree) more frequently than negative answers, regardless of the question, in the range of 10-50%. This effect is important for user interviews and research because it skews the data from opinion surveys and provides false confidence. The effect is also important for anyone involved in product management and stakeholder alignment, as it can generate a false sense of agreement.
This article explains the potential causes of the acquiescence effect and strategies for avoiding it or reducing it.
- Origins of the Acquiescence Effect
- How significant is the Acquiescence Effect?
- What causes acquiescence?
- How to reduce acquiescence
Origins of the Acquiescence Effect
Cronbach was the first to spot statistical anomalies of selecting positive answers in research, publishing a paper about it in 1942 (Studies of Acquiescence as a Factor in the True-False Test). Cronbach used the name ‘Acquiescence’ in the paper title but ultimately called the effect a “response set”. Psychologists in the 1950s and 1960s tried to explain the effect, mostly attributing it to personality traits, social status of the survey participants, or even the difference between the race of survey participants and survey researchers. By the early 1980s, the issue was well known, deserving a whole chapter in the canonical book on survey methods, Questions and Answers in Attitude Surveys by Schuman and Presser, where they called it “The Acquiescence Quagmire”.
In the late 1990s, Jon A. Krosnick published several research papers about acquiescence, including the first measurements of the magnitude of the effect, and tried to explain the causes with mental models. Billiet and McClendon documented several methods for countering the effect in the early 2000s. More recently, the phenomenon was studied by researchers in the context of political opinion polls, notably by Seth J. Hill and Margaret E. Roberts.
How significant is the Acquiescence Effect?
Krosnick measured the effect at about 10% in 1999. Hill and Roberts measured a much larger effect, up to 50% in some cases, researching surveys about controversial statements such as popular political conspiracy theories. Mario Callegaro and colleagues, in Yes–no Answers versus Check-All in Self-Administered Modes: A Systematic Review and Analyses, measured the effect at 42% by comparing a set of questions asked in the Yes/No format with asking people to select all options that apply instead of simply agreeing.
What causes acquiescence?
There are still no firm answers to what causes acquiescence, but there are several plausible theories.
- Satisficing is a theory with some research evidence, proposed by Jon A. Krosnick in Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys. Krosnick argues that people, when presented with a choice, usually do not carefully think about everything they know about a topic and evaluate options, but generally look for a shortcut that produces a satisfactory answer. In addition, Krosnick suggests that “most people typically begin by seeking reasons to agree rather than disagree”, and once they find enough of them, they agree and move on. This kind of effect is larger when the task is difficult and the participant’s ability and motivation are low.
- Deference and politeness. Another theory is that the difference in status between the person asking a question and the person answering may create acquiescence, because people agree “out of courtesy and respect”. Although earlier psychological research looked into factors such as social status, ethnic and racial differences, there was no conclusive proof of that. However, this kind of situation can occur in a workplace where company hierarchy or perceived expertise creates a tendency to agree with a person’s opinion.
- Agreeable personality. A long-standing interpretation holds that some people are simply predisposed to be agreeable across all situations, consistent with the “Big Five” personality trait of agreeableness. Krosnick is skeptical of this account, noting that acquiescence “is not highly consistent across items within a scale, across scales, or across unrelated tasks”, which is not what one would expect from a stable personality trait.
- Ambiguity and lack of knowledge. John J. Ray argued that “acquiescence depends on the ambiguity of the measures”. Effectively, the vaguer a statement is, the more room there is to simply agree. This is linked to how much people actually know about a topic. Hill and colleagues noted that acquiescence is a standard “heuristic response when subjects do not know how to answer”, particularly problematic for rare, unfamiliar or complex subjects.
- A transient “mood” state. Krosnick also found that acquiescence “appears to result partly from a transient, moodlike state within a single questionnaire”, noticing that if two questions are asked close together, people are more likely to answer both with the same degree of acquiescence.
Beyond social and psychological causes, one clear factor that impacts acquiescence is the format of the question. Forced choice in Dichotomous Questions significantly increases the tendency to agree. Agree/disagree, true/false and yes/no items are, in Krosnick’s words, “seriously problematic” precisely because they invite simple agreement. Krosnick measured the tendency to acquiesce at an average of 34% for agree/disagree questions and 37% for true/false questions (though a lower 16% for yes/no questions), and reported that across eight studies “an average of 14% more people agreed with an assertion than expressed the same view in a corresponding forced-choice question”. Mario Callegaro and colleagues, in Yes–no Answers versus Check-All in Self-Administered Modes: A Systematic Review and Analyses, similarly measured that “endorsement levels increase by a factor of 1.42 when questions are posed in a forced-choice rather than check-all format”.
How to reduce acquiescence
Since the causes of acquiescence are still not fully understood, there is no single universal fix, but there are several methods proven to reduce this tendency.
Balance the scale for a set of questions
When asking a set of related questions together, it’s possible to reduce the acquiescence effect by phrasing half the statements positively, and half the statements negatively, so that the “agree” option sometimes points one way and sometimes the opposite. A typical example of this kind of survey is the System Usability Scale questionnaire. It alternates praise and criticism: “I thought the system was easy to use” is immediately followed by “I think that I would need the support of a technical person to be able to use this system”. Another typical example from usability research is UMUX.
Someone who just agrees with everything ends up praising and criticising at the same time, producing a balanced score instead of an enthusiastic approval. John J. Ray, in Acquiescence and Problems With Forced-Choice Scales, argues that wording half the statements each way “will eliminate any systematic effect of acquiescence”. Jaak B. Billiet and McKee J. McClendon, in Modeling Acquiescence in Measurement Models for Two Balanced Sets of Items, explain why: “acquiescence to the negative items will offset acquiescence to the positive items”, so the average comes out unbiased.
Balanced-scale questions work only when a set of related questions gets asked together, so this approach will not work for a single question. It also only impacts the combined score of the set of questions, not the answer to any single question in the set.
Ask the same question both ways
When working with a larger group, an approach that works even with a single question is to just ask the same question but in two different ways. This applies the balanced scale approach but to people rather than to questions. Researchers split their respondents into two groups: one sees a statement worded positively, the other sees the same idea worded negatively, and the two groups are compared.
Hill and Roberts evaluate this approach in Acquiescence Bias Inflates Estimates of Conspiratorial Beliefs and Political Misperceptions. Because the negative wording flips which answer counts as “agreeing”, the tendency to agree pushes the two groups in opposite directions, and the gap between them reveals how big the bias is. One of their examples is striking: a claim about the FBI reopening the Clinton email investigation drew 64% agreement when worded one way, but only 22% when worded the other, with a 42-point swing that is almost entirely due to the acquiescence effect, with the truth sitting somewhere in between.
This differs from balancing the wording in a subtle but important way. Balancing mixes both directions into one person’s score and needs a survey with several questions; asking the same question both ways works even for a single standalone question, but it only corrects the overall figure across a large group of respondents. This approach does not produce a clean answer from any individual participant. Hill and Roberts also warn that the bias might not be the same for both statements, which also has an impact on the difference.
Spot the habitual agreers
Another approach is to try to measure the bias instead of just hoping that the bias cancels out in a set of questions. Billiet and McClendon recommend adding a second, unrelated set of balanced questions and checking how often each person agrees across both sets. Someone who keeps saying “yes” to two topics that have nothing to do with each other is revealing a personal habit of agreeing rather than a genuine opinion, and that habit can then be subtracted from their answers. Billiet and McClendon’s paper is a good reference for ideas on measuring the bias instead of just reducing it.
Offer both sides, not a single claim
One potential way to fix the issue even for a single question and a single participant is to avoid asking a Yes/No question, but to present two opposing statements and ask people to choose which one they agree with more. For example, instead of asking “Do you agree that the government should reduce immigration?”, ask people “Which option is closer to your view: the government should reduce immigration, or the government should not reduce immigration”. Such questions do not appear to include the opinion of the person asking them, so causes related to deference and politeness should be reduced. People still choose one of two options, but neither is presented as the “official” view to just agree with.
Krosnick’s data suggests this does reduce inflated agreement, but Ray warns it is not a complete solution, because it can cause a “donkey vote” where people simply pick the first option. When working with a larger group of people, this effect can also be minimized by shuffling the order in which options appear.
Let people pick from a list
A similar approach that works even for a smaller group of respondents, but requires a set of questions, is to turn a list of related Yes/No questions into a set of options for people to choose from, and tick all options that apply to them.
For example, instead of asking a dozen separate yes/no questions about which shops someone visited last month, you show one list and let them check the ones they actually went to (also called a check-all-that-apply format). Several studies show this lowers the tally of “yes” answers noticeably. Mario Callegaro and colleagues, in Yes–no Answers versus Check-All in Self-Administered Modes: A Systematic Review and Analyses, pooled many such experiments and found that going item by item produces about 1.42 times as many endorsements as a tick-all list.
Thomas and Klein, in Merely Incidental?: Effects of Response Format on Self-reported Behavior, report on several studies comparing lists of Yes/No questions and asking participants to select all options that apply to them, involving tens of thousands of participants across different countries, and conclude that there is a significant difference in “average endorsement” between the two formats, measuring between 5-10%. They also found that Yes/No variants took significantly longer to complete on average. Jolene D. Smyth, in Comparing Check-All and Forced-Choice Question Formats in Web Surveys, found the same gap but argued that the difference appears because answering one item at a time makes people think harder about each option, so the answers might be more relevant and truthful.
Switching from Yes/No to a list for people to select is a bit of a controversial technique. All the research points to the fact that selecting applicable items from a list results in fewer positive answers than “Agree/Disagree” or “Yes/No”. Whether those missing answers are noise or genuine opinion is up for discussion.
Avoid a forced choice
Adding a “Don’t know” or “Not Applicable” option is another potential way to reduce acquiescence, as there is no forced choice anymore. Schuman and Presser suggest in Questions and Answers in Attitude Surveys that having this option causes about one fifth of respondents to switch away from simple agreement.
Changing from a binary choice to a third option makes the survey analysis more complex, and if you decide to go in that direction, it might be worth expanding the scale to at least six or seven points for attitudinal research. Preston and Colman argue in Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences that scales of 2-4 points have significantly lower reliability than scales of 6 or more points.
Make the question easier to understand
Finally, one of the leading explanations for acquiescence is that tired or unmotivated people just choose the easiest acceptable answer. Krosnick, in Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys, suggests slowing the pace, using simpler language, and giving people a reason to take the task seriously.
For example, instead of a long grid of densely worded agree/disagree statements, show questions in smaller chunks, and remind the survey participants that their answers genuinely matter, to avoid pushing people onto autopilot and clicking the “yes” answers.