Kano Survey

The Kano Survey is a research tool aimed at discovering consumer expectations related to product functions and aspects of product quality, according to the Kano Model.

In theory, Kano Surveys can be useful to help shape the understanding about customer needs and preferences, choose between alternative options for a function, uncover hidden customer expectations about basic product functions and potential exciters, and help with prioritisation of product development work. They can also help to prevent over-engineering by identifying unnecessary features and improvements that would not contribute to customer satisfaction.

However, there is no universally accepted way of scoring and quantifying Kano survey results, and there are no empirical proofs for the validity of the most common scoring systems. As a result, it’s best to treat Kano survey results as guidelines with decent scepticism.

• Kano survey questions and responses
  • Interpreting Kano Survey Results
  • Quantifying Kano Surveys
• Required population samples
• Advantages of Kano surveys
• Downsides of Kano surveys
• Criticism

Kano survey questions and responses

The survey asks participants to select their attitudes towards a product feature from two opposite perspectives:

• Functional: how they would feel like if the feature was present (implemented)?
• Dysfunctional: how they would feel like if the feature was absent (not implemented)?

There are no universally accepted ways of formulating such questions, and multiple different variants are popular for different perspectives. For example, when researching whether some feature should be included in the product design, you can ask customers “How would you feel if this feature was present?” and “How would you feel if this feature did not exist?” Alternatively, when researching if you should improve some variable function, you can ask “How would you feel if this was faster?” and “How would you feel if this was slower?”

For each perspective, respondents can choose one of the following attitudes:

• like
• expect
• neutral
• can tolerate
• dislike

These are usually presented in the form of longer statements, such as:

1. I like it that way,
2. It must be that way,
3. I am neutral,
4. I can live with it that way,
5. I dislike it that way.

Or

1. I like it
2. I expect it
3. I feel neutral
4. I can tolerate it
5. I dislike it

The answers are combined, to eliminate confused or arbitrary responses, and discover true customer preferences.

Interpreting Kano Survey Results

Once the faulty responses are eliminated, you can use the remaining answers to categorise the target of your research into segments of the Kano model. For example:

• If customers expect a feature to be present, and dislike when it’s not present, then it is “must be” (also called “basic”), marked as M below
• If customers like when a feature is present, but dislike if it is not present, then it is a “one-dimensional” (also called “performance”), marked as O below)
• If customers like when a feature is present, but are neutral if it not present, then it is an “attractive” (also called “exciter”), marked as A below
• If customers dislike when a feature is present, and expect it not to be present, then it is a “reverse” (also called “dissatisfier”), marked as R below
• Lack of strong opinion about both perspectives is likely signalling that the customer is indifferent, marked as I below
• Inconsistent responses (both positive, or both negative) suggest that the customer is not interested or not paying attention, and should be eliminated (“questionable”), marked as Q below.

There are several alternative matrix systems for scoring, differing on what they declare questionable or not. The two most popular are:

And

Quantifying Kano Surveys

Kano Surveys were originally intended as a qualitative method, so there is no universally accepted way of quantitatively scoring them.

Ting Wang and Ping Ji propose calculating two factors: user satisfaction (CS) and user dissatisfaction (DS). CS represents a level of satisfaction if the feature would be fully implemented, DS represents the dissatisfaction if the feature is fully excluded from a product.

• CS = (A + O) / (A + O + M + I)
• DS = (O + M) / (A + O + M + I)

On a graph, different features can be visualized for easy comparison by plotting points (1, CS) and (0, -1 * DS); then drawing a satisfaction function for visualization and analysis. Wang and Ji recommend drawing different lines or curves through the two points for categories of features (line for one-dimensional/performance, exponential decay (trending towards 0) for must-be, and exponential for attractive.

Chapman and Callegaro, present a potential mapping into a continuous scale (although in the paper they warn about the validity of the results):

• Functional: -2 (Dislike), -1 (Tolerate), 0 (Neutral), 2 (Expect), 4 (Like)
• Dysfunctional: -2 (Like), -1 (Expect), 0 (Neutral), 2 (Tolerate), 4 (Dislike)

They also suggest calculating the degree of agreement between the answers using the Rand Index. They also suggest calculating a reliability coefficient, either using using the Kendall Tau Correlation Coefficient if you want to treat the data as non-parametric ranked (ordinal), or Pearson’s r coefficient if you want to treat the data as continuous.

In Empirical Research on Kano’s Model and Customer Satisfaction, Feng-Han Lin and colleagues propose combining Kano classification with Gain and Loss factors from Prospect Theory, establishing a satisfaction and a dissatisfaction index and proposing a model for statistically analyzing performance.

Required population samples

Chapman and Callegaro argue that category assignment may be unreliable for samples of fewer than 200 participants. (“With N < 200, we would expect at least 1/3 of features to be miscategorized more than 20% of the time. With N <= 15 respondents, the overall accuracy rate may be less than 50%.”) They did find that for larger samples (of more than 1500 respondents), the overall category associations became stable, but even as such they might not be valid for business decision making.

Advantages of Kano surveys

Kano surveys can be used for systematic quantitative research with large populations of users. The questions and responses are closed (customers can only select from predetermined options) so the responses can be easily analysed at scale.

A big advantage of Kano surveys is that they are self-controlling. The combination of answers from two perspectives helps to eliminate inconsistent responses, where the surveyed participants were confused or not paying attention. It the participant understood the questions and had a strong opinion about the topic, they would be expected to provide a positive answer in one category and a negative in another. If, for example, someone said that they would like to be able to arbitrarily rearrange notes on a mind map, and that it would bother them if the notes did not follow the ordering from the map, then the response is invalid and should be removed from the survey results. Because of that, there is no need to include additional control questions and complicate the survey. Fewer questions generally leads to a higher completion rate, especially for longer surveys.

Downsides of Kano surveys

As a closed survey, the Kano questionnaire forces answers into discrete categories, which may not always accurately reflect the nuanced nature of customer satisfaction and expectations. This makes the model useful for later-stage research, once product managers already have a solid understanding of the user needs, and want to classify them further. For early stage research, other methods such as open-ended questions, focus groups, interviews or observing users will likely bring better results.

Another big limitation of this type of survey is the limited scope for innovation. The model focuses on current customer expectations and satisfaction, potentially overlooking innovative features that customers might not yet recognize as valuable.

Criticism

Using more than 10,000 respondents, Chris Chapman and Mario Callegaro tried to evaluate the reliability and validity of 7 different Kano survey versions. They started with three features that would be expected to fit clearly into one of the main Kano categories, and found that the results only partially aligned with the Kano model theory. The pattern of item associations was “largely inconsistent with Kano theory and assumptions”, apart from the “Dislike” answers.

There are several list several potential reasons why the standard Kano Survey structure may not lead to reliable answers:

Hypothetical attitudes: The survey asks about potential hypothetical feelings, not about current attitudes or behaviour. In particular, the dysfunctional form (asking about the attitude towards the product if a feature would be missing) is very difficult to answer unless people already use a product or that specific feature. Attitudes towards the absence of a hypothetical feature that people do not currently use “tend to have low reliability”, according to Chapman and Callegaro.

Multiple concurrent attitudes: The respondents are asked to choose one attitude from a list, when they can be experiencing several such attitudes at the same time. Chapman and Callegaro use an example of introducing incompatible faster connection cables for existing mobile devices, where the same person might find an argument to support all five attitudes (liking it because it’s faster, expecting it because manufacturers introduce better cables periodically, tolerating it because they’d can buy new cables, and disliking it because they would need to switch from current usage, or being neutral because they can see both pros and cons of the new cables). The rating categories are usually presented as mutually exclusive, but they are not in reality.

Using different variants of the survey, Chapman and Callegaro found that the same people would often change choices between different attitudes towards the same question. For example, only 54% of the people who selected “tolerate” would select that again, and only 39% of the people who selected “expect” in one survey variant did so again for another (40% changed their answer to “like”). Using Pearson’s r coefficient, they rated the reliability of responses at 0.735, concluding that it falls into the lower range of “marginal” reliability (“good” reliability would require a coefficient between 0.8 and 0.9).

Chapman and Callegaro suggest that it’s better “to rate each attitude separately rather than as a single forced choice.”

Large variance in attitudes: In one category of their research, the most common categorization of a feature matched only 28% of the respondents (72% of the survey responses did not match the category that would be presented as a conclusion to the business stakeholders), raising serious concerns about reporting a single category to decision makers. “Features are unlikely to align cleanly with a single, modal category,” say Chapman and Callegaro, suggesting that the reports should include “the degree to which a feature aligns with every category, across the sample”.

Multiple factors: The result scale “conflates multiple dimensions”, but is usually presented as a single, nominal scale. Using exploratory factor analysis on the survey result correlation matrix, Chapman and Callegaro argue that there are least 2, and possibly up 4 or 5 factors in the model. Presenting them on a single-dimensional scale is misleading.

As a result, Chapman and Callegaro claim that the resulting conclusions will appear plausible, but it will not be possible to assess the validity: “In short, we may be unable to trust whether the story is true.”

Learn more about the Kano Survey

• Kano Analysis: A Critical Survey Science Review, Proceedings of the 2022 Sawtooth Software Conference, by C Chapman, M Callegaro (May 2022)
• Understanding customer needs through quantitative analysis of Kano's model, International Journal of Quality & Reliability Management by Ting Wang, Ping Ji (2010)
• Empirical Research on Kano's Model and Customer Satisfaction by Feng-Han Lin, Sang-Bing Tsai, Yu-Cheng Lee, Cheng-Fu Hsiao, Jie Zhou, Jiangtao Wang, Zhiwen Shang (2017)

Related and complementary tools to the Kano Survey

• Kano Model