Voice of Vovici

Weapons of Mass Collaboration

Dean Wiltse’s first book, Weapons of Mass Collaboration, is now available. In it, Dean demystifies the phenomenon of online communities and explains how businesses can derive lasting benefit from immersion in such communities. He discusses the importance of transforming customer relationships by promoting deeper engagement in a collaborative, online environment.

Dean’s market-research experience as CEO of Greenfield Online and Vovici offers him a unique viewpoint on the industry. Leading the merger of WebSurveyor and Perseus into Vovici, Dean championed the need to extend enterprise feedback management into online communities and was the executive sponsor of Vovici Community Builder.

The contents of Weapons of Mass Collaboration include:

1. Online Communities Defined: A Different Kind of Neighborhood
2. Why Build an Online Community? The Benefits of Holding the Conversation at Your Place
3. Doing It the Right Way: Best Practices for Online Communities
4. The What? and the Why? The Quantitative/Qualitative Divide
5. Bridging the Quantitative/Qualitative Divide Good Things Happen: When “What” Meets “Why”
6. Blending Online Communities and Surveys to Create a Weapon of Mass Collaboration: Reaping the rewards of being focused, faster and more-frequent
7. Becoming a Feedback Driven Organization: Not if, but when?

The book can now be purchased from Amazon in hard copy. For a limited time, Vovici is making an electronic copy of the book available for free, simply for completing this form.

Order Effects and Questions

Having looked at the order effects of choices in web surveys a few weeks ago, I thought it appropriate to look at the order effects of questions themselves. Does re-arranging the order of the questions affect responses?

The market researcher would prefer that the respondent consider each question in isolation, unrelated to any questions that have been asked before. Of course, respondents are not robots, and earlier questions will unfortunately bring topics to mind that can “contaminate” later answers. 

The example of such contamination that I have seen in my own surveys is the order of general questions vs. specific questions. If the general question is an open-ended question, many survey authors I’ve worked with prefer to put it after the closed-ended question, since open-ends are harder to answer (requiring thinking and typing rather than thinking and clicking a button). But when asking the verbatim question second, you will definitely get a greater percentage of respondents talking about the previous questions.

In the paper “Effects of Question Order on Survey Responses” by Sam McFarland, some respondents were asked general questions (describing their interest in politics and religion) and then specific questions (evaluating the state of the economy and the energy market) while others were asked the specific questions first. Asking the specific first increased the likelihood that respondents would report an interest in the general questions.

	Test A	Test B
Question Order	1. General 2. Specific	1. Specific 2. General
General Results	Control	Greater interest in specific items
Specific Results	No change	No change

As a result, my preference continues to be to ask an open-ended question first about how an organization can improve a product or service, then follow up with a closed-ended question presenting a range of items to be rated.

Because of the ability for early questions to contaminate later questions, sometimes one question order for every respondent is the wrong approach. When asking a respondent to rate two or more contrasting items (typically products, services or organizations), it is customary to rotate the order of the items, so that the consistent assessment of one item before another doesn’t introduce any bias into the results. In survey software, this is typically accomplished by setting up page rotations that randomly rotate pages or other blocks of questions. This is analogous to randomizing choices in a choice list.

A free subscription to this blog to the first person with an RSS reader who can tie the photo to this topic! (Clearly I need to hire the MR blogger Zebra Bites as a photo consultant.)

Marketing in a Web 2.0 World

Marketing In A Web 2.0 World:

View more presentations from Vovici.

CSAT, the Public Domain Customer-Satisfaction Question

Many organizations use the following question, often called CSAT, to measure customer satisfaction:

What is your overall satisfaction with our company?
1. Very dissatisfied
2. Somewhat dissatisfied
3. Neither satisfied nor dissatisfied
4. Somewhat satisfied
5. Very satisfied 

This question has the twin advantages of brevity and familiarity; it is recognized and easily answered by most respondents. Any adult who has taken a survey has most likely answered a form of this question before.

CSAT is traditionally analyzed by tracking over time the percentage of “Satisfied” respondents, i.e., the percent who answer 4 or 5. Many organizations are happy to see that 70-80% of their customers are satisfied and feel little sense of urgency to make improvements so that satisfaction exceeds this level.

When phrased as above, the satisfaction question ignores significant research into how to structure rating scales to provide the greatest reliability and validity (see the abstract of “The Optimal Length of Rating Scales to Maximize Reliability and Validity” by Jon Krosnick and Alex Tahk, which studied 706 tests). The CSAT question should instead be asked in one of the two following ways, with no numbers presented:

What is your overall satisfaction with our company?

• Not at all satisfied
• Slightly satisfied
• Moderately satisfied
• Very satisfied
• Completely satisfied 

Or:

What is your overall satisfaction with our company?

• Completely dissatisfied
• Mostly dissatisfied
• Somewhat dissatisfied
• Neither satisfied or dissatisfied
• Somewhat satisfied
• Mostly satisfied
• Completely satisfied 

Mere satisfaction alone is not enough: the key top-line number is the percent of respondents reporting themselves to be “Completely satisfied”. This is a far more important metric. In the seminal paper, “Why Satisfied Customers Defect” by Thomas O. Jones and W. Earl Sasser, Jr. (covered in this description of the "folk" Apostle Model), the authors report that for Xerox completely satisfied customers (rating of 5) were six times more likely to repurchase over the next 18 months than somewhat satisfied customers (ratings of 3-4).

Your first survey to report this statistic will certainly report a much lower percentage than you had hoped for, giving you a meaningful metric to track your performance against. Use the top-two percentage on the five-point scale for marketing purposes, and to compare yourself to other firms’ self-reported numbers, but use the “completely satisfied” percentage to measure the results of your initiatives and innovation.

If satisfaction is just one part of the customer experience and loyalty survey you are conducting, the CSAT question can be an effective measure.

By itself, though, a single question can be very volatile from measurement period to measurement period. As a result, most professionally commissioned customer-satisfaction reports provide a customer-satisfaction index, derived from two to four questions. The American Customer Satisfaction Index is the most famous of these and is useful if the principle concern of your research is customer satisfaction.

Survey Translations: The Translator may be a Traitor

In March, Scott Blacker joined us as our new senior director of product management. In his career, Scott has held multiple product management positions, most recently with Rosetta Stone, the leader in language-learning software. Appropriately, then, Scott’s first post is on the perils of translation and surveys.

In the race for global market share, many organizations with an international customer base require surveys to be deployed in multiple languages. Typically, the survey is written in the native language of the survey author, translated into the target language(s), and then deployed. Unfortunately, this process misses a critical, often-overlooked step: back-translating a survey into the native language of the survey author.

The reasons for skipping this step are easy enough to understand. Translation costs are expensive, and paying to both translate and back-translate a survey doubles these costs. Additionally, time demands on survey deployment are often intense, and back-translating can add valuable days to the survey deployment timeline. However, skipping this step can have serious consequences when ultimately analyzing survey response data, effectively killing the survey ROI.

The original survey author is the subject matter expert on the topic at hand. The nuance of how a question is posed – and the specific word choices involved –matter greatly in determining the nature and validity of the final data collected. Translators may have several linguistically correct options to choose from during the translation process, but may choose a nuance that misses the original intention of the survey author. Back-translating through a second translator (who has no affiliation with the original translator) greatly reduces the likelihood of this type of error. Back translating allows the original survey author to:

• Validate the quality of the initial translation
• Ensure that the nuance of the translation matches the original intent
• Open up a dialogue with multiple translators to build a consensus around the best possible translation.

This actually happened to an associate of mine just last year. In deploying a satisfaction survey into the Japanese market, he paid a premium for a top-of-the-line translator. While the survey was linguistically correct and made perfect sense in Japanese, the translator has chosen a word for “satisfaction” that was closely aligned with “happy” in the Japanese language. When the results came, it appeared as if the product was a success – nearly 80% of Japanese indicated that they were “very happy” or “somewhat happy” with the product.

However, when looking over some of the open-ended qualitative feedback, he realized that something was amiss. Japanese respondents had interpreted the word not as “happy”, but as “fun”. So yes...80% of Japanese respondents had indicated that the product was “fun” (which it was – it was a learning game), but fun in this case bore little correlation to satisfaction. Other elements, such as the ability to achieve a learning objective (after all, it was a learning game), turned out to be much more relevant to the customer’s overall satisfaction.

In this case, the problem was caught, but not before the results had been presented to the CEO. The survey had to be re-run, incurring additional costs and delaying decisions on whether a major marketing campaign should be run. It also didn’t reflect well on the market research department that was responsible for deploying the survey. More seriously though, had the problem not been caught, the company might have invested millions of marketing dollars into a product that had no chance of succeeding in that market.

As a survey author, you spend hours agonizing over diction when constructing questions in your native language…not investing the same time and resources into ensuring that the same nuance is appropriately reflected in your globally deployed surveys could cause you to fail just inches shy of the finish line.

Six-Sigma Survey Projects

One of the factors that distinguishes Six Sigma from TQM (Total Quality Management) and earlier quality movements is its reliance on measurable data.  Jiju Antony, in "Pros and cons of Six Sigma: an academic perspective", describes this difference like this:

Six Sigma emphasises the importance of decision making based on facts and data rather than assumptions and hunches. Six Sigma forces people to put measurements in place. Measurement must be considered as a part of the culture change.

Surveys are a key tool for transforming hypotheses and hunches about customer attitudes and outlooks into numbers and metrics. As a result, surveys are useful throughout Six Sigma work.

• Early in its own deployment of Six Sigma, Caterpillar conducted a Six Sigma Supplier Survey with its partners to understand how they had deployed Six Sigma and what lessons they had learned (see Lean Six Sigma: Combining Six Sigma Quality with Lean Production Speed by Michael George).
• When General Electric began its own use of Six Sigma, each GE division conducted detailed customer  surveys, asking customers to rate GE products and services on CTQ (Critical To Quality) issues and to rate best-in-class performance. This evolved into a quarterly customer-satisfaction process for many divisions, with low-scoring items in the quarterly updates becoming candidates for subsequent Six Sigma projects (see Managing Six Sigma: A Practical Guide to Understanding, Assessing, and Implementing the Strategy That Yields Bottom-Line Success by Forrest Breyfogle III, James Cupello and Becki Meadow).
• Voice-of-the-Customer research is often conducted as an input to QFD (Quality Function Deployment), with QFD transforming customer needs into engineering and quality assurance methods for developing new, high-quality products and services.
• One Six Sigma approach to web design involves an ongoing study of web-site effectiveness, which surveys visitors about their goals at the site and tracks the success rate of achieving those goals over time.  Regular incremental improvements to the web site are evaluated by their effect on improving goal completion rates.
• Another organization uses an employee survey to identify bottlenecks and excessive bureaucracy that reduce employee productivity, to highlight and prioritize areas for internal process improvement.
• The book Managing Six Sigma is noteworthy among Six Sigma books because it actually practices what it preaches and includes within itself a readership satisfaction survey! The authors assert the results of this survey will help them prepare the next edition of the book.

How have you used surveys in your Six Sigma projects?

Common Scales to Use when Writing Questions

One of the most frequent mistakes I see when reviewing questionnaires are poorly written scales. Novice survey authors often create their own scale rather than using the appropriate common scale. It’s hard to write a good scale; instead you are better off rewording your question slightly so that you can use one of the following.

Acceptability	Totally unacceptable, Unacceptable, Slightly unacceptable, Neutral, Slightly acceptable, Acceptable, Perfectly acceptable
Agreement	Strongly disagree, Disagree, Somewhat disagree, Neither agree or disagree, Somewhat agree, Agree, Strongly agree
Amount of Use	Never use, Almost never, Occasionally/Sometimes, Almost every time, Frequently use
Appropriateness	Absolutely inappropriate, Inappropriate, Slightly inappropriate, Neutral, Slightly appropriate, Appropriate, Absolutely appropriate
Awareness	Not at all aware, Slightly aware, Somewhat aware, Moderately aware, Extremely aware
Beliefs	Very untrue of what I believe, Untrue of what I believe, Somewhat untrue of what I believe, Neutral, Somewhat true of what I believe, True of what I believe, Very true of what I believe
Concern	Not at all concerned, Slightly concerned, Somewhat concerned, Moderately concerned, Extremely concerned
Familiarity	Not at all familiar, Slightly familiar, Somewhat familiar, Moderately familiar, Extremely familiar
Frequency	Never, Rarely, Sometimes, Often, Always
Influence	Not at all influential, Slightly influential, Somewhat influential, Very influential, Extremely influential
Likelihood	Not at all likely, Slightly likely, Somewhat likely, Moderately likely, Very likely
Priority	Not a priority, Low priority, Medium priority, High priority, Essential
Probability	Not probable, Somewhat improbable, Neutral, Somewhat probable, Very probable
Quality	Very poor, Poor, Fair, Good, Excellent
Reflect Me	Very untrue of me, Untrue of me, Somewhat untrue of me, Neutral, Somewhat true of me, True of me, Very true of me
Satisfaction (bipolar)	Completely dissatisfied, Mostly dissatisfied, Somewhat dissatisfied, Neither satisfied or dissatisfied, Somewhat satisfied, Mostly satisfied, Completely satisfied
Satisfaction (unipolar)	Not at all satisfied, Slightly satisfied, Moderately satisfied, Very satisfied, Extremely satisfied

This list follows Krosnick’s advice to use 5-point unipolar scales and 7-point bipolar scales.

Let me know any of your favorite scales that I omitted.

Customer-Service Survey Template using NPS

As I mentioned last Friday, I attended a webinar, “The Ultimate Question: How to Measure & Build Customer Loyalty in the Support Center”, presented by Fred Reichheld on the use of the Net Promoter Score® within customer support centers. Fred reviewed the thinking behind NPS and presented this questionnaire for use as a customer-service survey:

Considering only your most recent support experience, how likely would you be to recommend our customer support to a friend or colleague? (0 is least likely, 10 is most likely)
( ) 0  ( ) 1  ( ) 2  ( ) 3  ( ) 4  ( ) 5  ( ) 6  ( ) 7  ( ) 8  ( ) 9  ( ) 10

Please give your reasons for the rating above.
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________

Considering your complete experience with our company, how likely would you be to recommend our products to a friend or colleague? (0 is least likely, 10 is most likely)
( ) 0  ( ) 1  ( ) 2  ( ) 3  ( ) 4  ( ) 5  ( ) 6  ( ) 7  ( ) 8  ( ) 9  ( ) 10

Please give your reasons for the rating above.
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________

Fred described this as a bottom-up measure of NPS. He pointed out two sources of bias:  the most recent transaction biases the respondent’s thinking about the overall relationship, and the fact that the survey is sponsored by the firm gives scores an upward bias as respondents are less negative than they would be if the survey was anonymous.

In contrast to this, Fred described a top-down measure of NPS as a double-blind survey of customers of the sponsor and its competitors. This top-down average score will be different because of the biases inherent in the bottom-up method.  Just as accounting reports can legitimately differ, these two measures, to Fred’s mind, have valid, legitimate differences.

Given Fred’s advocacy of short surveys, I was disappointed that he didn’t point out that customer-support surveys really shine when additional data is integrated behind the scenes into the survey. This preserves the survey experience for respondents while enabling the survey analyst to study performance across many additional factors that affect customer loyalty, such as demographics and product ownership. This provides a much richer source of information for the organization to use to adapt and grow.

[Net Promoter Score is a registered trademark of Fred Reichheld, Bain & Company and Satmetrix.]

Hurdles in Race to Turn Recipients into Respondents

You’ve chosen the list of people you want to invite to your survey.  You still have a lot of hurdles to jump to get each recipient to actually take your survey:

• Make sure their email address is not on the unsubscribe list
• Keep your survey invitation from being flagged as spam
• Craft a survey subject line and From field that help get the invitation opened  
• Cover the six points important to respondents in an invitation
• Write an invitation that is compelling enough that recipient clicks on the link to take the survey

Order Effects: Early Choices in Long Choice Lists Are Selected More Often than Later Choices

We last looked at respondent behavior with the post Long Surveys Turn Respondents into Liars. Well, similarly, long choice lists turn respondents into satisficers, selecting a satisfactory answer rather than the optimal answer.

Jon Krosnick and Duane Alwin in the report “An evaluation of a cognitive theory of response order effects in survey measurement” provide an excellent summary of the past research that documented this behavior:

Studies of impression formation¹, the impact of persuasive communications², sequential processing of performance information³, and the serial position effect⁴ all suggest that when items are presented visually on "show cards," primacy effects are to be expected. This occurs for two main reasons. 

1. Items presented early may establish a cognitive framework or standard of comparison that guides interpretation of later items. Because of their role in establishing the framework, early items may be accorded special significance in subsequent judgments.
2. Items presented early in a list are likely to be subjected to deeper cognitive processing; by the time a respondent considers the final alternative, his or her mind is likely to be cluttered with thoughts about previous alternatives that inhibit extensive consideration of it. Research on problem-solving suggests that the deeper processing accorded to early items is likely to be dominated by generation of cognitions that justify selection of these early items⁵. Later items are less likely to stimulate generation of such justifications (because they are less carefully considered) and may therefore be selected less frequently.

So, now that we know that our respondents do this, how do we address this issue when constructing choice lists?

• If a long choice list can be structured into an outline, present the choices as a hierarchical question instead.
• Consolidate the long choice list into a shorter list that makes fewer distinctions.
• For long lists that can’t be modified, use randomization. While it would be too costly in a paper survey to have multiple versions of the questionnaire, each presenting choice lists in different orders, for a web survey the ability to randomize choice lists is a built-in capability of most survey software and has no added cost to use. Such randomization isn’t needed for long lists that respondents don’t have to read; for instance, alphabetized lists of states or countries, where the respondent knows the answer without reading the choice list and is simply finding the choice in the list. Nor is randomization appropriate for rating scales. Instead, randomize the choices for any long list that lacks an inherent order.
• Finally, Krosnick and Alwin advise attempting to “to increase respondent motivation in order to increase concentration and decrease satisficing. Motivation may be increased by adding special instructions informing respondents that the question they are about to answer is relatively difficult and requires extra concentration.” 

¹ Asch, 1946;Nisbett & Ross, 1980, p. 172-175; Anderson & Hubert, 1963; Sherif, 1935; 1936; Lingle & Ostrom, 1981; Anderson L Barrios, 1961; Dreben, Fiske, & Hastie,1979.

² Miller & Campbell, 1959; Ronis et al., 1977; Crano, 1977; Hovland et al., 1957; Insko, 1964.

³ Jones et al., 1968.

⁴ Bruce & Papay, 1970; Crowder, 1969; Rundus, 1971.

⁵ Koriat, Lichtenstein, & Fischhoff,1980; Hoch, 1984; Klayman & Ha, 1984; Tschirgi, 1980; Wason & Johnson-Laird,1972.