Voice of Vovici

Survey Software Success: Companion Presentation

This presentation on success with survey software was the inspiration for my ebook:

View more presentations from Vovici.

Community Platform for Market Research

According to an early peek at the 2009 Tribalization of Business survey, the top two reasons for creating an online community are “market insights/research” and “idea generation”. So market research (which includes ideation) has become the most popular reason for organizations to sponsor proprietary communities, yet rarely are organizations turning to online community software built specifically for this purpose.

The Vovici Community Builder is one of the few community platforms designed for gathering market research. A common database coordinates information for panelists and community members across major components:

• Panel Management – By empanelling everyone your organization can email, you can reach beyond just those who participate in your online community to build samples that are more representative of all your customers and prospects.
• Online Surveys – The ability to build and field sophisticated questionnaires to members and nonmembers allow you to conduct detailed, extensive survey research. Community members see the surveys they have been invited to from a pane within the community, while nonmembers receive only email invitations.
• Business Intelligence – Integrating operational data and CRM information enables you to perform sophisticated analysis.

Vovici Community Builder also has the features you would expect in a community platform:

• Customize communities to maintain your corporate identity
  • Add, delete, and modify pages, content, and modules using our visual interface—no coding required
  • IFrames embed dynamic content from other websites or documents inside your community
• Use a variety of social tools to connect with your audience and understand perceptions, promote co-creation, and keep a pulse on sentiment.
  • Forums – collect ideas, opinions and comments with moderated or un-moderated forums
  • Suggestion Boxes – gather sentiments and ideas directly from community members
  • Multimedia – add images, audio and videos directly into your community
  • Blogs – communicate with members and even enable members to post comments
  • Events – provide a social calendar where key events can be posted for your members to view

For those community platforms that lack built-in feedback management, the robust API of Vovici v4 can be used to integrate with standing communities to provide the robust market research capabilities that are driving the usage of proprietary communities.

Net Promoter Score’s Angels & Demons

There’s something devilish in the air. Wednesday night my wife and I watched a Dr. Who episode involving a demon, Thursday afternoon I attended a webinar where Fred Reichheld said market researchers thought he was the devil, and this morning my son complained that his camping trip meant he was going to miss tonight's premiere of Angels & Demons.

Since you probably don’t care about my family’s viewing habits, let’s turn to Fred’s presentation. Fred showed a slide with his face superimposed onto a cartoon devil, claiming that he has become the enemy of traditional market researchers. Among the criticisms he reported:

Net Promoter Score® oversimplifies customer loyalty

NPS is a score that doesn’t provide solutions

Market researchers have proprietary indices with greater validity

NPS is hazardous to your business

NPS does not correlate to growth

Fred is dumb or lying

Well, presenting an oversimplification of legitimate criticism of NPS is one way to attempt to inoculate your listeners from paying attention to that criticism. I thought it was amusing, but I will play along. In fact, I will play devil’s advocate and give Fred halos instead of horns for these contributions to loyalty research:

Discovering that customer retention is a key driver of profitability (q.v., Reichheld, Frederick F. (1993), “Loyalty-Based Management,” Harvard Business Review, 71 (2), 64–73)

Promoting the importance of very short surveys

Championing the need for a common loyalty measure across the enterprise

Pointing out that a 0-10 scale is an intuitive scale for telephone research, since—unlike a 1-10 scale—the 0 is unambiguously the worst score

Advocating the firing of staff who cheat on their customer-feedback numbers and discouraging organizations from tying that feedback to compensation

Making NPS free

That last is important. As he said: “NPS is an 'open source' system: you are all welcome to use it! You don't have to pay me to use it.” (I could have done without his next statement: “Therefore its bad business for market research, and why they call me a liar.”) But if you’ve ever gone to implement the five questions of the Secure Customer Index® or the one question of the Customer Effort Score™ and realized you couldn’t because they are proprietary, you’ll appreciate that there are no such restrictions on the Net Promoter Score.

Oh, and even if you liked the book (Angels & Demons, not The Ultimate Question), my advice is to skip the movie. This review of Angels & Demons said it all for me: "Hanks returns to the dullest role of his career, under the direction of Howard, who takes the material as seriously as a kidney stone on the way out." As for Fred, I’ll leave it to you to decide whether he is an angel or demon.

[All trademarks are properties of their respective holders. Vovici makes no claim to any of these registered or common-law marks.]

Departmental Use of Feedback Management

Most organizations get to the point where they are subjecting customers to too many surveys through the bottom-up adoption of survey tools. This frequently happens in an organization where no market-research function exists or where MR concentrates primarily on strategic or outsourced projects.

With MR uninvolved, surveying starts as an individual activity, where each individual may have selected a different survey software application to use. Over time, these individuals begin working together, and survey research within the department turns into a collaborative effort. The requirements become increasingly complex as users’ knowledge of surveying increases.

Each department or division often evolves through four stages in its use of survey systems:

1. Survey activity using simple survey tools is widely distributed.
2. De facto survey-domain experts emerge through experience with these tools and through peer recognition. Only rarely do these experts have any formal training in market research.
3. Survey development, deployment and analysis changes from a solitary effort to a collaborative approach within the department and occasionally across departments.
4. The need quickly arises for more sophisticated survey systems, greater collaborative capability and wider information sharing. 

Missing from this evolution, unfortunately, is a market-research department mentoring the users of these survey tools.

While enterprise feedback management systems are intended to provide cross-departmental standardization, they are adapted as frequently for standardization within a department. Departments often have all the same problems as the enterprise does when it comes to survey research, only smaller in scale.

Representative Web Surveys Require Good Email Lists of Customers

Some organizations, such as ecommerce sites like Amazon and EBay, are fortunate in that they have the email addresses of every single one of their customers. When they invite a random selection of customers to participate in a web survey, they can be assured that the results are truly representative of their customer base.

Most organizations are not so fortunate. A traditional B2B vendor might have the email addresses for perhaps half of its customers. A consumer brand might have email addresses for fewer than 10% of its customers. For these organizations, web surveys with random samples are only representative of the customers for whom they have email addresses. Such organizations must be very careful when presenting their results to not generalize them to all customers. Significant differences probably exist between the group of customers for which they have email addresses and the group for which they don’t, especially since email addresses are typically collected as part of "loyalty" (frequent-buyer) programs. As a result, surveys of such email lists will overstate satisfaction, repurchase likelihood and willingness to recommend.

Organizations with unrepresentative email lists need to take the following steps for web-survey success:

• Describe the survey not as representative of all customers but of the target population for which the organization has email addresses.
• Work with the marketing department to increase the percent of customers and prospects who have provided email addresses.
• Conduct a telephone study or paper survey of customers for which your organization doesn’t have email addresses to determine their demographics, firmographics and attitudes. This can be used to contrast this group with the group accessible by email.
• Conduct more-frequent surveys of members of the customer loyalty program. Such programs typically use email for most communication. 
• Don't put faith in polls placed on your corporate web site; such polls are unrepresentative.

What steps have you taken to make sure that your web surveys are representative of your customer base?

Survey Monkey Feature Grid vs. Vovici

Support Model	SurveyMonkey	Vovici, Individual Edition	Vovici, Enterprise Edition
Email support	Yes	Yes	Yes
Online knowledgebase support	Yes	Yes	Yes
Phone support	No	Yes	Yes
SAS-70 Secure datacenter hosting	No	Yes	Yes

Writing Objective Survey Questions

The saying Garbage In, Gospel Out reflects our willingness to believe computer output, even if it was generated from bad input. Survey researchers are no more immune to this tendency than computer scientists, as poorly worded questions can lead to suspect results and erroneous conclusions.

We recognize bad questions when we see them in other people’s surveys, but we don’t recognize them as easily when we write them ourselves. Each of the following examples are adapted from public surveys:

1. “How likely is it that you will attend the 2009 Expo at our new, low entry prices?”
2. “The 2009 Forum will be held July 24-26 in central Iowa. Please check all the reasons why you may choose not to attend.”
3. “What type of involvement would you like to have with the Celebration of Nations program?”
4. “What do you think of those titles? Yes, we are clearly not creative, but that's why we are asking for your help.”

So how do you write objective questions that don't bias the results one way or the other?

• Your questions should use nonjudgmental wording and neutral terms. Respondents should not be able to determine where you stand on any topic. (See this post for an example of how to research attitudes towards abortion.)
• Don’t presuppose anything (one type of leading question). For the third example above, for instance, ask instead “What type of involvement, if any, would you like to have with the Celebration of Nations program?”
• Don’t ask other types of leading questions. (See the links for details.)
• Avoid so-called “double barreled questions” by splitting them into two. Instead of “How would you rate our price and service?” ask “How would you rate our price?... How would you rate our service?” 
• Remove ambiguity in use of words and grammatical structures.  The question “Will a bimonthly schedule make you more or less likely to renew your subscription?” is useless, since bimonthly can mean either “twice a month” or “every other month”.
• Avoid industry jargon and acronyms; too often in my own surveys I’ve assumed people know what I mean by CRM or TQM when I should have defined and described them.
• Specify how you use general terms. If you need to map results back to industry figures, make sure you are using industry definitions. Instead of “Have you purchased a new big-screen television in the past year?” make sure you define the cut-off screen size that maps to your data (e.g., “Since January 1, 2008, have you purchased a new big screen (40-inch or larger) television?”).
• Open-ended questions should specify a unit of measure. Instead of “How far do you live from the nearest Acme store?” ask “How many minutes does it typically take you to travel to the Acme store that you typically shop at?”

In general, try to write from the respondent’s perspective rather than your perspective. Don’t make subtle distinctions that would be obvious to a coworker but not a customer. You spend 40, 50, 60 or more hours a week thinking about your company and its products and services; your customers don’t. Have others outside the organization proofread your questions for clarity. For strategic surveys, pre-test your survey with a segment of your audience.

Bad questions are the Garbage In, with their analysis being the Garbage Out. A question like “How likely is it that you will attend the 2009 Expo at our new, low entry prices?” will end up overstating actual likelihood to attend, while a question like the following will understate the likelihood to attend: “The 2009 Forum will be held July 24-26 in central Iowa. Please check all the reasons why you may choose not to attend.” 

If you don’t get the question right, you won’t get the analysis right.

Customer Effort Score™: A Loyalty Predictor for Customer Service Interactions

Though it was little noted at the time, in December the Corporate Executive Board introduced a new loyalty metric, the Customer Effort Score™:

Exceeding customer expectations has long been the measure of success in customer service interactions–89% of customer service executives believe that “delighting the customer” will lead to increased loyalty. New research by the Corporate Executive Board's Customer Contact Council, however, reveals the alarming truth: exceeding customer expectations results in virtually no loyalty gains. In fact, service and support centers have little stake in building customer loyalty at all.

“The probability that a service interaction will drive disloyalty is approximately four times greater than the chance it will create any positive loyalty impression. In other words, as a function, customer service typically plays on the ‘negative side’ of the loyalty field,” said Matthew Dixon, Ph.D., Managing Director of the Customer Contact Council. “Most service executives are using traditional customer satisfaction (CSAT) or the more recently popularized Net Promoter® Score (NPS) to gauge loyalty in service interactions, but we found these metrics fail to capture the most powerful driver of disloyalty–the amount of personal effort a customer has to put into the service experience.” In light of this new understanding, the Customer Contact Council has developed an original metric that is far more predictive of loyalty than either CSAT or NPS. This new metric, the Customer Effort Score™ (CES™), is based on a single question that determines the degree of required customer effort during a service request.

Unfortunately, the actual wording of the single question is only available to Corporate Executive Board subscribers. The Customer Contact Council has published a presentation, "Shifting the Loyalty Curve: Mitigating Disloyalty by Reducing Customer Effort", which provides an introduction to the research. In a blog post, the author of the study provides additional detail about customer effort by service category and industry. The blog Every Experience Counts reports on the metric in a little more detail.

The selection of the CES metric was derived from a survey using a convenience sample of almost 18,000 customers of CEB clients. CES suffers from similar problems to NPS: it’s proprietary and the data used for deriving it is not publicly available, making it difficult for third parties to verify the claims about the methodology. That said, I would encourage any contact center managers who are already CEB customers to check out the full report. It requires little effort (ahem) to integrate the Customer Effort Score into existing transactional surveys, which certainly makes it worth piloting.

Most surprising to me was the finding that 89% of customer-service executives look to customer satisfaction as the primary driver of customer loyalty. After all, it was over 13 years ago, Jones and Sasser published their landmark paper “Why Satisfied Customers Defect”, introducing the Apostle Model and demonstrating that customer satisfaction and loyalty were orthogonal. Clearly the word is not yet out that satisfaction is but one component that drives loyalty.

Ranking Questions vs. Rating Questions

Sometimes I hear clients use the words ranking and rating interchangeably, even though there is a distinction. The difference is simple: a rating question asks you to compare different items using a common scale (e.g., “Please rate each of the following items on a scale of 1-10, where 1 is ‘not at all important’ and 10 is ‘very important’") while a ranking question asks you to compare different items directly to one another (e.g., “Please rank each of the following items in order of importance, from the #1 most important item through the #10 least important item”).  Both types of questions have their strengths and weaknesses.

Ranking questions:

• Guarantee that each item ranked has a unique value
• Take on average three times longer to answer than rating questions (Munson and McIntyre, 1979)
• Mentally tax respondents, requiring them to compare multiple items against one another
• Increase the difficulty of answering disproportionately as choices are added
• Limit the range of statistical analysis available.

Rating questions:

• Lead to less differentiation among items, with the possibility that a respondent rates every item identically
• Often have a narrow distribution of ratings, which typically fall into an upper band
• Accept great personal variations in rating styles (e.g., respondents who never assign the highest rating)
• Produce possibly spurious positive correlations due to individuals’ personal variations.

(See “The measurement of values in surveys: A comparison of ratings and rankings” by Duane Alwin and Jon Krosnick for a more technical review.)

The mental effort required to answer a rating question is linear: the same effort is involved per item. The mental effort for a rank-order question is almost exponential –  N*(N-1)/2 – since each item has to be compared to every other item. Because the effort grows rapidly as more items are added, it is commonly advised to only use ranking questions when there are seven or fewer items to compare. For longer lists, use a rating question instead or ask respondents to select the three most important items from the list.

For other cases, think about whether the items should be similar in comparison or should be very different. For instance, when asking people to rate the importance of items about why they did business with your organization or why they purchased a product, many attributes are of similar importance, making a rating scale appropriate.  Alternatively, when asking people what features you should work on next, where you need to build a priority list for your development team, a ranking question is more appropriate.

When do you prefer to use ranking questions instead of rating questions, or vice versa?

Top 10 Reasons for Building an Online Community

A presentation on the top 10 reasons to build an online community:

View more presentations from Vovici.

Also see this related post: The Top Ten Reasons for Building an Online Community in 2009.