The Sham (and Shame) of CSI
(Customer Satisfaction Indexes)
(Customer Satisfaction Indexes)
by
Bill Kaye
Bill Kaye
It's not forensics; it's barely science.
We've all heard about CSI... not the TV series... not Crime Scene Investigation...it's Customer Satisfaction Index(es)... but there's something almost criminal about them!
A company's CSI is based on statistics. But it was Mark Twain who popularized a quote about the nature of statistics (after attributing its origin to Benjamin Disraeli), that "there are three kinds of lies: lies, damned lies and statistics."
Or, "garbage in, garbage out."
We've all taken part in a survey, we've all seen their results. They're at the bottom of a cash register receipt, requesting we call a number or go online to complete a survey. We take part in a telephone survey, or answered an "official" mailing from some business we've just patronized. These surveys help determine a company's overall CSI, and, it is hoped, improve their overall service.
But do they? How reliable are the results?
A heavy advertiser for auto insurance claims a 97% customer satisfaction rate... other companies claim that they "have the highest customer satisfaction rate in (their) industry." Perhaps they do. But how valid are the claims, how valid are the figures?
How long ago was the CSI at 97%? Is it still at 97%? Wouldn't this change, from month to month, and year to year? And is it really 97%? Do only a miniscule 3% disapprove? (Note: this particular insurer can boast of 10 million customers. That translates to 300,000 dissatisfied customers. I wonder how if they worry about those numbers in the corporate office.)
Let's examine how CSIs are determined. Recall the "surveys" you’ve actually answered.
One method: "on a scale of 1 to 5, with 5 being the highest and 1 being the lowest, how would you describe your experience?"
Point values are determined, and the cumulative results averaged to determine an overall CSI. The highest possible score is generally 100. Any result in the 90s is great; in the 80s it’s still good. A score in the 70s is only average... and anything below 60, well, they just aren't going to promote that, are they?
Another method is to translate the scoring into terms everyone understands. This type of survey asks you to describe your experience as "extremely satisfied," "somewhat satisfied," "satisfied," "somewhat dissatisfied" and "extremely dissatisfied." Just about every customer can easily describe their level of satisfaction.
Yet for some companies, it's still important to determine a more specific numerical figure, so often the extremely satisfied answer will be graded as a 5, somewhat satisfied as a 4, and so on.
But here's the dirty little secret.
Because such things are graded, no matter what the methodology, the overall CSI may have some usefulness, either promotionally, or internally.
One quibble with the "verbal" method of scoring: what constitutes a "satisfied" customer? Is it determined by adding all satisfactory comments, and disregarding any dissatisfied responses? In other words, if all satisfied, somewhat satisfied and extremely satisfied customers are lumped together, that would produce a larger overall satisfaction rate. So any company that satisfies 97% of its customers must be a pretty good company to do business with, right? After all, statistics don't lie, do they? Re-read that quote from Mark Twain.
This is more troubling: some companies use their CSI "results" as weapons against their own employees, grading them on the responses they get from their customers. The logic is simple: an employee consistently "earning" a lower CSI, is not an asset. Depending on the company, policy may dictate remedial retraining to help the employee raise the score, or result in a termination. There are other dictates: make the CSI score part of the employee's performance review, or dock the employee if the figure falls below a certain threshold. Often, bonuses are awarded to employees for consistently high scores. Lower scores can reduce or eliminate those, perhaps even require the employee to forego a portion of a base salary, as a penalty.
Logically, that policy should attain two results for the company: employees will consistently strive to better their CSI, and from that, the company's actual customer satisfaction improves.
But not so fast. I once sold cars for a major automaker. As in many industries, the salesmen, as well as the service writers and technicians, were "graded" by surveys. Recent customers were called by an independent company which asked a series of questions, all using the "extremely satisfied" to "extremely dissatisfied" scale. But because all dealerships were graded against one another, our management encouraged us to tell the customers that they would be getting the call. We would tell them the scope of the questions, or even show them the actual survey.
This reveal (effectively negating the validity of any responses) was softened by saying: "if for any reason you can't say you’re extremely satisfied about anything, please let me know as soon as possible so that I can correct the problem." We would personally follow up with a call within one day, making sure that everything was perfect, again reminding them about the upcoming call. The inference, of course, was that we were providing continuing service, and that we were genuinely concerned about the customer’s "extreme" satisfaction.
And we got our share of 100s... in fact, one of my compatriots scored perfect 100s for seven straight months. Others consistently scored in the high 90s. It was difficult. A single misstep... a nick in the paint that wasn’t caught prior to delivery, failure to point out how to open the gas flap, or even a late phone call to the customer could result in point loss. That was a disaster, as it cost us money in lost bonuses.
But overall dealer CSI could then be matched against our competing dealerships, and in "up times" we could boast of having a better CSI than any other competing dealer in the region.
Impressive.
And a lie.
Because it was based on faulty data; data that was deliberately influenced by the customer's contact person, its usefulness as a true gauge of customer satisfaction was negligible, and laughable as a promotional tool.
How many companies' CSIs are true measures of their actual customer satisfaction? Can we truly believe any of them? Unless the methodology is clearly explained and its results truly uninfluenced, I would have doubts about any such claims.
So why go through this unscientific process at all?
It is useful for any business or organization to know how their customers truly feel. All companies will hear from dissatisfied customers, but not everyone will complain. Sometimes it's too much trouble, or customers believe their voices won't be heard.
But every company can benefit if they can detect problems, identify them as possible future sources of dissatisfaction, and work to correct the problem or improve performance. It's just good business to reject what doesn’t work, otherwise, it can easily result in unprofitability.
Degrees of Satisfaction... Do they Mean Anything?
How often have you truly been "extremely satisfied" about anything?
If you purchase something and it's everything you thought it would be prior to the purchase, are you simply "satisfied," or do you feel better than that? Are you "somewhat" satisfied or "extremely" satisfied?
If it does what you thought it would do, you're satisfied. Period. Why does it matter how satisfied you are? It matters only if the experience is through the roof, above and beyond all expectations, or an orgasmic experience: perfection. I can't think of a single purchasing experience I've ever had that left me extremely satisfied, because if it is all that was advertised, if it does what I expected it to, I'm happy. Satisfied. And that's about it.
The Perfect CSI survey
It's just two questions:
- "Would you return to this company for future purchases?"
- "Would you recommend this company to others?"
Then, adding a comment box will give customers an opportunity to note any specific issues, or add compliments. Positive comments can then be the basis for effective testimonials.
That's all. Because if the customer will come back for repeat business, you've done everything right to satisfy them, and you're on your way to improving your methods, procedures, and eventually, the success of your business.
And that is a true measure of CSI.
