Statistics don’t work like that

I’m shortly going to be married, so for the first time in my life I’ve taken an interest in buying jewellery. After my fiancĂ©e and I struggled to find the right ring for her, we ended up getting a bespoke one made by a specialist jeweller. Despite the swanky Mayfair address it was an unpretentious experience, run on a relatively small scale and with a personal touch.

All went excellently until after the piece had been delivered, at which point I received a customer satisfaction survey. Nothing particularly wrong with this, but for some reason they decided to phrase the questions thus: “On a scale from 1 to 5, how happy were you with…”

Stop and think about this for a moment. The company has invested their entire business model in making me feel special. Individualising things is what sets them apart from the competition. And yet when it comes to assessing satisfaction, they inadvertently send the message: “Your satisfaction is a number to us. We’re going to add it up and divide by n, where n is the number of customers we have. If the result is a large enough number, we will consider our job done.” I’m not naive enough to think I’m their only customer, but retail is a world of reassuring fictions, and this struck the wrong note.

I don’ t have any facts, but I can’t help wondering whether their number of customers is large enough to support any statistically significant analysis, beyond a simple arithmetic mean. I very much doubt that it is. But this kind of analysis is the only reason that numerical data is better than free text. If you want to dig deeper into what went wrong for your dissatisfied customers, then numeric data is the worst place to start from. None of this even touches on the deeper problem that people tend to misuse the data, assuming that three 3s count for the same as two 4s and a 1.

So why do so many people reach for the discrete numeric ranges when they want to gather information? My guess is that it feels more “professional”, more sciency, more like “real work”. Dealing with numbers rather than messy human ideas adds a facade of objectivity, but it’s really only a facade: the translation is taking place just the same, and different people will make the translation differently. Objectivity is nice to have, all other things being equal, but it is no substitute for insight. In a small scale retail environment, ones judgment and experience ought to be good enough to draw conclusions without resorting to cargo-cult data analysis.

Leave a Reply

Your email address will not be published. Required fields are marked *