Author's Opinion

The views in this column are those of the author and do not necessarily reflect the views of iTWire.

Have your say and comment below.

Tuesday, 09 October 2018 16:18

Marketing people and useful statistics – n'er the twain shall meet

By
Marketing people and useful statistics – n'er the twain shall meet Image: Max Pixel (used with permission)

Word doesn't make you an author, Photoshop doesn't make you an artist, and SurveyMonkey most certainly doesn't make you a researcher.

I recently read an article noting that the expected feedback value for Uber riders and drivers was 5 (out of 5) unless there were major reasons to offer something lower. It seemed that a 4 was for unreasonable behaviour and a 3 meant disaster. Who knows how bad you'd have to be in order to rate a 1 or 2!!

There is something inherently wrong with this kind of assessment – the expectation that maximum marks will be given unless there was an exceptional reason not to (this smells a lot like modern primary schools' "everyone gets a prize" mentality that we hear of so often). Surely, there is almost always room for improvement, no matter how exceptional the service delivery.

Until recently, this seemed to be relatively "academic", I'd not experienced it personally (not being an Uber user – am I the only one left?). However, I came to have my car serviced at a local dealership that represented the vehicle's very well-known international brand.

At the time I paid the account and collected my keys, I was warned by the assistant that I may receive an email requesting a response to the brand's standard customer survey. As part of the conversation, the assistant warned me that any rating lower than a '9' (out of 10) would reflect poorly on the dealership. I elected to stay schtum and not reveal a surprised/aghast expression on my face. My surprise extended in two directions – that the survey might actually expect this, and secondly that the person felt the necessity to warn me of this likelihood.

Sure enough, a few days later I did receive the email and proceeded to open the survey. Equally sure enough, the opening salvo looked like this:

 Redacted Rating

I felt compelled to write to the team managing the survey to express my total dissatisfaction with their understanding of suitable data-gathering. The following is a slightly edited version of that email.

Whoever added the annotation that a 7 or 8 was "neutral" and a 9 or 10 was "extremely likely" has a remarkably limited understanding of maths and statistics.

On any reasonable 0-10 scale, "neutral" is a 4, 5 or 6, while 7 and 8 would be rated as "positive" and 9 or 10, as you suggest "extremely likely". Labelling a 6 as "not at all likely" is both condescending and highly inaccurate, not to mention mathematically misleading.

Any customer interaction will contain room for improvement, therefore insisting (as your annotations do) that the only acceptable rating is a 9 or 10 is the worst kind of virtue-signalling I've seen in a long time. Even worse that the on-side attendant felt the need to set his expectation in advance.

Worse, you are NOT collecting any useful data — all you will get is a collection of 9s and 10s everyone in your organisation is smiling and happy, safe in the knowledge that the entire customer body is satisfied. The marketing team achieves their KPIs and everyone's pay-packet is suitably enriched.

Unfortunately, if there is a level of dissatisfaction, you won't see it. Having been told that the dealership will be "penalised" for low ratings, I would reasonably assume that people will probably not respond (as I am doing) or they will grit their teeth and respond according to your expectations.

I can only assume there is no-one in your team with even rudimentary statistical training, otherwise this would surely never have been foisted upon an unsuspecting public.

By the way, I stepped through the remainder of the pages (with random responses and without submitting) and it became remarkably clear that EVERY QUESTION was directly linked to a specific measure of the performance of the workshop. Questions like, "did they greet me by name," or "was the work to be performed fully explained, along with a break-down of costs."

As they say, "what gets measured gets done," or to quote Goodhart's Law, "When a measure becomes a target, it ceases to be a good measure".

I wrote to the company in the hope that some statistical "moral fibre" might be instilled, but I'm not holding my breath. I write this on iTWire in the hope that the message might be seen more broadly; again, my breath is not bated.

As I noted in the opening, just because you know how to use a tool, doesn't mean you're an expert in the process for which that tool is used. Knowing how to use Microsoft Word does NOT make you a writer and knowing how to create a survey in SurveyMonkey (or any of the other tools available) doesn't mean you're going to get meaningful, actionable data.

Perhaps the company in question doesn't want to learn the truth – if so they've done an excellent job of ensuring that specific outcome, as absolutely nothing useful will be learned from the survey in its current form.

Addendum: Prior to publishing this comment, I caught up with a family member who worked for a multi-brand car dealership in a major Australian country town. His own observations were very similar, noting that they were very close to losing one of their brands due to "low" feedback ratings. He also mentioned that his own mother had told him (I paraphrase both statements), "I thought the service was really good – I gave you 7s and 8s for everything." He replied, "but Mum, but we need 9s and 10s to stay in the car company's good books."

CHIEF DATA & ANALYTICS OFFICER BRISBANE 2020

26-27 February 2020 | Hilton Brisbane

Connecting the region’s leading data analytics professionals to drive and inspire your future strategy

Leading the data analytics division has never been easy, but now the challenge is on to remain ahead of the competition and reap the massive rewards as a strategic executive.

Do you want to leverage data governance as an enabler?Are you working at driving AI/ML implementation?

Want to stay abreast of data privacy and AI ethics requirements? Are you working hard to push predictive analytics to the limits?

With so much to keep on top of in such a rapidly changing technology space, collaboration is key to success. You don't need to struggle alone, network and share your struggles as well as your tips for success at CDAO Brisbane.

Discover how your peers have tackled the very same issues you face daily. Network with over 140 of your peers and hear from the leading professionals in your industry. Leverage this community of data and analytics enthusiasts to advance your strategy to the next level.

Download the Agenda to find out more

DOWNLOAD NOW!

David Heath

David Heath has had a long and varied career in the IT industry having worked as a Pre-sales Network Engineer (remember Novell NetWare?), General Manager of IT&T for the TV Shopping Network, as a Technical manager in the Biometrics industry, and as a Technical Trainer and Instructional Designer in the industrial control sector. In all aspects, security has been a driving focus. Throughout his career, David has sought to inform and educate people and has done that through his writings and in more formal educational environments.

VENDOR NEWS & EVENTS

REVIEWS

Recent Comments