Why surveys should always be piloted

This morning I completed an almost incomprehensible marketing survey. Here’s an example of one of the questions.

Marketing survey

Like all of the other questions in the survey, you have to answer it to proceed to the next question. There’s a fixed range of answers that can be selected, with nowhere for me to indicate that I didn’t understand the question. Most of the questions were like this, so my best guess is that YouGov’s client will end up with statistical noise and a sprinkling of confirmation bias.

My suspicion is that the survey wasn’t piloted before release with its target audience. If it had been, simple ambiguities (does 2030 mean half past eight tonight or is it something due to happen in 14 years?) would have been picked up, questions would have been rephrased to make them comprehensible to the lay-person and the ability to answer ‘don’t know/don’t understand’ would have been provided.

But even if such changes had been made, it’s doubtful that anything insightful will result from the survey. The client would have been far better to employ a qualitative research method to explore such hypothetical questions. A good first question would be to ask for a definition of a luxury brand, rather than making the assumption that the client, YouGov and the survey’s audience all share the same perspective. As it stands they’re likely to get some nice charts with average scores to a couple of decimal places, but little insight into what consumers really think.

1,000 word essays are harder to write than 2,000 word essays

Six years ago when I first started to study psychology I would never have believed that a 1,000 word essay was more difficult to write than a 2,000 word essay. After all, a 1,000 word essay is half as long, so they have to be twice as easy to write, yes?

This weekend I’ve managed to finish and submit my first assignment on my MSc. It was a 1,000 word essay and I had more than enough material for a 2,000 word essay. Trying to understand specifically what was required and therefore what I should leave out was rather difficult.

No. Not difficult. Impossible.

In the end, I finished with 996 words which I hope are relevant enough to ensure that I pass the assignment. It would be rather embarrassing, not to say deflating if they weren’t. In my experience, 1,000 word essays are probably twice, if not four times as difficult to write as 2,000 word essays. This certainly wasn’t the best essay I’ve ever written. I just hope that it wasn’t the worst.

But it’s finished and submitted a massive 42.5 hours ahead of the deadline, so I need to stop worrying about it. After all, I don’t have another essay to write until next year. But I do have another assignment to complete before 9th December. I’ve just had a quick look at it and listened to its accompanying podcast. The best thing that I can say about it at the moment is that at least it’s not an essay.

Four questions, with multiple parts, covering psychometric, experimental and qualitative design and analysis, plus a literature review. There’s a lot of reading and work required between now and deadline, that’s for sure.

But, the thing is, it looks really interesting. Definitely the kind of thing I’d hoped for when I signed up for the course. And I now have a nice, new copy of SPSS installed on my computer ready for me to have a play with. Happy days …