On leading/ loaded questions and response bias (or: David Cameron wants to know my views on immigration)

Standard

I swear, I stated this blog to write about the practice of research, about how we can sociologically understand the world and to promote the latest interesting studies. NOT to pick on David Cameron.

Yet, just a week after writing about Voodoo Polls, while I was peacefully checking my Facebook…

facebook feedBlimey, so David Cameron wants me to click on his survey and tell him how I feel about immigration.

Now, kids, what was I saying last time?

In human language: sampling is the use of a subset of the population to represent the whole population. In probability sampling (random sampling), we have ways of calculating the probability of getting any particular sample, and therefore we can rigorously infer from the sample to the general population.In non-probability sampling, we do not; and therefore we need to use them with care.

Granted, there are reasons why you need to use non-probability samples sometimes: some populations are easier to sample randomly than others […] the fieldwork is much cheaper and less time-consuming […] you’re more concerned with understanding a complex phenomenon than with very accurate representativeness; but at the very least you can use quotas of gender, age, ethnicity etc. to ensure you have a diverse sample; and be very careful how you generalise it. […]

While you do , therefore, have to take any generalisations from polls using non-random samples with a grain of salt, not all of them are voodoo polls; rather, voodoo polls represent a particularly egregious example: where instead of respondents being selected by a researcher (by whatever method), they need to opt-in (for example by phoning a number, clicking a voting option on a website or mailing a coupon from a newspaper); and everyone is free to opt-in; but the opt-in “button” is conveniently placed somewhere where it’s much more likely to be seen by certain types of people than by others.

Granted, in this case the poll has been advertised to me: a Romanian immigrant trade unionist left-winger hippie; but nonetheless, from my knowledge of Facebook ads targeting (I’ve worked in online marketing for a while, so I’ve got a pretty good idea of the mechanics of it), it is not a useful tool for getting anything resembling even quota non-representative sampling; but it could be a very useful tool for directing a Voodoo poll at exactly the demographics you want (for instance, people who have liked anti-immigration pages or Conservative Party pages).

I have a few guesses for how the ad may have gotten to me; maybe because it was targeted at “anyone in the UK” (unlikely, it would be very costly advertising for a poll the results of which you could not generalise; if that’s what you’re after you may as well hire a polling company to do one with a random sample), because I am generally interested in issues around politics and migration (again, I’m not sure how much sense this makes from their point of view), or because I happen to have some (Facebook) friends who liked David Cameron (on Facebook); as the ad itself is helpfully reminding me. (Given all the rest of how the poll is being framed and presented, this does seem to me the most likely possibility; which would also very firmly put it in the voodoo poll category).

So, before we click, let’s look at the ad itself for a moment:

analysedSo, you’re minding your own business on Facebook, and I suddenly ask you what you think about immigration; but first I show you a picture of a policeman (with an “immigration enforcement” van behind him, just in case you thought maybe the picture meant the upstanding citizen that is the policeman in cause is an immigrant himself). Now, as we click to answer the surveys, what images of immigrants are being evoked in our head? What aspects of immigration are we reminded of?

Why was this particular image chosen rather than, say, this-one?

altered(Image alteration my own. source photo here)

Now, if we look at the original again…

facebook feed

… notice how while immigration  as a phenomenon (and a phenomenon that needs to be first and foremost forcibly controlled, no less) is present in the image, immigrants as human beings are very much absent.

I’m taking a fairly long detour before even looking at the survey questions for a reason: and it’s because response bias is a thing.

Response bias is a general term for a wide range of cognitive biases that influence the responses of participants away from an accurate or truthful response; and it can have a large impact on the validity of the questionnaire or survey to which the participant is responding.

Basically, what happens is that people don’t respond passively to stimuli: when prompted to answer a question like “What do you think about immigration?”, or “How happy do you feel today”?, they actively integrate multiple sources of information to generate a response: things like the phrasing of questions, the appearance and demeanor of a researcher, the unconscious desire to be a “good respondent” and provide socially desirable answers- or, in our case, being lead-on with a suggestive image and the reminder that “Hey, these people who are your friends like David Cameron and his policies”.

Well, now let’s click the survey (located here, if you’re curious) and look at the questions.

But before we even get to look at the questions:

frontpage(via)

So, the Conservatives (whom, remember, your Facebook friends like) are “controlling immigration so our economy and local services deliver for hardworking taxpayers”- notice how (1) taxpayers are not immigrants, never mind that migrants- and particularly recent migrants- contribute much more in taxes than they take as transfers, and (2) YOU are “a taxpayer”, rather than a citizen, a human being, a community member, a family member, a friend or anything else.

The text reads as if you already assume, as if it goes without saying, that immigration hurts rather than helps the local economy and services; it wants you to assume this, to start from this premise before you even started answering. Then: “Labour let immigration spiral out of control – and they have opposed every measure we’ve taken to reduce immigration”. So, if you didn’t know or didn’t think immigration is spiralling out of control, we’re telling you it is, with the strongest language possible, before we ask you: OK, now what do you think of immigration? With a good bit of “Us versus them” rhetoric thrown in for good measure:

usvsthem

So: here’s an image to help you think of immigrants as scary and menacing without giving them a human face at all; now we could solve your problems by controlling immigration, but Labour wouldn’t let us. Think about it from the perspective of a “hardworking taxpayer” .Now proceed.

Q1

This, boys and girls and variations thereof, is a leading question;  and at the same time a loaded question.
Leading questions suggest to the other person the answer you want them to give. In common law systems that rely on testimony by witnesses, a leading question is one that suggests the particular answer or contains the information the examiner is looking to have confirmed. (For example: “Were you at the crime scene yesterday at 1 am?” is a leading question; “Where were you last night” is not).

Why this could be a problem in a survey: in our case, for instance, this is not equal to asking first “Do you believe that immigration is more beneficial or more detrimental to the local economy?” and then “Out of the following policies, which do you agree with?”. It’s outright telling that immigration is something that spirals out of control and needs to be limited, and then only asking “Look, we have this solution for the problem: don’t you agree?”. This is much more effective as a hard-nosed sales techniques than as a tool for getting people’s actual opinion.

A loaded question is one which contains a controversial or unjustified assumption (like, for example “Have you stopped beating your wife?”, which presumes, regardless of whether you answer yes or no, that you have a wife and you have beaten her at least once in the past).

If we unpack this question, it basically contains three different questions:
(1) Do you believe that the local economy and services should deliver for hard-working taxpayers?

The answer to this is easy and obvious; the question is leading enough and it would be really hard to argue that they shouldn’t.

(2)Do you believe that immigrations controls, as promoted by the Conservative Party, would have a positive effect, a negative effect or no effect at all on the local economy and services delivering for hard-working taxpayers?

This question, if it were phrased like this, would not have the same kind of simple and straightforward answer; this is a complex question that debates have been held on, books have been written etc.

Except everything you’ve read on this page, including the question itself, tells you that yes, immigration controls help the local economy and services in delivering for hard-working taxpayers. This is the assumption that you need to accept for the question to make any sense; the wife that you are assumed to have beaten.

(3) Do you support immigration controls as promoted by the Conservative Party?

If you have answered affirmatively to (1) (because it’d be hard not to) and (2) (because you’ve been told to), it follows that all things being equal (which is an assumption as well: what if you think that immigration controls help the local economy and services, but violate human rights?) then you ought to support (3).

And this is how the Conservative party gets people whom they specifically target through Facebook ads to agree with their policy. When they will publish the results, remember to take them with the traditional grain of salt.

tequilashot(via)

Now, let’s do an exercise, shall we? Here are the rest of the questions in the Conservative Party questionnaire. Have fun with them. Look for emotionally loaded words, ambiguities, vagueness, assumptions, leading questions and loaded questions; and please leave your thoughts in the comments. Good luck!

rest of Qs

Advertisements

One thought on “On leading/ loaded questions and response bias (or: David Cameron wants to know my views on immigration)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s