Changing the headline about KLRVU Research poll

Posted by Jessica Smith

Former federal Green Party leader Jim Harris contacted the Mercury about the headline of the story in today's paper about the KLRVU Research poll.
Following his call the electronic version of the headline has been changed.
Here is an excerpt from an email he sent explaining his views:

In reference to Rob O’Flanagan’s very interesting article “Poll says Liberals will win in Guelph,” the headline is misleading.
If the headline read “Had election been held Aug 4, Liberals would have won Guelph” it would have been accurate.
The polling was conducted over two days – July 27 and August 13 – therefore the “average” time the poll was taken was August 4.
I don’t in any way dispute the accuracy of the poll. The important point to remember is that a poll is a snapshot in time.
The election wasn’t held on August 4 it’s on September 8 – and a lot can happen in five weeks. The second piece of Green Party literature is just going out to every household this week. And there are more surprises in the Green Party campaign to come!
At the time of the first date of this poll (July 27) the Campaign to Elect Mike Nagy had not even sent out its first piece of literature to every household. By contrast the old-line traditional parties had blanketed the riding with literature, ads, events, etc before the election was even called. So this poll is a snapshot in time before the significant Green Party campaigning efforts had begun!

The email continues and Harris mentioned he also wrote a blog post about the poll.

Comments

[JS]

He's bang on. Not only that, despite the fact that the poll sample was large, the results of it are highly suspect. Don't know about any of you, but I've never heard of a single poll being conducted on two different dates, more than 2 weeks apart. Who does that?

[saskboy]

JS, probably no one. KlrVu are in a league of their own. Just check the progressivebloggers.ca for information on KlrVu and their background ties.

[Kevin Ames]

Let me get this straight the Mercury changes its headlines at the request of the Green Party. Talk about revealing your bias.

[David Akin]

Jessica says she doesn't dispute the accuracy of the poll and notes that polls are snapshots. I'm waiting for more evidence before I feel so confident. [Reference: http://davidakin.blogware.com/blog/_archives/2008/8/21/3849950.html] For one thing, households were surveyed two weeks apart. That's not a snapshot. That's two snapshots. Did respondents feel the same way on the first one? Surely any difference in the two snapshots would tell us something about how the campaigns are doing. Secondly, this pollster uses a relatively new methodology. So far as we're able to tell, Klr Vu's methodology in a previous poll has produced significantly different results than the methodologies used by mainstream polling firm.

[Kermit]

I have said it before, and I'll say it again -
ITS NOT EASY BEING GREEN.
First of all the Liberals take the Green idea and call it Green Shift.
Then the NDP claim that it is a flawed version of their position which they have had since Tommy Douglas was in knee pants, or maybe it was washable cloth diapers..
And the CPC says that it is a carbon tax and will hurt Alberta where Stevey has his stongest support, and why would he bite the gooseneck pump that feeds him. That of course is after he makes the Green Shi_t statement which as my Limbo partner chants - How Low can you go?
And the Mariuana Party says that if it is too green you cannot possibly smoke it.
So in the meantime, who is really got a program to save the environment? I hate to say it but it is the Dion Liberals. Sorry Steven - Shift can really happen, but not from someone who has oil wells up to his aspidestra.
I repeat - IT IS NOT EASY BEING GREEN!

[Scott Piatkowski]

I e-mailed the reporter yesterday about this "poll" (no response yet).

In the interests of full disclosure, I'm an NDP activist in Kitchener and I'm helping on Tom King's campaign whenever I can (including last night). But, these remarks should not be taken as representing the campaign.

I can't believe that any serious news organization would give coverage to this "poll", or anything else eminating from this alleged "polling firm". Consider that:

1) They've done only TWO polls ever... and the other one was a push poll for Campaign Life on the Morgentaler Order-of-Canada recognition.
2) They use robo-call technology and simply blanketed the riding with automated calls until they found enough people who didn't hang up on them (rather than using touch tone responses as directed). There is no record of their existance before July 24 of this year.
3) The robotic voice apparently didn't mention candidate names (kind of important in a by-election campaign, especially one in which Tom King is a candidate for the NDP).
4) They brag that every respondent heard the same question identically. In fact, standard polling methodology is to rotate the names of the candidates and/or parties. That prevents the order from skewing the results.
5) They claim that the number of undecided voters is "insignificant" and therefore not reported. Not only would this be suspect in an election, but I can guarantee you that it's not the case in this by-election.

I strongly urge the Mercury to publish a retraction / clarification or, better yet, to commission its own poll of the riding, using a more reputable firm.

[Krago]

Actually Scott, the robo-poll did use candidate names; I received one of the calls. However, the recorded message mispronounced both "Valeriote" (va-lair-ee-oat-ee) and "Kovach" (rhymed with "patch").

By the way, for a really cool political map of Guelph, go to http://www.gpmurray-research.com/electoral-atlas/index.htm and click on the "GOOGLE MAP" button.

[Brian Van Tilborg]

The Mercury have been led down the garden path by this "polling firm". I think there is a story within the story, if the Merc chooses to follow it. Who commissioned the poll? Who is the pollster related too? Could their be some political bias going on here? Whats the background on this firm and its reputation?

How can a firm suggest that the undecided are insignificant during a by election???? Based upon what?

Even a reputable newspaper can be had by an activist political agenda.

[Fred Blailock]

Here's the Akin Article. it's interesting.

http://davidakin.blogware.com/blog/_archives/2008/8/21/3849950.html

The KLR VU poll in Guelph: Dirty tricks or business development?
by DavidAkin on Thu 21 Aug 2008 11:09 PM EDT

A poll came out yesterday purporting to show the voting intentions of people in Guelph, Ont., a medium-sized city where a by-election is underway.

The pollster said that, based on his survey, it appeared that the incumbent Liberals held a commanding lead. Liberal Brenda Chamberlain retired in the spring and now, Frank Valeriote wants to take her place. With the vote set for Sept. 8, this poll would suggest he has nothing to worry about.
The poll also showed that the Green Party is doing surprisingly well and is in third place in the riding, just ahead of the NDP. (The Greens, you won't be surprised to learn, are thrilled.) The Conservatives, who are running city councillor Gloria Kovach, are a distant second, the poll says.

So if you're a Liberal here, what's not to like, right?

Apparently plenty.

The poll was done by a firm whose principal happens to be the brother of a Conservative MP. The firm, KlrVu-Research of Winnipeg, is headed by Allan Bruinooge, the older brother of Rod Bruinooge, a first-term Conservative MP who scored one of the biggest upsets of the 2006 election, taking out Liberal cabinet minister Reg Alcock.

Some Liberals as well as some non-aligned political consultants I spoke to yesterday said they believe federal Conservative party, with too much money on its hands and staffed by a group of creative Evil Geniuses, has its hands all over the poll. The political "dirty trick" here is available only to an underdog and only available to an underdog in a byelection. Here's the thinking:
1. The Liberals have held the riding for 15 years. They are expected to win the riding.
2. The biggest problem for every party in a byelection is getting out the vote. Voter turnout for byelections is always low and that means a few hundred extra votes here or there can make a big difference.
3. A poll in mid-campaign comes out showing that the Liberals, as expected, should win it in a romp. Result: The Liberal vote goes to sleep.
4. A highly-motivated group of voters, like Conservatives in many parts of the country including Guelph, take advantage of the sleepy Liberal vote, go nuts on polling day, and, in doing so, overtake the Liberals.

OK, so that's the conspiracy theory explained to me by those who asked to remain anonymous in exchange for advancing the theory.

What do those who will go on the record say?
Well, first of all, we asked Conservative party director of communications Ryan Sparrow if his party, flush with cash it can't spend quickly enough, paid for this poll. "Absolutely not," said Sparrow. And, after explaining the conspiracy theory to him on the phone, there was a short burst of laughter. So, on the record, the federal Conservative party says they have nothing to do with this thing and, in any event, they certainly aren't in the habit of releasing polls they pay for.

What about Allan Bruinooge, the brother of the Conservative MP who did the poll?
Bruinooge, reached by phone yesterday in Winnipeg, also said that the Conservatives did not pay for the poll. Who did, I asked? He says he did. He did it to raise the profile of his young firm. He's looking to compete with likes of Ipsos-Reid, The Strategic Counsel, and Decima and thought a poll about Guelph would help with his firm's profile. I checked our databases and, so far as I can tell, only The Guelph Mercury, the daily in that city, picked up the poll and reported it. Mercury reporters indicate on the paper's blog that some campaigns complained about the headline, at least, with the story, and the paper responded by changing it.
What about the conspiracy theory? Bruinooge was not as definitely dismissive as Sparrow but did not agree with the assumptions behind the argument.
I asked Allan why pick Guelph? Why not one of the other two byelections underway? He hinted that he may very well poll those ridings, too, before the Sept. 8 vote.

Allan, incidentally, had let someone know much earlier this summer that we was going to poll in Guelph. In fact, in a comment posted to a blog on July 24, the day before the writ was dropped, commenter "Eric" makes note of this fact.

One of the things the campaigns complained about was the methodology used by Klr-Vu.

The big guys -- Ipsos-Reid, Decima and so on -- use real people and telephones to call you up and ask a few 'screening' questions to make sure you're a qualified voter. The phone numbers are drawn randomly from a geographic area but pollsters do some additional weeding to balance for gender, income levels, and other qualifiers to make they get a random sample. Typically, the big-name pollsters will make thousands of phone calls to be able to report the opinions of about 1,000 Canadians, which they claim will be a representative sample. The big firms will qualify their results by saying that the results are accurate to within three percentage points, 19 times out of 20.

Klr-Vu, on the other hand, does not use human beings to do its polling: It uses software. Here is Klr-Vu's own words:

This KLRVU poll was conducted by touchtone technology which polled households across Guelph. Using this technology with the voice of a professional announcer all respondents heard the questions asked identically, which queried a response on the candidate's name and their associated party. In theory, with the stated sample size, one can say with 95% certainty that the results would not vary by more than the stated margin of sampling error, in one direction or the other. There are other possible sources of error in all surveys that may be more serious than theoretical calculations of sampling error. These include refusals to be interviewed, question wording and question order, weighting by demographic control data and the manner in which respondents are filtered (such as, determining who is a likely participant). It is difficult to quantify the errors that may result from these factors.
So, essentially, a digital voice -- software -- posed the questions and respondents registered their preferences by pressing a button a telephone. Critics say this technique does not appropriately screen for non-voters and is prone to errors.

Bruinooge said his sample is large enough - nearly 3,400 in the Guelph poll -- that "outliers" or weird statistical anomalies are readily apparent and easy to adjust for non-voters. Klr-Vu's dialers, it should be noted, polled on two different dates, two weeks apart. The mainstream pollsters typically poll over three or four nights in a row in order to present a "snapshot" of opinion.

Does it work? I'm not a pollster or a statistician so all I can rely on is past behaviour. We all marvelled, for example, at Nik Nanos and how he and his army of "human beings" polling just before the general election seemed to get it just about right. (And his business took off as a result.) I asked Allan if his firms had a track record he could point us to. He did not. His firm, he said, is a young one just trying to establish its name.

There was a poll his firm did which made the news. It showed that 56 per cent of Canadians opposed Henry Morgentaler's ascension to the Order of Canada, a poll which, conveniently, made his brother, Rod, looked like he was on the leading edge of Canadian opinion. For that poll, KlrVu used the same methodology that it used in Guelph: robot diallers contacted a lot of Canadians and a computer read out the question and took the punched-in responses.

Ipsos-Reid, the polling firm used by Canwest News Service whose methodology is similar to Nanos Research, also polled on the Morgentaler issue and got a completely different result. Ipsos-Reid asked 1,023 Canadians between July 4-7 about the suitability of Morgentaler to receive the Order of Canada and found that 65 per cent were OK with it. The Toronto Star asked its pollster, Angus Reid Strategies, to poll on the Morgentaler question. Angus Reid found that 60 per cent of Canadians were OK with the Morgentaler award.

Again: Klr-Vu found 56 per cent opposed.

Here's something else that's important for this issue:

The Canada Elections Act (you'll want to flip to page 117 for the section on Election Opinion Surveys) has some very specific instructions for 'transmission' of a poll during an election period. It says pollsters, their sponsors, and news organizations that publish them must do the following:
1. Name the sponsor of the poll. Neither KlrVu nor the Guelph Mercury did that. There was no mention in the KlrVu press release who paid for or sponsored the poll. The Mercury, as well, did not report who sponsored or paid for the poll. Bruinooge when I asked him, said that he did the poll on his own accord and is, therefore, the sponsor. He would not say how much it cost him.
2. You must name the organization conducting the survey. KlrVu did that.
3. You must name the date of the survey. KlrVu did that, too.
4. You must describe the population from which the sample was drawn, the number of people contacted for the survey, and margin of error, if applicable. It is an arguable point that KlrVu satisfied these conditions. It says it polled "households across Guelph". You will note that many polls, particularly those in the U.S. right now, talk about "polling voters." KlrVu reported voting intentions but does not say if it polled actual voters. KlrVu does not provide a margin of error (though the Act seems to suggest this is only an option in any event) but instead has this somewhat ambiguous language: "In theory, with the stated sample size, one can say with 95% certainty that the results would not vary by more than the stated margin of sampling error, in one direction or the other." But KlrVu never says in its release what the "stated margin of sampling error" is. THe mainstream firms use that phrase "accurate to within three percentage points" which I take to mean that if a poll shows the Conservatives at 34% and the Liberals at 32% they are, statistically speaking, tied because they are each within the pollster's margin of error of three percentage points.

Section 323, subsection 3, of the Canada Elections Act, goes on to say that the sponsor of the poll must also be prepared to provide, on demand, the exact wording of the question asked; the number of people asked to participate in the survey, the number that were declared to be ineligible or declined; "any weighting factors or normalization procedures used in deriving the results of the survey"; and some other details. Now normally none of that would be reported by a media outlet but the guys we use, Ipsos-Reid, or any of other mainstream firms, routinely post all of that information and more on their Web sites as soon as the poll is released. No such information has yet been published, so far as I can tell, at KlrVu's site.

So what are we left with at end of the day?
We have a poll which shows the Liberals in good shape in Guelph and yet, Liberals are unhappy that this poll is out there because they believe it to be a Conservative dirty trick intended to put the Liberal vote to sleep. The pollster, it appears, is indeed a Conservative but neither he nor the party he supports say the Conservatives paid for the poll. The pollster said he did the poll for free in order to raise the profile of his firm. If that was the goal, there's not much to show for it so far. Only the Guelph Mercury - near and dear to my heart as it is -- reported the poll. It appears that some of the routine reporting checkpoints spelled out in the Canada Elections Act were missed. Now the big question: Will it make a difference on Sept. 8?

[Fred Blailock]

Here's the Akin article - it's interesting

The KLR VU poll in Guelph: Dirty tricks or business development?
by DavidAkin on Thu 21 Aug 2008 11:09 PM EDT

A poll came out yesterday purporting to show the voting intentions of people in Guelph, Ont., a medium-sized city where a by-election is underway.

The pollster said that, based on his survey, it appeared that the incumbent Liberals held a commanding lead. Liberal Brenda Chamberlain retired in the spring and now, Frank Valeriote wants to take her place. With the vote set for Sept. 8, this poll would suggest he has nothing to worry about.
The poll also showed that the Green Party is doing surprisingly well and is in third place in the riding, just ahead of the NDP. (The Greens, you won't be surprised to learn, are thrilled.) The Conservatives, who are running city councillor Gloria Kovach, are a distant second, the poll says.

So if you're a Liberal here, what's not to like, right?

Apparently plenty.

The poll was done by a firm whose principal happens to be the brother of a Conservative MP. The firm, KlrVu-Research of Winnipeg, is headed by Allan Bruinooge, the older brother of Rod Bruinooge, a first-term Conservative MP who scored one of the biggest upsets of the 2006 election, taking out Liberal cabinet minister Reg Alcock.

Some Liberals as well as some non-aligned political consultants I spoke to yesterday said they believe federal Conservative party, with too much money on its hands and staffed by a group of creative Evil Geniuses, has its hands all over the poll. The political "dirty trick" here is available only to an underdog and only available to an underdog in a byelection. Here's the thinking:
1. The Liberals have held the riding for 15 years. They are expected to win the riding.
2. The biggest problem for every party in a byelection is getting out the vote. Voter turnout for byelections is always low and that means a few hundred extra votes here or there can make a big difference.
3. A poll in mid-campaign comes out showing that the Liberals, as expected, should win it in a romp. Result: The Liberal vote goes to sleep.
4. A highly-motivated group of voters, like Conservatives in many parts of the country including Guelph, take advantage of the sleepy Liberal vote, go nuts on polling day, and, in doing so, overtake the Liberals.

OK, so that's the conspiracy theory explained to me by those who asked to remain anonymous in exchange for advancing the theory.

What do those who will go on the record say?
Well, first of all, we asked Conservative party director of communications Ryan Sparrow if his party, flush with cash it can't spend quickly enough, paid for this poll. "Absolutely not," said Sparrow. And, after explaining the conspiracy theory to him on the phone, there was a short burst of laughter. So, on the record, the federal Conservative party says they have nothing to do with this thing and, in any event, they certainly aren't in the habit of releasing polls they pay for.

What about Allan Bruinooge, the brother of the Conservative MP who did the poll?
Bruinooge, reached by phone yesterday in Winnipeg, also said that the Conservatives did not pay for the poll. Who did, I asked? He says he did. He did it to raise the profile of his young firm. He's looking to compete with likes of Ipsos-Reid, The Strategic Counsel, and Decima and thought a poll about Guelph would help with his firm's profile. I checked our databases and, so far as I can tell, only The Guelph Mercury, the daily in that city, picked up the poll and reported it. Mercury reporters indicate on the paper's blog that some campaigns complained about the headline, at least, with the story, and the paper responded by changing it.
What about the conspiracy theory? Bruinooge was not as definitely dismissive as Sparrow but did not agree with the assumptions behind the argument.
I asked Allan why pick Guelph? Why not one of the other two byelections underway? He hinted that he may very well poll those ridings, too, before the Sept. 8 vote.

Allan, incidentally, had let someone know much earlier this summer that we was going to poll in Guelph. In fact, in a comment posted to a blog on July 24, the day before the writ was dropped, commenter "Eric" makes note of this fact.

One of the things the campaigns complained about was the methodology used by Klr-Vu.

The big guys -- Ipsos-Reid, Decima and so on -- use real people and telephones to call you up and ask a few 'screening' questions to make sure you're a qualified voter. The phone numbers are drawn randomly from a geographic area but pollsters do some additional weeding to balance for gender, income levels, and other qualifiers to make they get a random sample. Typically, the big-name pollsters will make thousands of phone calls to be able to report the opinions of about 1,000 Canadians, which they claim will be a representative sample. The big firms will qualify their results by saying that the results are accurate to within three percentage points, 19 times out of 20.

Klr-Vu, on the other hand, does not use human beings to do its polling: It uses software. Here is Klr-Vu's own words:

This KLRVU poll was conducted by touchtone technology which polled households across Guelph. Using this technology with the voice of a professional announcer all respondents heard the questions asked identically, which queried a response on the candidate's name and their associated party. In theory, with the stated sample size, one can say with 95% certainty that the results would not vary by more than the stated margin of sampling error, in one direction or the other. There are other possible sources of error in all surveys that may be more serious than theoretical calculations of sampling error. These include refusals to be interviewed, question wording and question order, weighting by demographic control data and the manner in which respondents are filtered (such as, determining who is a likely participant). It is difficult to quantify the errors that may result from these factors.
So, essentially, a digital voice -- software -- posed the questions and respondents registered their preferences by pressing a button a telephone. Critics say this technique does not appropriately screen for non-voters and is prone to errors.

Bruinooge said his sample is large enough - nearly 3,400 in the Guelph poll -- that "outliers" or weird statistical anomalies are readily apparent and easy to adjust for non-voters. Klr-Vu's dialers, it should be noted, polled on two different dates, two weeks apart. The mainstream pollsters typically poll over three or four nights in a row in order to present a "snapshot" of opinion.

Does it work? I'm not a pollster or a statistician so all I can rely on is past behaviour. We all marvelled, for example, at Nik Nanos and how he and his army of "human beings" polling just before the general election seemed to get it just about right. (And his business took off as a result.) I asked Allan if his firms had a track record he could point us to. He did not. His firm, he said, is a young one just trying to establish its name.

There was a poll his firm did which made the news. It showed that 56 per cent of Canadians opposed Henry Morgentaler's ascension to the Order of Canada, a poll which, conveniently, made his brother, Rod, looked like he was on the leading edge of Canadian opinion. For that poll, KlrVu used the same methodology that it used in Guelph: robot diallers contacted a lot of Canadians and a computer read out the question and took the punched-in responses.

Ipsos-Reid, the polling firm used by Canwest News Service whose methodology is similar to Nanos Research, also polled on the Morgentaler issue and got a completely different result. Ipsos-Reid asked 1,023 Canadians between July 4-7 about the suitability of Morgentaler to receive the Order of Canada and found that 65 per cent were OK with it. The Toronto Star asked its pollster, Angus Reid Strategies, to poll on the Morgentaler question. Angus Reid found that 60 per cent of Canadians were OK with the Morgentaler award.

Again: Klr-Vu found 56 per cent opposed.

Here's something else that's important for this issue:

The Canada Elections Act (you'll want to flip to page 117 for the section on Election Opinion Surveys) has some very specific instructions for 'transmission' of a poll during an election period. It says pollsters, their sponsors, and news organizations that publish them must do the following:
1. Name the sponsor of the poll. Neither KlrVu nor the Guelph Mercury did that. There was no mention in the KlrVu press release who paid for or sponsored the poll. The Mercury, as well, did not report who sponsored or paid for the poll. Bruinooge when I asked him, said that he did the poll on his own accord and is, therefore, the sponsor. He would not say how much it cost him.
2. You must name the organization conducting the survey. KlrVu did that.
3. You must name the date of the survey. KlrVu did that, too.
4. You must describe the population from which the sample was drawn, the number of people contacted for the survey, and margin of error, if applicable. It is an arguable point that KlrVu satisfied these conditions. It says it polled "households across Guelph". You will note that many polls, particularly those in the U.S. right now, talk about "polling voters." KlrVu reported voting intentions but does not say if it polled actual voters. KlrVu does not provide a margin of error (though the Act seems to suggest this is only an option in any event) but instead has this somewhat ambiguous language: "In theory, with the stated sample size, one can say with 95% certainty that the results would not vary by more than the stated margin of sampling error, in one direction or the other." But KlrVu never says in its release what the "stated margin of sampling error" is. THe mainstream firms use that phrase "accurate to within three percentage points" which I take to mean that if a poll shows the Conservatives at 34% and the Liberals at 32% they are, statistically speaking, tied because they are each within the pollster's margin of error of three percentage points.

Section 323, subsection 3, of the Canada Elections Act, goes on to say that the sponsor of the poll must also be prepared to provide, on demand, the exact wording of the question asked; the number of people asked to participate in the survey, the number that were declared to be ineligible or declined; "any weighting factors or normalization procedures used in deriving the results of the survey"; and some other details. Now normally none of that would be reported by a media outlet but the guys we use, Ipsos-Reid, or any of other mainstream firms, routinely post all of that information and more on their Web sites as soon as the poll is released. No such information has yet been published, so far as I can tell, at KlrVu's site.

So what are we left with at end of the day?
We have a poll which shows the Liberals in good shape in Guelph and yet, Liberals are unhappy that this poll is out there because they believe it to be a Conservative dirty trick intended to put the Liberal vote to sleep. The pollster, it appears, is indeed a Conservative but neither he nor the party he supports say the Conservatives paid for the poll. The pollster said he did the poll for free in order to raise the profile of his firm. If that was the goal, there's not much to show for it so far. Only the Guelph Mercury - near and dear to my heart as it is -- reported the poll. It appears that some of the routine reporting checkpoints spelled out in the Canada Elections Act were missed. Now the big question: Will it make a difference on Sept. 8?