When R2K give up all the data and we can sift through it then I'll know who to trust. I'm still with Kos on this one.

If they don't give up the data it means they have something to hide.

^{"Sí che dal fatto il dir non sia diverso."}

-Dante

... but, within the context of the statistical arguments advanced by KOS and Silver and the analysis above, which clearly debunks it, I don't see the case that there was any deceptive practice. It's just not there.

Researchers are supposed to share data for the furtherance of knowledge. One of big biggest cases ever on sharing primary data was the 2004 exit polls for the presidential election. Mitofsky and company refused congressional requests for primary exit polling data and then, to deflect pressure, shared primary data with a chosen few. That didn't work. It created a host of questions. (As it turned out fraud was demonstrated using just the data presented by the polling company.)

So we're at an impasse. We're told that R2000 won't share data (although I have not seen their direct response). Regardless of any corporate policy etc., the methods and data need to be released.

As for the substantive arguments about the numbers, KOS and Silver's case is very weak.

And when the report comes, the outcome will be devastating for one or the other in this controversy. If R2000 produces a report that is satisfactory according to neutral polling professionals, it's going to be an unhappy day on the other side. If they produce something sketchy, then it's their head on the block. Big risk to make those accusations imho.

Has he published on this since July 1? He got a cease and desist order, according to the president of R2000. July 1 is the last post I've seen.

Good response. I'll fill in an answer with "edit" tomorrow. In the meantime, check out the new link at the top on "Volatility." It's fascinating.

On your question, See this

..."forcing" into "all odds" or "all evens" issue. If you're always forcing the responses from one category to the other, there should logically be a lot of 1% shifts in the distribution. I'd really like to know what the reason for this "forcing" is - seems unlikely to me to be related to fraud (well at least in a real simple, direct, commonly recognized way). Why, were one trying to make up data out of whole cloth, would one do something so obvious? Seems unlikely. Right now, I'm wondering whether we're not seeing the patterns of some rather "hard" smoothing [probably not actually smoothing, but some sort of post-processing].

As to the data being "too clean" - idly I wonder what would happen were one to take one's data and resample the hell out of it. I'm not a pollster (in the sense of political polling) so I don't have a good understanding of whether anyone's played around with this in the polling world, but resampling based analytical strategies are all the rage right now - wonder if some clever bunny decided that it would be an interesting basis for data presentation in a polling context (i.e., what we see isn't simple tabulation of responses, but the product of thousands of re-sampling runs based on the raw responses).

“The absence of any US-Iran bilateral channel...may have the perverse effect of reinforcing Iranian interest in progressing in the nuclear realm so that the US will be forced to take it seriously and engage it directly." ~ Richard Haass

In fact Silver posted a column that discusses the possibility that the human intervention demonstrated in the results is inadvertent rather than deliberate: Is Research 2000 Merely Mangling Its Data - Rather Than Fabricating It?

... Grebner et al. argue that this phenomenon could only be a reflection of human intervention - no naturally occurring statistical process could produce it. In my view, that conclusion is correct beyond the shadow of a doubt. This does not necessarily imply, however, that the human being is making up the numbers entirely. He could also be manipulating real data in a scientifically unsound way. Perhaps it feels abnormal to Ali when his raw data has not shown some change in Obama's ratings: he would therefore tweak the numbers upward or downward by a point, as he feels he has license to. In essence, he could be using real data and [e.g. inadvertently - ES] making it look fake...

I should add that I wouldn't have any idea if I'm presenting any of Silver's more compelling arguments - I just chose one that made a good visual example from my complete layman's perspective.

The example cited above - the uncanny degree of correlation between the expected outcomes and R2K's claimed captures, vs the far noisier data gathered by PPP - seems to merit an accompanying graphic too.

... Take a look at the attached chart, for example: these are the age breakdowns in the Democratic vote share for the last
20 contests surveyed by R2K and PPP, respectively. The age
breakdowns in Research 2000's numbers are almost always close
to "perfect" -- in 20 out of 20 cases, for instance, the
Democrat gets a lower vote share from among 30-44 year olds
than among 18-29 year olds. PPP's data, on the other hand,
is *much* messier -- which is what I think we should expect
when comparing small subsamples, particularly subsamples of
lots of different races that are subject to different
demographic patterns...

...

Why, were one trying to make up data out of whole cloth, would one do something so obvious? Seems unlikely. Right now, I'm wondering whether we're not seeing the patterns of some rather "hard" smoothing [probably not actually smoothing, but some sort of post-processing].

I don't disagree with your speculation here, but it's suggested that the potential answer to your first question might lie in human psychology. It's alleged that people falsifying strings of numbers don't simulate randomness very well at all - for example they tend to pick "zero" significantly less often than they pick the numbers one through nine. The answer if that proved to be the case would then be "because they couldn't help themselves". Of course this wouldn't rule out an alternative explanation - some form of data processing that inadvertently produced the same results.

[by the way - that's not to tell you things you probably know better than I, but for the benefit of others who might follow the thread]

They're suing KOS in Maryland and KOS is suing them in Northern California. R2000 said they have no further comment until trial so it's speculation all around, particularly on the odd-even issue. However, here's the link to some further analysis and part of what's there.

The R2000 polls seem to match up with Gallup in a favorable way.

From "Research 2000: A Closer Look at Volatility" July 5, 2010

R2K was a 3-day tracking poll of 1100 total respondents. It was essentially no different than standard polls that sample 360-370 voters a day but only report the total sample-size with a typical 3.0% margin of error. Most tracking polls report a continuous three-day moving average, but avoid providing the daily results. R2K provided the daily numbers. Why don’t the other pollsters do likewise?

Therefore, we can look at the R2K tracking poll as just another standard poll that provides daily and full 3-day samples. The volatility of the R2K poll is equivalent to standard polls of 1000-1200 respondents. R2K provided more information than other 2008 tracking polls that only reported the rolling 3-day average.

Table 1 displays R2K daily statistics.

The margin of error is 1.96 times the standard deviation (a measure of volatility) at the 95% confidence level.
The standard deviation of Obama’s daily poll shares was 1.83%. It was 1.59% for the 3-day moving average.

Table 2 is a comparison of Gallup vs. R2K.

Gallup was a registered voter (RV) poll. R2K was a likely voter (LV) poll.
The average shares and volatilities (standard deviation) closely match.
There was a strong 0.70 correlation between Obama’s Gallup and R2K shares.
There was a good 0.50 correlation between McCain’s Gallup and R2K shares.

If the R2000 poll were falsified, it originates anyway from something. Probably from other polls.

Exact timestamps are needed to check if big moves in R2000 polls were coincidental, lagging or leading other polls. Thus having a good correlation with another pollster could be a bad sign.

Additionally it looks very bad when R2000 doesn't release details on its polling. Who did what and when? There should be tons of traces like phone bills etc.

When you see that somebody calculates correlation, you have take in account that maybe he doesn't know how to calculate it. That's normal.

_{-- Thinking inside the box maximizes risk adjusted profits}

Completely falsified data would never lead other polls. I'm assuming that the falsification would be based on the results other polls. The results would have a smaller standard deviation.

Both polls show negative skew (higher bars on negative side).

R2K results have a couple more extreme values (kurtosis) hinting on a smaller sample or a more aggressive likely voter model.

_{-- Thinking inside the box maximizes risk adjusted profits}