Monday, May 28, 2018
In Search of... Lost Kentucky Derby Viewers

in-search-of-nimoyGrowing up I couldn't get enough of the old "In Search of..." series hosted by Leonard Nimoy. So with a nod to the verbal disclaimer that opened each episode's investigation into historical mysteries and the paranormal...

"This article presents information based in part on theory and conjecture. The author's purpose is to suggest some possible explanations, but not necessarily the only ones, to the mysteries we will examine."

The table was set. Thoroughbred racing's good friend NBC unleashed a beast of a cross-platform promotional effort for its Kentucky Derby telecast last Saturday. The 164,858 patrons who jammed Churchill Downs broke an attendance mark that has stood since 1974. Derby Day wagering volume from all sources was massive and increased from last year, 
making it the third highest total on record. Google search volume for "Kentucky Derby" increased over last year. Page-views on increased. Looking good, eh?

Not so fast. What about... "the number"? You know, the much anticipated Nielsen TV rating. Thud. We were surprised to learn that viewership plunged, a 12 percent decline from roughly 16.5 million viewers last year and the year before to 14.5 million viewers in 2011. Churchill CEO Bob Evans was admittedly "disappointed" with this all-important news. So would advertisers and sponsors.

Now those 14.5 million viewers still represent a strong number but...

Do you believe it? Do you believe that roughly 2 million people watched Calvin Borel and Super Saver cruise home in 2010 but had better things to do last Saturday? Presumably a huge chunk of these 2 million viewers also watched Calvin and 50-1 longshot Mine That Bird splash home in 2009. Do you really believe they weren't hooked on the first Saturday in May?

And in the interim since last year's Derby we've had Zenyatta and Disney's film "Secretariat" to catch the attention of the uninterested masses. Might they not have a look at the Kentucky Derby to see what all the fuss is about?

Close observers moved in quickly to explain the unexplainable...
"The Derby lacked star power  and compelling story lines."
"The NBA and NHL playoffs were stiff competition."
"There was no major sports lead-in to the broadcast."
"Coverage of the killing of Osama bin Laden made little room for typical Derby feel-good stories."
And of course the obligatory...
"It was a nice day to be outside."

Let's add another observation. There were no nationally televised Kentucky Derby prep races this year, which was unprecedented. Still, 2 million lost viewers?

And what about this, albeit heretical, possibility: Kentucky Derby viewership didn't decline. It was on par with last year. And just maybe, viewership increased over last year.

FamilyWatchingTV1958cropThe Nielsen ratings from Nielsen Media Research are surrounded by a peculiar aura of legitimacy. They are reported in the media as if they are as precise and trustworthy as the odometer in your car. In truth, these numbers are anything but precise and should be considered as only an estimate of the reality they purport to measure.

Stripped down to the basics, the Nielsen ratings are nothing more than survey-based estimates that have endured persistent nagging criticism. As such they are subject to a host of error sources that can creep in to mangle the results of any survey.

Of all the potential error sources in scientific surveys the most well-known is sampling error, commonly and erroneously referred to as "margin of error". In simple terms, sampling error is a function of the quality (random and representative) and size of a sample from a population. Sampling theory tells us that a random sample from a much larger population will usually have similar characteristics to the population.

The word "usually" is important since a researcher can do everything "right" and still come up with an unrepresentative sample. But that is unlikely. This is what the customary 95% confidence level means. 

In survey research, an aggregate "margin of error" is simply unknowable. On the other hand, a "margin of sampling error" is knowable since we have theoretical calculations to estimate its range. Other potentially more serious error sources like sample design issues, non-response bias, inappropriate or inadequate weighting of the data, are impossible to quantify and would collectively contribute to a theoretical overall margin of error if one could be calculated. But it defies calculation, so we (professional researchers) don't.

Not many people are aware of the margin of sampling error surrounding Nielsen ratings. There was a bit of a dust-up in the New York Times over this issue a few years back that is well-worth a careful read.

If we make the assumption, based on the margin of sampling error noted in the New York Times controversy, that the "true" viewership number is within +/- 0.3-rating point of Nielsen's estimate, a curious thing happens. The overnight rating for last year's Kentucky Derby was 10.3 compared to 9.7 this year. But these are not hard numbers, only estimates. Recognizing the margin of sampling error, the true number in 2010 could have been as low as 10.0 and as high as 10.6. The true number in 2011 could have been as low as 9.4 and as high as 10.0. In other words, maybe there was no meaningful difference in viewership.

There are other big issues in play with the Nielsen ratings. TV viewing is more fragmented and "time-delayed" than ever. Hulu and streaming and DVRs have wreaked havoc with a Nielsen methodology that worked well not so long ago. Did you know that "outside-the-home" TV viewing goes unmeasured by the Nielsen rating? How many watched the Kentucky Derby in a restaurant, bar, airport, hotel, dorm, health club? Not to mention the "communal" aspect of big-event sports viewing. How many watched the Kentucky Derby at the same party they've attended year after year?

It's easy to see the challenge in answering a seemingly straightforward question: How many people watched the Kentucky Derby?

But Nielsen is the only game in town and their estimate is the best we have. To their credit, they strive to innovate and capture more pieces of TV viewing reality into their model. But shortcomings abound, competitors are scheming, and clients are impatient.

For decades we have followed Nielsen down the rabbit hole into a fantasy world in which they tell us, with unjustified precision, which programs are popular and how many of us watched. It has never been more apparent that we should interpret these ratings with a most critical eye.

Journalists, even those in the horse racing industry, can help by not taking Nielsen ratings at face value. Invariably the media will run with a story about rating fluctuations using phrases like "slightly up" and "dipped somewhat" when in reality there is little to no meaningful difference. There is no practical reason why Nielsen does not issue their ratings within a commensurate margin of sampling error, just as political pollsters and survey organizations do. But that would take the shine off "the number".

You'll not find a bigger proponent of using science to solve problems than myself. But a cautionary note is always in order.

Survey-based research efforts, like the Nielsen ratings, are like weather forecasts. Listen to them and let them guide you, but also look out your window.