{"id":40802252298,"date":"2022-06-16T15:08:05","date_gmt":"2022-06-16T19:08:05","guid":{"rendered":"https:\/\/www.monmouth.edu\/polling-institute\/?p=40802252298"},"modified":"2023-07-10T10:11:01","modified_gmt":"2023-07-10T14:11:01","slug":"if-the-polls-aint-fixed-dont-break-them","status":"publish","type":"post","link":"https:\/\/www.monmouth.edu\/polling-institute\/2022\/06\/16\/if-the-polls-aint-fixed-dont-break-them\/","title":{"rendered":"If the Polls Ain\u2019t Fixed, Don\u2019t Break Them"},"content":{"rendered":"\n

By Patrick Murray<\/strong><\/p>\n\n\n\n

Is public opinion polling accurate?\u00a0 The answer to that question is driven in large part by whether a specific subset of polling \u2013 specifically, preelection \u201chorse race\u201d polling \u2013 is perceived as accurate. On that front, the polling industry has hit some bumps recently, particularly with the 2020 election. An evaluation of preelection polls<\/a>\u00a0conducted by the American Association for Public Opinion Research (AAPOR)\u00a0could not identify a clear culprit behind that year\u2019s polling miss.\u00a0The most likely explanation is differential nonresponse \u2013 i.e., that a cohort of Donald Trump supporters did not participate in the polls out of mistrust in political institutions.<\/p>\n\n\n\n

In other words, the poll samples may have in fact looked \u201cright\u201d from the perspective of age, gender, race, and education, but within each group was a small overrepresentation of Democratic support because of nonresponse from Trump backers across these demographics.  Keep in mind that these errors are small, and in fact would not be noticeable in most public polls.  But when taken together in the context of the presidential election, these group errors resulted in a 4-point skew in the average polling sample.<\/p>\n\n\n\n

We examined the polling Monmouth conducted in six states prior to the 2020 election. We validated turnout for 2020 for everyone selected for the sample \u2013 both those who participated in the survey and those who did not \u2013 in an attempt to identify patterns that could identify the nonresponse problem. This included demographic and vote history data for tens of thousands of voters in each state. Even with this enhanced data set, we could not find a \u201csilver bullet\u201d in the voter files that distinguishes a cohort of likely Trump supporters who refused to participate in the polls. Our review is summarized in a presentation prepared for the 2022 AAPOR conference here<\/a>.<\/p>\n\n\n\n

Was this suspected nonresponse problem also behind the polling miss in the New Jersey gubernatorial election the following year? The short answer is no.  It turns out \u2013 via post-election voter verification \u2013 that the underlying sample in the New Jersey polling was representative of the full voter population. The real culprit was an unusually large partisan skew in who actually turned out \u2013 a skew that was not captured by likely voter models. This skew was not present in the Virginia electorate that year, which is why the polls in that state were better barometers of the actual outcome there. In other words, the samples of registered voters in both states were accurately representative of all registered voters there. The problem in New Jersey was likely voter modeling. [This is also reviewed in our presentation<\/a>.] <\/p>\n\n\n\n

Prior to 2016, our first suspect in an election polling error would have been the likely voter model \u2013 which is exactly what we found to be the case here. But since 2016, we have to contend with the notion that every polling error is due to a systematic problem.<\/p>\n\n\n\n

Thus, we are left with a dilemma.  Polling is a social science methodology with an acknowledged potential for error. This means that polls can and do miss the mark, either individually or as a group, from time to time. The magnitude of those misses can be as much perceptual as statistical.  For example, an election poll that has Candidate A ahead by 2 points in a race that Candidate B actually wins by 3 points is perceived by the public as being more erroneous than a poll with Candidate A ahead by 10 points in a race she wins by 16 points \u2013 even though the absolute error in both instances is the same.<\/p>\n\n\n\n

Public pollsters, unlike campaign pollsters, have a mission to hold up a mirror to the public on important political, cultural, and behavioral issues of the day \u2013 e.g., think prevalence and rationale for COVID vaccine opposition. While it is important to survey voters about elections, the \u201chorse race\u201d question is fraught with problems and likely voter modeling pushes polling methodology to do things it is not designed to do \u2013 i.e., predict future behavior. It is a testament to the polling field that election polls tend to get things right more often than not.<\/p>\n\n\n\n

The 2020 election aside, there is little evidence that public opinion polling is measurably less accurate today than it was a decade or a generation ago. The polling industry has been proactive, and largely successful in addressing challenges as they arise (such as declining response rates, the growth in cell phone use, etc.). But in an era of distrust, we have to be even more vigilant and transparent.<\/p>\n\n\n\n

In that spirit, Monmouth is making available the full datasets<\/a> for its final state election polls in 2020 and 2021. The 2020 files include appended voter file information for both respondents and nonrespondents.  As mentioned earlier, the Monmouth team did not identify any \u201csilver bullet\u201d corrective \u2013 either for the 2020 nonresponse error or the 2021 likely voter model error.  We hope that by sharing these datasets, though, other researchers may be able to further our understanding of challenges in polling these elections.<\/p>\n","protected":false},"excerpt":{"rendered":"

By Patrick Murray Is public opinion polling accurate?\u00a0 The answer to that question is driven in large part by whether a specific subset of polling \u2013 specifically, preelection \u201chorse race\u201d polling \u2013 is perceived as accurate. On that front, the polling industry has hit some bumps recently, particularly with the 2020 election. An evaluation of […]<\/p>\n","protected":false},"author":25,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-40802252298","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/posts\/40802252298","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/users\/25"}],"replies":[{"embeddable":true,"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/comments?post=40802252298"}],"version-history":[{"count":5,"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/posts\/40802252298\/revisions"}],"predecessor-version":[{"id":40802255482,"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/posts\/40802252298\/revisions\/40802255482"}],"wp:attachment":[{"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/media?parent=40802252298"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/categories?post=40802252298"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/tags?post=40802252298"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}