{"id":40802236347,"date":"2009-02-19T14:15:00","date_gmt":"2009-02-19T19:15:00","guid":{"rendered":"https:\/\/www.monmouth.edu\/polling-institute\/2009\/02\/19\/question-wording-matters\/"},"modified":"2021-04-20T13:41:06","modified_gmt":"2021-04-20T17:41:06","slug":"question-wording-matters","status":"publish","type":"post","link":"https:\/\/www.monmouth.edu\/polling-institute\/2009\/02\/19\/question-wording-matters\/","title":{"rendered":"Question Wording Matters"},"content":{"rendered":"

Today, we released a Monmouth University\/Gannett New Jersey poll on same-sex marriage<\/a>. I characterized the poll results \u2013 48% in favor to 43% opposed \u2013 as showing a public divided on this issue. I have been asked why I did not frame the results in a more positive light for same-sex marriage, especially since support among registered voters was slightly higher at 50%. The political reality is that these results indicate this is an issue with little momentum right now.<\/p>\n

First, if this issue were put to a referendum, polling support would need to be above 50% for passage (referendum polls in New Jersey tend to understate the size of the \u201cNo\u201d vote). Second, 27% of Garden State voters \u201cstrongly\u201d oppose same-sex marriage, including 1-in-5 Democrats and 1-in-3 independents. Considering how tight the gubernatorial pre-election polls are right now, Governor Corzine needs nearly every one of these voters if he is to be re-elected. So he can\u2019t afford to turn this into a campaign issue.<\/p>\n

In the end, I chose to portray the results cautiously. By comparison, Garden State Equality released its own poll last August which obtained similar results, but under the headline: \u201cZogby Poll: New Jersey Wants to Dump Civil Unions for Marriage Equality, and Is Ready Now\u201d. This leads me to the larger point of this post \u2013 which is to pay attention to question wording.<\/p>\n

I was initially skeptical when I saw who conducted the poll. John Zogby has a reputation among credible pollsters for a lack of \u201cquality control\u201d when it comes to his client studies. For example, his firm recently conducted a poll on behalf of a conservative activist which purportedly measured Obama voters\u2019 \u201cknowledge\u201d of the presidential candidates. However, it posed questions with unequal information about the two parties\u2019 tickets that basically got the client what he wanted to hear. You can find critiques of this poll from the Wall Street Journal\u2019s Numbers Guy and Politico<\/a>. Zogby, for his part, first defended the poll but later said he was \u201cout of town\u201d when his firm accepted the contract.<\/p>\n

Back to the August poll, some news organizations in New Jersey chose to report the poll under the rationale that Zogby supposedly has a good track record at election forecasting. That\u2019s debatable, but really immaterial to this poll. The real issue is whether he asked objective questions.<\/p>\n

The Monmouth poll\u2019s same-sex marriage question, a variant of one first used by the Eagleton-Rutgers Poll in 2003, is: \u201cDo you favor or oppose allowing gays and lesbians to marry legally?\u201d<\/p>\n

The Zogby poll asked: \u201cCurrently, New Jersey lets same-sex couples enter only into civil unions, while California and Massachusetts give same-sex couples the freedom to marry. Do you support or oppose same-sex couples in New Jersey also getting the freedom to marry?\u201d<\/p>\n

On its face, this question violates a number of tenets of question wording. It uses appeals to the status quo (\u201ccurrently\u2026\u201d), external benchmarks (\u201cother states do it\u201d), and loaded wording (\u201cfreedom\u201d).<\/p>\n

So what is the impact of all this extra verbiage? To find out, I included the Zogby wording in a split-half design experiment on our last poll (acknowledging that California voters have since banned same-sex marriage).<\/p>\n

Among registered voters, the Zogby wording produced a 51% favor to 37% oppose result. That\u2019s not significantly different than the Monmouth question wording\u2019s 50% favor to 40% oppose among voters. However, a closer look at the internal numbers suggests that the Zogby question wording did indeed have an impact \u2013 but it was different for different groups. Among Democrats, support for same-sex marriage using the Zogby question stood at 66%, compared to 58% in the Monmouth question. Conversely, Republicans were less<\/em> supportive of same-sex marriage in the Zogby question (31%) than they were in the Monmouth wording (37%).<\/p>\n

My best guess is that the \u201cother states do it\u201d and \u201cfreedom\u201d appeals encouraged some Democratic poll respondents to be more supportive than they would have otherwise. But the reference to Massachusetts and California \u2013 seen as bastions of liberal thinking by some \u2013 seems to have moved a similar number of Republican poll respondents in exactly the opposite direction.<\/p>\n

I\u2019ve posted about the importance question wording before<\/a> . It\u2019s a lesson worth repeating. If you are a client who insists that certain words need<\/em> to be included in a question, be careful what you wish for. And if you are a journalist reporting on a poll, make sure you read \u2013 and are comfortable with \u2013 how the questions are asked.<\/p>\n","protected":false},"excerpt":{"rendered":"

Today, we released a Monmouth University\/Gannett New Jersey poll on same-sex marriage. I characterized the poll results \u2013 48% in favor to 43% opposed \u2013 as showing a public divided on this issue. I have been asked why I did not frame the results in a more positive light for same-sex marriage, especially since support […]<\/p>\n","protected":false},"author":939,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-40802236347","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/posts\/40802236347","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/users\/939"}],"replies":[{"embeddable":true,"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/comments?post=40802236347"}],"version-history":[{"count":5,"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/posts\/40802236347\/revisions"}],"predecessor-version":[{"id":40802248822,"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/posts\/40802236347\/revisions\/40802248822"}],"wp:attachment":[{"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/media?parent=40802236347"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/categories?post=40802236347"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.monmouth.edu\/polling-institute\/wp-json\/wp\/v2\/tags?post=40802236347"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}