J. Ann Selzer has earned a reputation as “the best pollster in politics” through “old-school rigor” and not adjusting her data to fit guesses about the structure of the electorate. Des Moines-based Selzer & Co. is one of only five polling firms in the country currently rated A+ by FiveThirtyEight. Like many media pollsters, the firm uses a random digit dial method to find respondents for surveys about a primary or Iowa caucus. Most internal polls commissioned by campaigns draw the sample from a registered voter list, with an emphasis on past participants in either a Democratic or Republican nominating contest.

I sought comment from Selzer on her methodology because of Fred Hubbell’s and Cindy Axne’s unexpectedly large margins of victory in this year’s Iowa Democratic primary. In a telephone interview with Bleeding Heartland last week, Selzer explained why she will stick with her sampling method for future primary elections.

First things first: polls are snapshots in time, not predictions. This year, Selzer’s poll for the Des Moines Register and Mediacom was in the field a full three weeks before the June 5 election. Since many voters decide late, no one should expect that survey to measure the final result accurately.

That said, the dominating showings of Hubbell and Axne raise questions about whether the Selzer poll misread the Democratic primary electorate.

POLLING THE GOVERNOR’S RACE

To pull the 501 respondents for the poll on the six-way Democratic contest for governor, “Interviewers with Quantel Research contacted 1,880 households with randomly selected landline and cell phone numbers supplied by Survey Sampling International” from May 13 through 15. Only those who said they would definitely vote or had already voted became part of the sample. Findings: 31 percent of respondents were for Hubbell, 24 percent unsure/did not answer, 20 percent for Nate Boulton, and 13 percent for Cathy Glasson. Three other candidates were below 10 percent.

Boulton ended his campaign on May 24. Hubbell received 55.4 percent of the vote in the primary, Glasson 20.6 percent, John Norris 11.4 percent, and other candidates single-digit support.

Selzer’s poll had a margin of error of plus or minus 4.4 percentage points, which means that assuming a perfectly representative sample, there’s a 95 percent chance Hubbell’s true level of support among likely Democratic voters on May 13-15 was between a little below 27 percent and a little above 35 percent.

Theoretically, Hubbell could have surged from that level to 55 percent in three weeks. But the front-runner’s extensive internal polling, conducted by the highly-regarded Democratic firm Anzalone Liszt Grove Research, was rumored to have him in a much stronger position by early May.

In addition, more than a week before Selzer’s poll was in the field, a Remington Research Group survey for KBUR radio in Burlington found 46 percent of likely primary voters were for Hubbell, 20 percent Boulton, 18 percent undecided, and all other contenders in single digits. That poll used automated phone calls–often considered a less rigorous method than live interviewers–to reach 2,315 respondents on May 5 and 6. According to KBUR’s political talk show host Robin Johnson, “The calling universe included past Democratic primary voters plus newly registered Democrat leaning voters. Data was sourced from registered voters. The screen asked if voters planned to vote in the Democratic primary election, [and] only those that indicated they would vote were included in the sample.”

Amy Simon wrote in 2006 that political pollsters draw from voter lists because the costs are lower and the results are “just as accurate, or even more so,” than surveys conducted through random digit dial. Many campaign operatives believe past behavior is the best indicator of whether someone will vote in a primary.

What might be the advantage to sampling a registered voter list? For one thing, the Hubbell campaign spent more than $900,000 on direct mail before the primary, sending a “boat-load” of mailings to reliable Democratic voters in all 99 counties. Hubbell’s large field staff also organized many events around the state over the past year, calling local Democrats ahead of time to invite them to meet the candidate.

Any person could answer the phone and say sure, I will vote on June 5. But respondents who hadn’t participated in an Iowa Democratic primary would probably be less familiar with Hubbell than those who had received many contacts from his campaign.

POLLING THE CONGRESSIONAL RACE

For Selzer’s poll on the three-way Democratic contest in Iowa’s third Congressional district, “Interviewers with Quantel Research contacted 1,326 households with randomly selected landline and cell phone numbers supplied by Survey Sampling International. Of the 400 likely voters interviewed, 143 3rd district respondents were pulled from the statewide poll; the remaining 257 respondents were collected via a 3rd district oversample.”

Callers found 36 percent of respondents were undecided, 27 percent supported Eddie Mauro, 26 percent Axne, and 11 percent Pete D’Alessandro. Selzer said later on Iowa Public Television that the race was close every day it was in the field, from May 13 through 16. The margin of error was plus or minus 4.9 percent.

Axne won the June 5 primary with 57.8 percent of the vote; Mauro received 26.4 percent and D’Alessandro 15.6 percent. Des Moines Register opinion editor and longtime political columnist Kathie Obradovich stands by Selzer’s data and has speculated that the May 31 televised debate may have changed the dynamic after the poll. But it doesn’t seem likely that Axne picked up almost all undecided Democrats to more than double her support during the last three weeks of the campaign. UPDATE: I should have mentioned that Women Vote!, the political arm of EMILY’s List, spent $137,309 on television air time, more than $75,000 on direct mail, and $28,000 on digital advertising supporting Axne after mid-May (full list of independent expenditures here). That could have contributed to a late surge.

What factors might a pollster miss by sampling randomly selected phone numbers? Axne outworked her competitors. Starting last summer, she dedicated a substantial amount of time to calling past primary voters or caucus-goers, not to raise money but to introduce herself and listen to people’s concerns. Some of my less politically active friends asked me about Axne after receiving one of those calls. Campaign manager Joe Diver estimated last week that the candidate reached some 3,500 Democrats in the third district by phone before the primary.

Axne’s campaign also held 100 house parties or other gatherings, far more than anyone else in the field. Diver didn’t have a number for total attendance but told me, “I feel confident in saying 1,700 people attended a meet and greet.” Any potential supporters were welcome, but staff and volunteers focused on inviting active Democrats. By January a long list of “grasstops”–activists who are well-known in their communities or among key constituencies–had lined up behind Axne. Many of them hosted events for her or otherwise spread the word to their friends.

Anyone might tell an interviewer, I will definitely vote in the primary. But more Iowans on a Democratic voter list would have interacted with Axne, compared to respondents reached through randomly selected phone numbers.

“I DON’T WANT A METHOD THAT WOULD BLIND ME TO ANYTHING THAT MIGHT BE HAPPENING”

Selzer defends the random digit dial method because it “ensures every Iowan has an equal chance to be contacted.” Her skepticism about polls that rely heavily on past participants has deep roots. Jim Duncan wrote in this Cityview profile last year,

She was working at The Des Moines Register in 1988 when she questioned a Register poll that predicted George H.W. Bush beating Bob Dole in the Iowa caucus. She objected to a method of only contacting previous voters. She isolated new voters and predicted Dole would win, which he did, with Bush finishing third. Her biggest breakthrough was in 2008 when she predicted Barack Obama would beat Hillary Clinton in the caucuses and that 60 percent of Obama voters would be first timers.

Near the beginning of our interview, Selzer told me, “The best moment in time to decide how you should have done it is after it’s over. The best predictor of future behavior is past behavior, as you said, until there is change.” She recalled Obama’s 2008 strategy of turning out new caucus-goers.

That opportunity is available for any candidate in a primary. So I don’t want a method that would blind me to anything that might be happening out there, in terms of people cultivating people who have not primaried before, but suddenly, they’re activated, and they’re going to vote in a primary. And especially this year, when the read of the culture would be, there are people who are motivated that have not been motivated in past primaries, perhaps.

Obama’s 2008 caucus campaign was an unusual case. Few Iowa Democratic candidates have won a primary by mobilizing large numbers of new participants. Did Selzer sense a major effort this year to activate people with no primary voting history?

Well, I think what you saw was sort of an electoral culture where there was activation. There were these marches. There were these events that were happening through Twitter. […] There was a certain arousal, if you will, in the electorate.

So my method is designed to not blind me to that. So it would be very difficult for me, and I think arrogant for me to say, you know what? I don’t think this seems like a particularly, you know, activated primary season. I’m going to ignore that. That would feel very arrogant to me.

What I will say is the difference between the actual outcome and what our poll was, is we published our poll early. And that was by design, in terms of the way that my client who pays for the polls [the Des Moines Register] wanted to do things. We’re not–we’re not making plans for when the poll publishes on the basis of what it will do to my accuracy record. And why should we? You know, we’re not trying to hit it. And we had other things in that poll to publish that we thought might be helpful in shaping the rest of the debate.

What we know about primaries is that they gel late. And you, I’m sure, noticed how fluid that race still was at the time that we were doing it. In fact, we still had a candidate in the race who soon dropped out.

But in the third Congressional district, where Axne finished with nearly 58 percent of the vote, I would guess she was already ahead by mid-May. Does Selzer disagree?

Well, what I’m saying is that the race at the time we were measuring was far from settled, and there were a lot of people who didn’t really know who any of the candidates were [….] everything gels late.

Selzer mentioned that she looked for evidence Mauro was particularly strong in Polk County, where his family name might be more familiar (due to relatives who long served as county auditor and county supervisor). She “did not see that in the data,” so concluded his strength among respondents wasn’t simply a function of name recognition.

It just was–it was early enough, just a lesson about how late all of those candidates wait to really push. They just save up their resources, and then–boom.

I challenged the idea that the candidates started pushing late, mentioning Axne’s outreach by phone. Drawing from a voter list might have produced more respondents who had spoken with Axne but not with Mauro or D’Alessandro.

And what I would say is, that’s one element that goes into shaping a voter’s decision. They may have a neighbor who’s very strong. They might have a member of their family who’s very strong.

You know, you’re sort of singling out that particular piece of it, and I’m not going to deny to you that that might be a strong piece, but it’s not the only thing happening in that campaign.

True. On the other hand, the fact that television commercials and mail reach voters relatively late (Mauro went up on tv in mid-April, Axne and D’Alessandro in May) doesn’t mean activists wouldn’t have heard of the candidates earlier.

“KEEP YOUR DIRTY FINGERS OFF YOUR DATA”

Does Selzer worry about respondents lying when they say they will definitely vote in a primary? Our culture promotes voting as a civic duty. Maybe people don’t want to admit to an interviewer that they don’t participate.

We try to offer something, another opportunity that would be socially appealing. So our screen is, will you definitely vote, probably vote, might or might not vote, or probably not vote. We only take you if you say definitely.

Even a high-turnout primary is so much lower than a general election, though. (The 176,700 Iowans who cast a ballot for a Democratic candidate for governor represent less than 30 percent of all registered Democrats in Iowa.) Couldn’t Selzer’s screen be pulling in a lot of people in who say they will vote, but won’t?

I don’t deny that. I don’t think that there is a change I would make to my sample that wouldn’t insert more of my own, sort of view of things. So I’m fond of saying, keep your dirty fingers off your data. Any decision that I might make to alter that would be saying, I know better. I know better.

And so, there are a lot of pollsters who have their own models. And when you sort of get down to it, it’s a judgment call. And I do my best not to take a judgment call.

Are the campaign pollsters on the wrong track, then?

The campaign pollster is not trying to predict with accuracy what’s going to happen in that election. Those polls are designed to test, with their idea of what they think the universe will be, how to persuade, what is–to learn what is the most persuasive with that group.

So their goal and my goal are not necessarily aligned. They don’t need to be aligned. So if what their plan is, is to go out after those people who have participated in past primaries, that’s their meaningful universe. Because that’s who they are going to spend their time going after.

Selzer added that one candidate in a race may choose to invest more in cultivating new voters, while others will focus on reliable voters. If you’re only going to target past primary participants, “you’d be silly” to poll other respondents. “That would be a waste of resources if the rest of your campaign resource is going after past voters.”

I can’t agree. Campaigns aren’t looking to validate some pollster’s turnout model. They want to know the candidate’s level of support among those who will cast ballots. If they believed random digit dial was a better way to draw a representative sample, the well-funded campaigns (like Hubbell’s) would pay for that kind of survey.

On a related note, Selzer parts ways with political pollsters who weight their results based on assumptions about which voters will show up. The New York Times Upshot team illustrated how important such decisions can be when they gave four pollsters the same raw data from a September 2016 poll of likely presidential election voters in Florida. They came up with four different results for Hillary Clinton and Donald Trump.

Shortly after the Des Moines Register published the latest Selzer poll, I sought clarification about this sentence in the note on methodology: “Responses were adjusted by age and sex to reflect the general population based on recent census data.” Was Selzer projecting that the universe of IA-03 Democratic primary voters would reflect the demographics of the general population? No, she told me by e-mail, “we were not projecting what the universe of IA-03 Democratic primary voters would be at all.”

We collect demographic information on all contacts as we screen for likely Democratic primary voters—not just those who pass our screen. For both the governor’s and CD3 races, that was well over 1,000 contacts. This approach yields a general population sample to compare to U.S. Census data. Once we adjust that larger pool of contacts, we extract the likely voters, so if a particular demographic group is more likely than average to report they will vote, they will show up in our smaller voter sample in above average numbers. The opposite is true as well, of course.

I believe in weighting to known populations—in this case, the general population. We do not know the makeup of the population who will cast votes in the Democratic primary, so we do not make assumptions about what that demographic profile will be. The best predictor of future behavior is past behavior—until there is change. We let our data tell us about any such change.

“STRANGE THINGS HAPPEN”

During our phone call, I asked Selzer about her final poll of the 2010 GOP primary for governor. That survey was in the field a week before the election and found Terry Branstad (who was coming out of political retirement) leading Bob Vander Plaats by 57 percent to 29 percent among likely Republican voters. The primary turned out to be much closer: about 50 percent for Branstad, 41 percent for Vander Plaats and 9 percent for Rod Roberts.

I asked Selzer whether sampling random phone numbers might have pulled in respondents who weren’t engaged in Republican politics but recognized Branstad’s name from his long tenure as governor during the 1980s and 1990s. Using a GOP voter list might have revealed Vander Plaats’ strength among hard-core activists.

Selzer recalled that Vander Plaats seemed competitive in what was then the third Congressional district. But her big takeaway was, “Never underestimate the Christian Coalition in terms of their ability to turn out votes.” She posited that Vander Plaats benefited from a surge among social conservative voters. “We’ve seen that since 1988,” when Pat Roberts was a surprise second-place finisher in the Iowa caucuses. It’s a disciplined community and “tends to be almost invisible to the naked eye.” Her sample included 41 percent of respondents who called themselves evangelicals or born-again Christians, a “hefty group.”

All in all, Selzer was “not dissatisfied with how that poll performed, next to the way it came out. Not surprised by it. You ask any pollster, they’re going to say it takes a very strong stomach and a steady hand,” she laughed.

A lot of polling firms stay out of politics, she added,

because strange things happen. If my assignment was to always get primaries exactly right, I might say, well I don’t know–if that’s what my salary is dependent upon, I don’t know that I’d take that job. It’s just, these things happen.

Returning to this year’s governors race, I mentioned that I tend to be skeptical of leaked information about internal polling. So when some sources told me last month that Hubbell’s numbers had him in better shape than the Des Moines Register poll, I had my doubts. With hindsight, it seems Hubbell’s data may have been closer to the mark. What is Selzer’s take?

She pointed to many events that happened after her poll. The biggest was Boulton dropping out, but there were also three debates (on May 13, 16, and 30). Those “tend to affect the conversation.” Finally, Selzer pointed to late endorsements, which she said can be influential in small towns. Attorney General Tom Miller, former Governor Chet Culver, and several state lawmakers who had backed Boulton endorsed Hubbell during the final week of the campaign.

How would Selzer respond to the idea that random sampling of phone numbers might work better for a general election, when turnout is so much higher than for a primary? She relayed her “cautionary tale” about the 2008 caucuses. Iowa Democratic power-broker Jerry Crawford told her he’d always trusted her polls before, but those findings were not consistent with his observations knocking doors for Hillary Clinton. “They were blind to what was happening” because they were only contacting past caucus-goers and registered Democrats.

As for what happened this year, Selzer thinks Hubbell and Axne had “well-functioning campaign machines.” When the Secretary of State’s office releases the list, we’ll know how many people who participated on June 5 were past primary voters and how many were newly engaged.

Final question: have I understood properly that Selzer has no plans to change her sampling methodology before the next primary or caucus? “That’s correct.”

UPDATE: Several readers with extensive polling experience have pointed out factors I neglected to mention. Charles Franklin, director of the Marquette Law School Poll and the founder of the Polls and Votes website, observed that “the voter list tells you what districts (Congressional, state Senate or House) and municipality the voter lives in. Those are critical to any campaign polling for a office that isn’t statewide.” In a random digit dial sample, “people can’t reliably tell you” what Congressional or state legislative district they live in. Conversely, it’s impossible to contact many people on a registered voter list, for lack of a phone number.

Washington Post polling director Scott Clement echoed that point: “while RDD surveys are able to reach anyone with an area code” in the relevant area, “a significant chunk of voter file records lack a working number (20%-50% is common). If those voters have systematically different views, it introduces error.” In addition, Clement said, “it’s common for random digit dial surveys to use (self-reported) past voting behavior to identify likely voters & for voter file samples to use respondents’ reported intent of voting, often in combination.”

I wondered how accurate self-reporting of past voting is, having heard anecdotes about people who claim, “I never miss an election,” when a campaign staffer can see from the database that the person missed one or two of the last three. Clement said the Washington Post “checked respondent self-reports in two 2017 Virginia polls against voter records” and found 95 percent of reports matched the person’s record for the 2016 election, 90 percent for the 2012 election, and 79 percent for the 2013 election.

Top image: Screen shot from J. Ann Selzer’s June 1 appearance on Iowa Public Television’s “Iowa Press” program.