Continuing a six-part series. Part 1 covered basic elements of the caucus system, part 2 explained why so many Iowans can’t or won’t attend their precinct caucus, part 3 discussed how Democratic caucus math can affect delegate counts, and part 4 described how precinct captains help campaigns.

Measuring the horse race ahead of the Iowa caucuses poses special challenges, particularly on the Democratic side. Those problems affect even the Des Moines Register’s longtime pollster Ann Selzer, whom FiveThirtyEight.com has given an A+ grade and called “the best pollster in politics.”

Follow me after the jump to see why polling expert Mark Blumenthal has described the caucuses as a “pollster’s nightmare.”

Identifying likely caucus-goers: a challenge for either party

Likely voter screens are among the most controversial elements of election surveys. They can vary widely from firm to firm and greatly shape a poll’s findings. Turnout for the Iowa caucuses has usually been lower than for Iowa primary elections, partly because attending a caucus requires a much bigger time commitment than voting in a typical election, and partly because the need to gather in a certain place at a specific time on a weekday evening keeps thousands of politically-engaged Iowans from participating.

After researching the methodology of all polling companies doing work in Iowa ahead of the 2008 caucuses, Blumenthal wrote,

No two Iowa pollsters select “likely caucus goers” in the same way. Moreover, each pollster has a unique conception — sometimes radically unique — of the likely electorate. […]

After choosing a sampling method, the pollster must decide how to select likely caucus goers from their sample. That task is not easy. The historical high for turnout in the Iowa Caucuses was 5.5% of adults for the Democrats in 2004 and 5.3% of adults for the Republicans in 1988. That means, in theory at least, that a pollster starting with an RDD [random digit dialing] sample would need to screen out nine out of ten otherwise willing adults in order to interview a combined population of Democratic and Republican caucus goers strictly comparable in size to past caucus turnouts. This daunting task is one big reason why polling for the Iowa caucuses is so challenging and why pollsters differ so much in their sampling methods.

Those that sample from lists do so partly so they can more accurately and efficiently reach known registered voters (although caucus rules in both parties allow for walk-in registration, so list-based polls will miss the small percentage that register on caucus night or just before caucuses). […]

A few pollsters (such as PPP, Garin/Hart and McLaughlin and Associates) use the actual vote history on the registered voter list to narrow the sample universe before dialing. In other words, the registered voter list includes a record — for each individual voter — of whether they voted in each recent election. So pollsters can choose to select, say, only those who have voted in a Democratic primary election in the last four years (or any other permutation you can think of). […]

However, for the heavy lifting of screening for likely caucus goers most of the public pollsters rely on self-reports about intent to participate. […] No two ask exactly the same questions, but most pose a scale that includes options like “definitely attend” and “probably attend,” using those categories to classify respondents as likely caucus goers.

After reviewing sixteen Iowa Democratic polls released in January, Nate Cohn pointed out that polls drawing respondents from a registered voter list have tended to produce better results for Hillary Clinton, while surveys using random digit dialing have produced the most promising numbers for Bernie Sanders. In a different 2007 commentary on Iowa caucus polling challenges, Blumenthal wrote,

In 1988, I worked for Paul Maslin, the pollster for Democratic Senator Paul Simon. Simon always did better on the samples we drew from lists of actual past [Iowa] caucus-goers, while Congressman Dick Gephardt did consistently better when when we included registered voters [who] had not participated in the previous caucuses in 1984. Gephardt also did consistently better on the RDD [random digit dialing] surveys in the public domain. As I recall, those differences persisted through the final round of polling, though they probably narrowed a bit toward the end. Of course, the challenge is that every election year, the caucuses attract large numbers of voters who did not participate in the prior election cycle. And true junkies will remember that Gephardt ultimately won the Caucuses, although as I recall, the actual result fell somewhere between the two methodologies.

Assumptions about likely caucus-goers can skew findings. Donald Trump’s worst Iowa polling numbers lately have come from surveys for Iowa State University and the WHO-HD television station in Des Moines. Iowa State political science professors David Andersen and David Peterson explained why their methodology produced anomalous results for Trump in this December commentary for the Washington Post’s “Monkey Cage” blog.

In our poll, Trump registers 10-15 points lower than in the three polls of Iowa Republicans who were in the field around the same time. Our poll recorded the lowest level of support for Trump in Iowa since July. […]

To build a sample, we began with the list of registered voters in Iowa and stratified our sample by factors like age and sex. (More details are here.) Other pollsters have done something similar.

However, what we did — but other[s] have not — is stratify the sample based on another factor: Whether or not people had voted in at least one primary election since 2006.

This is what we think makes our poll different. We assume that voters who have voted in primaries are more likely to attend the caucuses. Of the 486 Republican voters in our poll, 78 percent had previously participated in the primaries, compared with only 48 percent of all registered voters.

Why didn’t we use previous caucus participation, since we are ultimately trying to figure out what will happen in the Feb. 1 caucus? The answer is simple: Iowa doesn’t record that information, and it isn’t part of information in the list of registered voters. Instead, we assume that voters who have voted in primaries are more likely to attend the caucuses.

One theory about Trump’s success is that he is energizing people who have traditionally avoided politics and political participation. If he succeeds, then our sample will understate his support. But if he doesn’t, and participation in the February caucus does depend on whether voters have voted in earlier primaries, then other polls to date have likely overstated Trump’s support.

The method Andersen and Peterson selected sounds reasonable. Regardless of a respondent’s stated intention to caucus, a person who has not voted in a GOP primary during the last ten years does seem less likely to leave home for more than an hour on a cold Monday night. Yet Trump’s campaign has upended so much conventional wisdom, we should not discount his prospects to motivate unprecedented numbers of Iowans who have not been engaged in Republican Party politics.

One reason Ann Selzer nailed her final poll for the Des Moines Register before the 2008 caucuses was she “makes no assumptions about the size of the electorate.” Her data were telling her that nearly 60 percent of likely Iowa Democratic caucus-goers would be attending their precinct caucuses for the first time. Many old political hands laughed at that prediction, but in fact, nearly 240,000 people turned out for Democratic caucuses on January 3, 2008, nearly doubling the previous record attendance of under 123,000. Based on my voter contacts, working from a list of primarily people who had caucused before, I expected John Edwards and Hillary Clinton to have far more support than Barack Obama in my precinct. Instead, Obama had a plurality after the first division into preference groups.

Even with a perfect likely caucus-goer screen, a poll measuring the Democratic presidential race in Iowa will have other limitations.

Special problems with polling the Iowa Democratic caucuses

Surveys of likely Iowa caucus-goers report statewide results as if we had a primary, where every person votes for the preferred candidate. That’s a reasonable approximation for the Republican caucuses, which are essentially a statewide straw poll. A ballot for Trump or Ted Cruz or Marco Rubio counts the same in any precinct as in any other precinct.

But because the Iowa Democratic Party reports only county delegate totals and “state delegate equivalents” for each presidential candidate, not every Democratic caucus-goer will equally influence the reported results, for two reasons.

1. Turnout will be higher in some counties than others, but state delegate totals are fixed for all counties.

As Bleeding Heartland discussed here, approximately 41 Democrats attended the 2004 caucuses for every state delegate equivalent. But in Johnson County, containing Iowa City and the University of Iowa, turnout was so high that 79 people attended Democratic caucuses per state delegate. On the flip side, fewer than 30 voters per state delegate attended Democratic caucuses in more than 30 counties.

A similar pattern held true in 2008. Using the number of Democratic state delegates each county assigned that year and Democratic attendance figures by county, John Deeth created a table showing the ratio of caucus-goers to state delegate equivalents for all 99 counties. He concluded,

The easiest place to elect a delegate was Osceola County, where it took 59.67 attendees per state delegate equivalent. The bottom ten is filled with small, rural, GOP leaning counties, with the curious exception of Clinton.

And once again, the hardest places to elect state delegates were campus counties. In Winneshiek, home to Decorah’s Luther College, it took 138 people to elect a state delegate equivalent, 2.3 times as many as in Osceola. Johnson County, the hardest place to elect a delegate in 2004, is narrowly behind Winneshiek at 134 bodies per state delegate equivalent.

Put another way: A “vote” in Osceola County is worth more than twice as much as getting out a Luther undergrad. […]

The top ten of hardest places to elect delegates were college counties including Story and Jefferson (Fairfield) and high growth suburban areas like Dallas and Warren counties. And, oddly, a couple of the most Republican places in the state, Page and Sioux counties.

Click through to view Deeth’s table, showing all 99 counties ranked in descending order in terms of how many people attended the 2008 caucuses per state delegate equivalent. Using the same data on Democratic attendance, I made this smaller table to highlight the disparities among the eleven most delegate-rich counties. Those counties accounted for about half of the 2,500 state convention delegates the Iowa Democratic Party assigned in 2008.

County	attendance	2008 state delegates	2008 attendees per state delegate
Johnson	18,362	137	134.03
Polk	44,098	358	123.18
Story	9,227	77	119.13
Linn	18,139	202	89.80
Scott	12,552	142	88.39
Woodbury	5,836	68	85.82
Pottawattamie	4,723	55	85.87
Dubuque	7,627	90	84.74
Black Hawk	9,382	117	80.19
Cerro Gordo	3,416	46	74.26
Clinton	3,203	46	69.63

Polling firms have methods to obtain respondents geographically dispersed across Iowa. But they can’t account for the fact that it may take far more Democratic caucus-goers to elect a state delegate in Iowa City or Des Moines as it will in Mason City or Clinton.

To complicate matters further, even within the same county, not every Democrat’s “vote” will count the same.

2. Turnout will be higher in some precincts than others, but county delegate totals are fixed for all precincts.

In my own Polk County precinct, we needed 27 supporters to reach the 15 percent viability threshold in 2004. An acquaintance was in a Des Moines precinct where turnout was so low, they needed just six people to be viable. There were precincts on the west side of Des Moines where candidates needed more than 40 supporters to be viable. Consequently, a Democrat in the neighborhood where my acquaintance lived had as much influence over the Polk County delegates as four caucus-goers in my precinct or seven caucus-goers in the political hot spot of Beaverdale.

Deeth wrote in 2007,

For example in 2004, in Iowa City Precinct 18, a hotbed of activism full of liberal professors and students, 534 caucus-goers recreated the Black Hole Of Calcutta in the Longfellow Elementary School gym. In North Liberty Precinct 1, full of trailer courts, newly developed housing and independents who marked their fall ballots for the Democratic ticket, only 171 people showed up. But based on their general election voting in 2000 and 2002, North Liberty 1 and Iowa City 18 each elected the same total of 10 [Johnson County] delegates.

There is no way a pollster can weight for such disparities. Selzer’s final survey before this year’s caucuses found 45 percent support for Clinton and 42 percent for Sanders. The gap between the candidates was within the poll’s margin for error, but let’s assume for now that exactly 45 percent of Iowans who come to Democratic precinct caucuses tonight will support Clinton and 42 percent will support Sanders. Without knowing how those supporters are distributed across the state, we can’t say who will walk away with the most state delegates.

A wave of first-time caucus-goers for Sanders could deliver victory for him, as it did for Obama in 2008. However, I suspect that Iowans who “feel the Bern” are more concentrated in certain areas, which is why I expect Clinton to win tonight by a larger margin than her lead in the final Iowa polls.

Democratic caucus math can distort caucus-goers’ preferences

The third post in this series explained how small shifts during “realignment” may change the number of delegates allocated to each Democratic candidate in a given precinct. If you had surveyed my neighborhood’s Democrats before the 1988 caucuses, Paul Simon would have come out ahead. But a clever maneuver by supporters of Michael Dukakis and Bruce Babbitt altered the math so that our delegates split two each for Simon, Dukakis, and Babbitt. Such gamesmanship was not a factor in my neighborhood in 2008. Even so, my precinct gave two county delegates each to Edwards, Obama, and Clinton, despite the fact that after realignment, the Edwards group was the largest and the Clinton group smallest.

Recent polling indicates that Martin O’Malley will struggle to be viable in most Iowa precincts. O’Malley supporters may be able to add a county delegate to Clinton or Sanders if they move en masse to one of the leading candidates. Alternatively, if O’Malley is relatively close to 15 percent on the first division into preference groups, supporters of Clinton or Sanders may help O’Malley become viable in order to subtract a county delegate from the main rival.

Drew Miller’s caucus calculator lets you explore such scenarios, which should convince you that the dynamics of an Iowa Democratic caucus are beyond the scope of what any pollster can predict.

By tomorrow morning, we will know which of more than 20 surveys of likely Republican and likely Democratic caucus-goers released in January 2016 best predicted the outcome.

Nevertheless, the Iowa caucuses will remain a pollster’s nightmare.