The Political Scientist Who Warned Us About Polls

4

Every era enshrines its prophets. Politics today has whiz kids like Nate Silver, Nate Cohn, Harry Enten and Dave Wasserman, who have achieved varying degrees of cultural celebrity by telling us what to expect come Election Day — even when, as happened once again this week, their vision proves cloudy (or worse). In 1948 there was no greater prophet than George Gallup, whose face graced the cover of Time magazine in May of that year. The accompanying profile called him “the Babe Ruth of the polling profession.”

Gallup’s name had by then become synonymous with a wide-ranging new effort by survey-takers and statisticians who were striving to know, with scientific precision, the very nature of American mind, including which presidents the public meant to elect. Gallup had rocketed to fame in 1936 by confidently declaring that the Literary Digest — at the time the gold standard of polls, which was forecasting FDR’s defeat in the fall election — would be wrong. FDR won in a landslide, in what turned out to be (on Literary Digest’s part) a historic polling fail.

Though hailed for his clairvoyance in 1936, Gallup eventually found himself, too, attacked for arrogance and short-sightedness. Like the other big-name pollsters of the day, Elmo Roper and Archibald Crossley, Gallup predicted in 1948 that New York Attorney General Tom Dewey would rout President Harry Truman in the fall election. Although Gallup’s data had detected Truman making gains toward the end of the campaign, he had never placed Truman within five points of his Republican rival. Elmo Roper did even worse; his final survey for Fortune magazine, in October, said Dewey would take 44 percent of the vote, Truman 31 percent. “So decisive are the figures given here this month,” the editors wrote, “that Fortune, and Mr. Roper, plan no further detailed reports on the change of opinion in the forthcoming presidential campaign.” Other journalists also took the pollsters’ word for it. Newsweek asked 50 political writers who would win. All said Dewey. The New York Times predicted that Dewey would win with 345 electoral votes. Life put Dewey on its next cover, before Election Day was over.

But, of course, it was Truman who defeated Dewey, not the other way around. It was a polling failure worthy of 2016 or 2020. (To give a sense of the misfire this year: Going into Election Day, 538 put Biden’s Florida lead at 2.5 percent while Real Clear Politics put it at .9 percent; Trump is now ahead there by 3.4 percent with 96 percent of votes counted — an error of between 4.3 and 5.9 percent). And just as those who prognosticated a Biden romp this week are looking egg-faced, so journalists who hitched their reportage to Gallup’s data came in for jeers. Some cheered the pundits’ and the pollsters’ comeuppance. In the New Republic, Richard Strout, writing under his usual pseudonym “T.R.B.,” celebrated “a glowing and wonderful sense that the American people couldn’t be ticketed by polls [and] knew its own mind.”

No one relished this epic fail more than the distinguished Columbia University political scientist Lindsay Rogers. For years, Rogers had been thundering about the unreliability of polling and — more importantly —the groundless faith that people placed in it. In an exquisite bit of timing, Rogers published a book in 1949 entitled The Pollsters. (The coinage, wags noted, evoked the term hucksters, though Rogers denied any intentional allusion.) Rogers’ polemic was a rebuttal, of sort, to a book that Gallup himself had published a few years earlier, called The Pulse of Democracy.

In his book (written with Saul Forbes Rae) Gallup had insisted that “scientific” opinion surveys were the best means ever devised to gauge the public’s desires, and thus to serve democracy. Science was the key word. Gallup placed himself in the company of chemists and physicists, men in white coats. “Measuring public opinion calls for a certain ‘laboratory’ attitude of mind,” he wrote. “It needs people trained in the scientific method.” He boasted of his rarefied knowledge of statistics — his book cited the probability theories of the 17th-century mathematician Jacob Bernoulli — that placed his work beyond the ken of lay citizens. He insisted his work was purely a matter of number crunching, devoid of interpretative coloration. He scoffed at naysayers who put quotation marks around “scientific” when it modified “polling.” “If our work is not scientific,” Gallup wrote, “then no one in the field of social science, and few of those in the natural sciences, have a right to use that word.”

Rogers would have none of it. A PhD from Johns Hopkins, the holder of a named chair at Columbia University and a former journalist, he commanded authority among academics and the literate public alike. A courtly man with Old World airs, he ranged widely in his work and relished controversy. In November 1941, he had written a long article for Harper’s trashing Gallup and deflating “the exaggerated claims that are made as to what the data mean.” Since then he had been on a crusade to explode the pollsters’ pretensions that they could pin down the public mind with any sort of precision.

Many of Rogers arguments dealt not with forecasting but with issue polling — the question of whether elected leaders should mind the polls in deciding which policies to adopt. But another set of his complaints centered on the intractable difficulties involved in obtaining truly objective information from survey methods at all. By now it was well known that the nature of sampling, the wording of questions, the kinds of answers that respondents were allowed to offer and the methods of tabulating them were all capable of introducing errors or producing misleading outcomes.

Methods could of course be tweaked and even improved (though it should be noted that Gallup and other pollsters went on to misjudge, by margins wide and small, the elections of 1952, 1968, 1976, 1980, 1996, 2000, 2004, and 2012 — hardly a proud record). At bottom, though, Rogers’ critique wasn’t methodological. At a philosophical level, he rejected the very idea that public opinion was measurable in the concrete way that the pollsters alleged. Public opinion was too inchoate to lend itself to precise measurement, even when fine-tuned with open-ended questions, scales of intensity and other methodological tweaks that had been introduced over the years. Public opinion, he said, wasn’t like distance or mass or other scientifically measurable phenomena; it had no freestanding existence apart from the operation of measuring it. Polling thus pretended to quantify the unquantifiable. Like others in the increasingly data-driven social sciences, Rogers charged, the public opinion analysts were following false gods of methodology. Properly understanding the public required not pseudo-scientific methods but human insight.

Along with many others, Gallup pushed back against Lindsay, calling him “the last of the arm-chair philosophers in this field.” And while Gallup’s name, owing to his lucrative polling business, endured through the decades, Rogers’ faded into relative obscurity. Political science became inexorably more quantitative and data-driven, leaving behind his concerns about its pretensions to scientific status. Moreover, the profits that commercial pollsters reaped — alongside, perhaps, Gallup-like hopes of improving democracy — ensured that the practice of election-season survey-taking would not subside anytime soon. Over the years, critics from both the world of journalism (columnist Mike Royko, polemicist Christopher Hitchens) and academia (political scientist Benjamin Ginsberg, journalism historian W. Joseph Campbell) have kept alive Rogers’ skepticism, but on the whole Americans have continued to be seduced every election season by the pollsters’ allure.

Perhaps in light of two successive flameouts by today’s George Gallups, Lindsay Rogers’ arguments about the impossibility of measuring public opinion with scientific precision should get a new hearing.

In the post-mortems after the shocker of 2016, there were complaints about flawed models and technical adjustments; some pollsters defended themselves by pointing out their narrow calls for Hillary Clinton were within the margin of error. In 2020, with the pre-election polling in Florida and other contested states so far off the mark — and with outlandish recent results like Wisconsin +17 for Biden reported by even A-rated firms — there are cries that something deeper has gone wrong, that polling is now somehow broken, because of technology, or methods, or current politics.

But Lindsay Rogers might have had a more fundamental critique than that: The idea of political polling was broken to start with. It was a falsely scientific way to put numbers on a concept that can’t be measured in the first place, and which changes shape every time you try. And indeed, it is the very elusiveness of political opinion — its resistance to being pinned down — that makes democracy necessary. When we measure mass or distance, we know we can do so accurately. But our values, attitudes and opinions are not concrete but fluid. They change with time — in the days and weeks before an election, as well as in the years in between them. Which is precisely why democracy requires that every few years, we vote anew.

View original post