What you are about to read is not a comment on 2024 polling but rather a musing on how systems are slow to react to bad actors.
Polls in 2022 were wrong. I think that is a fairly well understood phenomenon but this recent Ars Technica article on how hard it is to do polls well in the modern age brought that fact to mind because of what it says about how our political press and world has or has not reacted to this difficulty and the recent failures of polls.
A brief digression. You will find articles like this one at 538 that suggest the polls did well. I don’t find their point convincing, in large part because they only look at polls close to the election, which, as you will see in a bit, is potentially missing the forest for the trees. They also state that predicting a winner is a poor measure of how accurate a poll is, and their measures of accuracy are largely based on how close to the margin of error a poll was, not how close to reality a poll was. There is some truth to these measures, but in a closely divided polity, being within the margin of error really has no substantive value. It is just an admission that polls are largely ineffective.
It is the problems with individual polls that supposedly drives the power of aggregates like Real Clear Politic and 538 itself. But those aggregates are also the root of the problem for political journalism. It is clear in retrospect (and people like Simon Rosenberg saw it in real time and thus were more accurate about 2022 than other media) that Republican pollsters spent much of the year flooding the aggregates with favorable polls. That effect diminished as the election got closer, but it certainly skewed the coverage of the political media in the run up to the election.
This appears to be a deliberate strategy, and it is one that makes sense. Our political media is terrible, focusing significantly more on horse race coverage — coverage of who is wining or losing — rather than on issues and consequences. They largely ignore the polices of the leading candidates and what effect those polices can be expected to have on the nation. Polls, whether their own in house or the aggregates, play a leading role in that coverage. They appear scientific and unbiased, despite their significant problems. By flooding the aggregates, then, dedicated people can skew the coverage.
In house polls are likely just as harmful. The Ars Technica article mentioned above shows just how hard it is to do good polling. I encourage you to read the entire article, as it is fascinating, but the core issues are that people simply do not answer phones as often as they did in the past and the methods used to get around that issue are either extremely expensive, subject to fraud or trolling, don’t generate the same randomness/accurate samples that polling requires on, or all three. Polling is just very hard. But because polling is very hard, and because the polity is so evenly split, the polls and aggregates that drive political coverage makes it likely less accurate rather than more accurate.
There is a concept called Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.” Given how skewed toward the horse-race political coverage remains, aggregates are largely useless. As far as I can tell, neither 538 nor Real Clear Politics, the two most widely used and accepted aggregates, has taken steps to weed out operations that appear to be operating in bad faith. It is easy for a firm to adjust its numbers closer to reality near the end and blunt the “punishment” they receive. 538 merely graded down some pollsters after 2022, as an example, rather than remove them. Polling aggregates have become a target, and so they are no longer a measurement.
This is not to say that all polls are skewed or part of a plot to make one side or the other look bad. Many organizations do try to produce reliable polls. But poll reliability is extremely difficult to achieve, and the aggregates are vulnerable to organizations deliberately flooding them with bad data. This should have resulted in changes in both how the aggregates work and how politics are covered. Reporters should spend much less time on the horse race, as they don’t have a good way to accurately judge that race, and more on policies and impacts. Aggregates should be ruthless about banning or otherwise discounting orgs that appear to be flooding the zone with poor quality polls for political advantage.
Neither of those appear to have happened. Neither of those groups — political reporters or polling aggregators — appear to be capable of making those changes. Aggregators talk about the value of polls in terms, as we have seen, not of usefulness to readers but in mathematical abstracts. Political reporters stay focused on horse race material, even pushing back on the idea that they should do things differently. And of course, making those kinds of changes are difficult. Aggregates would open themselves up to charges of bias, and reporters would have to learn about policy to report on it effectively. Those are hard things to achieve and likely would cost more money than what they are doing today. As a result, we get horse race coverage that cannot accurately see where the horses are.
Political coverage is a good example of how organizations and cultures change slower than reality. This is an obvious and ongoing issue, and there are not easy answers. Changing incentives can be difficult. A press insulated from commercial requirements, for example, might be more willing to look at tissues and less willing to indulge bad behavior by polling aggregates. The success of places like Pro Publica suggests that non-profit media might be one way out. Pressure by the public and journalists to reform the organizations can also help. But for now, media consumers are simply going to have to understand that the horse race reporting they are receiving is not based on a clear view of the track.
Spot on. The other aspect you ID (supine 'horserace' corporate media) reflects the sheer 'fad-ness' of American life, where much serious energy and serious money is expended on professional sports journalism. Example 'the sky is falling' pile-on after Biden's performance in no way reflected the simple fact that wasn't a debate. And, both considered, seems to me the determining factor is the remarkable dearth of critical thinking in American public life...and I'd include SCOTUS's latest, to me, in turn, a subset of American exceptionalism (GW Bush, I'm looking at you) in the face of war crimes prosecutions. Talk about slow (in the US case, utterly resistant!) to responding to bad actors, beginning with Pinochet through Putin and both Netanyahu and the Hamas leadership. Thank you.