Sean Trende: Polls Come With Error Margins and Response Bias, Let the Data Speak for Itself

Intro but Sean you wrote about a there's a debate that's going on there was actually a fight a bit of a fight between writer at the Washington Post and uh Jesse Waters of Fox News talk a little bit about this and what what is the debate about what's the fight about The Washington Post vs Jesse Waters and what can we learn from it so it seems like we have this fight every presidential election from one side or the other and every four years we say don't do that and people do it anyway uh it it has to do with the cross tabs of polls you know pollsters good pollsters will report okay here's the Trump Harris numbers and then you can look and say well but you know Trump is getting x% of Republicans Harris is getting x% of Democrats or here's how the black vote is shaking up and every year people who don't like the poll report results try to unsew what we call unsew them um and so Waters does a little bit of this he says well look at these samples it's like d Plus 7 d+4 last time I looked Republican Advantage was like r+2 2% more Republicans than Democrats so these polls are going to kill you know if this is what the polls are showing now Trump's going to kill Harris and bump weighs in and says well no you can't do that um but in the process I think uh The Washington Post reporter makes a little bit of an error and what is the error I mean what you know I see this all the time too that because part of what happens when pollsters you get your respondents and then you have to make the pollsters wait these respondents they're making basically you know uh educated guesses about who's going to show up in November what is the electric going to look like how many Republicans how many Democrats how many Independence how many young voters how many black voters how many Hispanic voters etc etc so what is the mistake of going in and people looking at this and saying oh well you know the the electorate last time is this and this poll has you know Oversampling is oversampling Democrats or oversampling Republicans and therefore if you just dial it to where it was you know in the last election then the number would be this yeah so I think the the not so good response as the one the Washington Post reporter makes which is basically trust the experts like posters do this a lot it's never a good response I mean sometimes look look sometimes it is if I'm going in for surgery the old thing like I'm I'm happy to defer to my doctor about how he'd like to conduct the surgery as a general matter as a general matter but polling but but here's the thing is that polling there's science behind it there is math but there's also an art to it um and so if your defense is you know well you know pollsters are really smart and make these educated guesses you're basically opening up yourselves to water saying well I'm really smart too and I think a D+ 7 electr it's just incredibly implausible where it gets where where I Rejecting outcomes think the smarter value is is saying what you really need to do is kind of let the data go where they lied first off if we're going to just reject outcomes that just don't seem right to us why do we do social science you know Democrats do the same thing when they say well there's no way Trump's going to get 15 or 20% of black respondents well it's shown up in poll after poll you know why are we taking polls if you're going to reject uh certain outcomes that don't feel right and the same thing is true if a poll comes out with a D+ 7 electorate especially since party identification is malleable it's something that can change from day to day um and is very sensitive to how we ask the question if we ask about registration what you consider yourself how you frequently vote you're going to get very different answers across the board so it's just it's ultimately a loser's bet uh trying to guess exactly what the partisan composition of the Losing faith in polls poll should be and adjust it accordingly uh you know Sean there is I I do think there is something to this idea though that that people have lost faith in polls and and because of some of the misses that have taken place over the last few cycles and you know 2016 2020 in particular I think it is fair to to question polls this time around because in 2016 you know some of the the A+ rated pollsters or some of the the better you know bigger named pollsters um had some issues and they took a look at it and thought they fixed it in 2020 but actually they didn't it was worse in some circumstances and so to just blindly say oh they fixed it in 2020 in these po you know I I get totally get that right um but at the same time you know I and I get this all the time I just when I report these numbers on Twitter I'll get people oh you can't trust it I'm like well you can't just you know three weeks ago in full said Donald Trump was leading by Polls come with error margins Four Points nobody questioned it and now it says KLA Harris is at Four Points and suddenly we need to throw them out of our average and never use them again because you know there have to be consequences for getting it wrong and so it is a tricky thing but I totally get it and I don't know if pollsters will ever get their the the trust back from you know a majority of the public do you think that's possible you know I think it's one of those things it's very hard to put the horse back in the barn right I mean it it's you're right 2016 and 2020 were were disasters for the polling industry although people forget that in 2012 the polls were also off the opposite direction um you know polls do come with error margins and I do feel that as an industry we don't do such a great job of explaining those error margins or why aggregation is so important and even after you aggregate you can have some systemic issues um and I think there are actually good reasons for having some skepticism of the partisan composition of the polls right now also right before the Convention as Response bias well which is that when the new cycle favors one party or the other strongly a lot of times the the partisans become a lot more receptive to talking to a poster right response bias yeah response bias after after the debate and the assassination attempts Republicans wanted nothing more than to talk about the presidential election right I found that in my day-to-day life on Facebook it's true for pollsters as well now the opposite is true like Democrats cannot stop talking about it because they just they just survived a Near Death Joy is back their own yeah Joy is back it's not hope and change it's Joy um and so I I wouldn't be surprised if right now polls are over sampling Democrats a bit but that's something that works itself out uh over time the caution I would put about 2016 and 2020 is you're right like it is it you you should not exclude the possibility that there's a hidden Trump vote that we're not accounting for at the same time pollsters do change their techniques to try to make up for certain um for bad elections uh and you're put Hidden Trump votes in an awful lot of eggs in a basket where there's ultimately two right 2016 2020 right that's our data set bad things ha I mean random things happen to in pairs all the time yeah it's and it's one of those things I I keep telling Carl it's like one of it's the Known Unknown right we know it's a problem but we won't know until after election day which whether whether they're right or not but how do you think that um that holsters should be um they should be providing all of their data and cross tabs and should be you know doing this or does that encourage people to get in there and dig because when you start digging in the in subgroups where the margin of Errors get you know ridiculously large that that isn't necessarily a good thing either yeah I mean when you're talking about if you have a poll of 500 people there's probably 60 black respondents in it which means your error margin is going to be like Plus - 8% Harris vs Trump it's silly um but or sorry like 16% but um and just so people know that that that means if if it let's say it's you know Harris 60 and Trump 30 or something right it could be as much as you know minus 16 on Harris's side and it's like a 32o swing potentially yeah we can't say with confidence that Harris really is above Trump or she could be like way way way ahead of trump but the key thing to keep in mind there um at the end of the day is that you know we do the social science for a reason and these error margins are real they can complicate things I do think it's good a best practice for pollsters to release things but at the same time you know if you have ABC News funding you it's a lot easier to just put it all out on the table but if you have private clients if you really are riant on your own uh kind of uh trademarked uh way of doing the poll if you release all your data and your questionnaire and everything then there's nothing special about you anybody can do what you do so I do understand the proprietary concerns for smaller pollsters not wanting to release that stuff all right we will leave it there for today that was pretty weedy discussion of the polls but that's what we do we're nerding out all right Sean treny senior elections analyst for Rook politics thanks for your time appreciate it thank you all right going to take a quick break and we'll we'll be right back after this you're listening to Rook Clair Politics on series 6m