Experiment: Figuring out why the statistics don’t seem to tell the whole story.

Here are two charts of the same question showing the responses according to political party affiliation.

These come from the PDK/Gallup Survey of Education.

I’ll admit some confirmation bias that I expected the results to be bad, but I did not guess the difference in responses by party affiliation. Unfortunately, even though this was conducted by Gallup with a huge sample, it was not a particularly robust survey. The question was not stated clearly enough and the survey has little construct validity. This means the findings are unlikely to generalize to the wider US population, even if the results seem very clear.

Let’s take a look at what the chart seems to explain. The overall figures are that, for all Americans, 56% favor providing education for the children of undocumented immigrants and 43% of Americans oppose with 1% responding that they don’t know or are confused (I did not include this chart separately, it is the lighter orange and blue bars in the background). When we split it up by party affiliation, 85% of Democrats favor and only 27% of Republicans favor. Only 14% of Democrats oppose and an incredible 73% of Republicans oppose. It seems clear, generally Americans are pretty mixed on providing education, and Republicans really hate it. It doesn’t “ring true to me” and there is at least one severe issue with the survey.

Construct validity is how well the way our measurement actually fits what we intend to measure. For example, it is highly flawed to ask people their opinion about the quality of food at a restaurant they’ve never visited. But, if the construct we hope to measure is the perception of the restaurant, then it is a fine question.

Making the connection to the question about education, the construct is not well enough defined and it is akin to asking whether or not a random person believes the restaurant should be allowed to stay in business or not. What opinion is the survey actually measuring? Both the restaurant example and the education survey have a huge oversight in the question and some big assumptions behind what is asked that threaten the validity of the results.

Here is what I wonder; Did the respondents in this survey realize that the “children of immigrants who are undocumented” might be American citizens? If it was not clear, then what was probably measured was how many respondents assumed that the children were illegal AND believed that illegal immigrants should not receive public education. Looking at the charts, I am much more likely to believe that 73% of Republicans assumed that the children in question are illegal residents of the US. But I doubt that 73% of Republicans believe that we should put a limit on who receives public education that includes denying education to citizens.

I wonder how Americans would respond if the survey asked “Should all Americans be provided with a public education regardless of their ethnic origin?” I feel like that would have very high agreement. Then a separate question could be “Should undocumented immigrant children be provided with a public education?” I can only guess at the results. This survey did not have a clear enough definition of the construct of “children of immigrants who are undocumented” to make any legitimate claims about the opinion of Americans.

The reason this is important is that statistics, even if they have low construct validity, are used to determine policy. Looking at these results, someone could claim that 43% of Americans oppose educating children of undocumented immigrants and put into place policies that force schools to check the legal standing of parents as they attempt to enroll their children in school. This would be a huge burden on the education system in the United States (ICE is already overwhelmed with this without having to also hold classes), would likely keep students who are legal citizens out of schools, and create  an uneducated underclass of children, many of whom would be immigrants. I can’t imagine anyone would favor that, but without construct validity, we have no way of knowing. Before we make any policy, we have to ask the right questions!