Bacon's Rebellion

Democracy Thrives in Sunlight

Be Careful with Survey Results, Especially About Education

—

by

sherlockj
in , ,

Credit: iStock

by James C. Sherlock

I am a prolific reader and analyst of statistics about education. I find it constantly necessary to sort the wheat from the chaff. Chaff is the term I have chosen for this article, not the one I use in private.

The results of the latest Education Next Survey of Public Opinion (2022 poll) will appear endlessly in the press. EdNext has even graciously provided headline-ready assessments based on those results.

  • “Partisan Rifts Widen, Perceptions of School Quality Decline“
  • “Parental Anxieties over Student Learning Dissipate as Schools Relax Anti-Covid Measures”
  • “Hunger for Stability Quells Appetite for Change: Results of the 2021 Education Next Survey of Public Opinion”
  • “Parent Poll Reveals Support for School Covid-Safety Measures Despite Vaccine Hesitancy, Partisan Polarization”

Read the “Methodological notes” under “Notes” on the survey. Please also note that the links within those notes are both broken.

The parent sample includes oversamples of parents with at least one child in a charter school (305 respondents), parents with at least one child in a private school (310 respondents), Black parents (283 respondents), and Hispanic parents (429 respondents). The completion rate for this survey is 50%.” [Emphasis added.]

The judgments made from those data are breathtaking.

But are they justified by the data?

This is the type of chart that is created from the data.

Partisan Rifts Widen, Perceptions of School Quality Decline: Results of the 2022 Education Next Survey of Public Opinion

Dazzling.

So I assessed the admitted sampling errors among just parents.

  • 1,327 of 1,857 (71%) of parent responses represented parent groups oversampled;
  • Parents with one child in a private school oversampled by a factor of 2: survey 16.7% versus census bureau 8.2%; and
  • parents with at least one child in charter school oversampled by at least a factor of three: survey 16.4% versus actual of less than 5.4%.

Note: the percentage of children in charter schools nationwide — 5.4% — does not account for parents with more than one child in charter school, which makes the parents with children in charter schools figure less than 5.4%.)

Those sampling errors seem formidable obstacles to producing such a chart.

Then there was the 50% completion rate. From the American Association for Public Opinion Research

Who refuses?

From the perspective of survey error, the principal concern regarding refusals is the possibility that sampled refusers are systematically different on the measures of interest in a particular survey than sampled participators across a range of demographic, attitudinal, and/or behavioral variables, so that their self-exclusion may bias results to a non-negligible extent.

The authors of Partisan Rifts Widen, Perceptions of School Quality Decline are successful academics. I am going to assume that they assessed this survey to be within the bounds of the rules of the American Association for Public Opinion Research when they ventured their sweeping analyses.

I personally don’t agree with using these flawed data for these purposes, but that is what makes a republic. People take in information, and then they vote.

As I wrote above, I am careful about surveys. I recommend being particularly careful about this one — and the press articles derived from it.

I put this survey in my personal “chaff” category.

The assessments derived from it I will call flares, burning bright. Briefly.

Share this article

ADVERTISEMENT

(comments below)

ADVERTISEMENT

(comments below)

Comments

28 responses to “Be Careful with Survey Results, Especially About Education”

  1. vicnicholls Avatar
    vicnicholls

    Another great article Capt.

  2. LarrytheG Avatar
    LarrytheG

    I actually agree with the premise here. There are polls and there are polls and some are so obviously skewed as to be a joke.

    That’s why it’s important to determine if the pollsters are professionals who do polling all the time, independent from the issues and groups, and fully disclose their methodology.

    Even then, how the questions are asked, who is asked, and response rate and context is important and look at several polls not just one.

    There are several well-known and reputable polling organizations including Pew and Gallup.

    Doesn’t mean they get it right 100% of the time either , that’s why you look at other polling.

    1. James McCarthy Avatar
      James McCarthy

      Polls are not taken or designed to “get it right,” only to take the temperature on a topic at a given time like Casey Kasem’s top ten.

      1. James C. Sherlock Avatar
        James C. Sherlock

        Good to know.

      2. Lefty665 Avatar
        Lefty665

        You are wrong. Good polls by good surveyors are designed to accurately reflect opinion on a topic at the time they are taken. That is also known as “get(ting) it right”.

        To the extent that Kasem’s top ten were based on data, record sales for example as opposed to payola, they were an accurate reflection.

        You do a disservice to a profession that includes skilled professionals as well as some who are shoddy or biased. In that way surveyors are much like lawyers.

        1. James McCarthy Avatar
          James McCarthy

          No poll indicates an error of opinion. If folks believe poll results are binary choices between right and wrong, this article is unnecessary. By and large, the 2016 presidential polls did not get it right regarding Trump. In 2019, Trump was characterizing negative poll results as phony. Sorting the “shoddy and biased” from the skilled professionals is not guided by a list, leaving consumers to make their own judgments. Some shoddy lawyers like Giuliani, Eastman, Powell, and Ellis will, in fact, be rated by their respective bar associations. That’s something.

          1. Lefty665 Avatar
            Lefty665

            No poll indicates an error of opinion. No one has suggested they do, that’s a straw man.

            A poll can tell you with reasonable accuracy, for example, that 17% of respondents believe the world is flat or that Kamala is doing a great job. We are all free to draw our own conclusions about those accurately reported opinions.

            It was interesting in the last couple of presidential races that the aggregates of polls did a pretty good job of predicting the popular vote. Some polls were predictably biased for either Repubs or Dems, and some were just wrong. But when aggregated the errors largely offset and the results were reasonably close.

            We can be guided by polling history, How has a poll fared over the years? Has it been relatively close to election results or has it consistently favored one party or another? That is a judgement based on observable facts that we can individually make with high confidence and a small margin of error.

            We can make the same kind of evaluation of high profile lawyers without the need of other lawyers assessments. I would add Elias and Sussman to your list and some others on both sides of the aisle.

          2. Lefty665 Avatar
            Lefty665

            No poll indicates an error of opinion. No one has suggested they do. That is either a straw man, or a failure to understand polling.

            A poll can tell you with reasonable accuracy, for example, that 17% of respondents believe the world is flat or that Kamala is doing a great job. We are all free to draw our own conclusions about the validity of those accurately reported opinions that may reflect “binary choices between right and wrong.”

            It was interesting in the last couple of presidential races that the aggregates of polls did a pretty good job of predicting the popular vote. Some polls were predictably biased for either Repubs or Dems, some were just wrong and a few were pretty close. When aggregated the errors largely offset and the results were reasonably close.

            We can be guided by polling history, How has a poll fared over the years? Has it been relatively close to election results, has it consistently favored one party or another, or has it just been wildly wrong? That is a judgement based on observable facts that we can individually make with high confidence and a small margin of error.

            We can make the same kind of evaluation of high profile lawyers without waiting for lawyers associations assessments. I would add Elias and Sussman to your list of shoddy (or worse) for balance and some others on both sides of the aisle.

          3. James McCarthy Avatar
            James McCarthy

            Again, polls exist to measure opinion at a given time. All employ a margin of error as recognition that the exactitude of the results is not 100%, right or wrong, thus not a simple binary choice. Polls taken simultaneously may produce different results on the identical topic. Taken at a different time, polls may produce radically different results on a previous topic. At every juncture, individuals making decisions based upon poll results are making a wager. All the polls in all the world on the same topic at the same time will produce different results. They may be designed scientifically to be accurate but cannot offer certainty except with a margin for error. Polls are expressions of opinion very much like material on BR.

    2. john b harvie Avatar
      john b harvie

      “how the questions are asked, who is asked”

      Bingo, Larry. You got it in the proverbial nutshell.

  3. Kathleen Smith Avatar
    Kathleen Smith

    Great analysis. Politics needs to stay out of Education. This is a great example of the spinning by the divide.

    1. James C. Sherlock Avatar
      James C. Sherlock

      I honestly don’t object to someone drawing conclusions from good data. I just find opinion survey standards to be generally beneath human standards of sufficiency.

      My research over a number of years is that those standards are cost-driven. Restated: If the survey doesn’t meet standards and you have spent all of your money, lower the standards.

      For these guys to draw such sweeping conclusions from this survey means the survey results should have been locked in a safe somewhere.

      The issue is on the one hand that surveys are expensive.

      On the other, the surveyor needs to make sure the response demographics match reality closer than these did or adjust them with an algorithm.

      If this survey manager had adjusted the results mathematically, she would have reported it. As you see, she did not. That is an extra step that costs more money.

      Surveyors have convinced themselves that what they have is good enough. They normally publish margins of error. I could not find that in the case of this survey – just bold headlines about the implications of the survey.

      As you have read, I do not concur with either the unadjusted results of this survey or the sweeping implications drawn.

      As I wrote. It is a free country. They can publish whatever they wish.

    2. James C. Sherlock Avatar
      James C. Sherlock

      I honestly don’t object to someone drawing conclusions from good data. I just find opinion survey standards to be generally beneath human standards of sufficiency.

      My research over a number of years is that those standards are cost-driven. Restated: If the survey doesn’t meet standards and you have spent all of your money, lower the standards.

      For these guys to draw such sweeping conclusions from this survey means the survey results should have been locked in a safe somewhere.

      The issue is on the one hand that surveys are expensive.

      On the other, the surveyor needs to make sure the response demographics match reality closer than these did or adjust them with an algorithm.

      If this survey manager had adjusted the results mathematically, she would have reported it. As you see, she did not. That is an extra step that costs more money.

      Surveyors have convinced themselves that what they have is good enough. They normally publish margins of error. I could not find that in the case of this survey – just bold headlines about the implications of the survey.

      As you have read, I do not concur with either the unadjusted results of this survey or the sweeping implications drawn.

      As I wrote. It is a free country. They can publish whatever they wish.

    3. James C. Sherlock Avatar
      James C. Sherlock

      I honestly don’t object to someone drawing conclusions from good data. I just find opinion survey standards to be generally beneath human standards of sufficiency.

      My research over a number of years is that those standards are cost-driven. Restated: If the survey doesn’t meet standards and you have spent all of your money, lower the standards.

      For these guys to draw such sweeping conclusions from this survey means the survey results should have been locked in a safe somewhere.

      The issue is on the one hand that surveys are expensive.

      On the other, the surveyor needs to make sure the response demographics match reality closer than these did or adjust them with an algorithm.

      If this survey manager had adjusted the results mathematically, she would have reported it. As you see, she did not. That is an extra step that costs more money.

      Surveyors have convinced themselves that what they have is good enough. They normally publish margins of error. I could not find that in the case of this survey – just bold headlines about the implications of the survey.

      As you have read, I do not concur with either the unadjusted results of this survey or the sweeping implications drawn.

      As I wrote. It is a free country. They can publish whatever they wish.

      1. Lefty665 Avatar
        Lefty665

        “As you have read, I do not concur with either the unadjusted results of this survey or the sweeping implications drawn.”

        But Admiral, that is your favorite MO, why are you so harsh when others adopt it? Did you think you had a copyright on the technique?

  4. Kathleen Smith Avatar
    Kathleen Smith

    Great analysis. Politics needs to stay out of Education. This is a great example of the spinning by the divide.

  5. Nancy Naive Avatar
    Nancy Naive

    Well, now that you know this, how does that affect your acceptance of the results of the VBSB superintendent’s ad hoc survey you touted not a week ago?

    1. Lefty665 Avatar
      Lefty665

      But, but dontchya know the Admiral said “I am careful about surveys” in his original post above and that makes it all ok.

      1. James McCarthy Avatar
        James McCarthy

        And this article is a Sherlock Exception to the BR principle publishing articles about Virginia. Now all have been carefully taught to be wary of opinion sruveys.

        1. Lefty665 Avatar
          Lefty665

          ” Now all have been carefully taught” too funny, another Jim McCarthy silly walk.

          Do any of us really need to be “carefully taught” to be wary of what we read by the Admiral? Don’t believe three quarters of what you hear and half of what you read. I learned that at my dad’s knee when Ike was president.

          Surveys like surveyors vary. Some are very good, others not so much. It’s not rocket science to tell them apart.

        2. Lefty665 Avatar
          Lefty665

          ” Now all have been carefully taught” too funny.

          Do any of us really need to be “carefully taught” to be wary of what we read by the Admiral? Don’t believe three quarters of what you hear and half of what you read. I learned that at my dad’s knee when Ike was president.

          Surveys like surveyors vary. Some are very good, others not so much. It’s not rocket science to tell them apart.

          1. Nancy Naive Avatar
            Nancy Naive

            Will Rogers — none of what you hear and half of what you see.

            Lucky the poor bastard never heard about CGI. But then, by his own scale, he wouldn’t have believed it.

          2. Lefty665 Avatar
            Lefty665

            Rogers was great! His commentary on modern graphics would be interesting, and scathing.

            Another of his that I have liked and is scarily appropriate today is “I used to worry about what he (a politician) didn’t know. But here recently it’s what he knows for sure that just ain’t so that scares the Hell out of me”.

  6. Eric the half a troll Avatar
    Eric the half a troll

    Methodologies aside, 95% have not attended a school board in the past year. Probably accurate but a shame, nonetheless. You want to “fix” the schools, start there.

    1. Baconator with extra cheese Avatar
      Baconator with extra cheese

      I agree. Everyone should watch a Richmond School Board meeting. You will quickly understand that the dysfunction comes from the top down.

    2. Nancy Naive Avatar
      Nancy Naive

      And the other 5% went to shout about something they saw on Twitter… about Texas.

  7. LarrytheG Avatar
    LarrytheG

    Much of the narrative seems fair and objective and based on trend surveys over the years and somewhat consistent with other survey’s from other organizations.

    I suspect their conclusions and judgements are not in line with Sherlocks “public education sucks and is failing” beliefs though.

  8. James McCarthy Avatar
    James McCarthy

    HELP!!! I can’t identify the connection of this article to Virginia.

Leave a Reply

Read More