Bacon's Rebellion

Democracy Thrives in Sunlight

Political Damage to Primary Instruments of Improvement in Virginia Schools

—

by

sherlockj
in ,

by James C. Sherlock

We need all the help we can get assessing Virginia schools and producing actionable information to make them better.  

The Standards of Learning exams show the results of poor learning, but do not identify actionable causes.

Directed to choose an additional measure of school quality by the federal ESSA (Every Student Succeeds Act) in 2015, Virginia, like most states, chose to use a school climate survey.

Virginia chose a hell of a good one developed in Virginia by University of Virginia scientists for the federal government. It was used here very successfully for several years.

Then Virginia Department of Education (VDOE) functionaries dumped it in 2020 for what were clearly political/philosophical reasons.

I have not seen any evidence that the new survey has been proven valid or reliable, even if one agrees with their woke philosophy. Frankly it could not have been validated in such a short span of time in the middle of COVID.

The new leadership at VDOE needs to return to the original.

Virginia’s pre-woke school climate survey was called by its developers Authoritative School Climate Survey (Authoritative School). 

It has proven to be an instrument of improvement for those that have acted on its results. There are dozens of studies supporting its validity, the U.S. Dept of Education placed it on its list of approved surveys, and researchers around the world have adopted it.

The results of that survey in 2019 at Walker-Grant Middle School in Fredericksburg were predictive of dreadful chronic absenteeism in that school in 2020-21, and massive learning losses in 2021-22.

In 2020, the same survey found the learning climate far better at Fredericksburg’s James Monroe High. It accurately forecast that COVID-year attendance would be much better at James Monroe and the learning losses far less extreme than at Walker-Grant.  

Then Virginia stopped using it.  

An opportunity to change Authoritative School was provided when VDOE and DCJS in 2020 combined their two existing survey efforts into a single survey instrument and administration cycle.

Richmond bureaucrats scrapped the scientifically developed and validated Authoritative School for a new, untested one they “developed” in the middle of COVID in 2020. They have administered it since 2021.  

That new one is Virginia School Survey of Climate and Working Conditions (Virginia School Survey).

It is woke, but it also appears useless for the purpose.  

Pre-COVID Authoritative School Climate Survey results  in Fredericksburg 

  • Walker-Grant Middle School 2019

  • James Monroe High School 2020

The key scales presented in the reports in Authoritative School are scaled scores designed to make them easier to interpret. For each scale, raw scores for student and staff participants within a school were averaged to create school means for students and for staff. The school means across the state were transformed so that the state average is 10 and the standard deviation is 1.

As a result, schools with scores between 9 and 11 are in the average range. Schools with scores of 11 or higher are statistically above average and schools below 9 are statistically below average and show a need for improvement.

Walker-Grant Middle was awarded student scores that averaged 7.8. James Monroe 9.54.

The staff-awarded scores in Walker-Grant Middle averaged 9.28.  At James Monroe 9.0. Both were in the lower quarter of the average zone, but they were in it. So, the staffs were relatively sanguine about their schools. Only those at James Monroe were proven relatively correct.

The huge disconnect between the staff and the students’ opinions of school climate at Walker-Grant was a big deal. The staff was clueless about how the students felt.  

And the students were proven prescient in their assessments.

Operational results during and after COVID – same schools

So let’s see if the survey results predicted operational results:

Chronic absenteeism in the 2020-21 school year. The entire division turned in terrible results, but James Monroe was by far the best.

  • Walker-Grant 73.6%.
  • James Monroe 46.9%.
  • Division as a whole 70.5%.
  • State 11%

Chronic absenteeism in 2021-22. James Monroe approached state norms. Walker-Grant doubled the state average rate of chronic absenteeism.

  • Walker-Grant 41%,
  • James Monroe 24%.
  • Division as a whole 36%.
  • State 20%

Math SOL pass rates 2021-22. James Monroe was near state averages.

  • Walker-Grant 27%
  • James Monroe 62%
  • Division 40%
  • State 66%

Reading SOL pass rates 2021-22. James Monroe exceeded state averages.

  • Walker-Grant 50%
  • James Monroe 82%
  • Division 55%
  • State 73%

So, there can be no question what happened here.

Walker-Grant Authoritative School results showed that the middle school was in trouble before COVID hit, and that such a weak and chaotic school climate could not sustain the stress of COVID.

The Walker-Grant students nailed the problems there in the 2019 school climate survey. The staff at that school missed them.

The Authoritative School surveys used from 2013 to 2020 had been painstakingly and iteratively developed and extensively assessed as both reliable and valid. And they worked.

The surveys at both Walker-Grant and James Monroe were predictive. They provided actionable information for Fredericksburg schools.

The politics of school climate surveys. 

Authoritative School  found a place on the list of climate surveys validated by the U.S. Department of Education.

We have the ability to use those surveys to make informed decisions on which schools need improvement and in what areas, not just from test scores but from excellent school climate assessments.

But in 2020 VDOE decided not to use it any more.

The Richmond coup. School climate surveys are high-stakes measurements. Careers rise and fall based upon the outcomes. Strengths and weaknesses are exposed. In the best schools and divisions, the information they provide is used to improve the learning climate.  

So, the results had better be valid and reliable. 

All education researchers and statisticians are trained to know that surveys used for high-stakes measurement should be rigorously evaluated for reliability and validity, and that a change in the items means that reliability and validity must be re-established.

VDOE, with the lead on designing the Virginia School Survey, seems to have ignored the Standards of Educational and Psychological Testing which is the Bible developed by the American Educational Research Association and the American Psychological Association.

The team that met in Richmond to redefine the climate survey, either from home or masked at work, had no chance during COVID to design and test a survey that measured school climate. So, they just started using it.

They appear clearly to have tried to make the new survey more reflective of progressive values. 

The Virginia School Survey questions document eliminated everything from Authoritative School that did not square with their sensitivities. And they added their own questions (items). 

Fine – elections have consequences.  

The bureaucrats were perhaps carrying water for the administration. And maybe a little of their own. But whatever they were trying to accomplish, they did not do it right.

Let’s examine some of the carnage:

  • The word “fair” is used once in the Virginia School Survey.  Agree/disagree:

The school rules are fair.

  • That question replaced the original question in Authoritative School Climate Survey: Agree/disagree:

This school consistently has high expectations for student behavior with strict and fair discipline.

  • The word strict is banished in the new survey.
  • The word “discipline” is used once in the new survey.  It is in a question about school resource officers, apparently the only adults who can be associated with discipline. In Authoritative School Climate Survey, “discipline” was used 35 times.
  • “Disciplinary” — gone in the new one. In the original, “Disciplinary Structure” was a major scale.
  • “Academic Expectations” was also a major scale in the original. There is no use of that term in the new student survey. Or even the term “expectations.” Dickens would be sad.

You get the point. VDOE and the Department of Criminal Justice Services (DCJS) took away an established tool in the middle of a crisis, showing at a minimum poor judgment and misplaced priorities.

Changing a reliable and valid survey in the middle of the COVID storm means that the schools are deprived of an instrument they could have used to measure the strengths and weaknesses of their schools and know what was needed to improve.

It is more than just unfortunate that we cannot compare the pre-COVID school climate survey results with the new ones administered starting in 2021. But we can’t. Apples and oranges.

The surveys are entirely different. But, also unfortunately, the results can be accessed on the same government web page as if they were coherent.

Elsewhere DCJS announces:

To reduce survey administrative burden, the Virginia School Survey combines two required survey efforts into a single survey instrument and administration cycle.

Informative.

Like telling Mrs. Lincoln that her husband would be delayed returning from the theater.

Maybe the new leadership of DCJS and VDOE will consider returning to the valid and reliable survey that the feds paid UVa scientists to develop and test — in Virginia.

The one that worked in Fredericksburg.

Share this article

ADVERTISEMENT

(comments below)

ADVERTISEMENT

(comments below)

Comments

31 responses to “Political Damage to Primary Instruments of Improvement in Virginia Schools”

  1. DJRippert Avatar
    DJRippert

    I get the distinct impression that almost everything done during the Northam Administration needs to now be undone.

    1. James C. Sherlock Avatar
      James C. Sherlock

      We, and I include myself who had 40 years of dealing with bureaucracies, sometimes forget how internally willful and patient they are.

      Bureaucracies can harbor grudges and favorite causes for years and wait until the iron is hot to strike.

      If they get an elected administration that gives them their head, they run for the roses.

      Same thing happens in reverse when they get bosses from an administration with which they disagree. They put up Klingon shields and wait them out.

      The new Superintendent of Public Instruction and Youngkin’s newly-appointed Board of Education will have to prove themselves street fighters to make progress in the face of a dug in bureaucracy.

      That starts with banning most ed school reps from the building, the folks from UVa that put together the Authoritative School Survey being honorable exceptions.

      1. Dick Hall-Sizemore Avatar
        Dick Hall-Sizemore

        This is not a question of bureaucratic recalcitrance. Both DOE and DCJS are headed by Youngkin appointees. All they have to do is say, “Use the climate survey that was used prior to 2020,” and it shall be done.

        1. LarrytheG Avatar
          LarrytheG

          We’re playing the “deep state” game where the new leader has to eradicate it root and branch before he can be assured of loyalty and compliance to do
          what he wants.

          Seen that playbook recently.

    2. James C. Sherlock Avatar
      James C. Sherlock

      This is about the Northam VDOE substituting their political sensitivities for a decade of science in survey development.

      I can’t say everything needs to be redone.

      But fortunately, all that needs to happen here is return to the valid and reliable survey they ditched. No additional time or money required.

      1. LarrytheG Avatar
        LarrytheG

        This is the same VDOE that Youngkin is leading and changing in other areas like transgender?

  2. Moderate Avatar
    Moderate

    Where is the middle of the road? Will we ever seek it again instead of making huge swings from extreme to extreme? Now that it seems that virtually everything is weighed through a political lens and folks feel a mandate to push the envelope as far as they can when they are in charge, I fear we’re just confusing kids and undermining real progress.

    1. James C. Sherlock Avatar
      James C. Sherlock

      All the administration needs to do is go back to the original survey. The government has already paid the bill. This is the middle of the road.

      There is no one drawing breath that thinks the Virginia School of Education and Human Development, which produced Authoritative School Survey, would produce a conservative product.

      It is just one that works. It was expensively and extensively developed and validated and has proven reliable.

      1. LarrytheG Avatar
        LarrytheG

        You like the data and you claim that if it is acted upon it can be beneficial but do you have an evidence of any schools that have acted on it?

        I just don’t see any real connection, just a “hope”.

        These survey’s are virtually unknown to the general public and parents, right?

        I bet less than 1 in 1000 in Fredericksburg are familiar with it.

        1. James C. Sherlock Avatar
          James C. Sherlock

          I’ll consider paying you to refrain from commenting on my work. Name a price.

          1. LarrytheG Avatar
            LarrytheG

            I made a very similar comment to Moderate made above. Go fish old grumpy man. You’ve got more than me chewing on you these days.

      2. Moderate Avatar
        Moderate

        As I read the discussion, it sounds like the original survey is defended as being against wokeness, etc. My point is if every time we switch administrations we have to first go back and change everything the prior administration did, progress will never occur. We’ll just use all our resources swinging from one direction to another. It costs money each time a survey is conducted – not just the first time.

        Reading both surveys, I’m not sure what either of them really help schools do. I’m not sure they identify the problems or effective ways to address them. The name “Authoritative” sounds ominous. I don’t think I would have responded well had I been asked to complete such a survey when in high school. How many kids take this seriously? How many even understand all the issues in play?

        Surveys aside, we’ve got to stop re-re-re-inventing the wheel, especially when we can’t figure out if any version creates useful and meaningful change.

        1. James C. Sherlock Avatar
          James C. Sherlock

          Glenn Youngkin ran and won on reforming educational policy. That is what he is doing. What kind of man would he be if he did not?

          You are not expected to be an expert on survey design and validation. That is the work of scientists.

          The original survey has been approved by the Departments of Justice and Education. Take their word for it.

          If you read the complete report on the original survey that took five years to finish validating, you will see that the designers put questions in there that identify students who were not taking the survey seriously and reject the rest of their answers. It really is a science.

  3. LarrytheG Avatar
    LarrytheG

    re:

    Walker-Grant 50%
    James Monroe 82%
    Division 55%
    State 73%

    So do Walker Grant kids, after they finish Middle school , go to high school – James Monroe?

    so they do terrible in Walker-Grant but wonderful when they attend James Monroe?

    1. James C. Sherlock Avatar
      James C. Sherlock

      Got it backwards, Larry. School climate surveys measure the school, not the students.

      They used to leave a dreadful environment at Walker-Grant and enter a good one at James Monroe.

      If your question is, will the enormous learning losses at Walker-Grant measured this year carry over to James Monroe, the answer is of course.

      Whether James Monroe can fix those issues is the question. Since we no longer understand from a flawed school climate survey the learning environment at James Monroe, we can only say we hope so.

      1. LarrytheG Avatar
        LarrytheG

        isn’t that what the SOL data is showing?

        Reading SOL pass rates 2021-22. James Monroe exceeded state averages.

        Walker-Grant 50%
        James Monroe 82%
        Division 55%
        State 73%

        so how can kids be FAILING in middle school and PASSING in high school if middle school is the founational courses for high school?

        1. James C. Sherlock Avatar
          James C. Sherlock

          Only the freshmen at James Monroe in 2021-22 when the learning loss was measured were at Walker-Grant in 2020-21 when the majority of the learning loss occurred.

          1. LarrytheG Avatar
            LarrytheG

            Besides picking an outlier year, do you think the other years at Walker-Grant and James Monroe are much different?

            Where do you think those kids with the good scores at JM came from if not Walker Grant like the other years?

          2. James C. Sherlock Avatar
            James C. Sherlock

            The “outlier year” was the COVID year. Please go away.

          3. LarrytheG Avatar
            LarrytheG

            No, you’re purposely evading the issue here.

            how do kids with bad SOL scores at Walker Grant suddenly get good SOL scores when they move to the JM High School?

            Is that a typical thing year to year? Don’t you want to know that?

            Reading SOL pass rates 2021-22. James Monroe exceeded state averages.

            Walker-Grant 50%
            James Monroe 82%
            Division 55%
            State 73%

  4. Matt Hurt Avatar
    Matt Hurt

    Here’s the problem when it comes to having people self-report their expectations. Everyone has high expectations, at least according to them. For example, one day while working with a middle school that was not accredited, I asked the teachers what kind of expectations they had for their students. They all assured me that they had very high expectations. Then I showed them the data from the previous year which indicated that over half of the kids who earned a C for the course failed their SOL test. When I asked them if that was acceptable, they replied “That sounds about right”. Just for reference, the average score every year on every Sol test in Virginia is significantly above the passing benchmark.

    We cannot accurately gauge expectations via a survey. We need more objective measures, such as the relationship between student final grades and their SOL scores. That’s where the rubber meets the road when it comes to expectations. If our expectations don’t align with the state’s expectations, too many of our kids are doomed to failure.

    1. James Wyatt Whitehead Avatar
      James Wyatt Whitehead

      “We need more objective measures, such as the relationship between student final grades and their SOL scores.”

      As long as school administrators push the de facto soft bigotry of grade inflation this relationship cannot be objectively measured. When you have a 50% floor for the lowest F, homework that cannot be collected for a grade, and retakes of class tests you can never determine the nature of that crucial relationship.

      1. LarrytheG Avatar
        LarrytheG

        so how do you fix this James, – institutionally?

        Do you think climate surveys can?

    2. James C. Sherlock Avatar
      James C. Sherlock

      Certainly worth doing, Matt. I would encourage every school principal to do that.

      But Virginia, offered an option by ESSA to pick a non-academic measure of school quality, which the schools had demanded, picked the school climate survey. This article is about using a validated one.

    3. LarrytheG Avatar
      LarrytheG

      re: ” We need more objective measures, such as the relationship between student final grades and their SOL scores.”

      This is coming from a Professional Educator and is simple and elegant in practice and should be in policy and one where Youngkin could actually make a real difference potentially without demonizing anyone or politicizing public education.

      The issue seems to be how to standardize grading that is consistent and adheres to what the SOLs are measuring. “Teaching to the test” – yes – which is no different from the way we test and certify everyone from network engineers to airline pilots to Doctors to Military occupations AND the way that most all other developed countries do education.

      1. Nancy Naive Avatar
        Nancy Naive

        It’s not just subjective versus objective, it’s the events surrounding the administration of the surveys as well.

        Take the differences Hiroshima HS would have had with ANY survey between 1944 and 1945 school years, for example.

    4. Nancy Naive Avatar
      Nancy Naive

      This assumption on the value of “self-reporting” would discount the original survey as well as the augmented survey.

      “It is difficult for the average student to make an “A” in this professor’s class.”

      1. LarrytheG Avatar
        LarrytheG

        The “value” itself is questionable but that’s easy compared to setting up what changes to make and how to ensure they are.

        All we never needed was Conservatives who hate the idea of public education, “educating” on how public education should be done.

        Their stock answer is that non-public schools with no rules or reporting will surely do better.

  5. Nancy Naive Avatar
    Nancy Naive

    “Arrrrrgh! They changed something! Woke! Woke!”

    Notice how the Captain decries the changes to a survey, as an effort to consolidate multiple surveys that were given for a handful of years, as “woke” because of result differences in one year to year result (never mind the pandemic), and yet not one month ago was all gung-ho on an ad hoc survey given by the VB Superintendent.

    1. LarrytheG Avatar
      LarrytheG

      The “woke” is the continued ignorant ruminations of those whose first reaction to things they do not agree with is name-calling.

      They start with “woke” and from that point on, its a rant against what they do not really understand but are more than willing to “reform” it.

      1. Nancy Naive Avatar
        Nancy Naive

        It’s age. That which is done once is “tradition”.

Leave a Reply

Read More