Bacon's Rebellion

Democracy Thrives in Sunlight

What the SGP Scores Reveal… and Don’t

—

by

James A. Bacon
in

by James A. Bacon

John Butcher, writing on Cranky’s Blog, has been on a tear recently as he’s plowed through the reams of “Student Growth Percentile” (SGP) data that the Virginia Department of Education has released recently under the prodding of the federal government.

The problems with using raw Standards of Learning (SOL) data to rate teachers, principals, schools and school divisions are well known. Roughly 60% of the variability in SOL scores between schools reflects the socio-economic status of the student body. It is patently unreasonable to compare the educational efficacy of a school teaching poor, inner-city kids with a school teaching affluent suburbanites on SOL scores alone. But the SGP gets around that problem by calculating the improvement in scores over time. Improvement is correlated with the quality of teaching and administration, not socioeconomic status.

In one recent post, Cranky… er, I mean John… demonstrated that there is almost no correlation between SGP and the divisional expenditure of money. The correlation coefficient between divisional expenditure per student and average SGP is less than 1% — meaning that less than 1% of the variability between school divisions can be explained by how much money they spend.

Graph credit: Cranky's Blog
Graph credit: Cranky’s Blog

For details, read the full post here.

John then took VDOE to task for suppressing the identity of individual teachers. The data anonymizes the data, identifying teachers only by a five-digit number. Delving into the City of Richmond data, John shows the wide variability in the ability of teachers to teach.

teacher_variability

The chart above shows a representative sampling of teacher SGPs. John homes in on Teacher 74415, seen in the purple line above:

The principal who allowed 25 kids (we have SGPs for 24 of the 25) to be subjected to this educational malpractice in 2014 should have been fired.  Yet VDOE deliberately makes it impossible for Richmond’s parents to know whether this situation has been corrected or whether, as is almost certain, another batch of kids is being similarly afflicted with this awful teacher.

Read the full post here.

Lastly, John makes an intriguing suggestion — using the SGP data to rate the quality of teachers from Virginia’s schools of education.

Just think, VDOE now can measure how well each college’s graduates perform as fledgling teachers and how quickly they improve (or not) in the job. In this time of increasing college costs, those data would be important for anyone considering a career in education. And the data should help our school divisions make hiring decisions.

In addition, VDOE could assess the effectiveness of the teacher training at VCU, which is spending $90,000 a year of your and my tax money to hire Richmond’s failed Superintendent as an Associate Professor in “Educational Leadership.” Wouldn’t it be interesting to see whether that kind of “leadership” can produce capable teachers (albeit it produced an educational disaster in Richmond).

Go, Cranky, go!

Share this article

ADVERTISEMENT

(comments below)

ADVERTISEMENT

(comments below)

Comments

20 responses to “What the SGP Scores Reveal… and Don’t”

  1. larryg Avatar
    larryg

    It all depends on what you really want to know. Measuring by school division is not going to tell you what measuring by a school within that division will tell you since neighborhood school often reflect the demographics of the neighborhoods as much as student or teacher performance measured at the district level.

    you have to want to know the truth not confirm you biases.

    a teacher right out of college – his/her first job at a school that no veteran teacher wants to teach at – is going to tell you what? that new teachers trying to teacher the hardest-to-teach don’t do that well ??? and you do what? fire her and replace her with yet another right out of college?

    what do you do with a teacher that has a mixed class and half of them progress on grade level and the other half does not?

    how are you going to measure individual teacher performance if half the kids are sent out to a title 1 teacher or other specialists?

    finally – there is no way in this world that any yahoo “on a tear” is going to see individual performance appraisals of ANY employee – period and that whole idea is loony to start with. Next thing you know these same folks are going to be demanding the performance reviews of the principles.. the administrators, then state police.. etc..

    you are NOT entitled to the names of the students NOR the names of the teachers… this is not how you do accountability… it’s misguided – for ANY citizens to want to see the performance appraisal of ANY EMPLOYEE – for ANY reason unless there is cause for a legal action and discover is used.

    you have one teacher who has 15 kids and 13 of them advance a grade level and 2 do not and one of them – Johnny’s father is convinced it’s the teacher who caused his kid to fail… and he’s going to hold that teacher accountable?

    come on guys.. if you want to be loons.. then at least try to look normal.

  2. LifeOnTheFallLine Avatar
    LifeOnTheFallLine

    “But the SGP gets around that problem by calculating the improvement in scores over time. Improvement is correlated with the quality of teaching and administration, not socioeconomic status.”

    Of course, the question arises as to whether these improvements can ever lead to the disadvantaged catching up to their advantaged peers. If not, at what point do we start perusing policies that address that 60% socioeconomic variability versus looking for shiny new tools we can use to clobber teachers?

    “John then took VDOE to task for suppressing the identity of individual teachers.”

    Yeah, it’s terrible that VDOE suppressed the identity of people when the sample size for the latest trend is three school years. What a bunch of jerks.

    “The chart above shows a representative sampling of teacher SGPs.”

    But is it a good representation? It looks at the best and the worst, two groups that are never representative of anything. I don’t judge running backs by looking at Marshawn Lynch and Ryan Mathews. The truly telling figures are the trend lines for the state and the city of Richmond. The state is close to flat, and the city would be, too if it weren’t for the system underpeforming in 2013 thanks to truly terrible leadership at the highest level.

    A more telling statistic in terms of performance comes from John’s blog where he points out that between 2012 and 2014 there were 304 teachers in 4th through 8th grade reading, but only 74 that stuck with it all three years. That’s a tremendous amount of turnover.

    Further, speaking of administration at the highest level, the American Statistical Association did an analysis of available studies of VAMs and found that individual teachers account for anywhere between 1 and 14 percent of variability and that system-wide changes mattered far more for student outcomes. They also warned against the negative impacts of using VAMs to rank individual teachers.

    http://www.amstat.org/policy/pdfs/ASA_VAM_Statement.pdf

    Now, I know from your discussion of global warming you think anytime someone turns to an actual expert in a complicated field of study it’s an appeal to authority so you’ll take that with a grain of salt, but I decided to leave that for you anyway.

    “The principal who allowed 25 kids (we have SGPs for 24 of the 25) to be subjected to this educational malpractice in 2014 should have been fired.”

    This is asinine. The teacher in question may be truly abysmal, but the data has exactly one more point than is needed to form a trend line. We have no idea what is going in that teacher’s school or the neighborhoods that feed it. But in a school system where 230 teachers couldn’t stick with the same subject matter for three years straight let’s pretend the problem is that one teacher or can be fixed by firing one teacher at a time.

    “Just think, VDOE now can measure how well each college’s graduates perform as fledgling teachers and how quickly they improve (or not) in the job.”

    This is intriguing only in so much as the SGP can actually be filtered to provide something meaningful. But teachers from more prestigious universities are probably already landing in more stable, prestigious schools and districts, which can lead to distortions. There’s so much more to being an effective teacher than just what one person is doing in one classroom.

    It also feels like there’s more to the story…

    “In addition, VDOE could assess the effectiveness of the teacher training at VCU, which is spending $90,000 a year of your and my tax money to hire Richmond’s failed Superintendent as an Associate Professor in ‘Educational Leadership.’ Wouldn’t it be interesting to see whether that kind of ‘leadership’ can produce capable teachers? ”

    There it is! It’s an intriguing idea because it can be used to go after maybe one school that hired one person the OP doesn’t like! And there’s the old tax dollars shibboleth! The efficacy of the failed superintendent to impart useful pedagogy to the students is – by the OP’s own writing – a secondary concern behind that taxes (I guess VCU students don’t pay tuition) fed salary.

    1. larryg Avatar
      larryg

      the “approach” says more about the folks behind the approach than the subject of their attention.

      like a lot of other science and data – one can choose to take a dumbed-down simplistic approach or one can at least acknowledge that the make-up of the classes is an integral and key aspect of measuring teaching performance.

      SGP measures A student over time – with more than one teacher.. so how do you handle that? how do you handle the fact that the kind is behind and gets put in a class where most are on grade level – and the teachers primary responsibility is to get a many of them to the next grade level as possible rather than stopping and getting the one or two up to snuff while the others drift?

      It’s NOT the teachers. It NEVER WAS the teachers. It’s the WAY the school system chooses to deal with the harder to teach kids and the neighborhood schools with substantial economically-disadvantaged demographics.

      I’ve said it before and I will continue. If the Conservatives think public school hires and keeps bad teachers and that is the reason for poor performance scores – then get ALEC and others to pilot real non-public schools teaching similar demographics included the economically disadvantaged – and demonstrate that schools and teachers “done-right” are superior.

      otherwise – get off the tear-down jihad and admit this is not about better schools but swinging that old conservative cudgel… or in this case – what appears to be a vendetta against particular teachers…

      If you REALLY want to gauge performance of individual teachers you have to do more than a dumbed down SGP approach. be honest. Do the job right if you are really serious about it. trying to get personal data on specific teachers is NOT the right way.

    2. James A. Bacon Avatar
      James A. Bacon

      You make some worthwhile points here. I’m glad to see an intelligent discussion on how the data can be usefully and fairly employed.

      1. larryg Avatar
        larryg

        I think it is “worthwhile” to point out that no matter the rest of the discussion that we don’t think we’re going to get personal data on anyone.. it’s a dumb concept.

        but I also say – that you need to recognize the realities of 15 kids in one classroom and the duty of the teacher is to get as many of them to the next grade level as possible – and if that teacher has some kids who are behind that teacher has an almost impossible job trying to keep the ones on grade level who are on grade level and at the same time catch up the ones who are behind. Think about if half the class is on grade level and half behind and you are rating this teacher on the same scale as a teacher in a “good” school where none of the kids are behind.

        how do you do this? It’s not my job to solve this issue – it’s the job of the folks who are proposing such a loony “solution” to start with and they don’t. All they do is try to advance the simple-minded option.. as the solution and it’s simply not …

        this is not folks who are looking for solutions to start with. These are folks who are opposed to the current operations. They are critics. Fine. But proposing wacko “solutions” is not a solution.

        they donj’t like the current system and apparently are willing to tear the current one down and do something different – that is totally unproven.. and has such obvious holes in it that it’s obvious they have not thought it through at all.

        these are not folks who are interested in making things better.. their solutions are not pointed to making things better but instead to assign blame for what they think are things that are wrong.

        1. James A. Bacon Avatar
          James A. Bacon

          Larry, I can understand the objections to releasing individual teacher data to the public. However, it is insane NOT to release it to the school principle he or she answers to. I can’t tell from John Butcher’s blog post whether that’s the case or not.

          1. larryg Avatar
            larryg

            Jim – you do not release ANY employee’s performance data to the public.

            this is nothing more than a cry for witch hunting.

          2. James A. Bacon Avatar
            James A. Bacon

            So, what are you saying? We shouldn’t release the data to school principals either?

          3. larryg Avatar
            larryg

            no. I’m all for performance data to be released to principals but let me also tell you, as I have before, not all classrooms are the same and the tendency is for the veteran teachers to get to pick first and they don’t usually pick the tougher situations and they often fall to those who have less seniority.

            that’s not a teacher issue. That’s an issue with how school districts administer neighborhood schools and especially those with the tougher demographics.

            I’m on the road now for 4-6 hours and will check back at the other end and continue the discussion.

          4. LifeOnTheFallLine Avatar
            LifeOnTheFallLine

            Unless, of course, the principal is the problem. I know we’re all weary of bureaucratic overhead for school systems, but this is a situation where if we’re going to use this data it has to be examined by people outside of the individual schools.

          5. larryg Avatar
            larryg

            It can be higher up than the principal also. It boils down to HoW a school administration staffs…and deals with things like class size, class make-up and special teaching assistance from other teachers.

            some of this is based on ignorance about how teaching is done with some folks thinking that it’s one teacher not several but worse that any class is equal to any other class therefore all teachers have the same task.

            then it goes one step further and seek personal performance reviews – a total violation of privacy for anyone…

            it sounds like one of those “we’r going to straighten this problem out” ideas but it’s based on simplistic, even ignorant thinking .. in my view.

  3. James A. Bacon Avatar
    James A. Bacon

    Posted on behalf of Jean Wight:

    Thank you for writing on this topic.

    From 13 years of volunteering in an “East-End” Richmond public elementary school, here are some observations you might want to consider:

    1. It has often arisen that a child does not live with a mom and/or a dad (incarceration or other reasons), but with an aunt or a grandmother – this is not the child’s fault at all;

    2. Food stamps cannot get kids healthy fresh foods if you do not have a car in the poorer sections of Richmond, and you cannot really use them at a farmer’s market;

    3. I have seen the good, the bad and the ugly when it comes to kid situations, whatever the economic means, but on the whole it has been just very economically challenged homes that bring a child hungry to school – especially with this harsh winter;

    4. The scenario portrayed of female heads of households “strung out on drugs” has been the exception, rather than the rule – I have suspected only about a handful in my thirteen years who actually brought kids to school – being poor does not mean your mom is a crack-headed bimbo.

    Come volunteer one day with me at Bellevue Elementary School. For the last 4 years I have adopted a pre-K Class. I think you would see poverty and food insecurity in a different light.

    I do not think that Schools should necessarily be feeding centers, but on the other hand, it is vital to education and brain development that young children especially have the food not only for their minds but also for their bodies to grow and learn. If it takes more food in our more impoverished areas to keep kids in school, and have them able to learn, I am all for it.

    1. larryg Avatar
      larryg

      Ms. Wright has it right. Go spend a few days (more than one) in a classroom for K-5 – and see the challenges that are presented to teachers, not the least of which is folks who’ve never seen the inside of a classroom talking about “bad” teachers.

      Bad Parents – yes…

      The very first thing a teacher has to do – to be successful is called classroom management and that means getting 15 of them to sit still and listen – go try it sometime.

  4. Peter Galuszka Avatar
    Peter Galuszka

    Jim,
    Didn’t know you have volunteered in Richmond’s East End for 13 years. Did I read that right?

    1. James A. Bacon Avatar
      James A. Bacon

      My bad. I forgot to add a note attributing the comment to Jean Wight.

  5. larryg Avatar
    larryg

    I call this – malpractice by data.

    it represents ignorance on the part of folks who think that every class is the same and every teacher has the same exact level of difficulty.

    If you put a low performing kid in a class of higher performing kids – who is that teacher going to teach to?

    how about if half the class was low performing – what then?

    what if the teacher is a 20 year veteran or right out of college?

    but the proponents of this – never have asked this question, much less addressed it.

    so the premise assumes it’s apples to apples front to back – and it’s simply not.

    If you REALLY wanted to see something truly significant – you’d do SGP by school – not teacher or district – and you’re going to find that some neighborhood schools in districts – like Henrico – and RIchmond – have very different performance profiles for the same demographics.. of kids.

    and that ought to be as much or more of a question than individual teachers.

    and what you’d also find is that new teachers right out of college put in chronically un-performing neighborhood schools with terrible SOL scores – for specific schools – is a disaster. Yes.. you’d see individual teachers doing terrible because the combination of entry-level skills with kids who are way, way behind… more than a grade level – is a disaster – that cannot be fixed by staffing the whole school with new teachers or teachers that just simply lack the skills to deal with chronically economically disadvantaged kids.

    we’re using the data wrong. we’re not looking at what the data really represents in terms of neighborhood school performance because that’s not the intended purpose here to start with – which is to “prove” there are “bad” teachers in the schools… no matter the circumstances..

    this, unfortunately, has become politics of the ignorant… and I do not mean anyone in particular here – just the mindset behind this.

  6. Hill City Jim Avatar
    Hill City Jim

    The data is available school by school. Stop foaming.

    1. larryg Avatar
      larryg

      I know it is. no foaming…

      but it does go back to what you actually want to know.. and it’s not about “bad” teachers per se unless the folks who are after that data – actually know how teachers get assigned their classes.

      if we are not careful – we are going to fix it so that no teacher is going to accept a tougher class.. they’ll just take the easiest ones they can get – in schools that are not in economically disadvantaged neighborhoods.

      what the SGP folks COULD BE doing INSTEAD is identify the high performers, the ones that beat the odds – and reward them and set up
      an incentive system that rewards teachers that excel.

      that’s my complaint.. no foaming.. I’m just incensed that we set out on these witch hunts and the folks running them don’t even know the first thing about how education works..

  7. Hill City Jim Avatar
    Hill City Jim

    Had you been following Brian Davison’s trials and tribulations, you would find out that is exactly why Brian sued to get the data. Reward the good ones, fire the bad ones. In Lynchburg they employ ~900 teachers, I was told they ran 2 off. I doubt the others are all high performers.

    1. larryg Avatar
      larryg

      part of what I was reading was some folks wanted to know about their kid’s teacher – right?

      but the trouble is SGI is not how you determine a “good” or “bad” teacher unless you really don’t care if your idea is ignorant.

      I’ve brought up several times how no two classes are alike in terms of high, middle and low kids and how some classes are almost impossible to have one teacher try to teach since the kids have such diverse education status – yet, often, in some schools, there are no options other than hiring more teachers to break up the kids into more reasonable skill sets.

      that’s on the class side.

      on the teacher side – there are 20 year veterans that are highly skilled and capable of teaching Title 1 – and there are brand new teachers right out of college.

      you’d just them both on SGP?

      what happens when the highly skilled teacher essentially has her pick where to teach and refuses the “impossible” group and the administration punts and puts a new college hire there?

      I’m not making excuses here – I’m relating the realities and HCJ – teachers who have half a brain will not teach in the Lynchburg school system because they know – most of the teachers in that system – no matter what – using SGP will be determined to be “bad” so they teach elsewhere…

      but this is what you get when you have folks using SGP who do not understand how education currently works.

      How do you “incentivize” a place like Lynchburg? explain that to me.

Leave a Reply

Read More