Bacon's Rebellion

Democracy Thrives in Sunlight

Identifying the Education X Factor

—

by

James A. Bacon
in

by James A. Bacon

The 2014 Standards of Learning (SOL) scores are in, and it appears that Virginia’s school divisions made decent improvements in mathematics over the past year while losing ground marginally in reading, writing, science and history. Bottom line: Virginia students tread water another year.

Here are the percentage pass rates across all grades and schools systems. (The cells highlighted in blue reflect old tests, which were changed in 2012-13 to make them more rigorous.)

SOLs
Another year running in place — that’s demoralizing. Can we find some seeds of succor? There are a few. I plowed through the data released by the Virginia Department of Education and compiled composite pass scores for every school district. The highest possible score — a 100% pass rate for all five subjects — is 500. I pulled out the school districts with an average pass rate of 80%, hardly a world-beating performance but at least sufficient to prosper in a knowledge-based economy.

top_SOLs

As one would expect, affluent Northern Virginia cities and counties, which have some of the highest median incomes and highest average levels of education in the country, stood out in this list of the top-24 performing school districts.

But there are some pleasant surprises, most notably West Point, a small mill town on the York River and one of only two towns in Virginia that maintains its own school district. The median household income in 2010 fell short of $50,000 — less than half that of Loudoun County, Virginia’s most prosperous locality. (Although incomes are not high, poverty is very low in the town — less than 3.0%.)

How do the school children of a small, southern mill town out-perform super-affluent localities such as Loudoun, Arlington and Fairfax Counties? It could be a fluke — the town’s population is only 3,300. Maybe the outstanding performance was the result of random variation, which create wider swings in smaller numbers. Or maybe West Point schools are doing something right that others could emulate. Some enterprising newspaper reporter should find out.

Other stand-outs are schools in the Roanoke Valley. Roanoke County, Salem and Botetourt County schools all scored in the top twelve. Those school districts are significantly larger than West Point’s, so it’s harder to attribute such consistently high scores to random fluctuations.

Even more surprising is the performance of school divisions in far Southwest Virginia. Wise County, which racked up scores equal to Fairfax County, sits in the heart of Virginia’s economically ravaged coalfields. This is deepest, darkest Appalachia. Scott County and Washington County, also in Southwest Virginia, performed in line with affluent exurban school districts in the Richmond and Hampton Roads regions.

Again, one must ask the question: How do these school systems beat the odds? They have fewer resources. Students’ parents have lower incomes and less education than in more affluent districts. The bromides about what determines school performance — spending per student, socio-economic status and education levels of the students’ parents — provide an incomplete explanation at best. What is that X factor? Can we capture it, bottle it and share it with other school districts?

I would love to crowd-source the analysis of these questions. For anyone who is interested in digging into the numbers, here they are:

Share this article

ADVERTISEMENT

(comments below)

ADVERTISEMENT

(comments below)

Comments

54 responses to “Identifying the Education X Factor”

  1. DonR Avatar
    DonR

    “How do the school children of a small, southern mill town out-perform super-affluent localities such as Loudoun, Arlington and Fairfax Counties?”

    By not having a significant percentage of newly arrived immigrants who came with whatever education they may have gotten back in their originating country and spent at least a few years trying to learn English?

    By not having a high percentage of their most academically talented children attending private schools and, therefore, not counted in the public school reporting?

    1. billsblots Avatar
      billsblots

      I wrote on here once, probably months ago, that my oldest daughter earned her Masters in Elementary Education from VCU, then moved to Winston-Salem as her husband was starting Wake Law. She was hired by a W-S school to teach 4th grade. She resigned after two years because the school was unmanageable. 40 % of her class qualified in English as Second Language (ESL) and were removed from her classroom in rotation throughout the day for language training. She was told she couldn’t teach new material when kids were out of the room, which she eventually ignored because that was a majority of the day. Parents of just 2 kids came to the initial open house for the school year, while she was in elementary school I remember the classrooms being full of parents so that some of us had to stand.
      She interviewed for a job with an insurance company starting at the same pay she was making and was hired and trained, has received a promotion and two raises. The education system is a mess and is not equipped and does not know how to handle the intolerable situation that the federal government has put them in, made much worse lately by the sitting administration. No worries, “These children are our future.” – BHO.

      1. larryg Avatar
        larryg

        so Billblots – would you blame that situation on “bad” teachers that cannot be fired?

        bonus question: – for a private/voucher schools that had to serve the SAME student population demographics – do you think they’d do it better?

        how?

        double bonus – why are brand NEW teachers assigned to such difficult duties ? why don’t the schools provide additional resources so that the two different student demographics can be more equitably dealt with?

        is this caused by “bad” hard to fire – union teachers?

      2. larryg Avatar
        larryg

        I thought this may be of interest:

        http://www.fairfaxcounty.gov/demogrph/langmaps/elemsch-non-english.pdf

        Fairfax County elementary school students speak a language other
        than English at home 47,484 (49.2%) of total school population.

        Fairfax has 180,000 students spread over 196 schools over an 400 sq miles.

    2. James A. Bacon Avatar
      James A. Bacon

      Those are plausible explanations. It should be possible to test the English-as-second-language hypothesis — the figures exist to do that. If anyone has the inclination to track down the data and run a regression analysis, I would love to see it.

      As for the private-school factor, that might be harder to test. As I recall, about 10% of Virginia’s kids attend private schools and/or are home schooled. I wonder if it it’s possible to get Census Bureau data of the school-age population and deduct the public school enrollment data to get a good estimate of the private/home-schooled population for each locality… and then run a regression analysis.

      1. larryg Avatar
        larryg

        re: ” I wonder if it it’s possible to get Census Bureau data of the school-age population and deduct the public school enrollment data to get a good estimate of the private/home-schooled population for each locality… and then run a regression analysis.”

        worth a try.. but sounds painfully manual… and what does it really prove anyhow – what’s the point?

        1. James A. Bacon Avatar
          James A. Bacon

          The point is to test Don’s hypothesis that the SOL performance of Fairfax and other affluent school systems is depressed by the fact that a disproportionate number of students (presumably from affluent households who likely to pass at a higher rate) attend private schools and, thus, are not counted.

          1. larryg Avatar
            larryg

            Fairfax
            Population definition and source info Population, 2010 1,081,726
            Persons under 5 years, percent 2013 6.7%
            Persons under 18 years, percent, 2013 24.0%
            Fairfax county schools = 188,000 students

            under 18 = 259000
            under 5 = 72000

            number of kids between 5 and 18 = 187,000

            hmmm…..

            but are you and Don ASSUMING that the kids NOT in public school – score higher? how would you show that?

            and are you assuming that Fairfax county has any more non-public school students as a percent than other counties?

            at best – you’d have pretty tenuous data much less sold conclusions.

            you’d have, at best, the usual sound-bite stuff that seems to feed/cause misconceptions..

            if you look at the Fairfax SOL data for the minorities – it pretty much tracks many other Virginia schools where blacks and Hispanics and economically disadvantaged score lower by 5-10 points and more.

            what’s the point – of doing this – in terms of what you’d do about it – policy wise?

            would you just advocate that these kinds of kids are a drag on the system and should not be accommodated?

            what choice do you have – other than to educate them?

          2. James A. Bacon Avatar
            James A. Bacon

            Thanks for looking up the numbers. Hmmm… Looks like that methodology might have some problems.

            Regarding the assumption that private-school kids out-perform public school kids…. Yes, I would make that assumption. Private school kids come from more affluent households; performance in school is correlated with socio-economic status. I think it’s reasonable to assume that private school kids on average out-perform public school kids academically on average.

          3. larryg Avatar
            larryg

            even kids with english as a second language? You _are_ aware of the diversity of the Fairfax county school population, right?

            but what you seem to be saying is that the economically well-off are going to do better – which is fine.. but what does that mean in terms of policy?

            along these lines – do you equate private schools with voucher schools?

            we still have the job of educating the kids, including the economically and culturally disadvantaged… still not sure what the point is.

            is it relevant to anything that we’d do about it?

          4. larryg Avatar
            larryg

            this is 2010 -from Fairfax County and it seems to say there are about 200,000 school aged kids:

            http://www.fairfaxcounty.gov/demogrph/demrpts/report/fullrpt.pdf

          5. DonR Avatar
            DonR

            Many of the private schools have entrance qualifications. Some are quite tough. So, yes, my assumption is that the private schools probably score better on standardized tests then the average public school.

            I also assume that Fairfax County has a higher percentage of kids in private school than the average in the US.

            The point of all this is to level the playing field across jurisdictions to determine how well each jurisdiction is educating their children.

            We also need to adjust the poverty measures to account for cost of living. The Weldon Cooper Institute of UVA has created the Virginia Poverty Measure which does just that. By that measure there is no difference in the percentage of children in or near poverty between NoVa and Richmond. In fact, the part of NoVa inside the beltway is considerably worse off than places like Chesterfield and Henrico Counties.

            http://www.coopercenter.org/demographics/interactive-map/citycounty/20063

          6. larryg Avatar
            larryg

            re: ” The point of all this is to level the playing field across jurisdictions to determine how well each jurisdiction is educating their children.”

            can you explain this? what is it you are saying should be done ?

            “We also need to adjust the poverty measures to account for cost of living. The Weldon Cooper Institute of UVA has created the Virginia Poverty Measure which does just that. By that measure there is no difference in the percentage of children in or near poverty between NoVa and Richmond. In fact, the part of NoVa inside the beltway is considerably worse off than places like Chesterfield and Henrico Counties.”

            I agree… on this.

      2. larryg Avatar
        larryg

        what is the purpose of any conversation about the difficulties of students with English as a second language anyhow?

        If you are Fairfax – you accept that responsibility and deal with it. If you are ANY school with that issue – what would you do otherwise?

        sure you can chew on the “immigration ain’t fair bone but really what’s the point?

        Are we really going to deny kids a basic education because they don’t speak English?

        Basically what we seem to be saying is that we’d score higher if we didn’t have all these kids with language and economically disadvantaged issues.. eh?

        okay. And we do what to deal with it?

        See this is the problem I have with Conservatives these days. They know what they don’t like. They can regale you with chapter and verse all the bad stuff going on – but when you ask them for the path forward – their response lacks…. rationality and clarity.. and instead seems to veer towards ‘the system is broke, trash it and start over”.

  2. larryg Avatar
    larryg

    what happened to the bad can’t fire unionized teachers narrative?

    Northern Virginia has a highly diverse population, wide diversity in economic status and culture/language status and it still manages to do a good job compared to schools with less diverse more monolithic populations.

    The smaller schools have the advantage of much few schools to try to allocate their resources to – equitably (not all schools do this well as NoVa does).

    The challenges come with multiple school districts spread across geography that is economically segregated into de-facto neighborhood schools that align with the economic demographics. In plain language – less affluent neighborhoods tend to be served by schools in those neighborhoods and ditto for more affluent neighborhoods. You end up with very different school snapshots on things like percentage of economically disadvantaged, SOL scores, percent enrolling in AP, percent passing AP, percent graduating… etc…

    but these numbers are generally reported on a School District basis.

    and we’re going to get insight into that issue when the state releases in a few weeks the actual schools within each county/district that did not make accreditation. (so maybe wait for the crowd source deal until that data comes out also)

    If you have 10 schools in a district and 3 failed accreditation – what does that mean in terms of the schools academic performance? Does it mean you have entire schools with bad teachers and/or bad kids?

    I have a theory and that theory is that the kids that are the easiest to teach are the ones that do best and the ones that are harder to teach – don’t get the resources they need to overcome that disadvantage.

    We like to blame the parents in these situations but think about how morally corrupt that attitude is. We’re saying that kids that are teachable (we know his from the schools that are successful at it) .. deserve to fail.. -even though it means the rest of us will end up paying their entitlements/incarceration.

    how does this happen? It happens because the school systems themselves do not report information that they are compelled by law to report to the state and the Feds … they don’t show it on their own school websites.

    go try to find most of these statistics .. like SOL scores per school – on your local school website.. Some do – many don’t-because obvious disparities would get more parents and taxpayer involved in asking why.

  3. larryg Avatar
    larryg

    here’s a comparison of Fairfax, Henrico, Chesterfield, Hampton, and Albemarle County schools… on some academic and demographic stats that are not SOLs:

    http://projects.propublica.org/schools/districts/5101260#5101890,5100090,5100840,5101800

    this reaffirms the view that Fairfax is a large diverse school system – both culturally and economically and it does pretty well compared to other smaller systems in Va.

    1. James A. Bacon Avatar
      James A. Bacon

      Interesting data. If nothing else, it shows that there are a lot more indicators than just overage SOL pass rate. For the most part, SOLs measure basic proficiency. They don’t measure mastery of advanced learning. But it is the AP courses and advanced courses where a school system like Fairfax may excel.

      1. larryg Avatar
        larryg

        For Fairfax, the high rate of AP enrollment and pass rate given the fact that they do have a lot of language and cultural diversity as well as substantial numbers of minorities and economically disadvantaged.. to me .. is amazing….

  4. Cville Resident Avatar
    Cville Resident

    I think SOLs can be fool’s gold for comparing school systems. I know a teacher at Atlee in Hanover (one of the better HSs in the state). She said they spend almost zero time prepping for SOLs. She tries to hit the SOL points in her curriculum, but she doesn’t focus her lessons on the SOLs. The prevailing attitude there is that kids don’t need to be taught for the SOLs. If they get a screwy result one year as a School Divsion, they’ll work to correct it for accreditation purposes, but they’re not designing an instructional package bases solely on SOL tests.

    Whereas other school systems spend their entire year preparing for SOLs. Trust me, Nelson’s last Sup’t seems to have basically foregone all other aspects of education in favor of raising SOL scores.

    If you want to dig deeper, I think you need to examine just how much of each division’s classroom time is devoted to SOL learning objectives and assessments.

    1. James A. Bacon Avatar
      James A. Bacon

      In other words, some school divisions teach to the (SOL) test, others don’t. Yeah, that could explain some of the seemingly strong performance of some downstate school divisions — they’re focusing on the SOLs to the exclusion of other stuff. I wonder if there are any metrics for teasing out the extent to which schools teach for the test.

      1. larryg Avatar
        larryg

        re: “teaching to the test”

        when you take AP – guess what – you have to take the AP exam so they’re teaching AP “to the test”.

        when you take the SATs for college, guess what? you have to learn what’s going to be on that test. You have to be taught how to answer the questions on that test.

        If you are in a State with Common Core – guess what? ditto

        if you want to go into the Military – guess what? ditto.

        If you want to become a Doctor or Lawyer or Professional Engineer – guess what?

        this “teaching to the test” dog – he don’t hunt in the real world.

        you can bet your fanny in Va.. if you don’t meet the numbers on the SOLs there are consequences… for the students, the teachers, the schools, and the school districts.

        and if you take a look at the Hanover SOLs – you will see that they look like a lot of other schools – including the deficits in blacks, Hispanics, and economically disadvantaged.

        if you look at MOST of the other smaller schools in Va – take the time to also look at their minorities and economically disadvantaged scores… so you can at least KNOW what you are talking about!

  5. James A. Bacon Avatar
    James A. Bacon

    Larry, nobody is being ideological here. I started off by asking the question, can we learn something from the downstate school systems that perform so well on SOL tests? Are they doing something that other school systems can learn from? I posed that question because we get zero — nada — of that kind of analysis from the Virginia Department of Education.

    I suggested a crowd-sourced approach to analyzing the data. Don jumped in and made the perfectly reasonable observations that English-as-a-second-language and private-school attendance could skew the SOL results against Northern Virginia schools. In other words, he was suggesting that their *adjusted* performance might make them look better. Those are valid hypotheses worth testing.

    There are no right-wing narratives here.

  6. Richard Avatar
    Richard

    Interesting.

    I would like to see the bottom performing schools. I suspect you’d see anomalies there, but also confirmation that poor communities are more likely to have poor schools. A comparison between Richmond and Henrico would certainly highlight some issues. Maybe that is too obvious and maybe it’s been done, but that does seem to me to be the more significant issue.

    1. larryg Avatar
      larryg

      More Than 40% of Low-Income Schools Don’t Get a Fair Share of State and Local Funds, Department of Education Research Finds

      http://www.ed.gov/news/press-releases/more-40-low-income-schools-dont-get-fair-share-state-and-local-funds-department-

      this is the problem with the bigger school districts with neighborhood geography aligned along along economic status.. the school demographics tend to reflect the neighborhood demographics and if you compare all the schools in that district -you will often (not always) see disparities percent of economically disadvantaged, English as a second language, percent of minorities black/Hispanic, in SOL scores, AP enrollment rates, high math participation rates, etc.

      DonR and Bacon worry about places like Fairfax getting a bigger share of the harder-to-teach kids because of a presumption that higher than typical numbers of the better scholars don’t go to public school but private.

      I worry about kids from disadvantaged circumstances not getting a sufficient education to end up in the workforce as a taxpayer rather than an entitlement burden.

      I think the data tends to show an equity problem with allocation of resources between schools in affluent areas verses schools in economically depressed areas…. which, if true, means we’re not dealing with a fundamental threat to our economic development potential.

      Should Fairfax get more school money? sure.

      will giving Fairfax more money fix the equity issue throughout the rest of the state? No.

      1. TooManyTaxes Avatar
        TooManyTaxes

        Fairfax County, without being honest with residents, goes well beyond spending federal and state money on schools with low income kids. I’ve been told FCPS spends c. $60 million extra. This means elementary school classes with as many as 35 kids in “affluent” areas of the country. FCPS administrators refuse to disclose class size information. PTAs and PTOs must compile the data.

  7. virginiagal2 Avatar
    virginiagal2

    Hi Larry –

    I think the question is, do you want to focus on the actual goal, or do you focus on test results?

    It’s nice to have high AP scores, good SATs, a high score on the bar exam, or whatever.

    Personally, I think we are better served to focus on things like , are we doing a good job of educating our students, are our students doing well in college, and are our lawyers reasonably well qualified.

    To me, there is a real danger of getting too wrapped up in the test scores and not looking at what those scores were originally intended to measure – which, in this case, are kids that have a good core of knowledge to function in society and are prepared to go on to college and succeed.

    1. larryg Avatar
      larryg

      re: ” I think the question is, do you want to focus on the actual goal, or do you focus on test results?

      It’s nice to have high AP scores, good SATs, a high score on the bar exam, or whatever.”

      if you can’t measure what have you got?

      “Personally, I think we are better served to focus on things like , are we doing a good job of educating our students, are our students doing well in college, and are our lawyers reasonably well qualified.”

      using what metrics? how do we know we are “doing a good job”?

      “To me, there is a real danger of getting too wrapped up in the test scores and not looking at what those scores were originally intended to measure – which, in this case, are kids that have a good core of knowledge to function in society and are prepared to go on to college and succeed.”

      you mean like on a driving test or an airline pilot test or a test to see if you know how to design a bridge properly?

      I do not believe in high stakes tests – I prefer assessments like PALS but when you can’t even get into college without taking a test – how do you justify not doing the same for other education?

      you have to be able to measure . you have to want to measure.

      without measuring – everyone has their own subjective standard

      the tires on your car – are tested for a load and speed spec. your life depends on that test. when you get your blood tested – you are totally dependent on an objective standard for measuring the parts of it. when you use your cell phone – you depend totally that the tower and phone were designed to a standard – and tested to assure that they met the standard.

      when you go to a doctor – who claims to be a board certified specialist -you are depending on the test he/she had to take.

      that’s the real world. everything requires a test.. including people. You do not want a guy using a scalpel on you or pointing a laser at your eyes that has not been tested.

      this “teaching to the test” stuff is just foolish blather.. we teach to the test – all the time.. for virtually everything of any importance at all..

      You likely depend on nutrition labels right? that’s testing…

      1. virginiagal2 Avatar
        virginiagal2

        Hi Larry –

        If your schools are doing well, but you can’t easily measure kids who will succeed in college and in life, you still have kids who will succeed in college and in life.

        The emphasis needs to be on the goal, not the metric, and shortcomings of the metric need to be kept in mind at all times. Where appropriate, adjust the metric or use alternate metrics.

        Direct measurement, rather than measurement of proxies, is always preferable. In this case, the tests are proxies, and not always terribly good ones.

        You actually can get into college without taking these types of tests.

        There are objective standards that aren’t being used because they are harder to measure. Those standards are the actual outcomes. How many of these kids finished school? How well did they do in school? How well did they do in life?

        How well they did on tests is not the actual measurement. You are confusing actual measurements of the actual things you are interested in – tire pressure, blood pressure, cell phone signal – with proxies that are not what you are actually interested in.

        In the case of SOL, SAT, and AP tests, these are proxies that have known weaknesses, but that were chosen for reasons of cost, convenience, and to some degree feasibility.

        Making decisions based on proxies is inherently problematic. The fact you are measuring a proxy, not what you are actually interested in, always needs to be kept in mind.

        Everything does not require a test. The fact that something is easy to measure does not make it important. And conversely, many of the most important things are not easy to measure.

        The issue of teaching to the test is not foolish blather – it is the key issue here. And it is inherently problematic. What appears to be happening in many cases is that we are getting more of what is easy to measure, and less of what we actually want.

        1. larryg Avatar
          larryg

          “If your schools are doing well, but can’t easily measure kids who will succeed in college and in life, you still have kids who will succeed in college and in life.”

          how do you know if you did not measure? how do you know what the kids do not yet know that they need to be taught if you don’t measure?

          “The emphasis needs to be on the goal, not the metric, and shortcomings of the metric need to be kept in mind at all times. Where appropriate, adjust the metric or use alternate metrics.”

          how do you know when a kid has learned/mastered and can move on to the next thing?

          “Direct measurement, rather than measurement of proxies, is always preferable. In this case, the tests are proxies, and not always terribly good ones.”

          continuous incremental assessments to ascertain what has been learned, what has not, what to go back on, what you can go forward on.

          “You actually can get into college without taking these types of tests.”

          sure you can – how many? you’re into deep blather now… why do kids take the SATs if they do not have to?

          “There are objective standards that aren’t being used because they are harder to measure. Those standards are the actual outcomes. How many of these kids finished school? How well did they do in school? How well did they do in life?”

          you purposely design objective standards – to measure… that’s the entire point.

          “waiting to see the outcomes” you’re kidding right? Really?

          “How well they did on tests is not the actual measurement. You are confusing actual measurements of the actual things you are interested in – tire pressure, blood pressure, cell phone signal – with proxies that are not what you are actually interested in.”

          no. it’s called incremental LEARNING. no measurement is perfect but to not measure, to not WANT to measure – is in my view – just plain ignorant.

          “In the case of SOL, SAT, and AP tests, these are , proxies that have known weaknesses, but that were chosen for reasons of cost, convenience, and to some degree feasibility.”

          no test or assessment is perfect. It’s not a reason not to measure. When you make it the reason not to measure – you’re using excuses to not measure.

          “Making decisions based on proxies is inherently problematic. The fact you are measuring a proxy, not what you are actually interested in, always needs to be kept in mind.”

          again – no metric or measurement is perfect but to use that as a reason to not measure is ignorant. if the metrics are not good enough -you improve them – you don’t abandon them all together.

          “Everything does not require a test. The fact that something is easy to measure does not make it important.”

          most of what you encounter in life that you depend on involves measurement and testing whether it’s the cereal you eat or the aspirin you take or the microwave you use – totally trusting it’s not going to fry your organs outside of it.

          “The issue of teaching to the test is not foolish blather – it is the key issue here. And it is inherently problematic. What appears to be happening in many cases is that we are getting more of what is easy to measure, and less of what we actually want.”

          you LEARN to take the SAT test. You go find out the questions that will be asked.
          you ARE… LEARNING TO THE TEST.

          You LEARN the questions you need to answer to pass the armed forces entrance test.

          You LEARN to take the tests you must pass to become a doctor, lawyer, professional engineer… and you learn the things that are going to be on the tests.

          someone who becomes very good knows much more than the test but the test is the minimums standard for certification that you know the minimum required. There are levels up from there – higher level tests for higher level certifications.

          you do not get a drivers license determined by “outcome”. You get it after you take a test and you study the questions that will be on that test.

          LIFE is – “teaching to the test”. we just totally disagree about this.

          education is fundamental and measuring progress and competency is mandatory.
          you do not get a College Degree based on “outcome”.

          you do not get in the pilot seat on a plane – to see what the “outcome” will be. Outcome is paramount but you don’t not measure and test and just wait for the “outcome”. too much movie watching here..

          you don’t become a doctor by reading a book.. and then start practicing.

          you appear to have veered off the reality train here.

          1. virginiagal2 Avatar
            virginiagal2

            Hi Larry –

            You appear to be missing my points, so I’m going to try to go over this again.

            First, I expressed concern about some of the measures you are using to evaluate schools, particularly your use of percent of kids taking AP in very small schools with graduating classes less than a hundred – in one case, less than 20. My point was that what measurement you use, and how you validate it, are vital.

            Any time you test, you need to validate your measurements.

            Things that are hard to measure can be as or more important than things that are easy to measure.

            The only way to validate measurement tools is to compare them against the goal – not a proxy metric. A test that you can’t validate is worthless – if you trying to measure whether or not kids can succeed in college, you must – not just should, but must – compare your test results against whether or not those kids did well in college.

            A test that actually does not correlate with the goals of the testing is worse than useless. It can be actively harmful because of opportunity cost. If you are going to spend many tens of millions getting kids to do well on various metrics, it’s kind of important that those metrics actually correlate with your goals.

            If your metric has shortcomings, you need to be aware of that, and adjust your metrics or use alternate metrics.

            Second, AP tests. Specifically with AP tests, you are using them as proxy measurements and assuming that schools that do not have large numbers of students taking AP are not good schools. That despite the fact that many of these are tiny schools in rural districts and AP classes are likely to be extremely small and may, in many cases, not be financially feasible for these small schools.

            I suggested using dual enrollment instead – and dual enrollment numbers are actually a metric on the state evaluation, if you look at them, and in the exact same section as AP – as potentially a better fit for these small districts.

            Third, you questioned the ability to get into college without taking AP tests – about 28 of the top 100 colleges on the US News reports do not require the SAT or ACT. The best known ones I can think of offhand are Sarah Lawrence and Mount Holyoke, but FYI, the University of California discussed eliminating the SAT requirement a few years back as well, although thus far it has not done so.

            Of Virginia universities, the best known ones that don’t require SAT or ACT for entering freshman who are qualified – meaning decent GPA – are George Mason and ODU. Further, qualified transfers from community colleges are admitted automatically, regardless of their SAT or ACT scores – to the best of my knowledge, that includes transfers to UVA and Tech.

            A number of traditional graduate programs will admit you with work experience without graduate tests like GRE and GMAT. Some do not even require the work experience – Northwestern and UCLA come to mind. The online programs almost never require them – including fully accredited programs and prestige programs like the MSCS from Georgia Tech.

            You keep referring to objective standards – but no matter how objective, if you are measuring the wrong things, you have GIGO – garbage in, garbage out.

            You also at a couple of points have put quotation marks around things I did not say. I did not say, waiting to see the outcomes. I did say that there are actual outcomes here, and whether or not measurements accurately correlate with those outcomes matter very much and needs to be validated.

            In particular, I am specifically and repeatedly questioning whether a low percentage of kids taking AP in a tiny district means that the school is not providing a good education.

            No one is saying, do not measure. I am saying, be aware of the limitations of the tools you are using, and realize that they do not fully encompass everything that is important to providing a quality education. Do not neglect important things that kids need to succeed in college and in life because they are not on a test.

            Things like ability to synthesize information from multiple sources, to form and argue complex concepts, and to do independent research on a topic are as important as knowing who was fighting in the war of 1812.

            Life is NOT teaching to the test. Many of the most important accomplishments in life are taking what you have learned and creating new things, where there is no book to fall back on. No one starts a business or makes a discovery learning to the test.

            Education is fundamental, but most of what we learn and what we use is not measured in a standardized way and most of it cannot be measured easily.

            Also, just to correct you, getting a college degree is an outcome – what on earth do you think outcome means?

            Outcome is the term used for the real life result. Did you graduate from high school? That’s an outcome. Did you get into college? That’s an outcome. Did you pass your class? That’s an outcome. Did you get a job? That’s an outcome. Did you get on the honor roll? That’s an outcome? What was your GPA? That’s an outcome.

            What did you think outcome meant in this context?

          2. larryg Avatar
            larryg

            “You appear to be missing my points, so I’m going to try to go over this again.

            First, I expressed concern about some of the measures you are using to evaluate schools, particularly your use of percent of kids taking AP in very small schools with graduating classes less than a hundred – in one case, less than 20. My point was that what measurement you use, and how you validate it, are vital.

            well it’s not MY measurement and it’s not the only one. The validation of it is also not mine.. it’s done by many others. I accept what they’ve done as largely legitimate.

            “Any time you test, you need to validate your measurements.”

            agree.

            “Things that are hard to measure can be as or more important than things that are easy to measure.”

            again, agree.

            “The only way to validate measurement tools is to compare them against the goal – not a proxy metric. A test that you can’t validate is worthless – if you trying to measure whether or not kids can succeed in college, you must – not just should, but must – compare your test results against whether or not those kids did well in college.”

            do you think this has not been done? It’s a WIDELY ACCEPTED measure that is used in conjunction with other validating information such as the high school subject matter scores and SAT… etc… If these various measures confirm each other.. then why would you argue otherwise?

            “A test that actually does not correlate with the goals of the testing is worse than useless. It can be actively harmful because of opportunity cost. If you are going to spend many tens of millions getting kids to do well on various metrics, it’s kind of important that those metrics actually correlate with your goals.”

            the goals have to be direct to the subject tested not generalized life goals.
            all tests whether 5 question quizes or final exams, or SAT or AP exams – all of them relate back to the specific subject matter taught and learned.

            “If your metric has shortcomings, you need to be aware of that, and adjust your metrics or use alternate metrics.”

            I totally agree but virtually all of the major testing regimes have done this and still undergo continuing re- examination of the things you refer to.

            “Second, AP tests. Specifically with AP tests, you are using them as proxy measurements and assuming that schools that do not have large numbers of students taking AP are not good schools.”

            No, not necessarily. If the school scores high on SOLS and still have low AP enrollment it may well be due to it being a smaller school or district with less resources available to offer AP.

            but if the school scores low on SOLs then it indicates lower achievement levels – which are not going to be qualified for AP – nor Gov School or dual enrollment or IB.

            “That despite the fact that many of these are tiny schools in rural districts and AP classes are likely to be extremely small and may, in many cases, not be financially feasible for these small schools.”

            yes. If the SOLs are high and AP courses offered are low -… you can see this in the stats though.

            “I suggested using dual enrollment instead – and dual enrollment numbers are actually a metric on the state evaluation, if you look at them, and in the exact same section as AP – as potentially a better fit for these small districts.”

            maybe… who pays the tuition for DE? If the school has a large number of economically disadvantaged… would DE still be a good proxy?

            “Third, you questioned the ability to get into college without taking AP tests – about 28 of the top 100 colleges on the US News reports do not require the SAT or ACT. The best known ones I can think of offhand are Sarah Lawrence and Mount Holyoke, but FYI, the University of California discussed eliminating the SAT requirement a few years back as well, although thus far it has not done so.”

            and I agree.. what I said is that you limit your options to far fewer schools if you do not take the measurements that such schools want as a condition of consideration. SAT and AP are not the only things.. that would be exclusively used – but most Universities want SOME KIND of measurement from which to judge your suitability – although I will admit there are colleges out there that have few if any requirements.. and in those cases you as a student – kinda of get what you pay for – because the quality of the degree from the college also becomes a metric used by prospective employers.

            “Of Virginia universities, the best known ones that don’t require SAT or ACT for entering freshman who are qualified – meaning decent GPA – are George Mason and ODU. Further, qualified transfers from community colleges are admitted automatically, regardless of their SAT or ACT scores – to the best of my knowledge, that includes transfers to UVA and Tech.”

            we BOTH LIKE – DE – for the same reason. I think if the family can afford the tuition that DE is a much better path for a lot of kids who are not straight A types or who simply do not yet know what they want to be… or possibly they want to get a 2yr certificate – get a job – and then finish up the last 2 when they can afford it I know this sounds crazy but not all kids or parents can afford college – and they refuse to go into heavy debt to do so.

            “A number of traditional graduate programs will admit you with work experience without graduate tests like GRE and GMAT. Some do not even require the work experience – Northwestern and UCLA come to mind. The online programs almost never require them – including fully accredited programs and prestige programs like the MSCS from Georgia Tech.”

            been there done that, liked it. Some require that you take an exam for the subject to prove you know it. That’s part of the concept behind AP, right?

            “You keep referring to objective standards – but no matter how objective, if you are measuring the wrong things, you have GIGO – garbage in, garbage out.”

            I say you have to measure. It goes without saying that it has to be objective and legitimate. That’s why standards that are widely accepted have legitimacy.. despite potential for “herding”… All of these measures are challenged and re-examined from time to time as they should be.

            what I challenge is the idea that if the measurement has flaws that it then becomes a reason not to measure at all.. an excuse to not measure.

            Teachers oppose “teach to the test” NOT because they are opposed to the concept of measuring but because they think the content of the tests need to be fixed. They WANT the measurement but non-teachers misunderstand and think it means that people oppose the basic concept of measurement.

            “You also at a couple of points have put quotation marks around things I did not say. I did not say, waiting to see the outcomes. I did say that there are actual outcomes here, and whether or not measurements accurately correlate with those outcomes matter very much and needs to be validated.”

            I’m known to screw up on the quotes and to miss important passages sometimes.. just keep at me…

            outcomes are obviously hugely important but they cannot be the rationale for not measuring.. what outcomes do is validate that you not only did learn and it did show up on the tests – but that you actually know how to use what you were taught – well – to do a good job.

            this actually directly relates to my point about using what you learn – to be able to solve real-world problems – a quality that employers want more and more – beyond you academic credentials.

            “In particular, I am specifically and repeatedly questioning whether a low percentage of kids taking AP in a tiny district means that the school is not providing a good education.”

            look at the academic scores.. for the school as a whole and see how many kids are not only passing but proficient. The kids who pass but do not master the material usually are not prepared for AP – nor gov school, or DE, or IB either.. you don’t use ONE measure ..to decide -you look at more to get a picture, a context of the academic performance of the school.

            If it is a GOOD school – most of the kids will be accomplished at the academics and be able to do AP – as well as GS, IB and DE. If it is not -you’ll see depressed levels of all 4.

            “No one is saying, do not measure. I am saying, be aware of the limitations of the tools you are using, and realize that they do not fully encompass everything that is important to providing a quality education. Do not neglect important things that kids need to succeed in college and in life because they are not on a test.”

            i totally agree but you can’t not measure then wait for outcomes.. to see.

            you have to measure – continuously – it’s the name of the game , always has been way before we had AP or SAT … or SOLs.. Surely, you too have suffered through those nasty “pop” quizzes as I have.. or have experienced the dread when you know you have an hour to do something or the truly nasty take-home tests that beat you to death trying to go back and find all the answers. I lost a letter grade on a geography test one time because the instructor made what I thought was an offhand remark about were corks were made in Italy.. and it was on the test – a 5 question test. It did not help that my friend had taken the notes and got an A.

            “Things like ability to synthesize information from multiple sources, to form and argue complex concepts, and to do independent research on a topic are as important as knowing who was fighting in the war of 1812.”

            see – we agree! you have to be able to 1. master the academics and 2 – be able to use them to do real problem solving!

            “Life is NOT teaching to the test. Many of the most important accomplishments in life are taking what you have learned and creating new things, where there is no book to fall back on. No one starts a business or makes a discovery learning to the test.”

            it’s minimum competence verses excellence.. – we agree.

            “Education is fundamental, but most of what we learn and what we use is not measured in a standardized way and most of it cannot be measured easily.”

            well…. okay

            “Also, just to correct you, getting a college degree is an outcome – what on earth do you think outcome means?”

            it means the things you did in College that could be measured that then entitled you to the degree. Without passing the tests for the subjects, you get no “outcome”.

            “Outcome is the term used for the real life result. Did you graduate from high school? That’s an outcome. Did you get into college? That’s an outcome. Did you pass your class? That’s an outcome. Did you get a job? That’s an outcome. Did you get on the honor roll? That’s an outcome? What was your GPA? That’s an outcome.”

            we agree.. but how do you get those things? you test then you get them. the test is what determines whether you get them or not.

            What did you think outcome meant in this context?

            see above.

            and I’ll agree with your premise to the extent that the founding fathers and scientific leaders (like Newton, Edison, etc) – were largely self-taught and did not succeed because they were taking tests… and SATs, etc.

            see.. we agree!

            😉

            I appreciate that we can go back and forth and keep it civil!
            thank you

          3. virginiagal2 Avatar
            virginiagal2

            I wrote -“Third, you questioned the ability to get into college without taking AP tests ”

            Yikes. That should read, “Third, you questioned the ability to get into college without taking standardized achievement tests.”

            That part of the discussion was not about AP, but about college admissions tests – not sure why I typed AP in there. I think I had just typed AP so many times, I was on auto-pilot.

            My apologies.

          4. larryg Avatar
            larryg

            well I understood what you meant, I think.

          5. virginiagal2 Avatar
            virginiagal2

            Hi Larry –

            I don’t accept anything unconditionally – it’s actually pretty common for educational changes to come in without a lot of upfront assessment – too much politics and blind hope, and not enough critical thinking in the implementation.

            Even when studied, it’s very common to see correlation and causation confused, with causation assumed when interpreting studies, and it’s very common to see studies that have major flaws used to drive policy decisions.

            For that matter, this isn’t just education – it applies to business, scientific, and medical studies too. Go back and look at the Brookings study we were talking about, that excluded businesses that don’t have employees, despite the fact that is becoming a more important model for new startups.

            I definitely don’t think that the newer testing protocols such as the ones for Common Core have been validated thoroughly – there is even some question about the developmental appropriateness of the early standards.

            For AP, for those that pass the test with a 3 or better, benefits include that it correlates with majoring in the subject. But, and I didn’t expect this, kids who don’t pass basically appear not to have benefited at all.

            There are questions, from educational researchers who are not opposed to AP, as to whether it’s better than old-fashioned advanced and honors classes, or dual enrollment.

            AP has the support of a corporation, which makes it easier to use, and the lure of college credits. However, some colleges do not accept AP credits, and others recommend against using AP to replace foundational courses especially in one’s major.

            For SAT and ACT, there is some correlation, but it doesn’t appear to be equally valid for all groups. For one for the first universities that made SATs optional, the differences both in completion and GPA was less than a percent between those that took the tests and those that chose not to.

            In general, re college achievement, test scores correlate about as well as high school grades. The two together raise accuracy, and tests can help pick out kids that are underachieving but talented.

            These critiques of tests shortcomings are fairly widespread. I’m not coming up with some weird fringe viewpoint that no one else has suggested. And I’m not arguing against all testing or even against testing in general, just for careful interpretation.

            See discussion at Harvard Graduate School of Education at http://hepg.org/hel-home/issues/26_3/helarticle/putting-ap-to-the-test_466#home and also the link “Do AP and IB Courses Have Merit?” to the right.

            For the cost of dual enrollment – I actually don’t know who pays for the cost today. I would think something could be worked out with state colleges. Especially if it allows more kids to take advanced classes at less cost, it seems like something worth investing in.

            I don’t disagree that measurement is important and helpful. I do have less trust in standardized tests, and I think subjecting them to critical analysis is pretty important.

            No one has suggested not to measure at all, or to just rely on outcomes. My points are actually more, “grain of salt” and “go back to ground truth periodically.”

            BTW, I first ran across the concept of ground truth in reading about geology – in the context of comparing air surveys to checking what is actually on the ground when looking for minerals – but the concept is nearly universally applicable.

          6. larryg Avatar
            larryg

            “I don’t accept anything unconditionally – it’s actually pretty common for educational changes to come in without a lot of upfront assessment – too much politics and blind hope, and not enough critical thinking in the implementation.

            Even when studied, it’s very common to see correlation and causation confused, with causation assumed when interpreting studies, and it’s very common to see studies that have major flaws used to drive policy decisions.”

            and rampant now days

            “For that matter, this isn’t just education – it applies to business, scientific, and medical studies too. Go back and look at the Brookings study we were talking about, that excluded businesses that don’t have employees, despite the fact that is becoming a more important model for new startups.”

            agree

            “I definitely don’t think that the newer testing protocols such as the ones for Common Core have been validated thoroughly – there is even some question about the developmental appropriateness of the early standards.”

            Common Core is not about testing. It’s about standards. You can test it in a number of different ways. You can have several different states adopt common core standards and use very different testing protocols.

            “For AP, for those that pass the test with a 3 or better, benefits include that it correlates with majoring in the subject. But, and I didn’t expect this, kids who don’t pass basically appear not to have benefited at all.”

            talk about high stakes!

            “There are questions, from educational researchers who are not opposed to AP, as to whether it’s better than old-fashioned advanced and honors classes, or dual enrollment.”

            and there should be – but it should not be an excuse to choose _nothing_ because you cannot find the perfect one.

            “AP has the support of a corporation, which makes it easier to use, and the lure of college credits. However, some colleges do not accept AP credits, and others recommend against using AP to replace foundational courses especially in one’s major.”

            agree.

            “For SAT and ACT, there is some correlation, but it doesn’t appear to be equally valid for all groups. For one for the first universities that made SATs optional, the differences both in completion and GPA was less than a percent.”

            none are mandatory standards – they are adopted by the institutions that choose the kinds of criteria they want for their institutions.

            “In general, re college achievement, test scores correlate about as well as high school grades. The two together raise accuracy, and tests can help pick out kids that are underachieving but talented.”

            they can, I agree.. again.. you’re dealing with others who have chosen their requirements.. for entrance…

            “These critiques of their shortcomings are fairly widespread. I’m not coming up with some weird fringe viewpoint that no one else has suggested. And I’m not arguing against all testing or even against testing in general, just for careful interpretation.”

            and again – it boils down to what you do pick – as opposed to picking none because they all have flaws.

            “See discussion at Harvard Graduate School of Education at http://hepg.org/hel-home/issues/26_3/helarticle/putting-ap-to-the-test_466#home and also the link “Do AP and IB Courses Have Merit?” to the right.”

            I’m actually not a fan of AP – my view of it is pragmatic.

            “For the cost of dual enrollment – I actually don’t know who pays for the cost today. I would think something could be worked out with state colleges. Especially if it allows more kids to take advanced classes at less cost, it seems like something worth investing in.”

            I agree but there are a bunch of players involved and the 4-year folks don’t like the Community colleges poaching their money.

            “I don’t disagree that measurement is important and helpful. I do have less trust in them, and I think subjecting them to critical analysis is pretty important.”

            pick the ones that though flawed .. do measure as opposed to picking none because all are flawed.

            “No one has suggested not to measure at all, or to just rely on outcomes. My points are actually more, “grain of salt” and “go back to ground truth periodically.”

            agree.. but look up formative vs summative assessments..

            “BTW, I first ran across the concept of ground truth in reading about geology – in the context of comparing air surveys to checking what is actually on the ground when looking for minerals – but the concept is nearly universally applicable.”

            archaeology now being done with drones … 3d printing of raptor drones – who chase real birds away from dangerous places… jobs jobs jobs..

            😉

            drones are the new internet… my very opinionated 2 cents…

      2. TooManyTaxes Avatar
        TooManyTaxes

        I agree with Larry; we need some type of measurement of results. Otherwise we measure school success by inputs. I’ve heard lots of parents argue schools need to spend whatever it takes to deliver the services they want for their children. Whether the money produces any positive, measurable results is immaterial. That’s insane. Lack of measurement in government is why we had $600 toilet seats. But it also keeps the professional caring class dutifully employed.

        1. larryg Avatar
          larryg

          “measurement”

          it’s everything.. how would you every know for anything whether you achieved what you intended to achieve – whether it’s education or the amount of salt in a product or the amount of effective ingredients in a drug?

          Everyone who drives a car is 100% reliant on the company to insure the brakes work as designed.. that the air bags don’t go off unless they are supposed to.. that when the speedometer says 55 – it IS 55 and not 65…

          it’s as fundamental as just about anything in our lives.

        2. virginiagal2 Avatar
          virginiagal2

          Hi TMT –

          I’m not saying don’t measure results. I’m saying, make sure the results you measure actually make sense, and don’t dismiss what you can’t measure.

          Things like ability to summarize and write up information, work ethic, ability to learn on their own, and so on are hugely important and shouldn’t be dismissed because there’s not a good way to measure them in a standardized testing environment.

          I’m also suggesting that we need to validate any measurements – in other words, when we’re using them to evaluate students and schools, do those measurements actually correlate with high school graduation, success in college, etc. If we’re not measuring the right thing, we can’t evaluate performance.

    2. larryg Avatar
      larryg

      re: AP

      virginiagal2

      look at this:

      National Rankings Best High Schools

      http://www.usnews.com/education/best-high-schools/national-rankings?int=974b08

      and notice 2 things:

      1. metric of AP the top schools in the country look like this:
      100% Tested (AP – 100% Passed (Ap)

      2. – look down the list of the top 100 and see how many Virginia schools are in that list – and again notice the AP metric…

      then GOOGLE ” does AP improve your choice of colleges”

      12 Tips for Getting into the College of Your Choice

      College admission officers carefully assess your high school grades, courses, test scores, essays, activities, recommendations, and interviews, if required. You will increase your chances of getting into the colleges of your choice by following these twelve tips:

      Get the best possible grades you can during ALL four years of high school. Grades are extremely important.

      Take academically rigorous classes ALL four years. You should carry as many challenging courses as you can handle—college prep, Advanced Placement (AP), honors, and International Baccalaureate (IB).

      Practice taking the SAT or ACT. Become familiar with the types of material covered and the test directions. Take the PSAT during your sophomore year. Determine what knowledge and skills you lack and master them for the actual tests. Take advantage of free online SAT or ACT materials, study guides, practice tests, tutors, and prep courses before or during your junior year.

      Try taking both the SAT and ACT. Colleges will accept either test. You may do better on one test than the other. This will boost your chances for admission. Take the SAT or ACT more than once if you are not satisfied with your scores.

      Take SAT Subject Tests and AP Tests. Competitive colleges may require you to take some of these exams and they take note of exam results. Only AP scores of five (5) are accepted by top colleges.

      Can a kid do none of the above and do well? Of course.

      is the advice above wrong? of course not.

      notice the heavy emphasis on tests and AP…

      1. virginiagal2 Avatar
        virginiagal2

        Hi Larry –

        You are kind of making my point for me.

        First of all, I honestly do not care, at all, what US News thinks are the best high schools. US News is not the king of high school evaluation, and some of the problems we are having with colleges and high schools are a direct result of our obsession with rankings.

        Second, I do not think 100% taking AP is a reasonable or desirable goal for the population at large. That’s the metric they chose for the top high schools because it’s what well educated professionals care about, and that’s who will buy their magazine with its list of rankings.

        You will not have 100% of kids wanting to go to college, and you do not have 100% of kids needing to go to college to do what they want to succeed in life. But someone who wants their kid to go to Harvard or Stanford is going to care, a lot, about rankings.

        Third, I actually do not remotely care how many Virginia schools are on a US News list that is essentially designed to sell magazines. I have no reason to think that metric matters.

        I care about how well our schools are doing. I do not care about lists that turn things into competitions, when the things they are ranking are inherently not competitive.

        A great high school in Massachusetts does not make my local high school worse. A bad high school in Connecticut does not make my local high school better. But both will affect where my local high school falls in the rankings.

        Fourth, AP can help you get into college – it can be used as a metric. So can many other things. For example, if you are going to measure college readiness, how about dual enrollment, where you take an actual college course, either in person or online?

        That seems likely to be more educationally beneficial, it automatically creates college credit without taking another test, and it actually measures something real – a real college course, as opposed to something that is supposed to be equivalent – directly, rather than by proxy with a single test.

        For very small high schools, dual enrollment may be more feasible than AP. Distance ed, with a teacher as backup – I’m not suggesting distance ed without a local resource – makes it possible for a kid anywhere to take a college course, no matter how remote the area they live in.

        Not every high school is in the same position. What is best for their students is not going to be the same for every high school. It’s more important to do a good job than to fit a standard metric.

        Academically rigorous courses are important. AP is not the only set of academically rigorous courses, although it has been crowding out advanced courses that do not have an interested corporation advocating for them.

        Our goal, as a state, is to educate our kids well. That includes taking measurements with a grain of salt where appropriate.

        There’s a nice and IMHO fair evaluation of AP, looking at strengths and weaknesses, over at http://hepg.org/hel-home/issues/26_3/helarticle/putting-ap-to-the-test_466#home

        I am NOT against AP. I am against the idea that it’s the only way for kids to get rigorous courses.

        1. larryg Avatar
          larryg

          AP is NOT the only way… there are other ways..but the point here is that all of those ways are measured.

          I also totally support Career and Tech certificate programs, dual enrollment, etc.. but again – the point is that whether you’re earning a College Degree or a vocational certificate – they measure your knowledge and in doing so it signifies that you have some minimum level of competence.

          You certainly have the option of NOT doing those – but then the burden is on you as an individual to demonstrate to a prospective employer that you DO have some DEFINED level of competence beyond your self opinion.

          Internships are excellent ways to demonstrate actual competence beyond what might be on a test.

          and some fields, especially businesses – what’s more important is not that you have an MBA from some prestigious University – but can you successfully run a business and on that point – you’ll get me to agree with you.

          High School rankings are information. The schools that proclaim 100% AP enrollment are information for people who want to go that route or for organizations to use as benchmarks for prospective applicants.

          it’s your choice but what you do – does affect your range of options and many folks want as many options as they can achieve.

          we cannot be all Einsteins.. I’m certain not.. but I worked for and with some very intelligent and educated people that showed me how I had limited my own options .. by not doing the things they had done to increase their own opportunities.

          1. virginiagal2 Avatar
            virginiagal2

            Hi Larry –

            I agree with a lot of what you say. Two quick points, and I’ve already said most of this –

            First, it’s really important to measure the right things, and measure them as accurately as possible.

            Second, some vitally important things can’t be easily measured.

            Offhand, examples would be work ethic, research skills, and ability to analyze and write a coherent analysis with new information. Those things are vitally important to real-life success in most fields – not just business. Schools should not forget these skills, which are vital to college and work success, and they are not easy to measure in standardized tests.

            To me, dual enrollment – where high school students take actual college classes and are enrolled in both college and high school – is a more direct measurement of college readiness than someone taking an AP class.

            If you take a college course and get a good grade in it, that’s an actual outcome where you’re directly measuring the goal of college-readiness. The AP exam is a a proxy for college work, rather than a direct measurement.

            Dual enrollment may – note may – be more feasible for very small schools, particularly if a way could be worked out to do it distance ed, perhaps in partnership with a flagship school like UVA that both does educational research and offers majors in most AP subjects?

          2. larryg Avatar
            larryg

            “Second, some vitally important things can’t be easily measured.”

            core academics are not one of them. Most tests do test what can be measured. but agree some things cannot.

            the point is – you still have to measure in order to determine competence. You can’t measure some things, but you can enough of the other things and should.

            “Offhand, examples would be work ethic, research skills, ability to analyze and write a coherent analysis with new information. Those things are vitally important to real-life success in most fields – not just business. Schools shoujld not forget these skills, which are vital to college and work success, and they are not easy to measure in standardized tests.”

            the tasks you construct to measure – can be designed to show if you can do some of these things. remember TERM PAPERS?

            the one test that they are having trouble automating on the SOLs is writing. A particular scenario is presented to the student and they have to write about it – to show that they understand it and can synthesize the concepts embodied it in to apply to a similar situation. so far these things have to be actually read by another human.

            “To me, dual enrollment – where high school students take actual college classes and are enrolled in both college and high school – is a more direct measurement of college readiness than someone taking an AP class.”

            I don’t think DE courses are as rigorous as AP… most community college subjects tend to be down a notch from 4-year – that’s why you have to get higher grade for credit transfer.

            “If you take a college course and get a good grade in it, that’s an actual outcome where you’re directly measuring the goal. The AP exam is a a proxy for college work, rather than a direct measurement.”

            it’s both. You are taught college level material and have to pass an exam if you are going to get credit – similar to DE.

            “Dual enrollment may – note may – be more feasible for very small schools, particularly if a way could be worked out to do it distance ed, perhaps in partnership with a flagship school like UVA that both does educational research and offers majors in most AP subjects?”

            I’m big on Community Colleges – especially for the students that are not on a straight 4-year track (for a wide variety of reasons) but many k-12 are not really close enough so they have to have satellite facilities – if they can justify the cost – for the space and instructor.

            but you know… one of the BIG issues with distance learning is whether or not the student actually learned… and deserves credit… right?

          3. virginiagal2 Avatar
            virginiagal2

            Hi Larry –

            I would argue that writing, and the ability to take new information, analyze it, and present that analysis coherently is in fact a core academic skill, and one of the most important ones for academic success in any discipline. I don’t think you can fully test that skill set in a standardized test setting.

            I agree, writing is almost impossible to grade automatically. Many of the efforts to do it are pretty bad – for example, the SAT writing test grading methodology rewarded length and complexity of words, and didn’t look at all to factual accuracy of any analysis. Thus the move away from it.

            There are things that you can measure pretty well. Then you get into the issue, how well does success with those things actually correlate with what you care about?

            Are we measuring the right things, and if we identify those things, are we measuring them right? Doesn’t mean don’t try, it means, clearly state your assumptions, validate your assumptions and where needed, adjust accordingly.

            Are kids with lower SOL scores able to achieve with later help? For those kids, do remedial classes help? Who does it help? Some of them? Most of them? Is there a way to identify who can succeed with more help or is it more, throw it against the wall and see what sticks?

            I hear a lot of criticism about students taking remedial classes, but if a remedial class is able to get a kid on a solid career path, it seems like a good investment.

            Re term papers – yes, that’s my point. Teaching to the SOL tests cannot be done to the exclusion of things like research papers, projects, etc. That’s what I’m talking about when I say teaching solely to the test can be problematic.

            In order to get kids to learn and actually absorb material, you need to present things in an engaging way. Do you know how I remember what few facts I have absorbed about the war of 1812? The novelty song “Battle of New Orleans.” Not kidding. The song doesn’t have all the facts about the war that I remember, but that’s the hook that my knowledge is hung on.

            Dual enrollment doesn’t have to mean community college – it may now in practice, but there are a lot of options. If you go with distance ed, that definitely does not have to mean community college – if we are pressing top state universities to move into more distance ed, why couldn’t we offer distance ed options for high school students on AP subjects? Why can’t kids do in person classes, for that matter?

            Distance ed – not MOOCs, but traditional distance ed – can be quite rigorous. IMHO depending on the university, DE can be far more rigorous than AP. It doesn’t have to be from a community college and for that matter, it doesn’t have to be entry level. From what I can tell, Mason and ODU both offer quite a lot of undergraduate classes online, including at least some that appear to be AP-equivalent and at least some full degree programs at ODU.

            Also, pretty rigorous distance ed is currently available from non-state universities – the issue I could see with those would be cost – it would seem likely to be easier to work out a deal with a state university than with a private one or an out-of-state one.

            The difference between AP and distance ed, to me, is that with distance ed, pass or fail is not dependent on one exam. Depending on the subject, you are likely to do projects, tests, papers, research, programming projects, and so on, answering questions each week on the work, submitting projects throughout the semester, and then getting a final exam for some but not all of your grade.

            Where it would get tricky are subjects that require lab work other than computer lab. In those cases, DE is probably not a good option.

            For those kids within driving distance of a university that would work with the high school, I think an in-person class might be even better, if the logistics could be worked out. That would also allow lab courses.

            Distance ed, properly done, does validate that the student has actually learned. Many colleges require proctors for tests – which, for a dual enrollment student, would be easy, because you could take the test in school. I’m not talking about MOOCs – I’m talking about traditional distance ed classes.

            Probably the best example of where I’m uncomfortable with AP is the CS exam – I don’t think you can evaluate programming knowledge as well with a test as you can just looking at the kid’s programming projects over a semester.

          4. larryg Avatar
            larryg

            “I would argue that writing, and the ability to take new information, analyze it, and present that analysis coherently is in fact a core academic skill, and one of the most important ones for academic success in any discipline. I don’t think you can fully test that skill set in a standardized test setting.”

            it’s the skill that many of our kids do not acquire that is now required in the world of global jobs.

            “I agree, writing is almost impossible to grade automatically. Many of the efforts to do it are pretty bad – for example, the SAT writing test grading methodology rewarded length and complexity of words, and didn’t look at all to factual accuracy of any analysis. Thus the move away from it.”

            I used to think this but I’m changing my mind – and you can prove it to yourself with GOOGLE.

            try keying in a phrase worded differently perhaps with similar or different key words and see what GOOGLE makes of it. It fascinates me that an algorithm in software is parsing my words – for meaning… Siri on phones is similar but more easily overwhelmed.

            “There are things that you can measure pretty well. Then you get into the issue, how well does success with those things actually correlate with what you care about?”

            well .. if you are training to become a brain surgeon or airline pilot…. you want SOMETHING!

            “Are we measuring the right things, and if we identify those things, are we measuring them right? Doesn’t mean don’t try, it means, clearly state your assumptions, validate your assumptions and where needed, adjust accordingly.”

            agree.. again.. you can go on forever trying to find the right thing.. but the failure mode is to do nothing while you search.. for perfection.

            pick something, do it.. look for better…when you find it – backfit it.

            “Are kids with lower SOL scores able to achieve with later help?”

            not if the stuff later on is built on earlier foundations

            “For those kids, do remedial classes help? Who does it help? Some of them? Most of them? Is there a way to identify who can succeed with more help or is it more, throw it against the wall and see what sticks?”

            they have a saying in software engineering. If you find an error in requirements or even design – it’s a thousand times cheaper to fix than finding it in the finished software..

            “I hear a lot of criticism about students taking remedial classes, but if a remedial class is able to get a kid on a solid career path, it seems like a good investment.”

            the longer they go without remediation – the harder it is.

            “Re term papers – yes, that’s my point. Teaching to the SOL tests cannot be done to the exclusion of things like research papers, projects, etc. That’s what I’m talking about when I say teaching solely to the test can be problematic.”

            “teaching to the test” .. is a reason to – 1. find better ways to test or 2. discredit testing as valid?

            “In order to get kids to learn and actually absorb material, you need to present things in an engaging way. Do you know how I remember what few facts I have absorbed about the war of 1812? The novelty song “Battle of New Orleans.” Not kidding. The song doesn’t have all the facts about the war that I remember, but that’s the hook that my knowledge is hung on.”

            I have teacher friends… they all say – it’s all about presenting the material so that kids maintain interest levels… some even joke that embedding lessons in video games might be an awesome solution! In other words..the kids don’t go up on the difficulty levels until they master the videos with the lower level lessons!

            Ironically (or not), the Military looks for kids who are good at video games to be drone pilots!

            “Dual enrollment doesn’t have to mean community college – it may now in practice, but there are a lot of options. If you go with distance ed, that definitely does not have to mean community college – if we are pressing top state universities to move into more distance ed, why couldn’t we offer distance ed options for high school students on AP subjects?”

            I like internships…. also but distance learning for AP – totally agree. It would be a boon for smaller schools with limited resources for the advanced kids.. let them excel!

            “Distance ed – not MOOCs, but traditional distance ed – can be quite rigorous. IMHO depending on the university, DE can be far more rigorous than AP. It doesn’t have to be from a community college and for that matter, it doesn’t have to be entry level. From what I can tell, Mason and ODU both offer quite a lot of undergraduate classes online, including at least some that appear to be AP-equivalent and at least some full degree programs at ODU.”

            agree. there is a problem with authenticating who is on the other end taking the material.. they have to fix… we don’t want surrogates taking tests for bogus students.

            “Also, pretty rigorous distance ed is currently available from non-state universities – the issue I could see with those would be cost – it would seem likely to be easier to work out a deal with a state university than with a private one or an out-of-state one.”

            well.. except.. the Universities are like newspapers these days..they don’t like the loss of revenues.. while delivering the same content!

            “The difference between AP and distance ed, to me, is that with distance ed, pass or fail is not dependent on one exam. Depending on the subject, you are likely to do projects, tests, papers, research, programming projects, and so on, answering questions each week on the work, submitting projects throughout the semester, and then getting a final exam for some but not all of your grade.”

            I support this. Take the final exam first. If you pass it, you get credit – and you move on to the next – until the final exam demonstrates you don’t know and need to go back and learn. I see no reason why any kid cannot do this – with any subject but especially those who don’t have financial resources.

            “Where it would get tricky are subjects that require lab work other than computer lab. In those cases, DE is probably not a good option.”

            agree… so you have to show up in person to do the labs?

            “Distance ed, properly done, does validate that the student has actually learned. Many colleges require proctors for tests – which, for a dual enrollment student, would be easy, because you could take the test in school. I’m not talking about MOOCs – I’m talking about traditional distance ed classes.”

            we’re preaching to each other here! but none of this is possible if the kid has a lousy core academic education. It’s fundamental to future opportunities whetherit is learning in person or by distance.

            “Probably the best example of where I’m uncomfortable with AP is the CS exam – I don’t think you can evaluate programming knowledge as well with a test as you can just looking at the kid’s programming projects over a semester.”

            Did I tell you that I graduated from high school and did not go to college and got a job then went to Community College – where they had entrance tests – and then informed me that I had CRAPPY core academic skills and had to take remedial classes before I could even take their entry level courses.

            My ego was destroyed… but it needed to be…

            At that time, I was a gopher for scientists and engineers and I knew full well that I

            … and watch this word… did not ….._measure_ up…

            😉

            I had the last laugh because as smart as they were – they totally sucked at configuration management (maintaining multiple versions of software and data) – implemented via networked database.

  8. ksmith8953 Avatar
    ksmith8953

    What many of the EDWONKS do is concerning themselves with why poverty is a factor and never consider why will is a more important factor. In Southwest Virginia, there is a “will” by the people in the community (educational community and other communities) for all students to do well, poor or not. There are other factors as well. In many places the school system is the largest employer and the “proudest” employer — teachers and principals and other administrators come from families with rich DNA from parents who were teachers and principals and administrators. Failure is not an option. Complaining is not an option. Get-er-done is the only option. I have worked with this community for 14 years and I find it to be a place that believes that the people attached to the school want to do and will do what is best for kids.

  9. ksmith8953 Avatar
    ksmith8953

    With regard to high-stakes assessments — the pendulum is swinging back, but it need not swing to far. Let’s not forget why we moved to high stakes testing in the first place. Have Nots were not making the grade to get into colleges like the Haves. I agree that testing is a little over the top. What we need to focus on is teacher effectiveness in knowing not only “IF” a student learned a specific standard, but “WHAT” to do if the student did not. This should be done every day as part of instruction. This is formative rather than summative assessment. The fact is, when you ask a 7th grade or 9th grade teacher how he/she is helping a student that cannot read well, most can tell you the student can’t read, few can tell you how to fix the reading problem. We are employing dentist to do brain surgery.

    1. larryg Avatar
      larryg

      re: high stakes – I distinguish between “test” and “assessment”.

      “assessments” are quiet information things that a teacher uses for each student to assess their progress and their needs. UVA has something called PALS that I have come to believe is a better way …

      but testing is also required but if you are also doing frequent assessments – the tests themselves reduce in “high stake”.

      re: ” … most can tell you the student can’t read, few can tell you how to fix the reading problem. ”

      in k-5 this is what PALS is for. It breaks down reading into constituent skills and tells the teacher WHY he/she can’t read… and from there – the kid goes to a reading specialist, sometimes a TITLE 1 specialist .. and gets back on grade level.

      If the kid does not catch up in fundamental reading skills by the time they go to Middle School – it becomes a much more difficult problem and as stated, at the MS level – the teacher may not have the skill to determine what exactly the deficit is – but even if they did – they’re not paid to rehabilitate .. usually.. they have other things they’re supposed to be doing for the whole class.

  10. ksmith8953 Avatar
    ksmith8953

    “the teacher may not have the skill to determine what exactly the deficit is – but even if they did – they’re not paid to rehabilitate .. usually.. they have other things they’re supposed to be doing for the whole class”—Really? Tell this to the parents of 60% of the 6th graders in a low performing middle school. This is the teacher’s job and they don’t know how to do it.

    For your information, although I think the PALS assessment is a good tool, it assesses phonemic awareness, not comprehension. Big difference. I can teach a kid to decode anything, but does he/she have the vocabulary knowledge to be able to understand what he/she is decoding. For high-poverty K-3 students, assessing only phonemic awareness is a big mistake. They don’t come to school with the same vocabulary. Also, when 70% of the students are not reading on grade level in K–3, often evident in high-poverty schools, there aren’t enough reading specialists to go around, thus we have the 60% not reading on grade level in 6th grade.

    Distinguish between summative and formative and you will hit the mark.

    Good discussion.

    1. larryg Avatar
      larryg

      re: ” For your information, although I think the PALS assessment is a good tool, it assesses phonemic awareness, not comprehension. Big difference.”

      you are correct.

      “I can teach a kid to decode anything, but does he/she have the vocabulary knowledge to be able to understand what he/she is decoding.”

      in the weeds for many but yes.. they manipulate great.. but do they understand? PALS is K-5 … and not yet in all higher k-5 grades, right?

      I’m told that math has no equivalent assessment protocol .. yet.. or because it’s so different… thoughts?

      ” For high-poverty K-3 students, assessing only phonemic awareness is a big mistake. They don’t come to school with the same vocabulary. Also, when 70% of the students are not reading on grade level in K–3, often evident in high-poverty schools, there aren’t enough reading specialists to go around, thus we have the 60% not reading on grade level in 6th grade.”

      My understanding is that this is what your Title 1 and Reading specialists are for.. but your point about 6th grade is at the heart of the issue for the economically disadvantaged demographic in my opinion. If they get there and are not proficient – they may be doomed depending on what the district does for MS kids that are behind.

      Distinguish between summative and formative and you will hit the mark.

      http://www.cmu.edu/teaching/assessment/howto/basics/formative-summative.html

      and YOU do know your STUFF!

      Good discussion.

      Bacon provides the Blog – and the people who contribute do provide informative discussions.. some of us are more talkative and opinionated than a body ought to be – my bad.. but I learn much from others.. and correct wrong ideas sometimes…

      thanks

  11. Elle D Avatar
    Elle D

    “This is deepest, darkest Appalachia.” Can you please clarify this comment?

    1. James A. Bacon Avatar
      James A. Bacon

      It’s hyperbole.

      If you’re worried that I buy into common stereotypes of Appalachia, I do not. I lived in Blacksburg and Roanoke for five years and I spent four years of my journalism career traveling to the SW Virginia coalfields. As editor and publisher of Virginia Business, I made many more trips to region. I have great fondness and respect for the people of Appalachia, as I hope I have made clear by my reference in this and other posts to the superior educational performance of many SW Virginia school districts — contrary to the stereotype.

  12. Elle D Avatar
    Elle D

    I was in fact worried, but your explanation makes better sense than what I feared. I appreciate the clarification!

  13. James A. Bacon Avatar
    James A. Bacon

    Comment submitted on behalf of Dave Johnson:

    I read your SOL articles and thought I might be able to shed some light on West Point.

    We have some kids in Eastern Henrico who actually go to WP schools. Apparently, out-of-district kids can apply to WP schools and, if accepted, pay tuition to attend.

    From their website:

    Demographics
    Enrollment: 806
    Residents: 661
    Non-Resident Tuition Students: 145

    So if an entire school system can choose nearly 20% of its student body, that may have some bearing on the numbers.

    I really enjoy the blog – especially the transit and urbanism articles. Thanks for what you do.

Leave a Reply

Read More