Northam Administration Information Technology Failures Continue

Image by Mediamodifier from Pixabay

Help! WJLA is reporting that the State of Virginia is using a 35-year-old computer system to process unemployment checks. The system has buckled, leaving 70,000 Virginians without their unemployment benefits.  In a stunning admission, Bill Walker, Director of Unemployment Insurance with the Virginia Employment Commission says, “We are right at the first of July now” when asked how far behind the process stands.

It seems obvious that ineffective processing of unemployment claims disproportionately impacts less affluent and minority Virginians. Yet this issue has been missing from the Ralph Northam COVID-19 updates I have watched.  Those press conferences have included discussions of the presidential election and a description of court cases involving Confederate statues but nothing about the real pain that the ineptitude of the Northam Administration is visiting on 70,000 Virginians, including many people of color.

Deja Vu. This is far from the first IT problem faced by the Northam Administration. The Daily Press outlined a series of errors, failures and missteps in July, 2019. Curiously, one of Governor-elect Northam’s first decisions was to disband the technology secretary role which had been a cabinet level position.  While some old school observers said that the technology secretary role was ceremonial and should be disbanded others contended that the role should have been revised and expanded rather than disbanded. Northam did add a diversity chief position to the state cabinet. One is left to wonder if the people now waiting five months for their unemployment benefits would rather have a chief diversity officer or a competent state IT operation that can deliver benefit checks on time.

The plantation elites don’t get it. In fairness to Northam, the problems with Virginia’s IT efforts go back to at least the Kaine Administration. Back then the battle centered around an outsourcing contract between Northrop Grumman and the state. State employees contended that the outsourcer was slow to deliver service. Virginia’s CIO at the time withheld payment from Northrop Grumman and was summarily fired  The matter was covered by Bacons Rebellion. As always, the solution going forward revolved more around The Virginia Way than competent management of IT. Northrop Grumman moved its corporate headquarters to Northern Virginia and everybody got back to work. Well, not really. The Luddites in the plantation elite who run the state considered the matter closed. But problems persisted and the Northrop Grumman contract was closed out in 2018.  “This is a new era in IT services for state agencies and VITA,” said Nelson Moe, Virginia’s chief information officer said at the time. Lawsuits between Northrop and Virginia were eventually settled with the state paying $35.8m of the $72m that Northrop claimed it was owed. Problem solved? Of course not.

More than a year after the legal settlement JLARC was once again blasting VITA for incompetence. Phone service was out to an animal safety testing field office for 27 days. Phone service! More than 40% of the equipment and services promised for delivery by the end of 2018 were not in place on July 1, 2019. On and on went the post-Northrop Grumman list of failures. “I’m flabbergasted,” JLARC vice chairman, Del. Steve Landes, R-Weyers Cave, said. He was flabbergasted before COVID. One wonders how Del. Landes might feel today about 70,000 Virginians waiting for their unemployment checks

New World Order. In 2011 Marc Andreessen famously quipped that “software is eating the world.” He claimed that every company needed to become a software company to keep up. More recently Forbes observed that “Software Ate The World, Now AI is Eating Software,” These trends are well understood and accepted in the executive suites and boardrooms of corporations around the world. Yet these concepts are completely foreign to our elected and appointed leaders in Virginia. Why? Because they are throwbacks to an era when technology was called “tabulating machines.” They run the state’s IT operation more like a candy store than a major entity.

If Virginia’s annual budget were annual revenues of a company, the state would rank #59 on the Fortune 500 list. Bigger than Goldman Sachs, Morgan Stanley, Cisco and Delta Airlines. It’s high time that technology vision and competent IT management become a significant part of what we expect from our elected officials. Certainly there are 70,000 Virginians waiting for unemployment benefits who would agree.


Share this article



ADVERTISEMENT

(comments below)



ADVERTISEMENT

(comments below)


Comments

138 responses to “Northam Administration Information Technology Failures Continue”

  1. Virginians are going hungry. Virginians are being evicted from their apartments. Naturally, some people blame food deserts or heartless landlords. I have yet to see a newspaper headline putting the blame exactly where Don has put it — with the antiquated IT system of Virginia’s unemployment insurance system.

    Next question: Northam allocated $50 million in federal CARES money to help Virginians facing evictions. How much of that money has actually been dispersed?

    I asked that question of the PR person in charge of the program but never got an answer. I suppose I’ll have to file a FOIA request.

    1. djrippert Avatar

      I strongly suspect that Northam is a technological illiterate. The kind of person who still marvels at digital watches. The Princes of the Byrd Machine were pretty much left in the dust by the technological revolution of the last 40 years. However, that doesn’t excuse his apparent inability to appoint competent people who are technologically literate. Which leads me to my sense that Northam has never managed anything bigger than a team of nurses. Competent leaders who manage organizations with 55,000 employees know they need to make sure the technology is working. This is 2020 not 1970 … no matter what the plantation elite may prefer.

      Beyond the obvious pain involved in being denied benefits that are deserved there are huge opportunities for improved efficiency in government through more effective automation. Like limiting Medicaid fraud. I know how the credit card companies check credit card fraud. Similar systems to limit Medicaid fraud ought to be implemented in Virginia (and every other state for that matter).

      Why aren’t the responsible parties for this disaster being terminated? The checks have been behind for many months and it’s still a disaster. Meanwhile, people are calling VEC 30 times a day just to try to get through.

      1. idiocracy Avatar

        Northam is probably what is known as a “12 o’clock flasher”. Every clock in his house is flashing 12:00 because he can’t figure out how to set them.

        (As opposed to me, where every clock is flashing 12:00 because I set them and then a 2-second power outage causes them to flash 12:00. Oh, the problems a bigger capacitor would have solved…)

      2. LarrytheG Avatar

        COVID has screwed everything up:

        ” IRS has 1 million unprocessed returns and 3 million pieces of unopened mail after COVID-19 wreaked havoc on tax season

        The Internal Revenue Service still has about one million 2019 tax returns to process after the coronavirus pandemic and distribution of millions of stimulus checks upended operations, the agency’s commissioner told federal lawmakers this month.

        “We have done all that we can really do,” said IRS Commissioner Charles Rettig, talking to members of the House Ways and Means Oversight Subcommittee. He said the committee still hears from constituents who haven’t received their refunds this year.

        The approximate one million still-unprocessed returns are down from 2.5 million unopened returns in October, the last time Rettig testified about IRS operations in a tax season like none other. There’s now 3 million pieces of unopened mail, down from 5.3 million.

        There are currently 6.8 million returns in process — which, Rettig explained, are returns that have gone through the IRS systems.

        “On behalf of the Internal Revenue Service and every employee, for literally every American, we appreciate the patience and understanding,” Rettig said. “This is not an excuse, but I will say that our employees went through the same exact thing as every other American during COVID with respect to health and safety concerns.”

        1. djrippert Avatar

          Internet providers kept the networks up despite an unprecedented hike in usage as people worked and shopped from home. Videoconferencing companies like Zoom keep their applications running with vastly expanded use. Amazon.com’s systems continued to process orders. Grocers and retailers implemented pick systems and curbside deliveries.

          No, Larry – COVID didn’t screw everything up. Government screwed everything up.

          1. idiocracy Avatar

            COVID is being used as an excuse for all sorts of poor performance.

          2. LarrytheG Avatar

            The Government is not good at change. They don’t have to respond to the market and competitors!

            OTOH – The govt creates things like the GPS satelitte system which truly makes the world go round including your cell phone!

            Or the NOAA satellites.

            Or the Air Traffic control system.

            And for that matter – Social Security and Medicare which both seem to function well.

            Everyday – most of us end up putting our ives in the hands of those government things called traffic signals – and how many folks have been hurt or killed because of “bad” traffic signals?

          3. idiocracy Avatar

            Those things were created by contractors hired by the government to develop them.

          4. LarrytheG Avatar

            Pretty much everything the govt does is by private sector contractor. Right?

          5. idiocracy Avatar

            Larry, here you go, lives lost due to traffic signal misconfiguration:

            https://en.wikipedia.org/wiki/1995_Fox_River_Grove_bus%E2%80%93train_collision

          6. LarrytheG Avatar

            Oh, it HAPPENS, but how often? Last time it happened to you?

          7. idiocracy Avatar

            Well, Larry, as a matter of fact, several years ago I observed a traffic signal in the City of Manassas operating with a “yellow trap”.

            Fortunately I was not involved in an accident due to the issue, but I came very close.

            I made the “Traffic Signals Supervisor” aware of the problem–or attempted to. He came back with “There is nothing wrong with that traffic signal, the problem is the drivers”.

            The traffic signal remained in that condition for at least several more months until I made the city manager aware of the issue.

            https://en.wikipedia.org/wiki/Yellow_trap

            I am unaware of how many accidents resulted from this configuration, but the potential was there.

            I’m sure that, given the indifference displayed by the “Traffic Signals Supervisor” towards the problem, there exist other signals in Manassas with the same problem, even to this day. However, it is not my job to find them.

          8. LarrytheG Avatar

            Not familiar with “yellow trap” but it sounds like a conceptual thing as opposed to a malfunctioning system.

            I’m not saying there are no issues but if you think of how many times we encounter traffic signals in our lives – and make no mistake, out lives are involved, 99.xxx % work without loss of life.

            In fact, most deaths are due to driver errors and irresponsible acts, not traffic signals improperly designed or operating.

            That’s my point. Yes, you can find exceptions but the vast, vast majority of GOVERNMENT – designed, operated and maintained traffic signals function correctly without people getting hurt or killed. That’s the same government we say is a “fail”. It’s not.

        2. idiocracy Avatar

          Larry, the only part of the traffic signal that government actually has to do anything with is that they employ the idiot who misconfigures it with a yellow trap.

          All the rest of it is designed, produced, and installed by the private sector.

          1. LarrytheG Avatar

            Indeed as is GPS, NOAA and most of the DOD weapon systems.

            Your point? Is it Govt screwing up or the private sector?

        3. idiocracy Avatar

          One big difference between the government and the private sector is that the private sector isn’t protected by the concept of “sovereign immunity” for their screwups.

          For better or worse, the threat of litigation often serves as a motivator to do the right thing.

          1. LarrytheG Avatar

            That’s true but my experience is that the govt is pretty careful with things that can cause injury and death and it goes into their regulations also.

            VDH spends significant time and effort on making sure restaurants keep hot foods hot and cold foods cold.

            Without the Govt doing that – what would happen?

        4. idiocracy Avatar

          My experience is that the government (and private sector’s) care with things that can cause injury or death varies from place to place and is largely dependent upon the local culture’s attitude towards such things.

  2. LarrytheG Avatar

    Shoot, Virginia can’t hold a candle to the Feds:

    ” The IRS System Processing Your Taxes is Almost 60 Years Old

    One of the IRS’ most important tax-processing applications is old enough to be a grandparent, and officials warn a failure during tax season could have dire economic ramifications or delay tax refunds for 100 million Americans.

    The Individual Master File, a massive application written in the antiquated and low-level Assembly programming language, is comprised of data from 1 billion taxpayer accounts going back decades, and chiefly responsible for receiving individual taxpayer data and dispensing refunds.

    Despite hundreds of millions in spending, plans to fully modernize the application are more than six years behind schedule, and in a statement to Nextgov, IRS revised its new timeline for a modernized IMF to 2022.”

    I guess that makes the Feds honorary Plantation Elites also? 😉

    1. djrippert Avatar

      The Feds get some things right. In the 1980s I had a clearance and worked as a contractor for the military. Tip of the spear stuff. They had their act together.

      As for assembly language – that was my first programming assignment out of college. System 360 / BAL. Used the TPF operating system from IBM. One of my assignment was to train new BAL programmers. I work with them until they finally got their program working. Then I’d say that I just noticed they were using some registers much too frequently. I’d tell them they had to re-write the program to use the registers more evenly. We didn’t want to wear any of the registers out. The ones who burst into tears became process designers instead of programmers. The ones who said “fine, I’ll do it” became application programmers. The ones who said’ “registers don’t work that way” became architects.

  3. PackerFan Avatar

    Great post Larry! That has to make those 70,000 people waiting for their checks feel better! ” Dang, I might be homeless and hungry, but at least the good Old Dominion isn’t using equipment as old as what the IRS is!”, said no one.

    1. djrippert Avatar

      Good point. Elections have consequences and so does failure. Our failure to elect a competent governor has the consequence of 70,000 Virginians not getting benefits they deserve.

      All I keep thinking is that if the system is 35 years old it had to be creaking during the recession of 2009. Nobody wondered what would happen the next time there was a recession?

  4. idiocracy Avatar

    It’s probably an IBM AS-400. The SCC still uses one, which is obvious if you use the website to look up corporate data.

    1. djrippert Avatar

      I worked on those too. Started with System/34, then System/36 then System/38. The AS/400 was a huge leap forward. I went to a two week IBM school in Chicago to learn about the cool AS/400. RPGII then RPGIII. I think they finally got COBOL on the platform but I was gone to other things by then. Like an HP3000 system written in FORTRAN to run HMOs.

      Software development was the wild west in the 1980s.

      1. idiocracy Avatar

        I worked at a place where I maintained an AS-400. It was around 10 years old when I started there. I had never once before worked on an AS-400 but I am capable of utilizing the Internet to do things beyond looking at stupid memes and videos….

        After I left, it went down because the contractor they replaced me with had no idea how to keep it running. Never heard the full story, but I’m guessing a drive went bad. They ignored it. Another drive went bad. Bye-bye RAID 5. So nice to have known you.

      2. LarrytheG Avatar

        If you did military, then you know about embedded systems…right?

        Wanna know how old some of the fire control systems are on our ships and other weaponry?

        😉

        1. djrippert Avatar

          What I worked on for the military was classified and I was never told that it has been de-classified. All I’m going to say is that the people I worked with knew their stuff and their stuff worked.

          1. LarrytheG Avatar

            Everything in weapon systems is classified.. 😉

            but the age of the computer equipment on our ships is so old that it’s no longer classified.

            I think the point is that across the government – Fed and State and the Military that computer systems tend to not advance like they do in the private sector and part of the problem is the knowledge and skill of the employees as they do not pay the salaries that the private sector does for top-notch technological employees.

            It’s a constant battle in the military to try to maintain older systems while trying to upgrade.

            It’s a gargantuan job to try to coordinate software for ships that are 30, 40, 50 years old. It’s also a configuration management problem of gargantuan dimensions because every ship has a unique array of equipment because they were built in different years.

            It’s like Hertz trying to maintain a fleet that has 1970’s vehicles all the way up to 2020 vehicles.

      3. John Harvie Avatar
        John Harvie

        I started with Big Blue in 1964 and worked on 7044/7094 scientific machines which required a 1401/1410 to produce output from tapes since there was only a small operators’ printer on the 7000 series. The 1400 series machines used SPSS and involved “word marks” to delineate the end of a data or instruction element … some memory details fade a bit after a while. RPG was a tremendous advance when it arrived. A popular program of the day was a surveying program, COGO which closed traverses.

        1. djrippert Avatar

          Wow. I assume you were at Big Blue on or about April 7, 1964 when the 360 architecture was announced. In many ways the birthday of the modern computer. Microcode, the 8 bit byte, backwards compatibility, 32 bit registers, the IBM floating point architecture. I read Fred Brooks’ book “The Mythical Man Month” in awe of the sheer guts and grit that IBM showed in building that architecture.

          I worked at IBM years later and could only wonder at the incredible power of Big Blue in its prime. I saw the somewhat empty buildings and campuses in Somers, Raleigh, Austin, Poughkeepsie and wondered what those places must have been like when they were packed with people.

          Another seminal event in computing happened on March 13, 2006. That was the day Amazon Web Services sold its first cloud service (Simple Storage Solution) to a third party. My youngest son had been born just a few days prior. I still look at him sometimes and think to myself, “the cloud is still only in its adolescence” .

          1. John Harvie Avatar
            John Harvie

            Did not know of Fred Bro0ks’ book … thanks and I’ll look it up on Amazon.

            In a small sense “what comes around goes around” (or is it the reverse?). Later on I worked in IMD division on a timesharing product named Call/360 (later Call/370) which in some ways is analogous to today’s cloud computing, except that in those days we called it “distributed computing”. Our profits were obscene since we were able to charge an arm and a leg for some of our proprietary software.

          2. Reed Fawell 3rd Avatar
            Reed Fawell 3rd

            “the cloud is still only in its adolescence”

            While,

            Virginia is in its senescence.

          3. Nancy_Naive Avatar
            Nancy_Naive

            I knew there was a natural revulsion… I was CDC in the mid-70s. 😉

      4. Nancy_Naive Avatar
        Nancy_Naive

        Started on Wang 2200B, PDP-8, and some AN/YUK something or other. Moved up to the CDC Star-100, Cray-1. After that, nonspecific machines. Oh no. There was a Data General and a DEC in there somewhere. All those exciting machines and here I sit with the wife’s 10-year old hand-me-down iPad with OS-10, cursing the touch keypad.

        1. John Harvie Avatar
          John Harvie

          OMG! We’re cousins! … I went to CDC when SBC/IBM/CDC settled the consent decree. I was in DC, Richmond, Hampton and the Beach. Did we cross paths?

          1. Nancy_Naive Avatar
            Nancy_Naive

            Your name has always struck a bell, but I think I may be confusing you with someone else. I started at CDC straight out of college. Contracted to NAVELEX. I wrote a database management system and a spreadsheet for the WANG called AutoBase. The Navy used it well into the late 80s. Used self-modifying code. What a hoot!

  5. UpAgnstTheWall Avatar
    UpAgnstTheWall

    The WJLA article rather assiduously did not draw a causal link between the “35 year old system”, and didn’t even bother to inform the reader what that system is. If it’s a Windows based system from 1995 then, yeah, there are probably some problems there, but if it’s built on something sturdier like COBOL then it’s more likely to be a manpower than technical problem, and if it’s something like Unix/Oracle translated across to a Windows system then it’s a potential technical problem most likely with 32-bit database architecture trying to liaise with a 64-bit operating system, which would explain – to Steve’s point – why cracks didn’t appear in 2009 since that’s when Windows 7 launched with loading 64-bit OS as the default and the state probably didn’t get Windows 7 machines until 2012 at the earliest.

    But again, we don’t know, and to assert we do is sheer hackery.

    1. LarrytheG Avatar

      Anyone remember Y2K and how we all were expecting all heck to happen?

      We had scads of people on “standby” – like they were going to fix something right away if it broke.

      🙂

      1. idiocracy Avatar

        I also remember all the hyper-ventilating about “Clocks could be affected by Y2K and we don’t even know where they all are!”

        My guess at the time was that if there is a clock that nobody knows about, then nobody bothered to set it to the correct date and time and therefore it’s irrelevant as far as Y2K is concerned.

        1. LarrytheG Avatar

          We were worried about it for weapon systems and other critical national defense infrastructure. We spent years/months combing through code looking for stuff but some of it was so old and in code we could not fix – the compilers for that code were long gone… etc..etc..

          Computers are a mess but they are also the way the world works now.

          1. idiocracy Avatar

            There appeared to be a complete lack of recognition that many systems don’t care what the date and time are.

            Which is how you got people concerned that their computer controlled fuel-injected car wouldn’t start on 01/01/2000.

          2. LarrytheG Avatar

            Well, yes, but it does matter quite a bit with satellite systems and weapon systems….

          3. idiocracy Avatar

            It depends. A satellite that operates as a satellite typically does is a satellite that doesn’t need to know the correct time and date. All it does is repeat a radio signal.

      2. djrippert Avatar

        Actually the Y2K remediation work was largely done in India and kicked off the Indian offshoring wave that continues to this day. The systems almost all worked but the labor model was changed forever.

    2. djrippert Avatar

      If the system is 35 years old then it wasn’t written in windows. Nobody was writing batch check generating systems in Windows in 1985.

      My guess is the original system was an IBM S/370 mainframe written in COBOL with VSAM as the “database”. No doubt whatever it was originally it has been hacked, pathed and kludged over the years.

      Given that the system can produce checks but not keep up with demand it’s reasonable to assume that the logic works but the system won’t scale. That points to a rigid underlying architecture. They’ve been behind since May and it’s now December. If adding more memory would have solved the problem even Virginia’s state government would have done that by now.

      The bigger point is that they were either too negligent to know the scalability of a mission critical system or too lazy to do anything about it. Recessions aren’t exactly black swan events. COVID may be unique but suddenly having to start printing unemployment checks is not.

      The biggest point is that a recession started in April and our state is still five months behind getting out employment checks in December.

      That represent gross incompetence and culpable negligence.

      1. idiocracy Avatar

        The only way Windows would have anything to do with this is that the user interface/terminal emulator they use to connect to the system operates on Windows.

        And if they haven’t figured out how to get a 64-bit version of that app by now, they have even bigger problems.

        1. John Harvie Avatar
          John Harvie

          Not needling you but you must have been around IT for a while to use the phrase “terminal emulator “.

          Takes me back…

  6. Peter Galuszka Avatar
    Peter Galuszka

    It didn’t

  7. The problems cited are not unique to Virginia, are not new, and are not unique to government.

    Oversimplifying – IT is often treated as an expense, rather than an investment and those that run IT (Chief Information Officers) are not “at the table” of decision making to determine how and when to update, change, discard information systems. Plus, people (administrators, users, managers) hate to change. They get used to doing things a certain way, and changing systems is hard stuff.

    That is not to excuse it, but managing organizations require good, effective, knowledgeable, motivated leaders that understand how to bring about change.

    Often, governments are hamstrung by legislative guidance and codified regulations that result from previous laws. IT system instantiate these inadequacies.

    https://www.cio.com/article/3018436/study-finds-companies-are-failing-to-modernize-it.html

    https://www.altexsoft.com/whitepapers/legacy-system-modernization-how-to-transform-the-enterprise-for-digital-future/

    1. idiocracy Avatar

      “managing organizations require good, effective, knowledgeable, motivated leaders that understand how to bring about change”

      Good luck finding many of those in Virginia.

      1. Nancy_Naive Avatar
        Nancy_Naive

        Yes, yes. But they can be found. Those people write half baked opinions and make snarky remarks on blogs, so if they’d just look.

  8. LarrytheG Avatar

    Management that understands the importance of IT is better than it used to be but very different between private sector and Govt. Govt management can be totally disconnected from realities and not suffer from it.

    1. idiocracy Avatar

      Government IT managers will never have the Big Boss screaming at them that the outage is costing them $10,000 a minute.

  9. Virginians are going hungry. Virginians are being evicted from their apartments. Naturally, some people blame food deserts or heartless landlords. I have yet to see a newspaper headline putting the blame exactly where Don has put it — with the antiquated IT system of Virginia’s unemployment insurance system.

    Next question: Northam allocated $50 million in federal CARES money to help Virginians facing evictions. How much of that money has actually been dispersed?

    I asked that question of the PR person in charge of the program but never got an answer. I suppose I’ll have to file a FOIA request.

    1. djrippert Avatar

      I strongly suspect that Northam is a technological illiterate. The kind of person who still marvels at digital watches. The Princes of the Byrd Machine were pretty much left in the dust by the technological revolution of the last 40 years. However, that doesn’t excuse his apparent inability to appoint competent people who are technologically literate. Which leads me to my sense that Northam has never managed anything bigger than a team of nurses. Competent leaders who manage organizations with 55,000 employees know they need to make sure the technology is working. This is 2020 not 1970 … no matter what the plantation elite may prefer.

      Beyond the obvious pain involved in being denied benefits that are deserved there are huge opportunities for improved efficiency in government through more effective automation. Like limiting Medicaid fraud. I know how the credit card companies check credit card fraud. Similar systems to limit Medicaid fraud ought to be implemented in Virginia (and every other state for that matter).

      Why aren’t the responsible parties for this disaster being terminated? The checks have been behind for many months and it’s still a disaster. Meanwhile, people are calling VEC 30 times a day just to try to get through.

      1. LarrytheG Avatar

        COVID has screwed everything up:

        ” IRS has 1 million unprocessed returns and 3 million pieces of unopened mail after COVID-19 wreaked havoc on tax season

        The Internal Revenue Service still has about one million 2019 tax returns to process after the coronavirus pandemic and distribution of millions of stimulus checks upended operations, the agency’s commissioner told federal lawmakers this month.

        “We have done all that we can really do,” said IRS Commissioner Charles Rettig, talking to members of the House Ways and Means Oversight Subcommittee. He said the committee still hears from constituents who haven’t received their refunds this year.

        The approximate one million still-unprocessed returns are down from 2.5 million unopened returns in October, the last time Rettig testified about IRS operations in a tax season like none other. There’s now 3 million pieces of unopened mail, down from 5.3 million.

        There are currently 6.8 million returns in process — which, Rettig explained, are returns that have gone through the IRS systems.

        “On behalf of the Internal Revenue Service and every employee, for literally every American, we appreciate the patience and understanding,” Rettig said. “This is not an excuse, but I will say that our employees went through the same exact thing as every other American during COVID with respect to health and safety concerns.”

        1. djrippert Avatar

          Internet providers kept the networks up despite an unprecedented hike in usage as people worked and shopped from home. Videoconferencing companies like Zoom keep their applications running with vastly expanded use. Amazon.com’s systems continued to process orders. Grocers and retailers implemented pick systems and curbside deliveries.

          No, Larry – COVID didn’t screw everything up. Government screwed everything up.

          1. idiocracy Avatar

            COVID is being used as an excuse for all sorts of poor performance.

          2. LarrytheG Avatar

            The Government is not good at change. They don’t have to respond to the market and competitors!

            OTOH – The govt creates things like the GPS satelitte system which truly makes the world go round including your cell phone!

            Or the NOAA satellites.

            Or the Air Traffic control system.

            And for that matter – Social Security and Medicare which both seem to function well.

            Everyday – most of us end up putting our ives in the hands of those government things called traffic signals – and how many folks have been hurt or killed because of “bad” traffic signals?

          3. idiocracy Avatar

            Those things were created by contractors hired by the government to develop them.

          4. LarrytheG Avatar

            Pretty much everything the govt does is by private sector contractor. Right?

          5. idiocracy Avatar

            Larry, here you go, lives lost due to traffic signal misconfiguration:

            https://en.wikipedia.org/wiki/1995_Fox_River_Grove_bus%E2%80%93train_collision

          6. LarrytheG Avatar

            Oh, it HAPPENS, but how often? Last time it happened to you?

          7. idiocracy Avatar

            Well, Larry, as a matter of fact, several years ago I observed a traffic signal in the City of Manassas operating with a “yellow trap”.

            Fortunately I was not involved in an accident due to the issue, but I came very close.

            I made the “Traffic Signals Supervisor” aware of the problem–or attempted to. He came back with “There is nothing wrong with that traffic signal, the problem is the drivers”.

            The traffic signal remained in that condition for at least several more months until I made the city manager aware of the issue.

            https://en.wikipedia.org/wiki/Yellow_trap

            I am unaware of how many accidents resulted from this configuration, but the potential was there.

            I’m sure that, given the indifference displayed by the “Traffic Signals Supervisor” towards the problem, there exist other signals in Manassas with the same problem, even to this day. However, it is not my job to find them.

          8. LarrytheG Avatar

            Not familiar with “yellow trap” but it sounds like a conceptual thing as opposed to a malfunctioning system.

            I’m not saying there are no issues but if you think of how many times we encounter traffic signals in our lives – and make no mistake, out lives are involved, 99.xxx % work without loss of life.

            In fact, most deaths are due to driver errors and irresponsible acts, not traffic signals improperly designed or operating.

            That’s my point. Yes, you can find exceptions but the vast, vast majority of GOVERNMENT – designed, operated and maintained traffic signals function correctly without people getting hurt or killed. That’s the same government we say is a “fail”. It’s not.

        2. idiocracy Avatar

          Larry, the only part of the traffic signal that government actually has to do anything with is that they employ the idiot who misconfigures it with a yellow trap.

          All the rest of it is designed, produced, and installed by the private sector.

          1. LarrytheG Avatar

            Indeed as is GPS, NOAA and most of the DOD weapon systems.

            Your point? Is it Govt screwing up or the private sector?

        3. idiocracy Avatar

          One big difference between the government and the private sector is that the private sector isn’t protected by the concept of “sovereign immunity” for their screwups.

          For better or worse, the threat of litigation often serves as a motivator to do the right thing.

          1. LarrytheG Avatar

            That’s true but my experience is that the govt is pretty careful with things that can cause injury and death and it goes into their regulations also.

            VDH spends significant time and effort on making sure restaurants keep hot foods hot and cold foods cold.

            Without the Govt doing that – what would happen?

        4. idiocracy Avatar

          My experience is that the government (and private sector’s) care with things that can cause injury or death varies from place to place and is largely dependent upon the local culture’s attitude towards such things.

      2. idiocracy Avatar

        Northam is probably what is known as a “12 o’clock flasher”. Every clock in his house is flashing 12:00 because he can’t figure out how to set them.

        (As opposed to me, where every clock is flashing 12:00 because I set them and then a 2-second power outage causes them to flash 12:00. Oh, the problems a bigger capacitor would have solved…)

  10. LarrytheG Avatar

    Shoot, Virginia can’t hold a candle to the Feds:

    ” The IRS System Processing Your Taxes is Almost 60 Years Old

    One of the IRS’ most important tax-processing applications is old enough to be a grandparent, and officials warn a failure during tax season could have dire economic ramifications or delay tax refunds for 100 million Americans.

    The Individual Master File, a massive application written in the antiquated and low-level Assembly programming language, is comprised of data from 1 billion taxpayer accounts going back decades, and chiefly responsible for receiving individual taxpayer data and dispensing refunds.

    Despite hundreds of millions in spending, plans to fully modernize the application are more than six years behind schedule, and in a statement to Nextgov, IRS revised its new timeline for a modernized IMF to 2022.”

    I guess that makes the Feds honorary Plantation Elites also? 😉

    1. djrippert Avatar

      The Feds get some things right. In the 1980s I had a clearance and worked as a contractor for the military. Tip of the spear stuff. They had their act together.

      As for assembly language – that was my first programming assignment out of college. System 360 / BAL. Used the TPF operating system from IBM. One of my assignment was to train new BAL programmers. I work with them until they finally got their program working. Then I’d say that I just noticed they were using some registers much too frequently. I’d tell them they had to re-write the program to use the registers more evenly. We didn’t want to wear any of the registers out. The ones who burst into tears became process designers instead of programmers. The ones who said “fine, I’ll do it” became application programmers. The ones who said’ “registers don’t work that way” became architects.

  11. PackerFan Avatar

    Great post Larry! That has to make those 70,000 people waiting for their checks feel better! “ Dang, I might be homeless and hungry, but at least the good Old Dominion isn’t using equipment as old as what the IRS is!”, said no one.

    1. djrippert Avatar

      Good point. Elections have consequences and so does failure. Our failure to elect a competent governor has the consequence of 70,000 Virginians not getting benefits they deserve.

      All I keep thinking is that if the system is 35 years old it had to be creaking during the recession of 2009. Nobody wondered what would happen the next time there was a recession?

  12. idiocracy Avatar

    It’s probably an IBM AS-400. The SCC still uses one, which is obvious if you use the website to look up corporate data.

    1. djrippert Avatar

      I worked on those too. Started with System/34, then System/36 then System/38. The AS/400 was a huge leap forward. I went to a two week IBM school in Chicago to learn about the cool AS/400. RPGII then RPGIII. I think they finally got COBOL on the platform but I was gone to other things by then. Like an HP3000 system written in FORTRAN to run HMOs.

      Software development was the wild west in the 1980s.

      1. LarrytheG Avatar

        If you did military, then you know about embedded systems…right?

        Wanna know how old some of the fire control systems are on our ships and other weaponry?

        😉

        1. djrippert Avatar

          What I worked on for the military was classified and I was never told that it has been de-classified. All I’m going to say is that the people I worked with knew their stuff and their stuff worked.

          1. LarrytheG Avatar

            Everything in weapon systems is classified.. 😉

            but the age of the computer equipment on our ships is so old that it’s no longer classified.

            I think the point is that across the government – Fed and State and the Military that computer systems tend to not advance like they do in the private sector and part of the problem is the knowledge and skill of the employees as they do not pay the salaries that the private sector does for top-notch technological employees.

            It’s a constant battle in the military to try to maintain older systems while trying to upgrade.

            It’s a gargantuan job to try to coordinate software for ships that are 30, 40, 50 years old. It’s also a configuration management problem of gargantuan dimensions because every ship has a unique array of equipment because they were built in different years.

            It’s like Hertz trying to maintain a fleet that has 1970’s vehicles all the way up to 2020 vehicles.

      2. John Harvie Avatar
        John Harvie

        I started with Big Blue in 1964 and worked on 7044/7094 scientific machines which required a 1401/1410 to produce output from tapes since there was only a small operators’ printer on the 7000 series. The 1400 series machines used SPSS and involved “word marks” to delineate the end of a data or instruction element … some memory details fade a bit after a while. RPG was a tremendous advance when it arrived. A popular program of the day was a surveying program, COGO which closed traverses.

        1. djrippert Avatar

          Wow. I assume you were at Big Blue on or about April 7, 1964 when the 360 architecture was announced. In many ways the birthday of the modern computer. Microcode, the 8 bit byte, backwards compatibility, 32 bit registers, the IBM floating point architecture. I read Fred Brooks’ book “The Mythical Man Month” in awe of the sheer guts and grit that IBM showed in building that architecture.

          I worked at IBM years later and could only wonder at the incredible power of Big Blue in its prime. I saw the somewhat empty buildings and campuses in Somers, Raleigh, Austin, Poughkeepsie and wondered what those places must have been like when they were packed with people.

          Another seminal event in computing happened on March 13, 2006. That was the day Amazon Web Services sold its first cloud service (Simple Storage Solution) to a third party. My youngest son had been born just a few days prior. I still look at him sometimes and think to myself, “the cloud is still only in its adolescence” .

          1. Reed Fawell 3rd Avatar
            Reed Fawell 3rd

            “the cloud is still only in its adolescence”

            While,

            Virginia is in its senescence.

          2. John Harvie Avatar
            John Harvie

            Did not know of Fred Bro0ks’ book … thanks and I’ll look it up on Amazon.

            In a small sense “what comes around goes around” (or is it the reverse?). Later on I worked in IMD division on a timesharing product named Call/360 (later Call/370) which in some ways is analogous to today’s cloud computing, except that in those days we called it “distributed computing”. Our profits were obscene since we were able to charge an arm and a leg for some of our proprietary software.

          3. Nancy_Naive Avatar
            Nancy_Naive

            I knew there was a natural revulsion… I was CDC in the mid-70s. 😉

      3. idiocracy Avatar

        I worked at a place where I maintained an AS-400. It was around 10 years old when I started there. I had never once before worked on an AS-400 but I am capable of utilizing the Internet to do things beyond looking at stupid memes and videos….

        After I left, it went down because the contractor they replaced me with had no idea how to keep it running. Never heard the full story, but I’m guessing a drive went bad. They ignored it. Another drive went bad. Bye-bye RAID 5. So nice to have known you.

      4. Nancy_Naive Avatar
        Nancy_Naive

        Started on Wang 2200B, PDP-8, and some AN/YUK something or other. Moved up to the CDC Star-100, Cray-1. After that, nonspecific machines. Oh no. There was a Data General and a DEC in there somewhere. All those exciting machines and here I sit with the wife’s 10-year old hand-me-down iPad with OS-10, cursing the touch keypad.

        1. John Harvie Avatar
          John Harvie

          OMG! We’re cousins! … I went to CDC when SBC/IBM/CDC settled the consent decree. I was in DC, Richmond, Hampton and the Beach. Did we cross paths?

          1. Nancy_Naive Avatar
            Nancy_Naive

            Your name has always struck a bell, but I think I may be confusing you with someone else. I started at CDC straight out of college. Contracted to NAVELEX. I wrote a database management system and a spreadsheet for the WANG called AutoBase. The Navy used it well into the late 80s. Used self-modifying code. What a hoot!

  13. Steve Haner Avatar
    Steve Haner

    Gee, a focus on competence in picking the next Governor. The failures of the last couple of administrations are evident. Might work…Naah, virtue signaling and promises of free stuff beats competence.

    1. djrippert Avatar

      You’re probably right. Who was the last governor of Virginia who ran at least a 1,000 person operation before becoming governor? Virginia state government has 55,000 employees. There’s a certain learned skill set required to make an organization that large work.

      Ineffective elected management means an entrenched bureaucracy is actually responsible for the nuts and bolts of operating a giant organization. And that’s how you get 35 year old systems that don’t work. Some bureaucrat in the past would have had to take on the risky business of rewriting that system. Why bother when nobody’s in charge and nobody’s accountable. Better to just ride along and make excuses when it fails.

      As an aside, Joe Biden has never run anything bigger than his congressional staff as far as I know. We’ll see how that goes.

  14. Steve Haner Avatar
    Steve Haner

    Gee, a focus on competence in picking the next Governor. The failures of the last couple of administrations are evident. Might work…Naah, virtue signaling and promises of free stuff beats competence.

    1. djrippert Avatar

      You’re probably right. Who was the last governor of Virginia who ran at least a 1,000 person operation before becoming governor? Virginia state government has 55,000 employees. There’s a certain learned skill set required to make an organization that large work.

      Ineffective elected management means an entrenched bureaucracy is actually responsible for the nuts and bolts of operating a giant organization. And that’s how you get 35 year old systems that don’t work. Some bureaucrat in the past would have had to take on the risky business of rewriting that system. Why bother when nobody’s in charge and nobody’s accountable. Better to just ride along and make excuses when it fails.

      As an aside, Joe Biden has never run anything bigger than his congressional staff as far as I know. We’ll see how that goes.

  15. UpAgnstTheWall Avatar
    UpAgnstTheWall

    The WJLA article rather assiduously did not draw a causal link between the “35 year old system”, and didn’t even bother to inform the reader what that system is. If it’s a Windows based system from 1995 then, yeah, there are probably some problems there, but if it’s built on something sturdier like COBOL then it’s more likely to be a manpower than technical problem, and if it’s something like Unix/Oracle translated across to a Windows system then it’s a potential technical problem most likely with 32-bit database architecture trying to liaise with a 64-bit operating system, which would explain – to Steve’s point – why cracks didn’t appear in 2009 since that’s when Windows 7 launched with loading 64-bit OS as the default and the state probably didn’t get Windows 7 machines until 2012 at the earliest.

    But again, we don’t know, and to assert we do is sheer hackery.

    1. LarrytheG Avatar

      Anyone remember Y2K and how we all were expecting all heck to happen?

      We had scads of people on “standby” – like they were going to fix something right away if it broke.

      🙂

      1. idiocracy Avatar

        I also remember all the hyper-ventilating about “Clocks could be affected by Y2K and we don’t even know where they all are!”

        My guess at the time was that if there is a clock that nobody knows about, then nobody bothered to set it to the correct date and time and therefore it’s irrelevant as far as Y2K is concerned.

        1. LarrytheG Avatar

          We were worried about it for weapon systems and other critical national defense infrastructure. We spent years/months combing through code looking for stuff but some of it was so old and in code we could not fix – the compilers for that code were long gone… etc..etc..

          Computers are a mess but they are also the way the world works now.

          1. idiocracy Avatar

            There appeared to be a complete lack of recognition that many systems don’t care what the date and time are.

            Which is how you got people concerned that their computer controlled fuel-injected car wouldn’t start on 01/01/2000.

          2. LarrytheG Avatar

            Well, yes, but it does matter quite a bit with satellite systems and weapon systems….

          3. idiocracy Avatar

            It depends. A satellite that operates as a satellite typically does is a satellite that doesn’t need to know the correct time and date. All it does is repeat a radio signal.

      2. djrippert Avatar

        Actually the Y2K remediation work was largely done in India and kicked off the Indian offshoring wave that continues to this day. The systems almost all worked but the labor model was changed forever.

    2. djrippert Avatar

      If the system is 35 years old then it wasn’t written in windows. Nobody was writing batch check generating systems in Windows in 1985.

      My guess is the original system was an IBM S/370 mainframe written in COBOL with VSAM as the “database”. No doubt whatever it was originally it has been hacked, pathed and kludged over the years.

      Given that the system can produce checks but not keep up with demand it’s reasonable to assume that the logic works but the system won’t scale. That points to a rigid underlying architecture. They’ve been behind since May and it’s now December. If adding more memory would have solved the problem even Virginia’s state government would have done that by now.

      The bigger point is that they were either too negligent to know the scalability of a mission critical system or too lazy to do anything about it. Recessions aren’t exactly black swan events. COVID may be unique but suddenly having to start printing unemployment checks is not.

      The biggest point is that a recession started in April and our state is still five months behind getting out employment checks in December.

      That represent gross incompetence and culpable negligence.

      1. idiocracy Avatar

        The only way Windows would have anything to do with this is that the user interface/terminal emulator they use to connect to the system operates on Windows.

        And if they haven’t figured out how to get a 64-bit version of that app by now, they have even bigger problems.

        1. John Harvie Avatar
          John Harvie

          Not needling you but you must have been around IT for a while to use the phrase “terminal emulator “.

          Takes me back…

      2. UpAgnstTheWall Avatar
        UpAgnstTheWall

        Yeah, my brain scrambled 25 and 35, although I wouldn’t be surprised to find someone, somewhere still using Lotus 1-2-3.

        And while recessions aren’t Black Swan events, this COVID stuff absolutely has been, especially in terms of the sheer volume of unemployment claims: https://www.vec.virginia.gov/node/12131

        It’s absolutely a problem that we’re processing these checks five months behind schedule, but actually knowing what the problem is and knowing if it’s solvable in a way that people would find reasonable are important parts of the discussion. This pandemic has exposed just how many of our systems have no slack built in, and if expanding the number of state employees is the answer a lot of the more conservative voters would balk.

      3. bodkinpoint Avatar
        bodkinpoint

        I worked in the VEC IT shop from 1980 to 1995. The Monetary Determination system and the Benefit Payment system technologies were System 370, COBOL, CICS, VSAM. I believe that the core code for the Benefit Payments system was shared by a number of other states, with tweaks for differing laws and regulations. I would not be surprised if that ancient mess was given a Windows veneer and declared to be “modernized.”

  16. Peter Galuszka Avatar
    Peter Galuszka

    It didn’t

  17. The problems cited are not unique to Virginia, are not new, and are not unique to government.

    Oversimplifying – IT is often treated as an expense, rather than an investment and those that run IT (Chief Information Officers) are not “at the table” of decision making to determine how and when to update, change, discard information systems. Plus, people (administrators, users, managers) hate to change. They get used to doing things a certain way, and changing systems is hard stuff.

    That is not to excuse it, but managing organizations require good, effective, knowledgeable, motivated leaders that understand how to bring about change.

    Often, governments are hamstrung by legislative guidance and codified regulations that result from previous laws. IT system instantiate these inadequacies.

    https://www.cio.com/article/3018436/study-finds-companies-are-failing-to-modernize-it.html

    https://www.altexsoft.com/whitepapers/legacy-system-modernization-how-to-transform-the-enterprise-for-digital-future/

    1. idiocracy Avatar

      “managing organizations require good, effective, knowledgeable, motivated leaders that understand how to bring about change”

      Good luck finding many of those in Virginia.

      1. Nancy_Naive Avatar
        Nancy_Naive

        Yes, yes. But they can be found. Those people write half baked opinions and make snarky remarks on blogs, so if they’d just look.

  18. LarrytheG Avatar

    Management that understands the importance of IT is better than it used to be but very different between private sector and Govt. Govt management can be totally disconnected from realities and not suffer from it.

    1. idiocracy Avatar

      Government IT managers will never have the Big Boss screaming at them that the outage is costing them $10,000 a minute.

  19. Dick Hall-Sizemore Avatar
    Dick Hall-Sizemore

    Three observations:
    1. Don continually complains about eliminating the Secretary of Technology position and links that to the troubles at VEC. They are not related because that is not how Virginia’s IT system is set up. VITA and, by extension the Secretary when the position existed, is responsible for system-wide and “enterprise” applications such as telephone, e-mail, and consolidated servers. It maintains the servers on which system wide applications are run, such as accounting, budgeting, and HR. Individual agencies are responsible for the software for their specific applications. Therefore, VEC, not VITA, is responsible for the system that processes unemployment claims and sends out the checks.
    2. As for VEC, that agency is a mess. As I pointed out in an earlier post, the agency was provided authorization in 2004 to upgrade its IT system. It took the agency 15 years to get around to starting the upgrade. That is inexcusable. The only reason that I can see for the present VEC management not being sacked is that the delay was not its fault; it inherited the mess and is now doing its best to cope. https://www.baconsrebellion.com/the-dog-ate-my-homework-does-not-work-for-vec/
    3. The main stream media has not ignored this problem. There has not been much coverage by the Richmond Times-Dispatch, but the Daily Press has run frequent articles. Here is the latest: https://dailyprogress.com/news/state-and-regional/govt-and-politics/virginia-ranks-worst-in-nation-for-timely-review-of-some-unemployment-claims/article_b3606704-1c7c-11eb-90a3-3fba2fce3c78.html

    1. Nancy_Naive Avatar
      Nancy_Naive

      Daily Progress. The Daily Press has become fish wrap.

      Pennsylvania’s Employment Agency appears to have their stuff together. Hey, here’s an idea. Let’s do what they do!

    2. djrippert Avatar

      You are very literal. If someone told you to put your shoes and socks on would you do it in that order? Just kidding, Dick. The question is how important technology should be in a state with 55,000 state employees with over $50B per year in spend? How important is IT at Morgan Stanley? The organizational structure of the Secretary of Technology may have been ineffective. However, that is not cured by failing to have the senior technology leader on the cabinet. You restructure the technology function so there is one executive with sufficient direct and indirect authority to get things done over a fairly prolonged period. That person belongs in the cabinet. This is particularly important in a government entity where the overall leader (governor) can only serve four years and many of the other cabinet members get replaced as governors come and go. Beyond that, many appointees to the cabinet are political philosopher kings rather than strategic thinkers and operational executors. For example, I would expect that a conversation about technology with Atif Qarni would be like talking to the white rabbit from Alice in Wonderland about subordinated debentures.

      Somebody needs to mind Virginia’s technology shop and that’s not happening.

      First everything was Northrop Grumman’s fault. After 10 years or more of trying to fix that contract it was abandoned. Now, there’s a multi-vendor approach with SAIC apparently playing a key role. According to JLARC that’s not working either. After you try enough usually successful vendors without success it’s time to start looking at the management of technology as the likely suspect. Who in the Northam Administration has the capabilities to do that?

      Every year technology becomes more important to large entities. Every year Virginia’s state government falls further behind.

    3. djrippert Avatar

      As far as the distributed divisional responsibilities of IT – that’s nothing new in government or in private enterprise. I’d be surprised if the structure were anything else. The advantage of this type of structure is accountability. It’s VEC’s budget so why wouldn’t VEC set the priorities? 70,000 Virginians could give their opinions on that I suppose.

      In a well tuned organization the overall CIO would not only own infrastructure but would also be the chairman of a committee that included the CIOs of the departments. The annual detailed IT budgets could not be approved unless both the overall CIO and the divisional / agency CIO agreed. Disputes would be “kicked upstairs” – usually to the COO. Once the budgets were approved by both the overall and agency CIO the responsibility for results would be shared. All progress and status for both infrastructure and the agency applications would be shared by the CIO committee.

      Under that structure the overall CIO would be on the management committee. Or, in this case – the cabinet. Otherwise, the governor would have less than the necessary visibility to the ever increasing importance of technology to the state.

      This technology governance issue is a problem faced by every corporation large enough to have multiple lines of business. When the technology governance approach fails over a period of years it is NEVER the right answer to lower the visibility of technology to top management and hope the problem goes away.

      Virginia’s ongoing IT fiascos speak for themselves. Wallowing in the mire of “that’s not how we do things here” should be unacceptable to everybody, including the electorate.

      70,000 Virginians are hurting badly right now because Ralph Northam lacks the experience required to run a large organization and apparently also lacks the humility to see the bitter impact of his own lack of experience.

      1. LarrytheG Avatar

        Othere agencies in Virginia have modernized but the problem with VEC may well be that everyone knows it’s on outmoded system in need of fixing but doing so in the pandemic may break it even more and so they’re hunkered down until later after things cool down.

        This happens across govt. I pointed out that IRS is still trying to update their systems but again in the middle of a pandemic probably not a good strategy.

        1. djrippert Avatar

          The problem, Larry is that government is a monopolistic, coercive organization that raises “revenue” through threat of force against its citizens. For those reasons the expectations of government as well as the limits on government need to be significant. If there were an “Option B” to The Imperial Clown Show in Richmond I’d imagine two things would happen:

          1. A number of people would decide to be governed by “Option B” and;
          2. The Imperial Clown Show in Richmond would get its act together very quickly

          However, there is no “Option B”. So, the best “we the people” can do is to demand excellence from our extremely expensive government entities. That includes, at ever rising levels, excellence in the application of technology to the opportunities and challenges facing our state government.

          One big role of blogs like BaconsRebellion, in my opinion, is to draw attention to areas where government is failing and refuse to accept the excuse – “well, it’s just government, you can’t expect excellence”.

          As an aside, I would have no problem with paying market rate salaries to government employees / managers / technicians if that was required to reach excellence in government. I firmly believe that our state government could spend considerably less than it spends while providing broader and better services if it topgraded its management personnel.

          1. LarrytheG Avatar

            We more or less agree on the ying and yang but there are some agencies in Virginia that do well and others not and in the private sector – if you don’t compete effectively, you close. In government, you just twist in the wind, forever!

            If you contracted with DOD, you might have heard of NMCI. It was a Navy-wide effort to modernize their IT including network security and it was not exactly welcomed with open arms as most agencies like to do things their way without interference.

            Every single device and every piece of software has a cost to maintain and operate to it. Some are incompataible with each other, some have significant security issues , etc.

            At any rate, I don’t know where Virginia stands on IT compared to other states. I would not be surprised if it ranks in the bottom third because tenured govt managers have the same issues as tenured lower-level employees.

          2. idiocracy Avatar

            There *IS* an “Option B” to the Imperial Clown Show in Richmond. 49 of them to be exact. Though you’ll probably need the services of a real estate agent or two to utilize it.

          3. “As an aside, I would have no problem with paying market rate salaries to government employees / managers / technicians if that was required to reach excellence in government.”

            The problem is more systemic. The state doesn’t even have the ability to properly audit what’s going on. I was so appalled at the lack of IT experience of state auditors that I once looked into openings there. As I recall, the pay was so minimal that anyone with real IT chops who would have the ability spot problems wouldn’t work there. I haven’t looked recently, however.

      2. LarrytheG Avatar

        The problem with “legacy” systems is the guys who created it are probably gone and back in the day, software was developed, often undocumented so for all practical purposes , it’s a black box in older programming languages that younger folks don’t know.

        Beyond that, is the underlying business processes that even good developlers may not well understand and you find out when they roll out their new software and then have to go back and re-jigger it, hopefully by dropping back to the older system instead of having done away with it at the time the new system came online.

        No matter what is claimed, new software often imposes changes on the business processes… rather than just emulate them and sometimes it’s a good thing because the business processes themselves are screwed up but with software development, the mantra is “first, do no harm” and that’s not an easy thing sometimes.

  20. Dick Hall-Sizemore Avatar
    Dick Hall-Sizemore

    Three observations:
    1. Don continually complains about eliminating the Secretary of Technology position and links that to the troubles at VEC. They are not related because that is not how Virginia’s IT system is set up. VITA and, by extension the Secretary when the position existed, is responsible for system-wide and “enterprise” applications such as telephone, e-mail, and consolidated servers. It maintains the servers on which system wide applications are run, such as accounting, budgeting, and HR. Individual agencies are responsible for the software for their specific applications. Therefore, VEC, not VITA, is responsible for the system that processes unemployment claims and sends out the checks.
    2. As for VEC, that agency is a mess. As I pointed out in an earlier post, the agency was provided authorization in 2004 to upgrade its IT system. It took the agency 15 years to get around to starting the upgrade. That is inexcusable. The only reason that I can see for the present VEC management not being sacked is that the delay was not its fault; it inherited the mess and is now doing its best to cope. https://www.baconsrebellion.com/the-dog-ate-my-homework-does-not-work-for-vec/
    3. The main stream media has not ignored this problem. There has not been much coverage by the Richmond Times-Dispatch, but the Daily Press has run frequent articles. Here is the latest: https://dailyprogress.com/news/state-and-regional/govt-and-politics/virginia-ranks-worst-in-nation-for-timely-review-of-some-unemployment-claims/article_b3606704-1c7c-11eb-90a3-3fba2fce3c78.html

    1. Nancy_Naive Avatar
      Nancy_Naive

      Daily Progress. The Daily Press has become fish wrap.

      Pennsylvania’s Employment Agency appears to have their stuff together. Hey, here’s an idea. Let’s do what they do!

    2. djrippert Avatar

      You are very literal. If someone told you to put your shoes and socks on would you do it in that order? Just kidding, Dick. The question is how important technology should be in a state with 55,000 state employees with over $50B per year in spend? How important is IT at Morgan Stanley? The organizational structure of the Secretary of Technology may have been ineffective. However, that is not cured by failing to have the senior technology leader on the cabinet. You restructure the technology function so there is one executive with sufficient direct and indirect authority to get things done over a fairly prolonged period. That person belongs in the cabinet. This is particularly important in a government entity where the overall leader (governor) can only serve four years and many of the other cabinet members get replaced as governors come and go. Beyond that, many appointees to the cabinet are political philosopher kings rather than strategic thinkers and operational executors. For example, I would expect that a conversation about technology with Atif Qarni would be like talking to the white rabbit from Alice in Wonderland about subordinated debentures.

      Somebody needs to mind Virginia’s technology shop and that’s not happening.

      First everything was Northrop Grumman’s fault. After 10 years or more of trying to fix that contract it was abandoned. Now, there’s a multi-vendor approach with SAIC apparently playing a key role. According to JLARC that’s not working either. After you try enough usually successful vendors without success it’s time to start looking at the management of technology as the likely suspect. Who in the Northam Administration has the capabilities to do that?

      Every year technology becomes more important to large entities. Every year Virginia’s state government falls further behind.

    3. djrippert Avatar

      As far as the distributed divisional responsibilities of IT – that’s nothing new in government or in private enterprise. I’d be surprised if the structure were anything else. The advantage of this type of structure is accountability. It’s VEC’s budget so why wouldn’t VEC set the priorities? 70,000 Virginians could give their opinions on that I suppose.

      In a well tuned organization the overall CIO would not only own infrastructure but would also be the chairman of a committee that included the CIOs of the departments. The annual detailed IT budgets could not be approved unless both the overall CIO and the divisional / agency CIO agreed. Disputes would be “kicked upstairs” – usually to the COO. Once the budgets were approved by both the overall and agency CIO the responsibility for results would be shared. All progress and status for both infrastructure and the agency applications would be shared by the CIO committee.

      Under that structure the overall CIO would be on the management committee. Or, in this case – the cabinet. Otherwise, the governor would have less than the necessary visibility to the ever increasing importance of technology to the state.

      This technology governance issue is a problem faced by every corporation large enough to have multiple lines of business. When the technology governance approach fails over a period of years it is NEVER the right answer to lower the visibility of technology to top management and hope the problem goes away.

      Virginia’s ongoing IT fiascos speak for themselves. Wallowing in the mire of “that’s not how we do things here” should be unacceptable to everybody, including the electorate.

      70,000 Virginians are hurting badly right now because Ralph Northam lacks the experience required to run a large organization and apparently also lacks the humility to see the bitter impact of his own lack of experience.

      1. LarrytheG Avatar

        Othere agencies in Virginia have modernized but the problem with VEC may well be that everyone knows it’s on outmoded system in need of fixing but doing so in the pandemic may break it even more and so they’re hunkered down until later after things cool down.

        This happens across govt. I pointed out that IRS is still trying to update their systems but again in the middle of a pandemic probably not a good strategy.

        1. djrippert Avatar

          The problem, Larry is that government is a monopolistic, coercive organization that raises “revenue” through threat of force against its citizens. For those reasons the expectations of government as well as the limits on government need to be significant. If there were an “Option B” to The Imperial Clown Show in Richmond I’d imagine two things would happen:

          1. A number of people would decide to be governed by “Option B” and;
          2. The Imperial Clown Show in Richmond would get its act together very quickly

          However, there is no “Option B”. So, the best “we the people” can do is to demand excellence from our extremely expensive government entities. That includes, at ever rising levels, excellence in the application of technology to the opportunities and challenges facing our state government.

          One big role of blogs like BaconsRebellion, in my opinion, is to draw attention to areas where government is failing and refuse to accept the excuse – “well, it’s just government, you can’t expect excellence”.

          As an aside, I would have no problem with paying market rate salaries to government employees / managers / technicians if that was required to reach excellence in government. I firmly believe that our state government could spend considerably less than it spends while providing broader and better services if it topgraded its management personnel.

          1. LarrytheG Avatar

            We more or less agree on the ying and yang but there are some agencies in Virginia that do well and others not and in the private sector – if you don’t compete effectively, you close. In government, you just twist in the wind, forever!

            If you contracted with DOD, you might have heard of NMCI. It was a Navy-wide effort to modernize their IT including network security and it was not exactly welcomed with open arms as most agencies like to do things their way without interference.

            Every single device and every piece of software has a cost to maintain and operate to it. Some are incompataible with each other, some have significant security issues , etc.

            At any rate, I don’t know where Virginia stands on IT compared to other states. I would not be surprised if it ranks in the bottom third because tenured govt managers have the same issues as tenured lower-level employees.

          2. idiocracy Avatar

            There *IS* an “Option B” to the Imperial Clown Show in Richmond. 49 of them to be exact. Though you’ll probably need the services of a real estate agent or two to utilize it.

          3. “As an aside, I would have no problem with paying market rate salaries to government employees / managers / technicians if that was required to reach excellence in government.”

            The problem is more systemic. The state doesn’t even have the ability to properly audit what’s going on. I was so appalled at the lack of IT experience of state auditors that I once looked into openings there. As I recall, the pay was so minimal that anyone with real IT chops who would have the ability spot problems wouldn’t work there. I haven’t looked recently, however.

      2. LarrytheG Avatar

        The problem with “legacy” systems is the guys who created it are probably gone and back in the day, software was developed, often undocumented so for all practical purposes , it’s a black box in older programming languages that younger folks don’t know.

        Beyond that, is the underlying business processes that even good developlers may not well understand and you find out when they roll out their new software and then have to go back and re-jigger it, hopefully by dropping back to the older system instead of having done away with it at the time the new system came online.

        No matter what is claimed, new software often imposes changes on the business processes… rather than just emulate them and sometimes it’s a good thing because the business processes themselves are screwed up but with software development, the mantra is “first, do no harm” and that’s not an easy thing sometimes.

  21. Peter Galuszka Avatar
    Peter Galuszka

    For what it’s worth, I have written somewhat on Virginia’s and Richmond’s problems dealing with handling large IT systems. The first is a cover story I did for Style Weekly in 2015 about how Richmond totally screwed up installing a new system. It took the city forever to get it right. Lots of staff turnover. Ironically Henrico County installed a working system that was very similar in a weekend or two. The second story goes back farther and reviews how the state’s obsession with privatizating government service doesn’t always work out.

    https://www.styleweekly.com/richmond/the-frustrating-embarrassing-story-behind-richmond-city-halls-mind-numbing-financial-mayhem/Content?oid=2265997

    https://www.styleweekly.com/richmond/the-myth-of-privatization/Content?oid=136894

  22. Peter Galuszka Avatar
    Peter Galuszka

    For what it’s worth, I have written somewhat on Virginia’s and Richmond’s problems dealing with handling large IT systems. The first is a cover story I did for Style Weekly in 2015 about how Richmond totally screwed up installing a new system. It took the city forever to get it right. Lots of staff turnover. Ironically Henrico County installed a working system that was very similar in a weekend or two. The second story goes back farther and reviews how the state’s obsession with privatizating government service doesn’t always work out.

    https://www.styleweekly.com/richmond/the-frustrating-embarrassing-story-behind-richmond-city-halls-mind-numbing-financial-mayhem/Content?oid=2265997

    https://www.styleweekly.com/richmond/the-myth-of-privatization/Content?oid=136894

  23. Check out the article below.

    Maybe this guy lives in Virginia and had to go back to robbing backs because he got tired of waiting for his unemployment checks to arrive.

    “Prolific bank robber dubbed the ‘Too Tall Bandit’ strikes after taking 2-year break: FBI”

    https://www.yahoo.com/gma/prolific-bank-robber-dubbed-too-224900912.html

  24. Check out the article below.

    Maybe this guy lives in Virginia and had to go back to robbing backs because he got tired of waiting for his unemployment checks to arrive.

    “Prolific bank robber dubbed the ‘Too Tall Bandit’ strikes after taking 2-year break: FBI”

    https://www.yahoo.com/gma/prolific-bank-robber-dubbed-too-224900912.html

Leave a Reply