The Bearable Lightness of Upgrading

I haven’t written about the Heartbleed vulnerability yet, mostly out of just being thoroughly busy with other concerns, but also because there isn’t too much to bloviate about. The Heartbleed vulnerability is not overly subtle, to the extent that even your average politician can figure out why it’s a bad thing. (As a sidelight, it is curious that there haven’t been Congressional hearings about this yet, or even any rumblings thereof. Or maybe not…I’m getting quite cynical in my advancing age.)

But the clean-up from Heartbleed is an excellent object lesson in the joys of upgrading. In addition to running penetration tests and security engineering projects, I have been involved in IT development and integration projects as well, and upgrading is the single biggest divide I see between the two sides. From a pen test perspective, one of the easiest problems to find in any system is out-of-date software. Oftentimes the upgrade issues are minor, and the security fixes missing from an older version are minor; sometimes one finds operating systems older than dirt. These are happy days for a pen tester! But regardless of the fun one can have with out-of-date software, it has another advantage for the pen tester; there is a 100% chance that it will still be there the next time you scan.

What Heartbleed shows us is basically how easy it is to upgrade software…when it is a priority. Many organizations were patched within a day or two of the vulnerability publication, and the exposure is rapidly diminishing. There will certainly be some corner cases where the upgrade is difficult, particularly due to too-tight integration in some packages, but there’s little indication these represent a significant fraction of deployments.

Now I don’t mean to say it’s easy, in the sense that I know there are many IT engineers and CIOs who were working unexpected long hours on a fix, and there’s always a “hold on to your butts” nervousness when patching a live system with minimal testing. But on the flip side, the upgrade project had only 2-3 major paths it could take, each with relatively well-defined steps.

So the natural question is: if upgrades really aren’t that hard, why is out-of-date software so common? I think there are four major reasons.

  1. If it ain’t broke, don’t fix it. This is probably the single biggest factor in the current dearth of Windows XP upgrades. It may not be modern, but it does what many people want from a computer.
  2. Upgrade horror stories. Windows XP is primarily responsible for this one too, but further back in its history when Service Pack 2 came out. SP2 provided many great new security features…that broke running applications. Every IT professional probably has at least one story of an upgrade gone bad, which makes them (rightly) leery of “just run this patch; it’s easy”. Plus some vendors often bundle different types of functionality into upgrades, not just security improvements; there is genuine risk in many upgrades.
  3. I was running that?!? OpenSSL is a component of many different systems; it is not typical for an IT engineer to think “I installed OpenSSL today.” Rather it was included as a library at O/S install, or included in an integrated package. I’ve discussed this problem before, and admit there aren’t any great solutions out there other than proactive IT management.
  4. I’ll get to it later. When there’s a risk that your company is going to start spewing PII over the Internet to all and sundry, it’s easy to get executive attention and resources for an upgrade. Preventive maintenance is a bit lower on the totem pole.

Actually I want to go into this last point a bit more heavily. Much as I joke about it, I’m not THAT old, and within my lifetime I’ve seen a significant shift in our society away from individuals doing their own repairs and maintenance. When I was growing up, repairs and maintenance were built in to the American lifestyle; admittedly this is probably because stuff used to break a lot more back then. Now when my kids get to sixth grade “tech ed”, they are in a small minority of the class who has ever even used a screwdriver.

In some ways this is good: products are becoming more reliable, and Ricardo’s theory of comparative advantage informs us that it is economically advantageous that I should focus on network security and let my mechanic maintain my car. But by becoming removed from the process of maintenance, it simply becomes a cost. People naturally try to minimize costs, so psychologically maintenance becomes something to avoid. Hence we make due with older software; “if it ain’t broke don’t fix it”. And then the crash comes.

Comments are closed.