Over the last week the broadcast news channels began reporting on a spy scandal centered reeking of international subterfuge. The culprit was a piece of software common among government agencies. No one should be surprised here. The only surprise is that it did not surface sooner.

In short, hackers were able to add code to a network monitoring software package named Orion from SolarWinds.com. The malicious code was distributed to their customers via update packages distributed from March through June of this year. The hack gave the hackers a “God’s eye view” of the infected networks and they were able to access any information available on the infected networks.

Hacks happen all the time, unfortunately. Well-intentioned users buy a product from a vendor found on the Gartner Magic Quadrant (more on this elsewhere) in the belief that the product is reliable and secure. This includes everyone from local cities to large federal government agencies. Unfortunately, many products found on the Magic Quadrant fail to live up to their promise. SolarWinds’ Orion is the latest such product. This specific hack is particularly worrisome because the compromised software was used by many federal agencies and the hacker is suspected of being the Russian government.

Electronic spying is not necessarily illegal. I suspect that the USA does a fair amount of its own spying. So the question is whether anyone is surprised that this happened. I hope not.

Government agencies, especially small local agencies, rely on 3rd party contractors and consultants to provide an ever increasing array of services and expertise. In an effort to “cut costs at all costs” these agencies look for any opportunity to eliminate a pension-eligible employee. The result of which is fewer and fewer on-staff personnel who can detect and identify problems such as hacks.

In this case, it was a cyber security consulting firm (FireEye) that detected the hack, when it finally discovered that it, itself, had been hacked. I’m not sure whether this will enhance or hinder their reputation. What does it say about your ability to defend against hacks when you yourself are hacked? And you want people to pay you for something you can’t even provide yourself?

What to do?

Is this (becoming a hacking victim) inevitable? Of course not. I won’t surrender the internet to the “bad guys.” There are things that can and should be done by agencies, large and small. But especially the small ones.

Stop the wholesale decimation of IT staff.

Rather than rely on the purchase of services and expertise, these agencies should invest in their staff so that they maintain the ability to detect and respond to hacks in real-time. Local, trained staff will notice unusual occurrences or patterns on established platforms more thoroughly than a software-only solution. Should the software solutions and consultants be abandoned? No. They usually provide solid reliable information that can be used to strengthen the defense against hacking. I prefer to think of them as a race car, and in-house, trained staff as the drivers.

Don’t rely on “magic.”

My section manager would buy rotten fruit out of a sewer if it had a “Gartner Magic Quadrant” sticker on it. When a new solution is needed, too many managers simply log on to the Gartner site and pick the product that is uppermost / rightmost on the Magic Quadrant. No questions are asked. No product comparisons. No consulting with the in-house team. After all, “no one ever got fired for using Gartner.” No, they haven’t. But maybe they should.

As a result, of Gartner-based purchases, we’ve had to implement (i.e. fight with) some software that was ill-suited to our needs. Think of taking a Ferrari to Home Depot to pick up roofing materials: you might get it done, but using the right vehicle from the start would be much better.

So avoid Gartner? Not necessarily. But you must still investigate ALL your product options. Contact other similar-sized agencies, especially ones using other products. Ask why they chose the other products? What do they know that Gartner doesn’t? You know: due diligence.

Hope for the Best; Prepare for the Worst

This won’t be the last time we hear about a hacking episode that reaches into government agencies. They’re rich targets. But the likely victims could be doing more to prepare and prevent these hacks, starting with employing able and capable staff members.