If you’ve been following along, then you know that Russian hackers slipped malware into Solarwinds updates that went out to 18,000 corporate customers and government agencies last year.
I hear you saying, whoa, slow down, cowboy! The hackers “slipped malware” into the updates? That doesn’t explain anything. How did they do that?
As of today, we don’t know the specific details. But the Solarwinds story has some unexpected twists that will take us into the dark side of American capitalism, and that might give us enough of an answer.
As to the specifics, it’s possible that an employee slipped up somewhere. Or (cue ominous music) perhaps an employee was working with the Russians. (Hold that thought. We’ll come back to it.) It’s the most common type of hacking attack: gain access to an admin account and work from there.
But not necessarily. This could have been an attack focused on network equipment used at Solarwinds corporate HQ, or an entry thru a vulnerability in a Solarwinds VPN appliance, or something else very obscure indeed. US and Russian intelligence agencies have super hacking tools that are not available to mere mortals, so perhaps the Russians cut through Solarwinds’ defenses with a military-grade icebreaker like something from Neuromancer. Remember Edward Snowden, the NSA contractor? He’s living in Russia now because he released details of a decade-long US program that employed thousands of engineers and contractors to build new cyber weapons that could be used for espionage or potentially to cripple or destroy foreign networks. The Russians have similar programs. Nation-states collect zero-day exploits and keep them secret for just this sort of operation.
A rogue employee? Vulnerability to an attack by super hackers? Magical hacking beans? Maybe it doesn’t matter. Solarwinds is a company with privileged access to tens of thousands of networks. Its first job is to be secure against attacks – full stop, end of story. It failed in spectacular fashion.
It opens up a wonderful opportunity to be snarky about Solarwinds, judging its every action for the last ten years in hindsight.
Solarwinds is a case study in the worst trends of modern capitalism. It made short-term decisions to cut expenses and push up profits, and now it’s paying a price when the cost-cutting turns out to harm the company’s products and reputation – and oh, by the way, endangers everyone in the US.
Let’s dive in.
The Solarwinds story
Solarwinds was founded in 1999 and grew quickly to become one of the best-known network monitoring companies.
Under a new CEO, Solarwinds dipped its toe in the private equity waters in 2007 when it raised money from Bain Capital, an investment firm already notorious for sucking all the money out of its targets at the expense of their long-term health. The private equity companies were on a rampage at that time, looting companies that could increase profits in the short term by raising prices, laying off workers, or selling assets. Bain Capital is one of the private equity companies called out in a book written a couple of years later, The Buyout of America: How Private Equity Is Destroying Jobs and Killing the American Economy.
Solarwinds went public in 2009 to raise money for acquisitions and became a merger machine, buying a dozen companies in the next five years. There are many ways for a company to become a market leader; the technology industry is particularly drawn to the method of buying all the competition and absorbing them or shutting them down.
In 2016, Solarwinds became a private company again when Thoma Bravo Partners and Silver Lake bought the company with Solarwinds’ own money, loading it up with two billion dollars of debt. Solarwinds continued its buying binge, acquiring companies who made threat monitoring software, email security, database performance monitoring, and IT support firms. Solarwinds went back into the public stock market in 2018, bringing in billions of dollars for its private equity backers.
Through it all, Solarwinds’ revenue and profits kept growing.
There are two ways for companies to keep profits rising.
The old-fashioned way to increase profits is to make a better product and sell it effectively.
The new-fangled way to increase profits is to cut costs, outsource labor, fire employees, and stop investing in expensive things – like, say, security.
Which one do you think Solarwinds chose? Ah, I see you’re following along.
“Employees say that under CEO Kevin Thompson, an accountant by training and a former chief financial officer, every part of the business was examined for cost savings and common security practices were eschewed because of their expense. . . .
“Employees said SolarWinds tacked on security only in 2017, under threat of penalty from a new European privacy law. Only then, employees say, did SolarWinds hire its first chief information officer and install a vice president of “security architecture.”
“Ian Thornton-Trump, a former cybersecurity adviser at SolarWinds, said he warned management that year that unless it took a more proactive approach to its internal security, a cybersecurity episode would be “catastrophic.” After his basic recommendations were ignored, Mr. Thornton-Trump left the company.”
Security is expensive, both in money and in the extra effort required to handle business processes safely, so okay, maybe Solarwinds cut a few corners here and there to improve the bottom line.
But that’s not all. There’s one more way that Solarwinds cut costs.
The New York Times reports that Solarwinds moved much of its engineering to satellite offices in the Czech Republic, Poland, and Belarus, “where engineers had broad access to the Orion network management software that Russia’s agents compromised.”
Remember I said we would come back to the possibility that a company insider had helped the Russians? There are fine engineers in eastern Europe and it’s great that they work cheap, but geographically, umm, they’re a lot closer to Russia than we are. It’s kind of hard to overlook that in hindsight.
So we don’t know whether Solarwinds bears any direct responsibility for the Russia hack, but that’s a lot of dirty laundry to have the New York Times airing in public. The stock market notices things like that.
Solarwinds in the aftermath of the Russia hack
Solarwinds is fighting for life. Its stock tumbled more than 40% in the few days following discovery of the Russia hack, and the stock price continues to be in the basement. CEO Kevin Thompson resigned almost immediately. The company filed a disclosure statement with the SEC laying out how deeply concerned the company was with the hack but carefully avoiding anything that would identify how the attackers had gotten into their systems.
Solarwinds has hired high-profile security consultants in a desperate effort to rebuild their credibility. Chris Krebs was the leader of the Homeland Security agency handling cybersecurity in the leadup to the 2020 election. Alex Stamos has become the go-to expert for companies dealing with embarrassing hacks, hired by Yahoo in 2014 after its devastating attack, then brought into Facebook in 2015 to deal with its many safety and security controversies. They are stars in the world of cybersecurity. Their advice will be valuable, but Solarwinds obviously hired their consulting firm for marquee value in a press release to reassure the stock market.
Solarwinds MSP is a separate division of Solarwinds that provides monitoring software for small and medium businesses. Plans were already in progress to spin off Solarwinds MSP into a separate company. Solarwinds MSP issued a statement making three points: (1) Solarwinds MSP operates 100% independently from the division responsible for the compromised Orion software, (2) Solarwinds MSP software was not hacked, not even a little bit, and (3) GET US OUT OF HERE WE DON’T WANT TO BE DRAGGED DOWN WITH SOLARWINDS PLEASE DEAR GOD WE DON’T WANT TO DIE.
Expect Solarwinds to issue a statement at some point – next week, next month? – declaring that they’ve identified the attack vector and no matter what else might be true, it was not their fault.
Solarwinds’ reputation may never quite recover but there are two reasons that things may settle back more or less to the pre-hack status quo: (1) It is difficult and expensive for customers to switch from one IT network monitoring product to another, and (2) there’s a good chance the Russia hack will be allowed to fade into the background. The Russia hack was unprecedented in its scope, as far as we know. The US may be reluctant to turn it into an international incident if we have the same penetration into Russian networks. Or Chinese or North Korean networks, for that matter.
In the next article we’ll take a quick look at how the Russians got caught by a single cell phone.
