This is mostly about scary Russians but don’t skip it – there’s also an important lesson for you about your own security.
The Russians hacked into Solarwinds in September 2019. (You didn’t know that. It’s six months earlier than all the reporting about this hack to date. Solarwinds tucked it into an online explanation of the hack a couple of weeks ago.)
They spent six months tiptoeing around, looking for all the tripwires and booby traps, before they inserted malicious code into the Orion updates sent out to Solarwinds customers in March 2020. Once they were in the large company and government agency networks, they hid in the shadows as they downloaded files, read the mail, and inserted additional back doors and cyber bombs. They never triggered a single alert anywhere until December 2020, more than a year after they began.
They were brought down by a cell phone, just one phone appearing where it didn’t belong, the equivalent of one lonely red light blinking on a control panel that is otherwise all green.
You could call it a mistake by the Russians, but given how successful they had been for more than a year, it’s more like an unfortunate bit of bad luck, like stepping on a board that creaks unexpectedly when you’ve almost finished robbing a house.
The security lesson
Before we get into the details, I want you to notice something important. When they were caught, the Russians already had a stolen employee password; they were trying to circumvent the two-factor authentication on the account.
Two-factor authentication is (1) a pain in the ass, and (2) magical. You can use it to keep the Russians out of your accounts. Also the Chinese and the teenagers in their basements and the low-level Eastern European cubicle scammers buying databases of stolen passwords.
Two-factor authentication works to keep you secure.
Two-factor authentication (2FA) adds an extra step to the process of logging into an account – LastPass or your Google account, for example. The extra thing might be a code sent as a text message to your phone or a number generated by an app on your phone.
The combination of 2FA and a password is far more secure than having just a password, even if the password is complex. This is the important part. From a security perspective, it’s like night and day.
When two-factor authentication is turned on, no one can log into your mailbox or Google account or LastPass vault, even if they have your password. And you’ll get an alert if anyone tries.
There’s more information here about two-factor authentication. If you want to be secure, set up two-factor authentication on your important accounts.
FireEye catches the Russians
FireEye is an important cybersecurity company, called in over the years to investigate high-profile attacks against Target, Sony, JP Morgan Chase, and many others.
FireEye monitored its own network with Solarwinds’ Orion platform. The FireEye network was compromised by the Russians in mid-2020 but like all the other targets, FireEye had no idea for months that its network was under attack.
The Russians proceeded to leverage its takeover of FireEye’s network just as it did with the thousands of other high-value networks it was burrowing into. Carefully, slowly, it began to insert malware and back doors in the FireEye network, stealing credentials for various network accounts along the way.
Some (or perhaps all) FireEye accounts are secured with two-factor authentication. With 2FA, the password isn’t enough to log in; an additional code is required, typically from a cell phone.
Imagine that you’ve set up two-factor authentication on your mail. A bad guy gets your password and tries to log into your mailbox. The bad guy is prompted for a code that will be sent by text message. If they could just type in a different cell phone number and get the code, you wouldn’t be very protected, would you? The extra security comes because your phone number, and only your phone number, gets the message. Your phone is the only approved device on your account.
Serious security at a big company, then, requires keeping close track of which cell phones are approved for use with each employee account.
The Russians wanted to connect to the FireEye VPN, the equivalent of a secure long-distance connection directly to the FireEye network. They could already use the malware on FireEye servers to accomplish many things on the network without the VPN connection, but their hacking would be easier if they could use the VPN.
So they took a chance – perhaps one they had done successfully before on other networks.
The Russians used their hidden back door access to add a cell phone as an approved device for two-factor authentication on the FireEye employee’s account.
Remember the suggestion that you think of security as a series of tripwires and cameras around your campsite, alerting you to anything that so much as twitches?
When a new device is added to an employee’s account at FireEye, it triggers an alert to the system admins. It’s a routine event. I’m sure it’s the kind of thing that happens every day.
But in this case, the sysadmins followed up and called the employee – and the employee said it wasn’t his phone.
I can only imagine what that must have felt like to the people running security at FireEye. Their network is so secure that there is simply no way for that phone to be on the list. They would have stared at that cell phone number over and over as they confirmed what they already knew in their gut: the only way for that number to be there is if their entire network is compromised.
FireEye CEO Kevin Mandia describes it this way:
Mandia said that method of attack was a big red flag. “The minute we saw that, we recognized that’s the kind of tradecraft advanced groups would do,” Mandia noted. No malware, and under the guise of a legitimate user, “doing exactly what your employees do when they go to work every day.”
FireEye then began a huge effort to decompile code and study packet captures and forensic software logs. FireEye has enough experience to recognize tradecraft of a nation-state whose primary goal is to stay “surreptitious and clandestine.” As its investigation proceeded, FireEye discovered just how extensive the hack had been in its own network, as well as realizing the implications for others.
Stage one of the attack planted the backdoor onto FireEye’s network via the SolarWinds platform, Mandia said. Stage two used the backdoor to access domain credentials, he said, such as user accounts and passphrases. “Stage three was to get the token signing-certs to access O365, likely for specific email accounts,” Mandia said. The final stage of the FireEye attack was the theft of its red-team tools.
FireEye quickly notified law enforcement, intelligence agencies, Solarwinds, and its customers about the hack. It worked with Solarwinds to make sure the original entry point was closed, but the Russians had long since removed the malware in the Solarwinds Orion updates. More important is the continuing work to identify malware installed on each network that might allow the Russians to monitor the mail and steal files – and possibly to blow up the networks someday.
All it took was a cell phone appearing in the wrong place to begin to unravel the Russia hack. Unfortunately that alert, the little twitch on the tripwire, didn’t happen until the Russians had been embedded in thousands of networks for months, leaving additional back doors and malware still to be discovered.
