That spurred the engineers of the Defense Digital Service — the so-called “SWAT team of nerds” that tackles the Pentagon’s thorniest IT problems — to make patching the vulnerability a top priority. Even then, it took nearly a year to complete what engineers consider a minor technical fix.
It’s a saga that illustrates the massive logistical challenges facing the world’s most powerful military as it tries to keep up with hackers intent on pilfering some of the country’s most sensitive data. As China, Russia and profit-seeking criminals ramp up their efforts to tunnel into U.S. systems, the federal government’s bureaucracy often stands in the way of its own efforts to be nimble on cybersecurity.
Informed of the fix by POLITICO, an aide to Sen. Ron Wyden (D-Ore.) called it welcome but long overdue.
“Anything that we can do to make life more difficult for our adversaries is a good thing,” the aide said. Wyden, who serves on the Intelligence Committee, called out the Pentagon four years ago for failing to protect employees’ emails from hackers and foreign spies.
The aide noted that Wyden’s office had recently reached out to DoD for an update on its efforts.
The flaw didn’t compromise the Pentagon’s classified communications or internal mail.mil emails. But it meant that DoD’s unclassified electronic conversations with outsiders were essentially naked as they traveled server to server across the internet.
That posed a risk for the vaccine push, opening the door for hackers to read trade secrets or launch spearphishing email attacks aimed at gaining access to other parts of DoD’s network. The Pentagon was already breached in such an attack in 2015, when suspected Russian hackers compromised an unclassified email server used by Joint Chiefs.
The root of the problem: The Pentagon never fully implemented a widely used security protocol, known as STARTTLS, that makes it easier for email servers to exchange encrypted messages. The protocol was created in 2002, but over the years the department enabled it only for communications with a handful of external agencies.
Even when the Pentagon overhauled its email safeguards in 2017 and 2018, its Defense Information Systems Agency opted not to buy a security certificate that would vouch for the authenticity of DoD emails — instead creating its own, less universally accepted version.
The setup ensured that Pentagon emails could be encrypted as long as they remained within the department’s networks. But messages lost that protection once they reached the outside world, where most email systems didn’t trust the department’s homegrown certificate.
The pandemic changed all that, by hastening efforts to adopt STARTTLS for all traffic crossing DoD’s email gateway.
“Government bureaucracy is often on a slippery slope that slides into the outdated reasoning that ‘Because we’ve always done it this way’ outweighs the better logic: ‘Because this is the right answer,’” said Goldstein, whose team highlighted the lack of basic email encryption in 2019. “Solutions that might otherwise seem obvious can get sidelined and forgotten, often because it is unfamiliar and foreign.”
Goldstein’s team got the go-ahead and the resources it needed in the early days of the pandemic. He assigned three engineers to the effort and recruited the Pentagon’s CIO for extra muscle to cut through layers of bureaucracy.
Cleghorn, the lead engineer, said that even then there were “lots of stop-and-go and odd hurdles that we had to overcome.”
They called the effort “Project Groot,” after a character from Marvel’s “Guardians of the Galaxy” movies.
“Groot is a tree-like character that’s resilient to fire and has the ability to regenerate, which is fitting for this project,” DDS chief Brett Goldstein said in an email. “He also has excellent taste in music!”
Even with buy-in from on high, enabling STARTTLS — something that should take minutes — became a nearly yearlong effort of testing and editing policies that hadn’t been implemented with a government-wide pandemic fight in mind.
DDS ultimately spent $3,000 to purchase a certificate from a company called Entrust. “Spending $3,000 to secure over 2 million email accounts was a drop in the bucket to resolve a lingering issue and significantly improve our security posture,” Goldstein said.
“From a technical perspective this is like an hour’s worth of work,” said Cleghorn. “It’s getting a certificate and installing it on the mail gateway — which is just ‘File, Browse, Click, Click, Upload’ — and then attaching it to that profile.”
Roger Greenwell, the risk management executive at the Defense Information Systems Agency responsible for signing off on the change, said most of the holdup wasn’t about instituting the fix, but in analyzing what impact hitching a new commercial certificate would have on DoD’s existing email system and network architecture.
“For all intents and purposes you can almost think of it as somewhat a relatively minor software upgrade,” Greenwell said.
The shift by DoD drew applause from people who have urged wider adoption of STARTTLS following former NSA contractor Edward Snowden’s revelations of government mass surveillance in 2013. But some had only limited praise for the department’s decision to finally catch up with the rest of the world.
Alexis Hancock, a technologist at the Electronic Frontier Foundation, said the move warrants only a “golf clap” because calls for adopting STARTTLS became more urgent and widespread post-Snowden.
DoD’s conversion also looks long overdue considering Google started an effort to shame organizations into switching to the protocol in 2014.
But now that it has adopted email encryption for itself, Hancock argued, DoD should support encryption efforts for the government and the public.
For now, she had just one message for the Pentagon: “Welcome to the encryption party.”