Not even virtual reality was safe from Facebook’s outage this morning, as still unexplained changes in the way Facebook links its web addresses to the rest of the Internet took the company offline.
All of its big services were taken offline, reportedly costing Facebook US $160 million per hour, while also reportedly wiping US $6 billion from Facebook founder Mark Zuckerberg’s personal fortune of well over US $100 billion.
And all of this came hours after former Facebook product manager, Frances Haugen, who worked on “civic integrity” issues appeared on the US TV version of 60 Minutes to expose that “almost no one outside of Facebook knows what happens inside Facebook,” and that Facebook makes the “toxic” decision to prioritise information in its algorithm to further its own interests in making money, rather than furthering the public good, with Haugen stating Facebook had created a “system that amplifies division, extremism and polarisation.”
It also came on the 10th anniversary of Steve Jobs’ untimely passing, but the outage has nothing to do with the ghost of Steve Jobs as in the US where this was unfolding, it is still October 4 as I type, while in Australia, it was already the early hours of October 5 due to time zone difference.
So, at about 2.30AM AEDT, Facebook started going offline, with iTWire’s Sam Varghese publishing two articles with the details, the first titled: “BGP woes: Facebook, Instagram and WhatsApp disappear from the Web” and “Facebook starts to come back, but Cloudflare tells the tale”.
In layman’s terms, Facebook somehow disconnected itself from the Internet being able to find its servers, and it took several hours for Facebook to get itself back online, something which has now occurred.
Facebook’s Australian PR representatives issued a statement, which reads: “To everyone who was affected by the outages on our platforms today: we’re sorry. We know billions of people and businesses around the world depend on our products and services to stay connected. We appreciate your patience as we come back online.”
Now given Facebook’s propensity to track its users extremely closely in what some have called clear breaches of privacy, with even Mark Zuckerberg stating at one of Facebook’s big conferences recently that “the future is private,” one might cynically say the Internet and its netizens did indeed enjoy a few hours of privacy this morning with Facebook having erased itself, albeit temporarily, from our online lives.
iTWire’s Sam Varghese also made the salient point that neither Facebook’s global blog, nor its Australian blog, nor its global newsroom, at time of publication (and even now), had any mention of the outage at all.
But naturally, there has been plenty of comment out there about the outages, with commentary from Acronis, Forrester, Macquarie Telecom Group and Localsearch below.
Let’s start with Localsearch Co-Founder and Chairman, Daniel Stoten, who said:
“The recent outages experienced across the Facebook network highlights no matter how big a player you are in the technology space, anyone can – regardless of scale or size – experience difficulty.
“For small business owners who critically rely on social media platforms as a means of connecting with and selling to customers, this outage shows that you must be multi-approached with your marketing efforts. This outage is proof you can’t have all your eggs in one basket.
“As systems look to be restored for now, we can’t be certain an outage like this won’t happen again. For SMBs, it’s best to be prepared for that by ensuring you have a presence across all social media platforms where your customers are present, while also having an organic Google search presence through SEO optimised content, listing your business using the Google My Business profile tool, a mature eDM list, or whatever stream is most relevant to you.
“The proof is in the pudding here – Facebook had to announce the outage on competitor social media network Twitter… even they themselves market across other platforms.
“In the last year, SMBs have battled through Facebook and Google news bans where businesses were effectively banned on the platform, changes to privacy limits and iOS updates restricting advertising opportunities, and continued outages across platforms.
“The scale of these incidents is only growing with time as we all become increasingly reliant on the tech giants. Facebook is a business too, who rely on the same technologies we all do to operate. This means that they too can cease to one day exist,” Stoten concluded.
Meanwhile, Forrester analysts published a blog post titled: “Facebook’s Outage: Breaking The Ad Empire — For A Day?” which starts off by stating: “It’s a sure tell that something must be pretty bad at Facebook, Inc. when the company is forced to turn to its competitor, Twitter, to communicate with its users.”
The blog post goes on to note: “acebook has a risk-management problem. Just on the heels of a(nother) whistleblower condemning the social media giant for its business practices, the Facebook ecosystem experienced a global outage at 11:40am ET today – affecting its core platform as well as Instagram, WhatsApp, Messenger, and Oculus VR. While the company has yet to confirm the exact cause of the ongoing issue, sources are pointing to an issue with DNS where BGP routes have disappeared.
“Over the past two years, Facebook consolidated its disparate app ecosystem onto one backend infrastructure. It’s a move that creates some operational efficiencies for the company and insulation from a potential breakup by regulators. But it also exposes Facebook to concentration risk: A single risk event that produces a cascading effect – like old school Christmas lights where one goes out, they all go out. This strategy comes at the expense of redundancy and impairs the company’s resilience. It also irritates consumers who don’t want a unified social media profile across Facebook’s family of apps.
“Although many brands had a “real-time marketing” moment on Twitter today, this outage has widespread implications to the advertising ecosystem given the fact that ads weren’t being served for over six hours across Facebook and Instagram, which command the lion’s share of social media ad revenue. This not only affects Facebook’s revenue (and stock price) but also brands’ bottom lines.
“It’s a platform that advertisers and users continue to rely on. Facebook reported a 47% year-over-year increase in Q2 ad revenue amounting to $28 billion. And Forrester’s 2021 Consumer Technographics® Benchmark data shows Facebook’s core app continues to rank as the top-used social media platform weekly among global audiences except China, including 76% in Metro India, 66% in the US, and 64% in Europe 5 countries (UK, France, Germany, Italy, Spain). And other apps within the Facebook ecosystem (Instagram, Messenger, and WhatsApp) also dominate globally over non-Facebook social media platforms.”
You can read the rest of the blog post here. but as the blog post notes, “Today’s Facebook outage wasn’t the first and it won’t be the last. It’s a reminder to advertisers to have proactive mitigation plans in place to avoid the scramble of trying to figure out what to do in the moment,” with the post concluding with some advice on what brands should do to avoid being stung by such outages in the future.
Barry Silic, the CTO of Cloud and Cybersecurity at Macquarie Telecom Group said:
“As we wait to hear the cause of the Facebook outage, it’s important we focus less on pointing fingers and more on starting a meaningful conversation.
“Every business will experience a cyber-attack or a human error at some stage, and if organisations become too afraid to speak up in case their name is splashed on a headline, we will see an increasing amount of incidents go un-reported.
“Cyber-crime losses across Australia totalled $33 billion last year, and this figure will not decrease until we encourage leaders to move past their fear of reputational damage, realise that even the best companies have holes in their cyber-security posture and ask for help,” Silic concluded.
And finally, two comments on what could have caused the problem, from Candid Wuest, the VP of Cyber Protection Research at Acronis, and the second from Topher Tebow, Acronis Cybersecurity analyst:
“While there’s no confirmation on what caused the incident from Facebook Inc, it’s possible that the issue lies with the BGP or DNS protocol – which happen to be popular targets among cyber criminals.
“There are various potential attacks against DNS infrastructure – from DDoS attacks to local DNS rebinding or hijacking a DNS with social engineering against the registrar. Looking at overall attack statistics, they are a lot less popular then common malware and ransomware attacks, but they can be extremely devastating if successful in a sophisticated attack. It’s like pulling the electric cable to your server room – whole enterprise suddenly goes dark.
“Protection against DNS attacks is not trivial as they come in multiple facets. It requires strong authentication and patching to guard your own services, training against social engineering attacks, as well as classical DDoS mitigations from providers, such as Cloudflare. Naturally, configuration issues should be avoided as well. Depending on what service is attacked – for example, if it’s a central authentication server shared between multiple brands, like in this case, then such a single outage can lead to multiple brands going offline.
“To be fair, we must note that most commonly such outages are caused by non-malicious actions – suspect it to be the case here too,”
Here’s what Topher Tebow, Acronis Cybersecurity analyst had to say in a Q&A style commentary:
How popular are cyberattacks on DNS servers? how sophisticated does the attacker need to be to execute?
Denial of service attack is the most common type of DNS attack, and is easily accomplished by attackers, as it relies on simply overloading a server with requests. Other attacks like DNS hijacking and DNS poisoning, where a domain’s records are replaced or spoofed by an attacker, are more difficult to pull off, but can be accomplished by an attacker familiar with potential vulnerabilities in the DNS system.
Have you seen the growth of such attacks since the pandemic hit?
Attackers are always looking at new ways to accomplish their goals. In the last couple of years, we have seen some DNS attacks used as part of a multi-extortion scheme when ransomware victims do not pay the ransom. These attacks have not seen quite the increase that other types of attacks have, but as with other types of attacks, they do seem to be happening more frequently – with DDoS attacks leading the DNS attacks.
In case of a cyberattack, what’s the recommended course of action?
As with any attack, it is important to remain calm, and have a response plan in place ahead of time. For a DNS attack, this plan will include who communicates what, how, and when – as well as having a backup DNS solution planned that can be quickly implemented, if not automatically switched to in the event of an attack on the main DNS servers. Direct communication with the DNS provider will be helpful in most cases.
How do businesses protect from such attacks?
DNS monitoring, CDNs, and redundancy are some of the best ways to protect against DNS attacks. Nothing is a full guarantee that an attack won’t be successful, but with proper monitoring, redundant DNS, and utilization of a CDN, the damage of an attack can be minimized.
For companies like Facebook Inc, housing multiple brands – does an attack on DNS servers mean outage for all their brands? or could it be avoided?
For companies that house multiple brands, the effect on subsidiaries will really depend on how the companies are configured. If they are all using the same DNS servers, and the attack is on those servers, then services will go down for all of the associated companies.