Facebook outage: Zuckerberg apologises as sites come back online

Who sent Facebook dark for at least SEVEN hours? Zuckerberg apologises after bungled server update causes global outage – and insider claims WFH and ‘lower staffing due to pandemic’ made problem worse

  • Facebook suffered a seven-hour meltdown on Monday evening that started with a bungled server update 
  • Faulty update file was sent to company’s core systems, effectively disconnecting them from the internet 
  • Facebook, Instagram, WhatsApp and Facebook Messenger all went down – along with all of the internal systems needed to fix the problem, right down to key-fob door locks 
  • Mark Zuckerberg, who saw $7billion wiped off his fortune during the outage, has apologised for the outage today as the hunt begins for the person responsible

Mark Zuckerberg has apologised after all of Facebook’s services were knocked offline at least seven hours due to a faulty update of the company’s servers – as the hunt begins for the person responsible.  

‘Sorry for the disruption,’ wrote Zuckerberg, who saw an estimated $7billion wiped off his personal fortune during the outage as share prices plunged. ‘I know how much you rely on our services.’

Problems began around midday Eastern Time (5pm GMT) when Facebook, Instagram, WhatsApp and Facebook Messenger all stopped updating – forcing billions of users, including businesses, offline. 

Facebook explained that the problem was caused by a faulty update that was sent to their core servers which effectively disconnected them from the internet. The same update also disabled all the systems Facebook needed to fix the issue, from digital engineering tools to messaging services, even key-fob door locks.

Engineers were rushed to the company’s data centres to reset the servers manually, but it took until 5.45pm Eastern Time (10.45pm GMT) for the servers to be reconnected.

While some users saw their apps start working immediately, it took a while longer before others were able to get back online – with some reporting they were locked out for seven hours or longer. 

One insider writing on Reddit claimed Facebook’s problems were exacerbated because large numbers of staff are still working from home in the wake of Covid, meaning it took longer for them to get to the data centres.

During that time, Facebook shares plunged five per cent while millions of users took to alternative sites such as Reddit and Twitter to lambast the company for its blunder.

A person claiming to be a Facebook employee said on Reddit that high numbers of staff working from home made the problem worse. The account was later deleted 

 

Users around the world reported problems with Facebook, Instagram and WhatsApp on Downdetector

Twitter founder Jack Dorsey appeared to make light of Facebook’s plight on Monday. Responding to a post which appeared to show how the facebook.com domain is for sale as a result of the outage, he jokingly asked: ‘How much?’

A Facebook staff member reportedly accidentally deleted large sections of the code (pictured) which keeps the website online

The above Tweet read: ‘So, someone deleted large sections of the routing….that doesn’t mean Facebook is just down, from the looks of it….that means Facebook is GONE’

Facebook shares are down by more than 6 percent from last week as a result of the outage on Monday

John Graham-Cunningham, the chief technology officer of web security firm CloudFlare, said Facebook made a series of updates to its border gateway protocol (BGP) which caused it to ‘disappear’ from the internet. 

The BGP allows for the exchange of routing information on the internet and takes people to the websites they want to access.  

Dane Knecht, senior vice president of the firm, said earlier the Facebook Border Gateway Protocol (BGP) routes had been ‘withdrawn from the internet.’ 

Cybersecurity expert, Kevin Beaumont, wrote on Twitter: ‘This one looks like a pretty epic configuration error, Facebook basically don’t exist on the internet right now. Even their authoritative name server ranges have been BGP withdrawn.’   

Whatsapp, Instagram and Facebook Messenger, run on a shared back-end infrastructure, creating a ‘single point of failure’ according to experts.

It wasn’t just the main Facebook apps going down, other services, including Facebook Workplace and the Oculus website were also down. 

Facebook’s Chief Technology Officer, Mike Schroepfer, offered his ‘sincere apologies’ for the outage on Monday afternoon.  The scandal-hit company’s shares had dipped by 5 percent on Monday amid the outage and after a whistleblower went public on Sunday night with claims that the firm prioritises ‘growth over safety’. 

There have been a number of social media outages in recent months, with Instagram going down for 16 hours just last month, and all Facebook platforms going offline in June. 

Twitter founder Jack Dorsey appeared to make light of Facebook’s plight on Monday. Responding to a post which appeared to show how the facebook.com domain is for sale as a result of the outage, he jokingly asked: ‘How much?’ 

The cause of the outage remains unconfirmed and it’s unclear if all are linked but not long before Facebook’s entities went down, entries for Facebook and Instagram were removed from the Domain Name System (DNS) it uses. 

A DNS is essentially an internet directory. Whenever someone opens a link or an app, their device has to search the DNS used by the service they are trying to access to find it and then connect them to it. 

WHAT IS THE DOMAIN NAME SYSTEM AND HOW DOES IT WORK? 

The Domain Name System, or DNS, is the directory of the internet.

Whenever you click on a link, send an email, open a mobile app, often one of the first things that has to happen is your device needs to look up the address of a domain. 

There are two sides of the DNS network: the authoritative side, ie webpages and other content, and the resolver side, devices that are trying to access this content.

Every domain needs to have an authoritative DNS provider, servers which store DNS records. Amazon, Cloudflare and Google are among the bigger names in authoritative DNS server provision. 

On the other side of the DNS system are resolvers. Every device that connects to the Internet needs a DNS resolver. 

By default, these resolvers are automatically set by whatever network you’re connecting to. 

So, for most Internet users, when they connect to an ISP, or a WiFi hot spot, or a mobile network, the network operator will dictate what DNS resolver to use.

The problem is that these DNS services are often slow and don’t respect your privacy. 

What many Internet users don’t realise is that even if you’re visiting a website that is encrypted, indicated by the green padlock in your browser’s address bar, that doesn’t keep your DNS resolver from knowing the identity of all the sites you visit. 

That means, by default, your ISP, every WiFi network you’ve connected to, and your mobile network provider have a list of every site you’ve visited while using them. 

Major DNS providers are Google, Amazon and CloudFlare. It’s unclear if all of the sites and services that went down on Monday use the same DNS or not. 

A similar outage at cloud company Akamai Technologies Inc took down multiple websites in July.

Cloudflare’s Mr Graham-Cunningham tweeted on Monday that Facebook accidentally ‘disappeared’ from the internet after making a ‘flurry’ of updates to its BGP – Border Gateway Protocol.   

‘Between 15:50 UTC and 15:52 UTC [4.50-4.52pm UK time] Facebook and related properties disappeared from the Internet in a flurry of BGP updates,’ he said.   

When sites go down because of failures in DNS systems, CloudFare tries to repair them. 

Usman Muzaffar, SVP of engineering at CloudFlare, explained to DailyMail.com: ‘Humans access information online through domain names, like facebook.com and DNS converts it into numbers, called an IP address, computers use. 

‘From what we understand of the actual issue —it is a globalized BGP configuration issue. In our experience, these usually are mistakes, not attacks.

‘Border Gateway Protocol (BGP) is the routing protocol for the Internet. Much like the post office processing mail, BGP picks the most efficient routes for delivering Internet traffic. 

‘Today, the directions for how to get to Facebook’s DNS server’s addresses weren’t available (and seem to still be unavailable). Without being able to contact the DNS servers, visitors trying to reach a Facebook property, like facebook.com, will not get an answer and so the page won’t load.’

According to Reuters news agency, security experts tracking the situation said the outage could have been triggered by a configuration error, which could be the result of an internal mistake, though sabotage by an insider would be theoretically possible.

An outside hack was viewed as less likely. A massive denial-of-service attack that could overwhelm one of the world’s most popular sites, on the other hand, would require either coordination among powerful criminal groups or a very innovative technique.

Shares of Facebook, which has nearly 2 billion daily active users, fell 5.5 per cent in afternoon trading on Monday, inching towards its worst day in nearly a year. 

It means that the company’s founder Mark Zuckerberg – who owns around 14 per cent of the firm – has seen his wealth plummet by nearly $7billion in a matter of hours, Bloomberg reported. 

Some users of UK phone network EE were also reporting that they were having difficulty accessing mobile internet services. However, the firm told MailOnline that there were no problems with the network.

Cyber security specialist Jake Moore said there is a ‘chance’ the issue affecting the firms could be related to a cyber attack.

He said: ‘There have been many reports and I’m struggling to find out exactly what has happened- I’m reading it could be DNS related, which means there is an issue with the connection not knowing where to go to your device.

‘It could well be a human error or a software bug lurking in the shadows but whatever it is Facebook needs to do its best to mitigate the problem of causing more panic about this.

‘The biggest problem is fears over a cyber attack but as we saw from Fastly in the summer I would hedge my bets on that not being the case as we’re talking about one of the biggest companies in the world, but there’s always a chance.’

Apologising on Twitter for the outage, Mr Schroepfer said: *’Sincere* apologies to everyone impacted by outages of Facebook powered services right now. We are experiencing networking issues and teams are working as fast as possible to debug and restore as fast as possible.’

Facebook was already in the throes of a separate major crisis after whistleblower Frances Haugen, a former Facebook product manager, provided The Wall Street Journal with internal documents that exposed the company’s awareness of harms caused by its products and decisions. 

Haugen went public on CBS’s ’60 Minutes’ program Sunday and is scheduled to testify before a Senate subcommittee Tuesday.

Haugen had also anonymously filed complaints with federal law enforcement alleging Facebook’s own research shows how it magnifies hate and misinformation and leads to increased polarization. It also showed that the company was aware that Instagram can harm teenage girls’ mental health.

The Journal’s stories, called ‘The Facebook Files,’ painted a picture of a company focused on growth and its own interests over the public good. Facebook has tried to play down the research. 

Former Deputy Prime Minister Nick Clegg, the company’s vice president of policy and public affairs, wrote to Facebook employees in a memo Friday that ‘social media has had a big impact on society in recent years, and Facebook is often a place where much of this debate plays out.’

Earlier on Twitter, Facebook communications executive, Andy Stone said they were aware some people were having trouble accessing Facebook apps and products.

‘We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience,’ the executive said in a tweet.

Soon after the first report came through, the hashtag #facebookdown was trending on Twitter, with users worldwide reporting issues connecting. 

The hashtag #instagramisdown and ‘WhatsApp’ were both also trending on Twitter, with a number of users saying they checked their internet connection when they couldn’t get on Facebook. 

Instagram comms tweeted: ‘Instagram and friends are having a little bit of a hard time right now, and you may be having issues using them. Bear with us, we’re on it!’

They’re some of the most popular social media apps around the world, but it appears that Facebook, Instagram, WhatsApp and Facebook Messenger have all crashed this afternoon. Above: The reports of Facebook outages reported on DownDetector

Down Detector also showed how problems with Whatsapp began being reported just before 5pm on Monday

Facebook Messenger’s outage was also reported on DownDetector at a similar time on Monday

At 11.33pm on Monday, Facebook tweeted to apologise about the global outage of its services. They added that they were ‘happy to report’ that they were coming back online

On Twitter, Facebook communications executive, Andy Stone said they were aware some people were having trouble accessing Facebook apps and products

The various Facebook-owned platforms, including WhatsApp, Instagram and Facebook itself, took to Twitter to explain the issues and say they were working on a solution

RECENT FACEBOOK OUTAGES 

Last month, a technical issue with Facebook owned Instagram caused an outage that plagued users around the world for 16 hours.

Problems started just after 8am on Thursday. About 18 hours later, at 2am on Friday, Instagram announced the problem had been fixed.

However, the last time Facebook, Instagram and WhatsApp went down at the same time was in June. 

In June more than a thousand people in countries including the United States, Morocco, Mexico, Bolivia and Brazil reported outages.

There were also two Facebook platform outages in March, with Instagram down on March 30, and all three down on March 19. 

WhatsApp tweeted: ‘We’re aware that some people are experiencing issues with WhatsApp at the moment. We’re working to get things back to normal and will send an update here as soon as possible. Thanks for your patience!’

Even Oculus, the virtual reality gaming platform owned by Facebook was having problems, with one user describing their headset as being ‘like a paperweight’.

Oculus tweeted: ‘We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.’ 

Every time Facebook and Instagram are down, it draws people to Twitter. 

On Monday, one user shared a meme of Homer Simpson jumping from his house to Moe’s bar, with the Twitter logo over the door.

Even Google got in on the action, tweeting: ‘Everyone going to Google to check if Instagram is down.’ 

There were multiple jokes along the same lines, with one showing a fast track race and the caption: ‘Me and my friends running to twitter to see if fb, whatsapp and insta are down.’ 

It is unclear what has caused the issue, although it has disrupted all Facebook owned platforms, including the Oculus Virtual Reality gaming website and Facebook Workplace.

NetBlocks, which tracks internet outages, tweeted: ‘Facebook, WhatsApp, Instagram and Messenger are currently experiencing outages in multiple countries.’

Adding that the ‘incident not related to country-level internet disruptions or filtering.’

When attempting to visit Instagram using a desktop web browser, it gives up a ‘5xx Server Error’ and Facebook simply says ‘this site can’t be reached.’ 

The last major outage of Facebook platforms was in June 2021, when people in the US, Morocco, Mexico and Brazil all reported not being able to connect. 

However, there were also problems last month, when Instagram went down for a whopping 16 hours. 

Jake Moore, cybersecurity specialist at ESET said outages are increasing in volume and are becoming increasingly harder to predict.

He said initially, a major problem with a website or app can point towards a cyber attack – but that can add to confusion and be misleading.

‘With recent issues such as what happened with Fastly [the web service platform that saw a major global outage on June 8] it highlights the power of an undiscovered software bug or even human error,’ Moore explained.

Soon after the first report came through, the hashtag #facebookdown was trending on Twitter, with users worldwide reporting issues connecting

‘Although these are increasing in frequency and require more failsafes in place, predicting these issues is increasingly more difficult as it was never thought possible before. Luckily, most outages only last under an hour.’

This latest outage highlights the major issues with using centralised systems, according to Matthew Hodgson, Co-founder and CEO of Element and Technical Co-founder of Matrix.

‘The ongoing outage of WhatsApp, Instagram and Facebook (including Facebook Messenger and Facebook Workplace) highlights that global outages are one of the major downsides of a centralised system,’ he said.

Centralised apps, like having a single back end for Facebook products, means putting ‘all the eggs in one basket,’ Hodgson explained.

‘Decentralised systems are far more reliable. There’s no single point of failure so they can withstand significant disruption and still keep people and businesses communicating.’

NetBlocks, which tracks internet outages, tweeted: ‘Facebook, WhatsApp, Instagram and Messenger are currently experiencing outages in multiple countries’

They’re some of the most popular social media apps around the world, but it appears that Facebook, Instagram, WhatsApp and Facebook Messenger have all crashed this afternoon

Source: Read Full Article