{"id":106400,"date":"2021-10-05T01:55:34","date_gmt":"2021-10-04T20:25:34","guid":{"rendered":"https:\/\/districtchronicles.com\/?p=106400"},"modified":"2021-10-05T01:55:34","modified_gmt":"2021-10-04T20:25:34","slug":"what-took-facebook-down-major-global-outage-drags-on","status":"publish","type":"post","link":"https:\/\/districtchronicles.com\/what-took-facebook-down-major-global-outage-drags-on\/","title":{"rendered":"What took Facebook down: Major global outage drags on"},"content":{"rendered":"

The old network troubleshooting saying is, when anything goes wrong, “It’s DNS.” This time\u00a0Domain Name Server (DNS)<\/a>\u00a0appears to be the symptom of the root cause of the\u00a0Facebook global failure.\u00a0The true cause is that there are no working\u00a0Border Gateway Protocol (BGP)<\/a>\u00a0routes into Facebook’s sites.<\/p>\n

BGP is the standardized exterior gateway protocol used to exchange routing and reachability information between the internet top-level\u00a0autonomous systems (AS)<\/a>. Most people, indeed most network administrators, never need to deal with BGP.<\/p>\n

Many people spotted that\u00a0Facebook was no longer listed on DNS<\/a>. Indeed, there were\u00a0joke posts offering to sell you the Facebook.com domain<\/a>.<\/p>\n

Cloudflare<\/a>\u00a0VP Dane Knecht was the first to report\u00a0the underlying BGP problem<\/a>. This meant, as Kevin Beaumont, former Microsoft’s Head of Security Operations Centre, tweeted, “By not having BGP announcements for your DNS name servers,\u00a0DNS falls apart<\/a>\u00a0= nobody can find you on the internet. Same with WhatsApp btw. Facebook have basically deplatformed themselves from their own platform.”<\/p>\n

Whoops.<\/p>\n

\"\"<\/p>\n

As annoying as this is to you, it may be even more annoying to Facebook employees. There are reports that\u00a0Facebook employees can’t enter their buildings<\/a>\u00a0because their “smart” badges and doors were also disabled by this network failure. If true, Facebook’s people literally can’t enter the building to fix things.<\/p>\n

In the meantime,\u00a0Reddit<\/a>\u00a0user u\/ramenporn, who claimed to be a Facebook employee working on bringing the social network back from the dead, reported, before he deleted his account and his messages, that “DNS for FB services has been affected<\/a>\u00a0and this is likely a symptom of the actual issue, and that’s that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).”<\/p>\n

\"\"<\/p>\n

He continued, “There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified. Part of this is also due to lower staffing in data centers due to pandemic measures.”<\/p>\n

Ramenporn also stated that it wasn’t an attack, but a mistaken configuration change made via a web interface. What really stinks — and why Facebook is still down hours later — is that since both BGP and DNS are down, the “connection to the outside world is down, remote access to those tools don’t exist anymore, so the emergency procedure is to gain physical access to the peering routers and do all the configuration locally.” Of course, the technicians on site don’t know how to do that and senior network administrators aren’t on site. This is, in short, one big mess.<\/p>\n

As a former network admin who worked on the internet at this level, I anticipate Facebook will be down for hours more. I suspect it will end up being Facebook’s longest and most severe failure to date before it’s fixed.<\/p>\n","protected":false},"excerpt":{"rendered":"

The old network troubleshooting saying is, when anything goes wrong, “It’s DNS.” This time\u00a0Domain Name Server (DNS)\u00a0appears to be the symptom of the root cause … Read more<\/a><\/p>\n","protected":false},"author":60,"featured_media":82566,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[340,5,358],"tags":[694,696,697,695],"_links":{"self":[{"href":"https:\/\/districtchronicles.com\/wp-json\/wp\/v2\/posts\/106400"}],"collection":[{"href":"https:\/\/districtchronicles.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/districtchronicles.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/districtchronicles.com\/wp-json\/wp\/v2\/users\/60"}],"replies":[{"embeddable":true,"href":"https:\/\/districtchronicles.com\/wp-json\/wp\/v2\/comments?post=106400"}],"version-history":[{"count":0,"href":"https:\/\/districtchronicles.com\/wp-json\/wp\/v2\/posts\/106400\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/districtchronicles.com\/wp-json\/wp\/v2\/media\/82566"}],"wp:attachment":[{"href":"https:\/\/districtchronicles.com\/wp-json\/wp\/v2\/media?parent=106400"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/districtchronicles.com\/wp-json\/wp\/v2\/categories?post=106400"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/districtchronicles.com\/wp-json\/wp\/v2\/tags?post=106400"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}