Google and Verizon jointly committed an error on Friday that caused internet connections throughout Japan to slow to a crawl and, in many cases, go out entirely. The somewhat simple issue boiled down to a misconfiguration of Google’s border gateway protocol (BGP) that was not caught by Verizon, who most of the traffic ended up going through. This, in turn, caused a leak of sorts. Essentially, a number of Google IP addresses were advertised as free for internet traffic, and thus had traffic routed to them by local and foreign service providers, so long as the traffic was meant for Japan. Not only were Google’s servers not equipped to forward traffic, but they could not handle the collective strain meant for all combined service providers in Japan. Google rooted out the issue and corrected it within minutes, but getting the traffic flowing correctly again resulted in hours of slow internet throughout the nation. The issue is currently under investigation not only by Google, but by the Internal Affairs and Communications Ministry of Japan.
The incident had somewhat long-lasting knock-on effects for a range of network providers and internet entities around the world, including Japanese entities like KDDI Corp and NTT Communications, and more global entities like the German arm of Vodafone. Many of these providers ended up having traffic routed through upwards of 1,000 of Google’s ports, which meant that the traffic needed to find that many open ports in order to right itself once the issue was fixed. NTT Communications was hit the hardest by the ordeal, having its traffic jump on 24,834 Google ports.
The widespread BGP leaks and misconfigurations led to somewhere in the area of 135,000 ports being erroneously used. The issue started with Google’s erroneous assignment of traffic into Verizon’s network. Verizon did not have filters on its BGP handler, which resulted in it accepting this erroneous traffic from Google seamlessly. This, in turn, caused Verizon’s network to blast the Google ports and prefixes that it had received as available, since it did not see them saturated with traffic. According to BGPMon, having filters on both ends of the transaction would have prevented this disaster.