After a thorough investigation, the Federal Communications Commission has identified the cause of the 911 outage that affected AT&T subscribers on March 8th. The outage that occurred in the carrier’s VoLTE network affected 12,600 users, making this one of the largest 911 outages that ever occurred. Upon FCC’s investigation, the Commission determined that the outage was due to the lack of additional checks implemented by the carrier that resulted in the loss of connection between AT&T and the subcontractors handling the routing for 911 calls.
Once a subscriber calls the 911, the carrier transfers the calls to subcontractors Comtech and West to determine the appropriate Public Safety Answering Points (PSAP) to handle the calls. The subcontractors use geographical data to determine the appropriate PSAP and then include the information about the chosen PSAP to the call’s metadata. Comtech and West will then transfer the call with the metadata back to the carrier. In an effort to exclude malicious traffic from the 911 network, AT&T has formulated a whitelist of trusted IP addresses. If a 911 call comes from an IP address that is not included in the whitelist, the caller is prevented from accessing the emergency services. This is where things went wrong for AT&T, which resulted in the inability of subscribers to access the network.
According to the FCC, AT&T placed in its customer provisioning system an incorrect list of IP addresses to include in the whitelist. This list did not include the IP addresses for Comtech, which resulted in AT&T losing connection with the subcontractor once the incorrect whitelist was pushed to its live network. This caused a chain reaction that led to the carrier losing contact not only with Comtech but also with West, which resulted in the carrier’s inability to route 911 calls properly. In addition, while the people responsible for maintaining the connections with the 911 network were alerted, the IP team responsible for the whitelist was among the last to be contacted, resulting the outage to last for around 5 hours. As a result of this outage, the carrier has taken steps to prevent the same outage from happening. Meanwhile, the FCC has urged service providers to improve notifications to public safety agencies and consumers when a 911 outage takes place.