When Your “backup” fails..


Last month a close associate of mine experienced two outages at one of his PSAP’s. The first event lasted over three hours, the second 15 minutes. All 9-1-1 calls were rerouted.

The PSAP in question is served by two circuits. Supposedly physically diverse. During the conversion to IP in 2011, each circuit was physically routed into the PSAP facility in a manner to ensure diversity.

This latest outage revealed that, in the providers central office, both circuits connected to the same equipment rack, same shelf and same card. Back in 2013 he had a similar scenario, only then the outage took down six PSAP’s. All circuits, primary and backup, terminated onto a single card. This sounds impossible, but I have read the official ‘Outage Report’.

An important fact- they spent tens of thousands of dollars throughout their county in ‘outside plant’ construction costs, with their service provider, to make sure there was physical diversity into the PSAP.

He also educated me on a fact he recently learned- In 2013 the Federal Communications Commission (FCC) issued the Derecho Report. This was in response to major storms and 9-1-1 outages in the midwest. The report listed a number of findings regarding 9-1-1 providers. Here is a quote from that document: “In most cases, the 9-1-1 and other problems could and would have been avoided if providers had followed industry best practices and available guidance.”

Derecho Report and Recommendations

This report resulted in the FCC issuing an Order:

FCC Order to Improve Reliability       Note -Appendix B- Part 12 Final Rules

The bottom line- as a customer you can request an audit, to include the physical path of  your 9-1-1 circuits. Here in Palm Beach County we now have an audit in progress.

Remember- it does not matter how big your circuits are  or how many you have (2, 3). When it comes to the last mile, the connection from the PSAP to the service provider, it is a PHYSICAL world.

A great way to address this issue is to contract with two providers. But only if they do not share the same telco facilities.

We are transitioning to a new world in 9-1-1. Like it or not, we occasionally will need to become more actively involved in order to understand how services are provided.


One thought on “When Your “backup” fails..

  1. Pierce Power

    The following best practice, were it fully implemented would prevent future outages in traditional, transitional and Next Generation 9-1-1 environments. A minimum of four completely diverse circuit, network and provider connections are required to attain 99.999% reliability.

    CSRIC Best Practices # 9-9-3238 Network Operators, Service Providers, and Public Safety should consider using wireless public or private networks as a backup to dedicated trunks for the 9-1-1 network during periods of network failure. In cases where the ability to deliver 9-1-1 calls to the Public Safety Answering Point (PSAP) through normal routing is interrupted by a failure (not all trunks busy conditions) consider forwarding the call over wireless public, private networks, or satellite-based services to provide an additional alternate path to the PSTN, providing IP multimedia connectivity for next generation networks, or used solely as an alternate call delivery path for the voice component of 9-1-1 calls.


Comments are closed