Backup

Critical Infrastructure

 

cr

 

There are 16 specific sectors in the U.S. that are considered critical infrastructure. 9-1-1 is  part of the Emergency Services sector, as defined by Homeland Security.

If you have a legacy PSAP (Public Safety Answering Point or 9-1-1 Center), you depend on the local telephone carrier to keep their central office equipment up and running during any event. The challenge then becomes the local connections- analog phone lines, T1, PRI, etc that connect your PSAP. These are typically single threaded, even if you have two connections from the same provider, they can end up on the same fiber or the same central office.

If you are planning to implement a Next Generation 9-1-1 system, you CAN have much more control over this situation.

Commercial business, especially large companies, understand the critical importance of ‘uptime’ to their business model. It is common practice to have a least two facilities that house their critical servers (data centers) and multiple  telecom providers. 

Consider putting a multi-telecom carrier requirement in your RFP. A primary and backup (or active/active model) for your NG9-1-1 Core Services equipment should be standard. Your NG9-1-1 PSAP could have have a circuit from your local Telco, maybe a wireless LTE connection (router card) to a different vendor and if you have an alternate provider (maybe your local cable company offers business broadband), include that in the mix.

Satellite  is also being used by the ARK-TEX Council of Governments.

Bottom line, you protect your PSAP operations by not tying yourself to single vendor.

The culture, for years in the 9-1-1 community, has been one of trusting the local telco to take care of everything outside of the building.  If we truly want to maintain as much uptime as possible and see our PSAPs as part of the nations Critical Infrastructure, then we need to step up and get involved in defining these requirements.

 

As they say, it’s not personal, it’s business.

 

 

 

 

 

When Your “backup” fails..

netmon_rev1

Last month a close associate of mine experienced two outages at one of his PSAP’s. The first event lasted over three hours, the second 15 minutes. All 9-1-1 calls were rerouted.

The PSAP in question is served by two circuits. Supposedly physically diverse. During the conversion to IP in 2011, each circuit was physically routed into the PSAP facility in a manner to ensure diversity.

This latest outage revealed that, in the providers central office, both circuits connected to the same equipment rack, same shelf and same card. Back in 2013 he had a similar scenario, only then the outage took down six PSAP’s. All circuits, primary and backup, terminated onto a single card. This sounds impossible, but I have read the official ‘Outage Report’.

An important fact- they spent tens of thousands of dollars throughout their county in ‘outside plant’ construction costs, with their service provider, to make sure there was physical diversity into the PSAP.

He also educated me on a fact he recently learned- In 2013 the Federal Communications Commission (FCC) issued the Derecho Report. This was in response to major storms and 9-1-1 outages in the midwest. The report listed a number of findings regarding 9-1-1 providers. Here is a quote from that document: “In most cases, the 9-1-1 and other problems could and would have been avoided if providers had followed industry best practices and available guidance.”

Derecho Report and Recommendations

This report resulted in the FCC issuing an Order:

FCC Order to Improve Reliability       Note -Appendix B- Part 12 Final Rules

The bottom line- as a customer you can request an audit, to include the physical path of  your 9-1-1 circuits. Here in Palm Beach County we now have an audit in progress.

Remember- it does not matter how big your circuits are  or how many you have (2, 3). When it comes to the last mile, the connection from the PSAP to the service provider, it is a PHYSICAL world.

A great way to address this issue is to contract with two providers. But only if they do not share the same telco facilities.

We are transitioning to a new world in 9-1-1. Like it or not, we occasionally will need to become more actively involved in order to understand how services are provided.