ESInet

FirstNet and NextGen 9-1-1

There is discussion in the 9-1-1 community regarding FirstNet and how it might relate to Next Generation 9-1-1. Here in Florida, we recently received a formal briefing on FirstNet.

As a reminder- in the world of the First Responder, the current Land Mobile Radio (LMR) system for voice will remain.  The initial FirstNet deployments will be data only.

SO…

table

Disclaimer- The following are my personal thoughts.

FirstNet may become much more than a wireless network. They have the ability to become THE leader in specialized public safety applications, applications that could be used by First Responders nationwide, regardless of whether their state has chosen to ‘opt-in’ or ‘opt-out’.

Core

There is also the potential that they could host existing software applications, maybe providing a value add by obtaining a larger volume licensing agreement from the vendor, an incentive to utilize the FirstNet Core.

In ‘NextGen’ 9-1-1, voice is an application. By this I mean that it utilizes SIP (session initiation protocol), which operates at Layer 7, the application layer of the OSI model. Translation- voice is an application.

FirstNet could offer VOICE services for the 9-1-1 community.  Simply add a hosted voice server to the graphic above. This could be of tremendous value,  especially to those states (mostly home rule) who are still putting together their NextGen 9-1-1 strategy.  FirstNet needs a core backbone network, why not provide voice services? Voice uses very little bandwidth.

The other aspect is that this lays the foundation for a real Public Safety Broadband Network. We do not need to pay for and operate TWO networks- it certainly does not happen in the business world.

Connecting from the core network to the 9-1-1 center (PSAP), it would make sense to have two types of connections, one land based and one wireless. Diversity..

And the critical aspect of security– we expect to utilize pictures and videos on the FirstNet wireless network. What better way to control the pictures and videos planned to be coming inbound to 9-1-1? Have them ‘land’ in the FirstNet core, when they can be dealt with and controlled prior to potentially being pushed out to First Responders.

Intrado (now West) pioneered the concept of hosted 9-1-1 services and the use of LTE wireless as a backup for 9-1-1 Centers (PSAPs). It’s all possible.

Instead of congress funding a separate NextGen 9-1-1 initiative, maybe there could be incremental funding to FirstNet to include the NextGen 9-1-1 services.

 

GeoSpatial (location based) Routing- We Tried

When I took over this project in 2013, one feature that appealed to me was geospatial  routing. Instead of wireless 9-1-1 calls being routed to a PSAP based on the cell tower/sector database, it would now be routed to the correct PSAP based on the location of the caller. (You NG9-1-1 technical types out there- you know the acronyms and flow)

We have hundreds of cell towers and 18 PSAPs, so the idea of avoiding/reducing 9-1-1 transfers between PSAPs  made tremendous sense.

There were plans to create a geofence around our main courthouse and the airport. Both of these facilities are located within the City of West Palm Beach, yet – as County facilities- are staffed by the County Sheriffs Office. Placing a geofence around these properties and routing 9-1-1 calls from within the geofence directly to the Sheriffs Office makes sense and, as we were told, easy to do.

courthouse

logo-courthouse

Palm Beach County Courthouse West Palm Beach

Our GIS team attended training, reviewed the standards and went to work on preparing our data. A few months ago, after having the data professionally ‘vetted’, we felt that we were ready to move forward with location based- geospatial- routing.

Around this same time we were asked to look at the current routing of 9-1-1 calls FROM the Town of Palm Beach. Donald Trump has a ocean front home on the island (not far from  our offices).  The town is a 16 mile long, narrow, barrier island. There are only a few cell towers. As a result, a number of 9-1-1 calls that originate on the island are connected to cell tower sectors across the intracoastal waterway. They are routed first to a ‘mainland’ PSAP and then transferred back to the island.

The ideal scenario would be to route all 9-1-1 calls directly to the Town of Palm Beach PSAP. In this scenario, turning on geospatial routing made sense.

In our industry there is a lot of talk about implementing this feature. So- when we reached out to the wireless carriers to let them know we were ready- we were surprised at the response.

Today, wireless carriers in our area are not ready to transition away from the MSAG and cell sector routing. It appears to be a complex issue. A portion of the 9-1-1 fee is returned to the carriers for providing  9-1-1 services (including the MSAG), so moving away from this may take time.

I have been told that there are NG9-1-1 deployments out there that are doing geospatial routing. I do not mean holding the call and waiting for the ‘Phase 2’ data as the initial input..

If you are reading this and are truly doing geospatial, please comment below – I’d love to speak with your wireless provider. In the meantime there are options being discussed in certain working groups, led by the FCC.

 

State 9-1-1 Boards and NextGen

boardmeeting-300x195

 

One aspect of FirstNet that I truly respect is the fact that they are funded, organized and empowered regarding their mission. They have the potential to positively impact Public Safety on a national level.

Things are not as well defined with NextGen 9-1-1. At the state level, we currently have extremes regarding  9-1-1 Boards and their authority. While some states are up and running with NextGen, others are struggling with the initial planning. Two States (Wisconsin and Missouri) do not even have a state level 9-1-1 Board.

Laurie Flaherty and her team @ 911.GOV have done a great job collecting and interpreting data from states.

Depending on the specific details, Home Rule  can play a major role in the lack of centralized authority for 9-1-1.

There are initiatives today at the federal level regarding funding for Next Generation 9-1-1. We do not want the scenario of a state receiving funding for NextGen without a definitive plan. For those states that do not yet have a plan in place, one option is to engage the Department of Homeland Security Office of Emergency Communications.

This VIDEO may prove of interest. I participate as a subject matter expert (SME) for DHS and believe this program to be of tremendous value.

 

Critical Infrastructure

 

cr

 

There are 16 specific sectors in the U.S. that are considered critical infrastructure. 9-1-1 is  part of the Emergency Services sector, as defined by Homeland Security.

If you have a legacy PSAP (Public Safety Answering Point or 9-1-1 Center), you depend on the local telephone carrier to keep their central office equipment up and running during any event. The challenge then becomes the local connections- analog phone lines, T1, PRI, etc that connect your PSAP. These are typically single threaded, even if you have two connections from the same provider, they can end up on the same fiber or the same central office.

If you are planning to implement a Next Generation 9-1-1 system, you CAN have much more control over this situation.

Commercial business, especially large companies, understand the critical importance of ‘uptime’ to their business model. It is common practice to have a least two facilities that house their critical servers (data centers) and multiple  telecom providers. 

Consider putting a multi-telecom carrier requirement in your RFP. A primary and backup (or active/active model) for your NG9-1-1 Core Services equipment should be standard. Your NG9-1-1 PSAP could have have a circuit from your local Telco, maybe a wireless LTE connection (router card) to a different vendor and if you have an alternate provider (maybe your local cable company offers business broadband), include that in the mix.

Satellite  is also being used by the ARK-TEX Council of Governments.

Bottom line, you protect your PSAP operations by not tying yourself to single vendor.

The culture, for years in the 9-1-1 community, has been one of trusting the local telco to take care of everything outside of the building.  If we truly want to maintain as much uptime as possible and see our PSAPs as part of the nations Critical Infrastructure, then we need to step up and get involved in defining these requirements.

 

As they say, it’s not personal, it’s business.

 

 

 

 

 

Ransom Attacks

OrElse

 

There are three broad categories of hackers:

  1. Destructive (crash a website, destroy data)

  2. Social Justice (Edward Snowden, WikiLeaks)

  3. Those that want to make a Profit 

One of the popular tactics used by those in category 3  is to hack into a company database (such as Target) and steal credit card information. The hacker can then take these to a “Cyber Pawn Shop” where the list would be published for sale on the ‘Deep Web’

yjYbypU

These ‘Cyber Pawn Shops’ sites will sell credit cards in bulk, the price is normally reduced as they ‘age’ in time. Bottom line- it is better to to steal LOTS of credit card info, which is why a hack such as Target was possibly a large financial score.

Now we are seeing a dramatic increase in Ransom Attacks. The hacker goes into the computer system, takes control and encrypts operational data, demanding payment for a password.

On one level this approach makes better business sense:

  1. Eliminate the middleman (Cyber Pawn Shops)

  2. Hackers set their own price

In some of the recent attacks, systems were simply shut down. Earlier this year a hospital had its medical records encrypted- basically shutting down the hospital (Hollywood Presbyterian Medical Center) until a ransom was paid. Hospitals, schools and cities are estimated to have paid hundreds of millions of dollars to date (source- CHRIS FRANCESCANI- NBC News).

Hackers are also targeting Police Departments nationwide.

A quote from the August 19, 2016 Wall Street Journal article by Robert McMillan:

“According to the U.S. Department of Justice, ransomware attacks have quadrupled this year from a year ago, averaging 4,000 a day. Typical ransomware payments range from $500 to $1,000, according to cyberrisk data firm Cyence Inc., but some hackers have demanded as much as $30,000.” Link

One aspect of this that is of great concern- in many cases these organizations have an IT department and security policy. They have purchased the typical ‘products’ such as firewalls, etc. So how can this be happening?

Today it is estimated that over 90% of the Ransom Hacks enter through a ‘phishing’ email, which an employee ‘clicks on’.

SO- based on the above, we can see two issues:

  1. The phishing email made it through the system

  2. Employees may not be properly trained regarding email security

If you’ve already transitioned to an IP based NG9-1-1 system you are safe- for the moment- as email is not directly connected.

But how about future hacking techniques or 9-1-1 text messages with hyperlinks? 

APCO Project 43, NENA and the FCC Task Force on Optimal PSAP Architecture (TFOPA) are all discussing the issue of security as we transition to IP. We should stay plugged into their ongoing recommendations…

 

 

 

A Shout Out to NextGen Tech Support!

tech-support

One of the greatest impacts the transition to Next Generation 9-1-1 will have is to those individuals providing technical support.

So, let’s put this in perspective- below shows the traditional 9-1-1 system today.Legacy

When a PSAP calls for support, the first thing the provider might ask for is the circuit information. Based on the diagram above, this makes sense- I check the circuit, then the equipment on either end. All 9-1-1 calls are send to the Local Telephone Company and its Selective Router. So, if things look good on the telco end and the circuit (CAMA Trunk) appears to be functioning, the next step is to check the equipment in the PSAP. Maybe a  dispatch to the PSAP to address the issue.

An important factor- the above picture shows that everything is:

  • Local

  • Linear and connected

  • Logical to troubleshoot

  • 9-1-1 Calls flow from left to right

  • Problems and outages are normally isolated to a single PSAP– there is no ‘network’

So imagine you’ve been in your role for awhile. You understand how things are connected, the different vendors, you are good at your job. You understand the critical nature and the importance of resolving any 9-1-1 issue as quickly as possible.

One day you receive a call from a PSAP that is connected to an ESInet (Emergency Services IP Network). Hello NextGen..

Our model here in Palm Beach County, FL now looks like this:

ESInet

  • There is more than one selective router and they are located over a hundred miles apart

  • The call processing has been removed from the local PSAP and is now ‘hosted’, in our case, in Data Centers – one five counties north. There are now two Call Processing systems and they support ALL of the PSAPs

  • A technical issue might affect one, many or ALL PSAPs

  • In the world of Internet Protocol (IP), things are connectionless, meaning that you are going to need additional technical support to see how the specific 9-1-1 call flowed through the system. There is no dedicated path a 9-1-1 call must follow. GIS and other I3 functions may engage

  • The complexity of the support model has now changed dramatically

  • The “Time to Repair” a problem has the possibility, especially during these times of transition, to be extended

If you provide technical support in this environment- thank you. Management has hopefully provided you and your team with a diagram that depicts each customers layout…

Training, staffing and procedures are all issues our vendors and partners are working to address.

 

 

NextGen 9-1-1 Inside the PSAP

If your career is INSIDE the Public Safety Answering Point (PSAP), one question might be “How will NG9-1-1 help us do our job?”

In speaking with Telecommunicators (those dedicated individuals that answer your 9-1-1 call), many identify the following to be important:

  • Reduce the number of transfers– in other words, have the network route wireless 9-1-1 calls to the correct PSAP the first time, eliminating the need for 9-1-1 Call Takers to transfer the call elsewhere. note: I’ve listened to a lot of 9-1-1 calls, it can literally freak people out to tell them that you cannot help them, that they are located in another jurisdiction and you will need to transfer the call. Recently, a mother whose child needed assistance called 9-1-1 and was told that her call needed to be transferred. She became extremely upset. Later, she went to a local TV station to complain. Dan Koenig, from my team, and I were interviewed for a segment to explain how the current system works. 

Today most wireless 9-1-1 calls are routed, not by the callers location, but by which cell tower and cell tower face (sector) their mobile phone is connected to..

In the example below, the 9-1-1 caller is connected to a cell tower and sector that is ‘programmed’ to send all 9-1-1 calls to Municipality A. This is simply a limitation of how the traditional 9-1-1 network functions today.

PSAP A will receive this 9-1-1 call.  Once they determine the 9-1-1 callers location, they will transfer the call to PSAP B.

Cell Sector

  • NextGen impact:

With Geospatial Routing, the location of the caller is determined first. This information is then passed on to the GIS mapping function, which should then route the caller to the correct PSAP. In the picture above, the NextGen system should route the 9-1-1 caller directly to PSAP B.

Update: Admiral David Simpson of the FCC (Bio Here) posted a comment on this blog, mentioning that there are numerous occasions, throughout the country, where the mapping capabilities of (in this scenario) PSAP A would not provide the location of the 9-1-1 caller, so the Call Taker would not know where to transfer the call to provide assistance.

It is important to note that the above example addresses only a single  aspect, routing the call to the correct PSAP. Next post, we will look at the Holy Grail of identifying the exact dispatchable address of the 9-1-1 caller…

FCC Chairman Testifies on NG9-1-1

HECC

Today there was a meeting regarding the Federal Communication Commission’s (FCC) oversight responsibilities, conducted by the U.S. House of Representatives Energy and Commerce Committee.

I am encouraged by the recommendation regarding funding for NG9-1-1.

Below is the testimony of FCC Chairman Wheeler:

Text of Wheeler Testimony on NG9-1-1

During my tenure as FCC Chairman, and in my prior testimony before this Subcommittee, I have been very vocal about the urgent need to improve our 911 system. The recent tragedies in Orlando, San Bernardino, and too many other cities highlight the importance of 911 in times of crisis. 

The Commission has taken action to improve the quality and accuracy of 911, and there is good news to report. We see industry is stepping up to many of the challenges, improving 911 location accuracy, supporting text-to-911, and generally investing to improve network reliability and resiliency. 

But effective 911 service depends on our nation’s 911 call centers. These Public Safety Answering Points, or PSAPs, must have technology to receive and process calls quickly, accurately locate callers, and dispatch an appropriate response. The unfortunate fact is that 911, designed originally for analog voice, doesn’t scale effortlessly to the advanced digital, wireless, and multi-media technology landscape. In too many communities, the PSAPs are relying on dangerously out of date technology, and the transition to Next Generation 911 (NG911) – envisioned by Congress in 1999 when it established 911 as the national emergency number – has not started or is stalled. Resource-strapped local jurisdictions struggle to maintain existing 911 service, let alone to achieve Congress’s NG911 vision. 

Industry and many states, counties, and cities are working hard to address transition risk and achieve NG911 capabilities. Nearly 20 percent of counties now support text-to-911. Many jurisdictions are building out their Emergency Services IP Networks – the basic backbone for NG911 in their communities.

But these islands of progress are the exception, not the rule. Unless we find a way to help the nation’s PSAPs overcome the funding, planning, and operational challenges they face as commercial communications networks evolve, NG911 will remain beyond reach for much of the nation. Let me be clear on this point: 911 service quality will not stay where it is today, it will degrade if we don’t invest in NG911. 

Congress has the unique ability to accelerate the transition to NG911. A clear national call to action, with timely application of resources, would actually lower NG911 transition costs by shortening the transition period and enabling 911 authorities to retire costly legacy facilities more quickly. Here are three ways that Congress could help:

  • National 911 Map: PSAPs are increasingly dependent on electronic maps for 911 routing and location, but the maps that they rely on should not end at the county or state line. Congress could authorize and fund the FCC (in collaboration with DOT) to create a national 911 map that would be available to every PSAP and would eliminate the seams between commercial communications network infrastructure and emergency response dispatch systems. 

  • Cybersecurity Defenses for PSAPs: PSAPs face the same cyber vulnerabilities that have proven so challenging to both government and commercial organizations, but most lack trained workforce and the necessary tools for cyber defense. Congress could bring PSAP IP Networks under the protective umbrella of DHS’s “Einstein” program by funding the deployment of intrusion detection sensors for NG911 networks.

  • National NG911 Implementation Date with Matching Funds: Currently, there is no national timetable or target date for completing the transition to NG911. Congress could establish a nationwide NG911 implementation date (e.g., to complete the transition by the end of 2020) and authorize matching funds to help state and local communities achieve this goal. Congress can further jump start this effort by ensuring that federally run PSAPs and Emergency Operations Centers make achievement of NG911 capability a funding priority.

This Committee has commendably made public safety a priority, and I urge you to do everything in your power to make sure our nation’s 911 system evolves safely as it adjusts to achieve your NG911 vision and that PSAPs have the tools and support they need to avoid undue risk in the transition. 

APCO Broadband Summit

img_0449

This week I participated in my first ‘Broadband Summit”.  I gave a presentation on the network management tools (SNMP, Wireshark, Dashboard) that we have implemented within our NextGen 911  ESInet in Palm Beach County.

FirstNet was, of course,  a major topic. A number of their executive team members were present. During the two day summit, there was occasional discussion regarding the potential synergy between FirstNet and NG9-1-1.

During my presentation, I put up the following slide to depict the PSAP’s  relationship to both FirstNet and NG9-1-1.

APCO Broadband Summit Slide

FirstNet (on the right) will directly connect to the PSAP, mainly for dispatch  purposes. The other role the PSAP plays, call taking (inbound on the left), I left blank. This is simply to depict that there is no nationally funded broadband program to ensure we can implement NextGen 911 across the country. It is possible that FirstNet might be rolled out in a region with no corresponding NG9-1-1 system.  Pictures, video, etc. ‘from the scene’ would not be available prior to first responder arrival.

Another topic of interest was FirstNet Apps,  applications that will run on the new broadband  first responder handsets:

firstnet app

The app depicted above could provide visual, real time info on data such as field intelligence, nearest officers, local maps, photos of a person of interest, etc. This is, of course, what we expect as we provide a hardened, ‘smart device’ to Public Safety teams.

And finally- FCC Commissioner  Jessica Rosenworcel addressed the group. Her comments include the following:

“..take jurisdiction. Federal authority over 911 is limited and with 6000 public safety answering points nationwide, we have a system that is vast—and vastly different in different parts of the country. There are 375 call centers in Mississippi but only 12 in Nevada. Yet both states have populations of just under 3 million. In other words, we have very different ways of managing emergency calling in different parts of the country. It makes a uniform effort hard.”

A positive event. Lots of information on ‘what’s coming’ and a number of dedicated individuals pushing to ensure we ‘get there.’

 

When Your “backup” fails..

netmon_rev1

Last month a close associate of mine experienced two outages at one of his PSAP’s. The first event lasted over three hours, the second 15 minutes. All 9-1-1 calls were rerouted.

The PSAP in question is served by two circuits. Supposedly physically diverse. During the conversion to IP in 2011, each circuit was physically routed into the PSAP facility in a manner to ensure diversity.

This latest outage revealed that, in the providers central office, both circuits connected to the same equipment rack, same shelf and same card. Back in 2013 he had a similar scenario, only then the outage took down six PSAP’s. All circuits, primary and backup, terminated onto a single card. This sounds impossible, but I have read the official ‘Outage Report’.

An important fact- they spent tens of thousands of dollars throughout their county in ‘outside plant’ construction costs, with their service provider, to make sure there was physical diversity into the PSAP.

He also educated me on a fact he recently learned- In 2013 the Federal Communications Commission (FCC) issued the Derecho Report. This was in response to major storms and 9-1-1 outages in the midwest. The report listed a number of findings regarding 9-1-1 providers. Here is a quote from that document: “In most cases, the 9-1-1 and other problems could and would have been avoided if providers had followed industry best practices and available guidance.”

Derecho Report and Recommendations

This report resulted in the FCC issuing an Order:

FCC Order to Improve Reliability       Note -Appendix B- Part 12 Final Rules

The bottom line- as a customer you can request an audit, to include the physical path of  your 9-1-1 circuits. Here in Palm Beach County we now have an audit in progress.

Remember- it does not matter how big your circuits are  or how many you have (2, 3). When it comes to the last mile, the connection from the PSAP to the service provider, it is a PHYSICAL world.

A great way to address this issue is to contract with two providers. But only if they do not share the same telco facilities.

We are transitioning to a new world in 9-1-1. Like it or not, we occasionally will need to become more actively involved in order to understand how services are provided.