Review Your Network Traffic

Sweden

I spoke last week with a colleague regarding Cyber Security. While reviewing network traffic with his security contractor, they noticed a vendor server was reaching out to known criminal (out of respect for Chris Roberts I will not use the Hac*er word) sites for Domain Name System services- DNS. These sites were located in Sweden and Finland.

It appears that the rogue DNS code was injected into the 9-1-1 vendors software. The vendor was unaware that this breach had occurred (but is currently resolving the issue) . Fortunately, the firewall was properly configured and did not allow the DNS response, coming from our friends in Finland and Sweden, to invade his network.

Many 9-1-1 sites have a vendor or IT contractor administering their firewall. They will open and close ports on request. They are typically not under contract to monitor/evaluate outbound traffic.

Understanding the data traffic on your network is critical. There are other stories out there regarding 9-1-1 centers having active two way traffic with foreign nations (I’ve read the reports) that should induce a wake-up call.

High blood pressure is often called “the silent killer” because it typically has no symptoms until after it has caused significant damage. Putting together an overall cyber security strategy should include, at a minimum, conducting a cyber benchmark- sort of like checking your blood pressure…

 

 

 

Securing CAD in 9-1-1 Centers

 ~Cyber Attacks and Computer Aided Dispatch (CAD)~

800px-petyaa

9-1-1 Centers (PSAPs) serve two main Public Safety functions:

  • INBOUND– Answering a 9-1-1 call or text from the public
  • OUTBOUND– Dispatching first responders (fire, EMS, law enforcement)

In most centers, these systems exist on two separate networks. The inbound network is typically installed and maintained by the vendor or channel partner who provides the Call/Text product (TriTech, Motorola, West, Solacom, AT&T, etc..).  The outbound (Computer Aided Dispatch) CAD system, in many cases, resides on a network managed by the local municipality or county government.

We currently have a PSAP in South Florida that has been without CAD for three weeks. A ransomware attack via the city email system made its way through the municipal network, into the 9-1-1 center and locked down a number of law enforcement systems, including CAD. For the past weeks, communication with first responders is a manual, paper and pen process.  The attack was not directed @ the 9-1-1 Center, but the collateral damage is a major hit to operations.

ALSO: Currently, in a Mid-Atlantic state, there is a PSAP whose CAD system has been down for weeks as the result of a cyber attack.

I have spoken to numerous centers across the country who have experienced similar CAD outages. Most of these were not as well publicized as major cities like Baltimore.

Can’t Patch Me

There are still government entities out there running Really old stuff.. and in some cases REALLY REALLY OLD STUFF (e.g.  MS SQL 2003-  Microsoft only supports back to SQL 2008). We should not run applications that are considered critical infrastructure (9-1-1 CAD) on the same network as these systems…

9-1-1 Center  Managers and Directors may have no clue regarding this connectivity or how to patch and protect these outbound networks.

We need to rethink how we deploy mission-critical CAD in 9-1-1 Centers.

Today in the 9-1-1 community there is a lot of excitement around new vendors and product offerings, including enhanced location accuracy and cloud-based applications.  In my opinion, Cyber awareness needs to be included.

 

 

 

 

 

 

 

Crypto-Jacking: Using 9-1-1 Center Computers for Profit

CPU 4

Most of us have heard the term ‘crypto mining’. Cryptojacking is simply using the computer power and electricity of another person or company (in this case a 9-1-1 center or PSAP) without their knowledge to ‘mine coins’.

Simple Explanation

Cryptocurrency is, by design, decentralized. Transactions, which are encrypted, are added to a “block’, the block then gets added to a chain (blockchain). With a computer CPU, or graphics cards, crypto miners run a process to verify these transactions and keep the cryptocurrency world running. For this validation service, crypto miners receive a small payment.

It is less expensive for crypto miners to use ‘someone else’s’ (meaning your) computer processing and electricity.

Cryptojacking is becoming more popular. It is easy money and cryptojacking ’kits’ are available on the dark web for as little as $30.

The actual code can run in the background for a long time without detection. Unlike ransomware, the goal is not to lock up your computers and hold them hostage for payment, the goal is to use as much of your systems CPU as possible without detection. The culprit could be an unknown ‘bad guy’, an employee, contractor, etc.

Public Safety Answering Points (PSAPs) Compromised

We are starting to receive reports of 9-1-1 Centers conducting ‘Cyber Security Benchmarks’ and identifying cryptojacking software. For example, one 9-1-1 center found that cryptojacking was using 60% of the CPU on their Computer Aided Dispatch (CAD) workstations.

In addition to compromising the performance of ‘critical infrastructure’ systems,  whoever installed this software OWNS YOU. They may be content with simply using your PSAP to make money, but the fact is they have established two-way communications and are running hidden, malicious software. They could easily take your infected systems down hard.

In the 9-1-1 community, we need to step up our awareness of cybersecurity.  While this may be a line item PSAPs will consider putting in their budget for next year- the fact is every center should consider conducting a cyber benchmark asap.

A final thought:

APCO BLACKHAT DEFCON are all in Las Vegas next week.

Be careful out there.. turn Bluetooth off on your devices and please do not connect to unknown or free WiFi.

TDoS Pilot Project

slc_logo_60c

S&T Logo

Today we finalized a two-year agreement to work with Securelogix on a pilot to research Telephony Denial of Service (TDoS). The Department of Homeland Security Science and Technology Directorate is funding the pilot.

Podcast with Mark Fletcher 

The article below provides a good overview:

TDoS Pilot

Securelogix will install equipment at our two core node locations, initially to collect call data. Here in Palm Beach County, we have a backup PSAP that will be used for testing purposes in the future as the pilot project evolves.

Attacks on our national 9-1-1 infrastructure are incredibly serious- overwhelming the system with ‘fake calls’ can prevent a legitimate call from being answered and dispatched.

Today there is no easy answer on how to address an active TDoS attack. We are hopeful that this project will produce useful data to formulate a response that can benefit the entire 9-1-1 community.

FirstNet and NextGen 9-1-1

There is discussion in the 9-1-1 community regarding FirstNet and how it might relate to Next Generation 9-1-1. Here in Florida, we recently received a formal briefing on FirstNet.

As a reminder- in the world of the First Responder, the current Land Mobile Radio (LMR) system for voice will remain.  The initial FirstNet deployments will be data only.

SO…

table

Disclaimer- The following are my personal thoughts.

FirstNet may become much more than a wireless network. They have the ability to become THE leader in specialized public safety applications, applications that could be used by First Responders nationwide, regardless of whether their state has chosen to ‘opt-in’ or ‘opt-out’.

Core

There is also the potential that they could host existing software applications, maybe providing a value add by obtaining a larger volume licensing agreement from the vendor, an incentive to utilize the FirstNet Core.

In ‘NextGen’ 9-1-1, voice is an application. By this I mean that it utilizes SIP (session initiation protocol), which operates at Layer 7, the application layer of the OSI model. Translation- voice is an application.

FirstNet could offer VOICE services for the 9-1-1 community.  Simply add a hosted voice server to the graphic above. This could be of tremendous value,  especially to those states (mostly home rule) who are still putting together their NextGen 9-1-1 strategy.  FirstNet needs a core backbone network, why not provide voice services? Voice uses very little bandwidth.

The other aspect is that this lays the foundation for a real Public Safety Broadband Network. We do not need to pay for and operate TWO networks- it certainly does not happen in the business world.

Connecting from the core network to the 9-1-1 center (PSAP), it would make sense to have two types of connections, one land based and one wireless. Diversity..

And the critical aspect of security– we expect to utilize pictures and videos on the FirstNet wireless network. What better way to control the pictures and videos planned to be coming inbound to 9-1-1? Have them ‘land’ in the FirstNet core, when they can be dealt with and controlled prior to potentially being pushed out to First Responders.

Intrado (now West) pioneered the concept of hosted 9-1-1 services and the use of LTE wireless as a backup for 9-1-1 Centers (PSAPs). It’s all possible.

Instead of congress funding a separate NextGen 9-1-1 initiative, maybe there could be incremental funding to FirstNet to include the NextGen 9-1-1 services.

 

Telemarketing Calls Invade 9-1-1

Vector icons set call center avatars in a flat style

 

Call Taker:  9-1-1, what is the location of your emergency?

Caller: Hello! how are you?

The caller then breaks into a sales pitch for:

  • Dental Insurance
  • Priceline
  • Hilton Resorts
  • Marriott Resorts
  • Orlando Theme Parks
  • Cancun resorts
  • Travel Dollars

and many more….. The telemarketers did not understand that their Robo-Dialer had routed their call to 9-1-1. This specific issue began in South Florida in September and is still in the process of being resolved.

Wall Street Journal Article

HOW IS THIS POSSIBLE?

If you work in a 9-1-1 Center you know the term pANI.

When a wireless caller dials 9-1-1, the system inserts a fake or “pseudo’ phone number (automatic number identifier or ANI) into the 9-1-1 call flow while the wireless callers real phone number is being identified. This pANI is sent, along with the audio, to the PSAP and presented on the call takers screen.

If you look at a cell tower, there are normally three sides or sectors and each sector has a group of these ‘fake’ or ‘pseudo’ numbers associated with it.

By design, you should not be able to directly dial these ‘fake’ phone numbers- Example below-

pani

What we now understand is that one of the major wireless carriers utilized phone numbers that can be dialed directly in their 9-1-1 configuration.

SO- A telemarketing company loaded a series of sequential phone numbers into their dialer, and the fake or pANI phone numbers were included. Because the wireless carrier had these numbers configured wrong, the robo caller dialed the number and directly connected their telemarketer to 9-1-1.

It took a lot of research and time to identify the root cause. Tremendous frustration on the part of 9-1-1 call takers who endured this issue for weeks.

Eventually the root cause was identified and the wireless carrier began making the appropriate configuration changes (requesting non-dialable numbers to be used as the pANI)  throughout their network.

Now- if we actually had Geospatial routing, the need for the pANI would go away…

 

GeoSpatial (location based) Routing- We Tried

soup1

When I took over this project in 2013, one feature that appealed to me was geospatial  routing. Instead of wireless 9-1-1 calls being routed to a PSAP based on the cell tower/sector database, it would now be routed to the correct PSAP based on the location of the caller. (You NG9-1-1 technical types out there- you know the acronyms and flow)

We have hundreds of cell towers and 18 PSAPs, so the idea of avoiding/reducing 9-1-1 transfers between PSAPs  made tremendous sense.

There were plans to create a geofence around our main courthouse and the airport. Both of these facilities are located within the City of West Palm Beach, yet – as County facilities- are staffed by the County Sheriffs Office. Placing a geofence around these properties and routing 9-1-1 calls from within the geofence directly to the Sheriffs Office makes sense and, as we were told, easy to do.

courthouse

logo-courthouse

Palm Beach County Courthouse West Palm Beach

Our GIS team attended training, reviewed the standards and went to work on preparing our data. A few months ago, after having the data professionally ‘vetted’, we felt that we were ready to move forward with location based- geospatial- routing.

Around this same time we were asked to look at the current routing of 9-1-1 calls FROM the Town of Palm Beach. Donald Trump has a ocean front home on the island (not far from  our offices).  The town is a 16 mile long, narrow, barrier island. There are only a few cell towers. As a result, a number of 9-1-1 calls that originate on the island are connected to cell tower sectors across the intracoastal waterway. They are routed first to a ‘mainland’ PSAP and then transferred back to the island.

The ideal scenario would be to route all 9-1-1 calls directly to the Town of Palm Beach PSAP. In this scenario, turning on geospatial routing made sense.

In our industry there is a lot of talk about implementing this feature. So- when we reached out to the wireless carriers to let them know we were ready- we were surprised at the response.

Today, wireless carriers in our area are not ready to transition away from the MSAG and cell sector routing. It appears to be a complex issue. A portion of the 9-1-1 fee is returned to the carriers for providing  9-1-1 services (including the MSAG), so moving away from this may take time.

I have been told that there are NG9-1-1 deployments out there that are doing geospatial routing. I do not mean holding the call and waiting for the ‘Phase 2’ data as the initial input..

If you are reading this and are truly doing geospatial, please comment below – I’d love to speak with your wireless provider. In the meantime there are options being discussed in certain working groups, led by the FCC.

 

Campus Safety- What Part Does a PSAP Play?

fau

Boca Raton is a beautiful city of approx. 91,000 located in southern Palm Beach County. Florida Atlantic University (FAU) sits on 850 acres within the city limits-  a mere 2 miles from the ocean. About 31,000 students attend this campus.

The university has its own sworn officers, giving it legal jurisdiction over the campus. 9-1-1 calls originating from the campus, however, are routed to the Boca Raton Police Department PSAP.

Here is an example scenario- a student on the FAU campus calls 9-1-1 for assistance. The call is answered by Boca Raton PD.  Medical and fire calls were (and still are) handled by Boca PD, while all others (law enforcement) were transferred to the FAU Campus Police. The call was answered on a traditional desktop phone.

There was no call back number displayed and no map to provide the callers location.

In 2015, FAU submitted a formal request to my department to become a Secondary PSAP. Their rationale was student safety.

fauboca

It was not uncommon for a campus 9-1-1 caller to NOT know their exact location (I’m in the parking lot!!!). After a few visits to the campus and meetings with both the Boca Raton PD and Campus Police, it was decided to move forward with the request. The State of Florida gave the project final approval and we recently went ‘live’ with the new PSAP.

FAU Campus Telecommunicators can now see the 9-1-1  callers location and their phone number.

west

Palm Beach Post Article

The University plans to integrate building floor plans into the 9-1-1 system- which could be of great benefit. One of the positive aspects of working with the University is that they own the buildings, so we do not need additional permissions, etc. (such as with, for example a regular business).

And so, while there is continued discussion across the country regarding primary PSAP consolidation, we need to also concern ourselves with safety.

There is no simple answer.

 

 

 

State 9-1-1 Boards and NextGen

boardmeeting-300x195

 

One aspect of FirstNet that I truly respect is the fact that they are funded, organized and empowered regarding their mission. They have the potential to positively impact Public Safety on a national level.

Things are not as well defined with NextGen 9-1-1. At the state level, we currently have extremes regarding  9-1-1 Boards and their authority. While some states are up and running with NextGen, others are struggling with the initial planning. Two States (Wisconsin and Missouri) do not even have a state level 9-1-1 Board.

Laurie Flaherty and her team @ 911.GOV have done a great job collecting and interpreting data from states.

Depending on the specific details, Home Rule  can play a major role in the lack of centralized authority for 9-1-1.

There are initiatives today at the federal level regarding funding for Next Generation 9-1-1. We do not want the scenario of a state receiving funding for NextGen without a definitive plan. For those states that do not yet have a plan in place, one option is to engage the Department of Homeland Security Office of Emergency Communications.

This VIDEO may prove of interest. I participate as a subject matter expert (SME) for DHS and believe this program to be of tremendous value.

 

Critical Infrastructure

 

cr

 

There are 16 specific sectors in the U.S. that are considered critical infrastructure. 9-1-1 is  part of the Emergency Services sector, as defined by Homeland Security.

If you have a legacy PSAP (Public Safety Answering Point or 9-1-1 Center), you depend on the local telephone carrier to keep their central office equipment up and running during any event. The challenge then becomes the local connections- analog phone lines, T1, PRI, etc that connect your PSAP. These are typically single threaded, even if you have two connections from the same provider, they can end up on the same fiber or the same central office.

If you are planning to implement a Next Generation 9-1-1 system, you CAN have much more control over this situation.

Commercial business, especially large companies, understand the critical importance of ‘uptime’ to their business model. It is common practice to have a least two facilities that house their critical servers (data centers) and multiple  telecom providers. 

Consider putting a multi-telecom carrier requirement in your RFP. A primary and backup (or active/active model) for your NG9-1-1 Core Services equipment should be standard. Your NG9-1-1 PSAP could have have a circuit from your local Telco, maybe a wireless LTE connection (router card) to a different vendor and if you have an alternate provider (maybe your local cable company offers business broadband), include that in the mix.

Satellite  is also being used by the ARK-TEX Council of Governments.

Bottom line, you protect your PSAP operations by not tying yourself to single vendor.

The culture, for years in the 9-1-1 community, has been one of trusting the local telco to take care of everything outside of the building.  If we truly want to maintain as much uptime as possible and see our PSAPs as part of the nations Critical Infrastructure, then we need to step up and get involved in defining these requirements.

 

As they say, it’s not personal, it’s business.