FCC Report on Derecho Telecom Failures: It's the Back-Up Power

As I mentioned in a post after Hurricane Sandy, the lack of back-up power, more than anything else, was a likely main culprit for communications failure after that natural disaster.  Today, the Federal Communications Commission (FCC) released a report about communications failures not surrounding Hurricane Sandy, but those that occurred during and after the earlier derecho events in June 2012, which hit the mid-Atlantic hard, causing days of power outages and communications failures during an extreme heat wave. Altogether, more than 1.2 million wireline users across twelve states experienced outages as a result of the derecho and 11% of cell sites were knocked off-line.

The FCC's report is surprisingly harsh on the seven communications providers examined in the investigation (I say surprisingly because the Commission's reports and opinions are usually over-lawyered and strive for a boring sense of balance and have in recent years rarely adopted a critical tone). The regulator raps the providers for failing to follow industry best practices established by the agency's Communications Security, Reliability and Interoperability Council (CSRIC), which is composed of industry representatives.  The report notes:

For reasons explained below, communications failures during the derecho revealed that many providers failed to implement crucial best practices developed by CSRIC that could have mitigated or prevented many of the storm’s most serious effects on communications networks,  including 9-1-1 service outages.  This failure, and the resulting damage, was costly.

Specifically, the chief reason for both wireline and wireless outages was the failure by the providers to ensure adequate back-up power.  Around 7% of Verizon's generators in central offices failed to operate properly when needed and the telco admits that generator failures in its Arlington and Fairfax, VA central offices directly resulted in the loss of 911 service to residents in northern Virginia .  Indeed, Verizon had suspended testing of the Arlington generators back in 2011 due to "a problem with the uninterruptible power supply in that  office."

The report, written by the FCC's Public Safety and Homeland Security Bureau, comes down hard on Verizon, the major communications provider in the affected area, citing shoddy maintenance procedures.  "The Bureau has serious concerns regarding Verizon’s actual repair practices and compliance with its own maintenance policies."

Verizon, however, is not alone in bearing the blame.  Another major area provider, Frontier Communications, gets its knuckles rapped too.  (Some Frontier Communications remote terminals weren't even equipped with back-up generators.)

Although this report offers a series of recommendations to fix the problems, when it comes to back-up power, the recommendations echo those of a similar report by an FCC task force issued in 2006 to assess the communications outages that occurred in the wake of Hurricane Katrina.  Namely, the FCC asks the industry to follow its own self-devised best practices.

The CSRIC best practices have been developed on the basis of widespread industry participation.  This creates a strong presumption that providers would be inclined to implement them—particularly those recommendations that are deemed most vital.  Still, many providers failed to implement crucial  best practices throughout the area affected by the derecho, which includes the densely populated National Capital Region.  We call on providers again to review and implement CSRIC and other best practices and emphasize the importance of doing so.  The proper implementation of CSRIC best practices could have prevented many of the derecho’s most serious effects on communications networks, including 9-1-1 service outages.

The earlier report also asked the communications industry to follow industry best practices. If the industry fails to pay attention to what, in essence, is a very minor task, certainly in comparison to running massive, global digital broadband networks, the third such report out of the unexpectedly irritated FCC may well go beyond simply urging the industry to follow best practices.

Image from FCC report.  NORS stands for network outage reporting system.

Posted in: Crisis Communications,Critical Infrastructure