Challenges and Solutions for Incident Response in OT Networks

Industrial control system networks are among the most important networks in industrial enterprises. In the worst case, the consequences of compromising these networks are unacceptable physical consequences, as well as business and reputational consequences. The highest security priority for essentially all physical operations is safety – safe operation of everything from elevators, railway switching systems and building-sized boilers to catalytic crackers and petrochemical pipelines. The second priority for most physical operations is reliability – from preventing damage to large, costly equipment, to keeping our lights on, keeping clean drinking water in our taps and keeping gasoline in our gas stations.

These priorities make incident response in OT environments challenging for many reasons. By the time an incident response team arrives at a physical site, the site has typically already suffered a safety shutdown and possibly suffered other physical consequences. Before physical operations can resume, the incident response team must identify both the nature and extent of the compromise at the site, and the attack paths used to bring about the compromise. All of this is essential to determining which backups of equipment configurations may be trustworthy, restoring configurations at the site to a consistent, trustworthy state, and eliminating the offending attack paths. Engineering teams at the site will authorize the resumption of physical operations only when they are convinced that such resumption is safe.

The Key = Visibility into OT Networks

Visibility into OT networks is vital to security programs and to OT intrusion response, for three reasons:

1. Visibility into OT networks contributes to understanding residual risks at industrial sites, so that those risks can be remediated before attacks occur, preventing unacceptable physical consequences.
2. Visibility into OT systems and networks is vital to early detection of attacks in progress, so that safety shutdowns can be triggered and incident response teams can be dispatched promptly. Prompt diagnosis and response can reduce the physical consequences of attacks.
3. Reliable OT forensic data is essential to understanding the timing of an attack, identifying how the attack was carried out, and identifying which conventional computers or specialized control system equipment may have been compromised. This information is vital to determining which backups are trustworthy, which attack paths must be remediated and which conventional or specialized control computers must be restored from known-good backups.

However, while Intrusion detection and security monitoring are the backbone of enterprise security programs, what we see at Waterfall Security is that many industrial enterprises are largely blind to the security status of their most important industrial equipment. Worse, when industrial enterprises do deploy security monitoring and intrusion detection systems for their industrial operations, those monitoring systems are deployed and configured in ways that introduce new threats to safe and reliable operations. At too many sites, “the cure is worse than the disease.”

Security Monitoring Issues

Most industrial control systems are designed to operate with minimal connectivity to external systems. This has been the practice since industrial control systems were first invented – not for security, but for safety and reliability reasons. External connections have always posed a risk of introducing unexpected or incorrect messages into industrial networks and thus causing physical malfunctions. The cybersecurity concerns of the last decade mean only that the potential for such messages has become much more commonplace: the result of deliberate attacks rather than inadvertent malfunction.

The problem with OT security monitoring and network visibility initiatives is simple: increased visibility demands increased connectivity, and increased connectivity into industrial networks is itself a threat both to security and to safe and reliable physical operations.

Take network monitoring for example.  Today, OT-focused Intrusion Detection System (IDS) sensors are becoming widely available. Such sensors have at least two kinds of network interfaces – one set of interfaces to connect to the mirror ports on network switches to acquire network traffic to analyze, and a second set of more conventional network interfaces to manage the devices.

Most IDS sensors  need regular adjustment to provide human oversight of machine learning algorithms, reduce false positives, update signatures, update software, and so on. The question when deploying OT IDS sensors is this: to which network do we connect the management ports? If we connect the ports to OT networks, now security analysts from our central SOC need permissions and access paths to log into every one of our industrial networks, throughout our enterprise.

Operations departments generally and legitimately resist such configurations. “You want who to get access to all of our plants?” they ask us, “To do what? To do it to what on our networks?” Operations networks are sensitive – they control costly, powerful and often dangerous physical processes. One false move on those networks and we have physical consequences – the same consequences that our security systems are supposed to prevent.

If, instead, we connect IDS sensors’ management ports to enterprise networks, operations still push back. Any such connection represents an opportunity to attack the sensor from that external network. A successful attacker can then pivot their attack through the compromised sensor into the OT network through the mirror port(s). Most mirror ports on managed switches are bi-directional by design, because many IDS sensors can also be configured to be intrusion prevention systems. This way, when the sensor detects an attack, the sensor can send packets back into the mirror port to close the TCP connections observed to be carrying the attacks. Even when switch vendors claim their mirror ports are unidirectional, that unidirectionality is controlled by the switch software, which may itself be vulnerable to stolen credentials, known software vulnerabilities and zero-days.

Safe Visibility

To enable safe visibility into OT networks, industrial enterprises are increasingly deploying industrial-grade, hardware-enforced unidirectional gateways. For example, Waterfall for Intrusion Detection Systems (IDS) is a solution that addresses the OT IDS sensor concerns. Waterfall for IDS enables safe deployment of new-generation OT IDS sensors, such as those from FireEye, Indegy, Dragos, CyberX, ForeScout and others. Waterfall for IDS unidirectionally emulates OT mirror ports to IDS sensors. This unidirectional emulation allows OT IDS sensors to be managed and updated directly from enterprise networks, by enterprise SOC analysts, without risk to operations.

How does this work? Waterfall for IDS is a type of unidirectional gateway. Deployed between an OT IDS sensor and an OT mirror port, the product hardware is physically able to send information in only one direction – from the mirror port into the IDS sensor through two circuit boards connected by a short fiber-optic cable. One board has a fiber-optic transmitter/laser, while the other has a receiver but no transmitter. This means there is physically no way to send any information back into the OT network from the OT sensor. It does not matter how clever our attackers or how sophisticated their malware, there is no laser on the receiving circuit board. If no information can flow from a compromised sensor back into an OT network, then no attacks can flow back either. Those sensors though, still see a clean feed of OT network traffic to analyze and so can alert our central SOCs to any problems that arise on OT networks.

More generally, Waterfall’s wider family of Unidirectional Security Gateway products enable not only mirror ports to be connected to enterprise sensors, but a wide variety of other security monitoring data sources, including Syslog servers, SNMP-capable devices, security policy monitoring data and operations data such as OPC and historian data for OT-aware SIEMs. With Unidirectional Gateways, operations resistance to enterprise security monitoring programs vanishes. Industrial enterprises can monitor their OT networks and manage OT sensors from their central SOC, all without risk to the monitored OT networks.

Waterfall for IDS and Waterfall’s Unidirectional Gateways use common unidirectional hardware that is Common Criteria certified to level EAL4+ (high attack potential). This means that the entire solution is certified to resist even the most sophisticated cyber attacks and adversaries. Waterfall products are also certified by the French ANSSI authority, Singapore NITES, Israeli NISA and other authorities.

Competing solutions have no such certifications or assurances:

• Firewalls are software – steal a password and all of the software’s defenses are defeated.
• Network switches may claim to have unidirectional mirror ports, but vendor documentation in this regard is notoriously unreliable. No network switch is certified for unidirectionality.
• Network taps often claim unidirectionality, but we have only the vendor’s claims here. Again, no network tap is Common Criteria certified for unidirectionality.

In addition, Waterfall’s product line is recognized by industrial authorities, such as NERC CIP, US DHS and other standards and best practices, as enabling the highest degrees of protection for monitored OT networks.

Effective OT Incident Response

Visibility into OT networks is vital to preventing very costly security incidents in OT networks, as well as to effective incident response for those networks. Fortunately, we are at the beginning of a new era for OT security monitoring and visibility. A host of vendors are finally producing credible OT IDS sensors, SIEM vendors are updating their systems to understand OT networks and enterprise security teams are integrating these technologies into their systems, people and processes. Waterfall for IDS and Waterfall’s other Unidirectional Security Gateway products support this evolution.

The way to bring operations and engineering teams on board with enterprise security programs and security monitoring initiatives is to relate cybersecurity to those teams’ safety and reliability priorities and to demonstrate conclusively that security monitoring initiatives introduce no new attack paths into vital operations networks. Waterfall’s Unidirectional Gateway products have been serving OT data safely and reliably to enterprise networks for over a decade. Applying these products to the emerging imperative to bring OT networks into enterprise security monitoring programs is a natural application of the technology, one that operations teams embrace rather than resist.