According to the ICS-CERT (Industrial Control Systems Cyber Emergency Response Team) fiscal year 2015 final incident response statistics, the food and agriculture segment reported only two cyberattacks last year. Still, food and beverage processors should not rest easy. Though the probability of a cyberattack may be small, its cost can be high.
Fifteen years ago, industrial espionage could be accomplished via phone calls and “human engineering.” (See “Corporate Spies: The Pizza Plot” by Adam L. Penenberg and Marc Barry.1) Today, corporate espionage often involves computer system break-ins that can result in the loss of intellectual property such as recipes, vital brand information, customer data, inventory and production records, supplier data—and the list goes on. At the industrial controls level, a cyberattack could shut down your production, destroy processing equipment, hurt product quality or even adversely affect food safety. The question is: While your plant may recover from a break-in, will your brand survive?
Although you may think an intentional break-in invariably comes from the outside, that isn’t true. Disgruntled employees can do a lot of damage, if they have access to things they shouldn’t. Other attacks can result from just doing “dumb things.” For example, email viruses and embedded viruses on memory sticks can find their way into the control system, if it’s not properly protected. In a Computer Weekly.com interview, former FBI Computer Intrusion Unit Head Don Codling said of internal threats, “There is no [software] patch for careless, greedy or stupid.”2
So, what’s a processor to do? To prevent problems down the road, systems should be planned out, so they’re not spaghetti-wired and coded together as time goes by. “Planning, planning and more planning,” says Tricore CEO David McCarthy. “Even if a plant’s control systems don’t communicate to enterprise systems, we don’t want to shut the door on the possibility of such communications in the future. This often involves making sure the right hardware and security protocols are put in place upfront.” While this may entail a higher initial upfront cost, doing the job right the first time is often easier and less costly overall since items don’t need to be reconfigured/replaced at a later date.
“As part of the planning process, we typically meet with the operations and IT staff to discuss what data is important to enterprise users and then develop a network architecture capable of providing this data in a secure manner. Traditionally, this involves moving data through an Industrial Demilitarized Zone (IDMZ) similar to those defined in the Rockwell/Cisco jointly developed Converged Plantwide Ethernet (CPwE) Design and Implementation Guide,” adds McCarthy.3
The importance of a layered approach
“Everything starts with proper design, where functionally distinct zones [cells/areas] are segmented, and well-defined data paths [conduits] are present,” according to Robert Albach, Cisco Systems product line manager—IoT security. “The second step is implementing access controls and defining the proper communications flows. Although the process equipment should continue to feed outputs to the control layers as normal, chances are, ERP systems would not be able to directly consume those outputs. Consequently, ERP consumable data should be a secured unidirectional flow from the control systems to the ERP system through the IDMZ.”
“[Defense in depth] is a military principle that has been adapted to cybersecurity. It means the network should be set up to have many layers of defenses...[With DiD], if any single defense is breached, there are more layers of protection an attacker would need to circumvent.”
In other words, the data flow should be outbound only since this minimizes applications’ and users’ access rights into the operational side, reducing the likelihood of intrusions from the enterprise layer. In this way, the ERP system will be only a data recipient, satisfying the general guidance to reduce access to need-only scenarios, adds Albach.
“No single security product, technology or service is sufficient for today’s abundance of internal and external threats,” warns Gregory Wilcox, Rockwell Automation global business development manager, networks. “Consequently, securing industrial networks should be part of a holistic, multilayered security approach that encompasses policy and procedure, a facility’s physical security and computer hardening, as well as application and device security. ISA/IEC, NIST and DHS/Idaho National Laboratory standards all recommend using a defense-in-depth [DiD] security approach.”
“[Defense in depth] is a military principle that has been adapted to cybersecurity. It means the network should be set up to have many layers of defenses,” says Eric Braun, Emerson Process Management principal systems engineer. “[With DiD], if any single defense is breached, there are more layers of protection an attacker would need to circumvent.”
“We utilize the concept of defense in depth, which compounds multiple layers of security measures to protect the integrity of the network and information therein,” says Scott McCausland, senior solution architect for Process and Data Automation, a member of the Control System Integrators Association (CSIA). “By configuring industrial networks to be isolated from outward-facing networks [or office networks], we can control the flow of information in and out through the creation of a DMZ that is provisioned with specific instructions on what can go to each network and allows only authorized assets to send data to and get data through it.”
“To implement defense in depth, a network architect will likely layer the network, with field devices and PLCs at the lower layers and Internet-facing devices at higher layers,” explains Braun. “Firewalls can be used to segregate any given layer from the other layers, and network intrusion detection systems [NIDS] and network intrusion prevention systems [NIPS] can be used to detect and prevent network intrusions. Routers and switches can be used to create a segregated network, either with physically different networks or virtual networks. Then, if an attacker gains access to a segregated network, he or she will likely have access to only a portion of the network rather than the entire thing.”
I think I’m in trouble, but not sure
If your industrial network seems sluggish, your data transmission seems to take forever, or some of your dashboards’ info seems incongruous, how can you tell if an intruder is to blame?
According to Eric Byres, noted industrial network security analyst, “All intruders leave tracks on the network and on the computers they infect, no matter how skilled they are. Very skilled attackers are better at hiding their tracks, but none are perfect. Even the Stuxnet worm was chatty on the network and sent lots of messages between infected nodes using a protocol known as remote procedure call [RPC]. Using RPC is a clever move by the worm’s designers because it is the same protocol that underlies the ubiquitous industrial protocol called OPC. So, if all you know about your control network is that OPC is being used, you will never spot the Stuxnet traffic. But, if you know exactly which computers are supposed to share data using RPC/OPC and monitor the network for RPC messages between computers not using OPC, Stuxnet sticks out like a sore thumb.”
Sometimes, processors invite attackers into their facility by doing “stupid things.” Richard Clark, InduSoft-Wonderware/Schneider Electric applications and control systems security engineer, recounts an all-too-familiar story. “In the early days of the Internet, I went to a milk formula processing plant in Denmark when our software ‘wasn’t acting right.’ It turns out the plant maintenance guy had connected one of the HMIs to the Internet, and the personnel were using it to surf, etc., and the plant contracted a Trojan horse program. The program ran from machine to machine, selectively destroying operating systems and hard drives. A complete reformat of every machine in the process control system was required, resulting in a couple of weeks of lost production.”
When trouble is suspected on an ICS network, certain steps must be taken. “If the symptoms are severe enough, isolate the control network from other networks—physically, if possible,” suggests Darin Heitman, senior engineer at Frakes Engineering, a CSIA member. “Review any available network or systems architecture documents or drawings. Generally, check any available logs on network, computer and control equipment, and try to isolate the origin of the earliest problem logged. A packet sniffer can help identify what kind of network traffic is taking place. Also, use antivirus and/or antimalware to scan computer equipment.”
Heitman also recommends verifying the equipment’s physical integrity and looking for signs of tampering and rogue connections. In addition, engineers should determine if the behavior is malicious, unintentional or the result of an equipment malfunction. If there is an intruder, severing possible connections should eliminate the threat in the short term, but a configuration change may be needed for a long-term solution, adds Heitman.
“First, we contact IT and look through logs from network equipment and existing control system templates to access the Syslog Server,” says Jacob Haugen, director of communication, Portland Engineering Inc., a CSIA member. “From there, we check traffic against known/whitelisted IP addresses. We also talk with personnel to get an idea of what’s happening. For instance, if a particular instrument is behaving badly, we start there and work deductively. If we identify an intrusion and the nature of its purpose, we isolate and quarantine the system, moving through each constituent subsystem to ‘clean’ or ‘repair’ it.”
Intrusion detection tools are available; some are open source and/or relatively inexpensive to use. “For example, Snort [www.snort.org] is a network intrusion detection and prevention tool commonly used in industry,” says Emerson’s Braun. “OSSEC [http://ossec.github.io] is an open source host intrusion detection system, while OpenWIPS-NG [www.openwips-ng.org] is a wireless IPS/IDS.”
Establish a baseline of network activity
Detecting intruders is easier if you have a baseline of your control system traffic and can detect when network traffic moves away from that baseline. “Most companies in the food and beverage industry have no baseline for their control network and little or no monitoring capability,” states Byres. “In fact, the industry’s capabilities are so weak, most groups attacking control systems today appear to have decided not to spend much effort on being stealthy.” For example, the DragonFly (based on the Havex family of worms) attacks in 2014 were successful, at least in part, due to industry weaknesses.4 “Finding the Havex malware and its related traffic in a plant was easy once you looked,” recalls Byres. “Unfortunately, no one looked for almost six months.”
Thankfully, more sophisticated tools are becoming available to help industrial control engineers with the task of monitoring networks for abnormal traffic. The Sophia software solution from NexDefense, Inc. is one of these. “Sophia’s passive approach to fingerprinting a system identifies connected devices; monitors communications, connections and activities; and separates unusual events from a normal baseline,” says Doug Wylie, NexDefense vice president, product marketing and strategy. “It notifies users of possible trouble, so steps can be taken to better protect automation equipment and processes against cybersecurity threats throughout all phases of the system’s operation.”
Another system that can monitor ICS networks comes from Darktrace. “Darktrace’s Industrial Immune System uses advanced Bayesian mathematics to create a baseline for the normal behavior of users and devices within industrial control systems,” says Dave Palmer, director of technology. “This allows Darktrace to detect the slightest anomaly and notify the user before a crisis occurs. The Industrial Immune System can provide coverage with a singular view across all networks—both corporate and industrial.” The system also helps a business’s converged threat analyst teams evaluate what has been traditionally viewed as separate networks.
Protection for PLCs and automation controllers
Since control systems and their programming systems typically operate on a Windows-based OS, manufacturers should use some basic form of virus protection, recommends Portland Engineering’s Haugen. And, when router/firewall devices are used for communications between machinery and control points, all the protection measures should be enabled. Having all password-protected PLCs on a given system can help protect these devices, offers Haugen. However, this can be risky, because while the procedure can lock out intruders, there is no “backdoor.” So, if the PLCs’ passwords are lost, there isn’t any way to recover them.
For a higher level of protection, a processor may want to consider employing a professional control system integrator that uses vendor-licensed products, says Haugen. “A control system integrator will charge around $150 per hour, significantly less than the cost of rebuilding your system from the ground up after a polymorphic RAT [remote-access Trojan] infiltrates all your VFDs and turns them into scrap metal.”
According to Oliver Gruner, ICONICS director for cloud business development, there are definite advantages to hiring outside firms. “Our enterprise customers often hire outside firms to test installed applications for network vulnerabilities. This is certainly more costly, but very effective. And, it can uncover areas of potential vulnerability a processor may not look for, if it does its own audit.”
HMI software can also be built to shield PLCs from invaders. For instance, InduSoft applications can serve as gateway devices to help provide the layers of security necessary for defense in depth, says Richard Clark. “Specifically, PLCs should be updated with the latest firmware. Also, any concerns about hackers changing code should be addressed during the single-point failure analysis and risk mitigation phase of the project.”
Not only should a PLC’s firmware be updated, it should be digitally signed, says Rockwell’s Wilcox. Rockwell Automation products can verify the firmware they are asked to install/update is authentic Rockwell code and has not been modified or tampered with. “We also provide features that allow customers to encrypt programs to protect intellectual property or insure integrity,” adds Wilcox.
Network architecture and least-level access are the best methods to protect PLCs, says Frakes Engineering’s Heitman. “An OEM’s proprietary security solution [like controller-integrated security] is one form of prevention. A revision control system that tracks changes and can notify authorized users is another. However, these systems must be managed to be effective and come at a premium. And, whether you have systems like these or not, you should always maintain a backup set of configurations, off site if possible and updated at regular intervals, so you can have them restored, if they’re tampered with.”
In addition, a number of hardware options can be used to protect the lower layers of the network architecture, says Emerson’s Braun. For example, the military uses data diodes, optically isolated devices that allow traffic to flow in only one direction, e.g., from lower layers to higher layers of the network architecture. “Their use may complicate the network architecture. Plus, whenever a device needs a configuration change, the operator must be physically connected on the low side of the data diode,” explains Braun.
When it comes to restricting data flow, network firewalls are absolutely necessary. “[For example,] the DeltaV controller firewall prevents unwanted access to embedded nodes [DeltaV hardware connected to the DeltaV area control network],” says Alexandre Peixoto, Emerson Process Management DeltaV product marketing manager.
One controller that combines the best of several worlds is Bedrock’s automation system that incorporates security in several ways. (See “A secure, reliable automation controller from the silicon up,” FE, May 2015.) “Any bolt-on or plug-in device is inherently insecure,” says Bob Honor, CEO, Bedrock Automation. “We designed the Bedrock system for use in PLC applications as well as DCS applications. It is, in fact, the first control technology with native cybersecurity protection. It will not run code that has not been signed and authenticated to our controller root of trust.”
Beyond the PLC and automation controller
Several solutions can be used outside a PLC to protect it from attacks. “We recommend managed switches when dealing with automation devices,” says Process and Data Automation’s McCausland. “They provide the ability to use access control lists to explicitly configure who is trusted to be on the network and who is denied access.” Managed switches also allow the creation of virtual local area networks (VLANs) that can isolate the traffic inside from accessing devices outside of the network. Plus, they provide the ability for logging and port analysis, which traces the activity of the switch and usage of devices on the network.
According to McCausland, “Managed switches should be used at the panel level, directly on the plant floors. You may have multiple PLCs, VFDs and other types of I/O devices on the network. A managed switch connecting these devices offers flexibility, traffic monitoring and maintenance, and a heightened level of security. When the devices need access to a larger network, we work our way upstream, making sure the conduits of communication are secure, depending on the organization’s acceptable level of risk.”
In addition, unified threat management (UTM) devices and firewalls can be used at the perimeter of the control network, says Frakes Engineering’s Heitman. “These devices include routing/gateway functions with increased security.”
Today’s suppliers and system integrators have the equipment, knowledge and experience to protect your intellectual property—and keep your plant running safely and smoothly. So, with so much at risk, why wait to have a risk analysis done to find out where your control system and enterprise stand in terms of cybersecurity?
Working together for a common cause: Cybersecurity
People are the most critical asset to a successful cybersecurity program. Doug Wylie of NexDefense recounts a real-world situation the likes of which all too often still occur. In this case, there’s a happy ending.
I once worked with a large global food manufacturer with significant information technology (IT) and operational technology (OT) resources and talent. Both teams were well funded and well staffed, and each had significant expertise managing its respective equipment and overall systems. I was in a kick-off corporate cybersecurity meeting where members of each team were instructed to talk through how to build a company-wide security program to link IT and production. It became apparent almost no one in the room knew his or her functional counterpart on the other team, even though there were clear parallels in their duties. In fact, there had been past incidents where one team member made changes without realizing how they would affect the other team.
As the meeting progressed, both teams began to realize the amount of energy they were putting forth just to maintain organizational separation. Yet, neither team was willing to take on full responsibility. The team members began to recognize they shared common ground and the common goals of protecting their customers and company. They understood they already had the technical expertise to overcome many challenges, but to move forward, they would first have to address people challenges and develop a common view and language around risks to their manufacturing systems. Fortunately for them, their breakthrough moment came during this first meeting, and it set the stage for what has become a very respectable, cross-functional security team that’s a mixture of both IT and OT and now works together as one instead of working against each other.
References:
-
“Corporate Spies: The Pizza Plot,”
Adam L. Penenberg and Marc Barry, December 2000, The New York Times,
http://partners.nytimes.com/library/magazine/home/20001203mag-penenberg.html. - “Internal threat among biggest cyber security challenges, says former FBI investigator,” Warwick Ashford, ComputerWeekly.com, June 29, 2015.
-
Converged Plantwide Ethernet (CPwE) Design and Implementation Guide,
Cisco Systems and Rockwell Automation, html and PDF, September 9, 2011,
www.cisco.com/c/en/us/td/docs/ solutions/Verticals/CPwE/CPwE_DIG.html. -
“Dragonfly: Western Energy Companies Under Sabotage Threat,”
Symantec Official Blog, June 30, 2014, Symantec Security Response,
www.symantec.com/connect/blogs/dragonfly-western-energy-companies-under-sabotage-threat.
Resources:
“Seven Strategies to Defend ICSs,” ICS-CERT/Department of Homeland Security,
Download PDF. ICS-CERT (Industrial Control Systems Cyber Emergency Response Team), Department of Homeland Security, https://ics-cert.us-cert.gov/.
ISASecure (Certifying industrial control system devices and systems), www.isasecure.org/.
“Recommended Practice: Improving Industrial Control Systems Cybersecurity with Defense-In-Depth Strategies,” Homeland Security, Control Systems Security Program, National Cyber Security Division, PDF, October 2009
“Defense in Depth: An Impractical Strategy for a Cyber World,”
Prescott E. Small, SANS Institute InfoSec Reading Room, 2012.
For more information:
Gregory Wilcox, Rockwell Automation, 440-646-3434,
gswilcox@ra.rockwell.com, www.rockwellautomation.com
David McCarthy, Tricore, 262-886-3630,
dmmccarthy@tricore.com, www.tricore.com
Robert Albach, Cisco Systems, Inc., 800-553-6387,
ralbach@cisco.com, www.cisco.com
Scott McCausland, Process and Data Automation, 814-866-9600,
chris.rudd@processanddata.com, www.processanddata.com
Eric Braun, Emerson Process Management,
eric.s.braun@emerson.com, www.emersonprocess.com
Ben Jackman, Emerson Process Management, 512-832-3774,
ben.jackman@emerson.com, www.emersonprocess.com/deltav
Eric Byres, ICS Secure, 604-897-9980,
eric.byres@ics-secure.com
Doug Wylie, NexDefense, Inc., 404-600-1117,
doug.wylie@nexdefense.com, www.nexdefense.com
Dave Palmer, Darktrace, 415-802-9422,
info@darktrace.com, www.darktrace.com
Darin Heitman, Frakes Engineering, 317-577-3000,
darinh@frakes-eng.com, www.frakesengineering.com
Jacob S. Haugen, Portland Engineering Inc., 503-256-7718,
jhaugen@portlandengineers.com. www.portlandengineers.com
Oliver Gruner, ICONICS Inc., 508-543-8600,
oliver@iconics.com, www.iconics.com
Richard Clark, InduSoft-Wonderware/Schneider Electric, 512-349-0334,
richard.h.clark@schneider-electric.com, www.indusoft.com
Alexandre Peixoto, Emerson Process Management, 512-832-3774,
alexandre.peixoto@emerson.com, www.emersonprocess.com/deltav
Bob Honor, Bedrock Automation, 781-821-0280,
bob.honor@bedrockautomation.com, www.bedrockautomation.com
