Managing and Surviving Software Updates

So you put off upgrades and updates as long as possible because you can’t afford the downtime. But doing nothing can put you in a darker place—one where you could be down for much longer than you thought because a hacker got into your old and unprotected system or you can’t find a replacement PLC on eBay for that 30-year old device with an “intermittent CPU or memory.”

Updates may not support older equipment and software

Over the Memorial Day weekend, I built a new Intel-based Core i9 computer to replace my aging 7-year old Core i7. Opting for a new Windows 10 Professional license and Ubuntu Linux 20.04 OS (dual boot), I knew that if I wanted to use an aging film scanner, I’d have to install a Virtual Machine (VM) on Windows 10 or Ubuntu to run Windows XP to support the old device—maybe you’ve faced a similar situation with older industrial equipment. Unfortunately, my 20-year old audio recorder’s software was plain obsolete, but my new TASCAM is compatible with Windows 10.

I now face a “free Windows 11 update” in the future, which I hear occupies 60 gigabytes of disk space. Seriously, do I need all that clutter? At some point, your IT department will be faced with the same upgrade decision—while your OT group struggles to keep its controls operating without downtime. Building and maintaining hardware is one thing, but dealing with Windows forced updates and migrating application software can be a painstaking process—and full of pitfalls—but when successful, with rewards.

Managing Software Updates: Windows updates

Scheduling Windows Updates is not necessarily so easy as you can only put them off for so long. Admins have more control by making changes in the Windows Group Policy Editor or the system registry. Image courtesy of: Screen dump, Wayne Labs (Click on image to enlarge.)

One way around the Windows forced-updates issue is to choose industrial applications that also are available on Linux, for example, Inductive Automation’s Ignition. Though most Linux distributions are available out of the box with automatic updates turned on (for example, I use Ubuntu 18.04 Server and 18.04/20.04 workstations), they’re so configurable that the user controls the OS—not the other way around.

I asked automation hardware suppliers, process control software folks and system integrators for some advice on dealing with Windows, control application software and device firmware updates.

Dealing—or not dealing—with Windows updates

In dealing with Windows updates, the experts I interviewed had varying opinions, but one thing they agreed on is to approach upgrading Windows-based workstations on the plant floor with caution.

Staying on top of Windows system updates is critical to security and performance, but many organizations have to be careful about update timing to avoid production interruptions, says Sesh Natarajan, Emerson DeltaV product director. Organizations need the flexibility to be able to decide when and what to update to minimize the impact to operations.

“With system updates, the best offense is a good defense. If automation leaders are strategic about how they handle operating system updates and perform due diligence to test a wide variety of installations, they can mitigate many of the risks of an update breaking an application or disrupting production,” says Natarajan.

ADISRA is a Windows-based HMI/SCADA package especially designed for machine builders and OEMs. According to Bruno Armond Crepaldi, chief technology officer at ADISRA, major Windows updates should be kept off production systems until the updates are tested for stability and compatibility issues. “Many times, we recommend that customers stay one [Windows] release behind so their production system remains stable, and early issues with a new released update can be identified and resolved before being placed on production systems. This advice is given with the assumption that the update is not a major security release and that the production machines are adequately secured from outside intrusions.” Crepaldi advises turning off auto-updates for production machines, or at least delaying them as long as possible.

Chris Schulze, VP sales at CODESYS Corporation, a Control System Integrators Association (CSIA) member, is adamant about shutting down Windows semi-annual updates. “No semi-annual or annual changes.” Only if major hardware (major machine system) gets changed or updated. It should be an IT and OT responsibility. The update decision should come from the OT alone. Schulze thinks Windows shouldn’t even be a plant floor consideration: “Please note, the number one OS for PC-based automation is Linux, and not Windows anymore.”

Some automation suppliers have gone the non-Windows route for their devices and controllers. “In many of its data acquisition units, controllers, PLCs and other products, Yokogawa has opted NOT to use the Windows operating systems due to the frequent upgrades and security patches required over time,” says Gerald Hardesty, product marketing manager, Yokogawa Corporation of America, Industrial Automation Products. Instead, Yokogawa has opted to use an alternative real-time embedded operating system to avoid these security issues and threats. Yokogawa products are less susceptible to the vulnerabilities that Windows systems are exposed to, and thus the routine security upgrades that Windows users undergo are required far less often, or not at all.

Blanket policies, automatic updates can result in big losses

Compliance and standard software updates are essential—they keep IT and OT functioning at peak levels and maintain the best cybersecurity standards, says Tony Baker, chief product safety and security officer at Rockwell Automation. However, blanket policies that force restarts at certain times regardless of the OT context should be avoided. There are multiple reasons why blanket policies and rigid update times are problematic. For example:

Rigidity is not a fit for OT. Every OT application functions differently and should be treated on a case-by-case basis. If, as an example, a brewery operation had to force restart, that would throw off the careful timing of the brewing process, ruining the batch process—hence a major financial loss.
Blanket policies could limit the ability of the OT team to consult a manufacturer on the best process for their control system. Vendors and manufacturers are experts in OT hardware and software, so allowing their voice to be heard in the patch rollout process will only help in keeping OT running smoothly.
What works for IT may not work for OT. In the OT environment, an all-at-once update could create a situation where all the control room’s screens go dark at once—severely limiting visibility into OT. The IT team should not make decisions for the OT team. Instead, the IT and OT teams should work together on a situational basis to determine software updates and rollouts.

Windows updates, no matter how big or small, should not generally be “automatic” on most control and monitoring systems, such as process, packaging, utilities, and building automation, says Dan Malyszko, director of Denver operations at Malisko Engineering, CSIA Certified Member. Contemporary control and monitoring systems typically run application software that can highly rely on various functionality and operational behavior of Windows.

Software manufacturers, such as Rockwell, Schneider and AVEVA, have rigorous testing protocols to test new Microsoft hot fixes and patches to ensure compatibility with their industrial software suites, says Malyszko. “Bad things can happen”—such as critical systems or equipment shutting down and/or negatively affecting safety and product integrity—should an update be deployed without prior verification from the software manufacturer.

In general, Windows updates and Windows Server patching should be an IT/OT shared responsibility. And that means IT must have visibility to those machines and be willing to not have automatic updates turned on. This also means that IT needs to stay up on the Windows patch qualifications websites, such as Rockwell.

Oracle’s VM VirtualBox is available for non-commercial and commercial applications and allows the running of one operating system (shown here, Windows XP) inside another OS—in this case Windows 10. The virtual machine (VM) allows Windows XP to run older applications—such as an old film scanner or an I/O system—while enjoying the protection of the host operating system, Windows 10. VirtualBox is available for Mac OS, Windows, Linux and Oracle Solaris. Image courtesy of: Screen dump, Wayne Labs (Click on image to enlarge.)

Scheduling application and Windows updates

Interestingly enough, application providers with cloud-based systems can make their own upgrades easier on users because often all that is required to use them is a web browser. Nevertheless, IFS is very sensitive to updates in regulated industries. IFS provides integrated HACCP and quality control, advanced demand planning and forecasting, supply chain management systems and much more. “IFS Cloud marks the start of a new, twice-yearly feature update cadence for an ‘evergreen’ customer experience,” says Antony Bourne, IFS senior vice president - industries.

“Historically we have made large core releases every two to three years, with quarterly updates containing a combination of fixes and new features,” says Bourne. “With IFS Cloud we now have twice-yearly releases of new functionality, each supported through monthly service updates that only contain fixes.”

In many cases, processors operating under particularly rigid regulatory requirements (e.g. FDA), have requested flexibility regarding the timing of update adoption in their production environment. IFS has chosen to offer the flexibility required for these customers, (via a defined time window where IFS provides access to the update), to determine when the time is right for them to push a new version to their environments. Having this flexibility is essential where any change to IT systems is likely to trigger the requirement for a new third-party audit before rolling it out into production.

ADISRA SmartView provides modern visualization and deployment options for HMI/SCADA functionality. Image courtesy of: ADISRA (Click on image to enlarge.)

Back at the machine/plant level, Allpax, maker of retorts and other production equipment, is careful with Windows updates. “Updates to the PCs (both servers and clients) in our system are scheduled during production downtime windows,” says Jonathan Watkins, VP of technology. Tests are performed to ensure that the updates do not adversely affect the system before production is resumed. These updates are coordinated with the local group that manages IT system for the customer.

Automated Systems Group, a CSIA member, stresses the importance of keeping automation up and running. “In our experience at AMT, any time you update Windows, there is a chance that other installed programs may not continue to work,” says Terry Meister, controls engineering manager. Some may encounter “bugs” when the Windows software is updated. “Also, we have found that the manufacturers of industrial hardware equipment lag behind the Windows updates, making sure their software will work well with the updates before releasing their own updates. For this reason, it is important that the admin and PLC programmers/maintenance discuss when the appropriate time is to update. Also, making sure that enough downtime is scheduled with production—in case issues arise—can save all a lot of heartache,” says Meister.

Typical system vendors for industrial applications will evaluate updates and roll out changes as appropriate, says Steve Pflantz, P.E./P. Eng., CRB associate. “Make sure you understand if it is ever advised to do any updates to an industrial system without them reviewing the update.” Industrial applications are a more substantial application than most, and updates of any kind to the operating system need to be verified to not cause a problem. This is the fundamental reason to manage a system according to the vendor’s guidelines.

While Travis Cox, co-director of sales engineering at Inductive Automation says that major updates should not be put on hold, it’s extremely important to stay up to date and avoid OS obsolescence. The problem is, however, that most OT applications rely on specific OS versions and likely won’t run on the newest version. Often the upgrade path is difficult and expensive, leading manufacturers to put the major updates on hold and putting themselves at risk. Software vendors need to take this into consideration and provide simple upgrade paths.

Replace ancient hardware, update firmware

When you find yourself in a situation where you have equipment dating back 20-30 years, there are a couple of things to note, says Creager. Because they are generally less connected (technology has advanced incredibly in the last three decades), your risk of a cyberattack may in general be lower, but the risk of a complete shutdown due to aging parts is incredibly high.

With parts that old, generally replacements are only found on eBay and risking your entire operation to whether or not eBay can find a part is generally not a wise business decision. The older equipment was probably less connected when installed but could be vulnerable if any new connections were made to them since so the cyber risk would still be high.

Jason Anson, automation manager at Interstates, a CSIA certified member provides some basic pointers on hardware updates:

Avoiding upgrades can lead to obsolete hardware, which can increase downtime and decrease productivity.
As software continues to be updated, the hardware will eventually need to match the software’s sophistication.
Scheduling downtime is a lot more cost effective than waiting for something to break.
Not every software update requires new hardware. VMs—as already noted—can allow older software to keep functioning.

IT and OT must work as a team

Communication between IT and OT professionals is key, especially when it comes to updates, says Keith Mandachit, P.E., engineering manager at Huffman Engineering, a Certified CSIA member. From an OT perspective, all updates should really be delayed until there has been an opportunity to check with the manufacturer and have it cleared to be installed. In an ideal world, organizations would have a separate test environment that wouldn’t potentially disrupt the entire production line. This is one reason it is vital to bring in a control system integrator at the beginning of the project so these discussions can be held prior to design, updates or upgrades.

Managing software updates: Bosch Rexroth ctrlX Portal

More savings in maintenance and servicing of the device software are made possible via the cloud by the ctrlX Device Portal powered by NEXEED. Image courtesy of: Bosch-Rexroth (Click on image to enlarge.)

OT and IT should work together on a patching strategy, says Cox. OT can help determine when to perform the updates or deploy strategies to allow updates without disrupting operations, such as redundancy. IT can provide OT sandbox environments to test out the latest update to understand whether they will be affected. Developing a procedure and strategy is critical and allows the organization to stay ahead of the updates. It also buys time to work with software vendors when issues have been identified. “You don’t want to find out about issues after it’s too late, and be forced to stay on older versions because of incompatibilities,” he adds.

This seems like a great opportunity to introduce collaboration between OT and IT teams as a Windows HMI is usually a shared asset among the key stakeholders, says Luis Narvaez, Siemens product marketing manager, basic automation & industrial security. “While it is important to ensure that your Microsoft/Windows products are always up-to-date with all of the latest security/functional patches, it is also equally important to verify with the vendor of whatever industrial applications whether those updates will be compatible with their software in order to avoid potential downtime and thus security vulnerabilities.” Siemens has a web page where users can verify compatibility of their software products with tested Microsoft or other third-party products via www.siemens.com/kompatool.

Instrumentation/hardware manufacturers make upgrades easier

Most instrumentation companies support older equipment through updates and give their users reasonable choices. “Endress+Hauser identifies software revisions for our instruments so our customers can choose whether to buy current or older versions,” says Ola Wesstrom, industry marketing manager - food & beverage. This is very important for validated systems, like within the life-sciences industry. “We also make it easy to use our FieldXpert SMT50/70 by automatically providing a device driver library, which is updated via cloud services. This eliminates common problems of not being able to communicate with instruments because of incorrect or outdated device drivers.”

Control hardware vendors have gone all out to help their users stay up to date by using the latest web-based tools. Bosch Rexroth - Electric Drives and Controls provides a full range of machine controls and software to automate food and packaging machinery and processes, says Allen Tubbs, product manager. “Our newest platform, ctrlX AUTOMATION, integrates all these products holistically, into a blend of components and services to address design, programming, commissioning and troubleshooting needs.”

One part of ctrlX AUTOMATION, the Device Portal, is designed specifically to help customers manage their software and hardware remotely through a cloud-based software management tool. The Device Portal tracks the last known connection time and maintains a digital twin of the control so that the state of the controller is always known, regardless of connection status.

When it comes to aging equipment, PLC vendors like Rockwell can help processors identify risks and create a replacement plan that works for them, says Baker. Besides that, Rockwell provides timely information about firmware updates as well as data to help back up the importance of updates plus information on which industrial controllers and PLCs are still receiving firmware updates and projected end dates for these updates. A PLC vendor can also help users with free tools to see what firmware versions are installed on their control devices and which need updates.

Siemens has a variety of means of communicating to users about their automation products, says Narvaez. Siemens Industry Online Support (SIOS) site allows users to create custom filters and setup email notifications for products that are installed on their site - free of charge. Users can get notified of product lifecycle announcements, updates to documentation, firmware updates or setup custom filters, just to name a few examples.

In addition to product notifications, Siemens also encourages users to subscribe to security advisories via its ProductCERT webpage. Siemens ProductCERT is comprised of a dedicated team of seasoned security experts that manages the receipt, investigation, internal coordination, and public reporting of security issues related to Siemens products, solutions, or services.

Managing software updates: Yokogawa GX GP 07

A variety of devices, each not dependent on frequent software updates, are available from Yokogawa for securely storing data. Source: Yokogawa (Click on image to enlarge.)

Let system integrators handle updates

We’ve already seen that application developers work behind the scenes to keep their software up to date so Windows updates don’t break a control system. “We develop and test all of our systems with current OS and software with current updates (patches),” says Jerry Leuthold, senior project manager, Bachelor Controls Inc, a CSIA Certified Member. “Then we install the system at the customer site and recommend that it is secured and has no access to the internet. Only tested OS and software patches would then be applied to the system manually.”

For those processors who have broken their system by downloading a Windows update, Leuthold says VMs and good backups can get a manufacturer back in operation to a state before a system was broken.

“This is a hard lesson,” says Meister. “If an end user is looking to update software on a machine connected to a PC or PLC, they should contact the integrator before proceeding. Before they shut down for the update, we can investigate and ensure what is needed to get them back up and running.”

Meister also suggests a good reason for using a VM. “We will use VMWare with older versions of Windows as necessary. This allows us to keep our main computer system up to date for security reasons and have a VM for PLC programs as needed.”

OS and application updates can break systems as Huffman’s Sean Creager, senior electrical engineer, describes one situation where an update crashed an application. “At the time we just had to roll back and uninstall the update until a Hot Fix was provided by the manufacturer. In very specific situations we have actually set up the system to turn off Windows updates because of the potential consequences of interference.”

Control system updates

In this article we’ve looked at dealing with Windows system updates. In another article, “System integrators and hardware suppliers look at the mechanics of controls updates,” I ask system integrators and automation suppliers how they work with users to deliver automation updates safely and efficiently. We also look at the importance of backups in updating automation software and equipment.

For more information: