In food and beverage operations, downtime to install all-new equipment can add significant cost to the project overall. This may tip the scale in favor of retrofit in some situations. Source: Bosch Rexroth Corporation.
Legacy automation systems will soon require replacement
If your plant has a legacy control system, it might be running on borrowed time.
You’ll soon be celebrating your control system’s 30-year anniversary, and it’s been, for the most part, a smooth ride. There might have been a situation when dozens of alarms showed up on the shift operator’s supervisory HMI—and you had to call in your maintenance and engineering staff at 3 a.m. only to find a defective sensor, stuck valve and an overheated motor.
Or once, when trying to balance year-end equations, missing ingredients and missing product couldn’t be accounted for, and the books showed many more dollars spent on ingredients that couldn’t be recouped in product. Quality assurance kept finding batches were a little off, but your control system didn’t detect any anomalies.
Article Index:
- Aging systems: Everybody’s headache and responsibility
- Role of automation suppliers
- Upgrade, gain transparency, connect with maintenance
- Old is not new again
- Sidebar: Security: Everyone’s responsibility
- Sidebar: Legacy systems: Costs and issues
- Sidebar: Real-world examples of upgrading
Since it’s a proprietary system, it has kept intruders out and prevented hacks, right? But because it lacks communication capabilities, unlike higher-level ERP and quality systems, this might explain why you can’t account for slightly out-of-spec products and missing ingredients. The outdated system is incapable of running some advanced control algorithms (like model predictive control) that might have caught ingredient over-usage or too dry a product.
While you might wait until a small motor burns out before you replace it, do you want to take a similar chance with a control system failure? After all, planning, installing and running trials on a new system are a little more time consuming than replacing a motor.
Aging systems: Everybody’s headache and responsibility
Keeping control systems from growing old before their time has many challenges. Oftentimes, it’ll come down to underlying operating systems. Windows XP is no longer supported, and neither is Windows Server 2003, both of which were susceptible to the recent cyberattack called WannaCry.
Bühler Aeroglide, a builder of high-end dryers, roasters, ovens and coolers, works hard to stay current. But what drives food processors to move ahead?
“Processing advantages due to availability and cost reduction in relevant [new] technology,” says Doug Beloskur, product manager automation. Improved process control can drive down energy use, scrap and/or downtime.
Also, updating a system can be spurred by a crisis. For example, Beloskur saw instances of obsolete infrastructure exposed during the WannaCry cyberattack that drove investments in updating systems. But even knowing the issues, some processors still don’t understand the risks involved with older systems.
“The two obvious key issues are lack of available replacement hardware for aging systems and security vulnerabilities for non-supported computer operating systems,” says Kevin Moss, group manager at McEnery Automation, a certified member of the Control System Integrators Association (CSIA).
He says these risks typically don’t convince small to mid-sized food and beverage clients to take action. Usually, these companies put off replacement until a capital project requires significant process changes that are not possible with existing systems.
“It specifically makes sense to upgrade when you have end-of-life components in your system … anything where you cannot go out and buy new parts off the shelf. That presents an inherent risk to your operation,” says Shawn Mott, president and founder, Digitronik Labs, a CSIA member. “We’ve seen customers experience week-long shutdowns because of a legacy component failing and not being able to procure a replacement—and so now, there’s a lot of lead time added on for redesign costs on the fly.”
Mott’s advice: Do a planned obsolescence study, examine existing equipment and have the systems integrator (SI) make recommendations on how best to mitigate risk.
The most logical reason for doing an upgrade/conversion is when a processor has considered its options and the unexpected downtime and costs associated with a not-so-simple conversion, says Steve Pflantz, senior instrumentation & controls engineer and president of the International Society of Automation (ISA).
“With a proper plant maintenance program, you really should never get to that point,” he says. “When a plant no longer has ready access to spare parts, it should have a near-term plan to replace the equipment. The lowest-risk way to do so is to plan the downtime versus unexpected downtime and loss of production.”
A lack of available spare parts—due to running older control systems past their supported lifecycle—puts companies’ production at risk.
“If a client is purchasing spare parts on eBay or any other second-hand source, they are risking their companies’ production on untrusted and unknown sources,” says Chris Wahl, project manager at Stone Technologies, a certified CSIA member.
Second, many older PLC programming applications simply are not supported on modern operating systems, which causes a conflict with the IT department, as it wants to stay current.
“From a software standpoint, we strongly encourage users to stay on a supported version,” says Alicia Millinger, GE Digital product marketing manager, automation software. “Even if a company is not ready to modernize, today’s security requirements make supported software versions imperative. This includes all HMI/SCADA-related software, as an example, as well as the OS.”
This is a basic, but often neglected, way to improve the security of a system. Modern software is built to be secure by design, adds Millinger.
Modern control systems take advantage of better network integration capabilities to provide more process data, alarming, reporting and interfacing to desktop applications. Modern programming systems allow faster development of projects, and increased control system memory allows for more system flexibility, says Wahl. Many older controllers have no memory left to accommodate new projects.
Last, but not least, many companies are finding a generation of maintenance technicians who supported the older system are retiring, and younger engineers are not familiar with the older technologies. For system integrators and engineering houses like CRB, staying current means keeping in touch with automation suppliers.
“We work with the major [automation] system vendors to know what their plans are for system migrations, phasing out of one model to the next,” says CRB’s Pflantz. “This allows us to advise our clients early on as to what they will have to deal with when implementing a project that will require changes or additions to an existing system.”
Role of automation suppliers
Automation suppliers recognize that processors want to use equipment for as long as they can, and they work with both end users and SIs to plan when upgrades are needed and how they can be managed.
“Many of our end-user customers have had systems that have been running successfully for years, and it can sometimes be hard to convince them to update their systems, because they don’t want to interrupt production for any length of time at all,” says Dave Cameron, Bosch Rexroth automation and electrification solutions director of sales.
The automation supplier, a partner member of the CSIA, keeps its parts available for at least 10 years after the parts have been announced as service items only, so often, it’s easier for the user to keep on ordering spare parts. But when these systems reach a critical point, processors must decide whether to retrofit or replace. In many cases, the cost of a new machine leads to a retrofit, says Cameron.
“Typically, we need to ask our users where they are on their journey with automation,” says Aaron Crews, director, global modernization solutions, Emerson Automation Solutions.
There is a cost-benefit ratio that a user needs to take a hard look at and consider. Is keeping the automation system healthy becoming more expensive? Typically, users will notice these situations: Replacement parts are expensive or non-existent. Legacy expertise is expensive and hard to find. Onsite support dwindles as individuals retire. Expenses get higher to support a system with decreasing value.
When food processors opt for replacing old systems with new, Emerson uses a method called Project Certainty, which is an approach for transforming capital project execution. It applies technologies and rethinks work processes to help eliminate cost, reduce complexity and better accommodate changes to maximize capital efficiency. Since a project affects an existing operation, risk reduction is most important. Emerson’s brownfield solutions include checklists to ensure there are no project surprises, software tools to reduce human error in the controls conversion, wiring solutions to assist changeover and reduce downtime, hot cutovers to eliminate downtime and integration solutions at the I/O, controller and HMI levels for phasing the projects and easing the transition for operators, says Crews.
Automation suppliers also work with machine builders that are faced with similar but differing upgrade issues.
“We are always updating products with great new capabilities, and it’s up to us to demonstrate to the machine builder that these new features add value for their end users,” says Bosch’s Cameron.
There’s also another design decision: If an OEM is making a series machine (where several models make up a series), switching to new hardware and software is more complicated than it is for a builder that specializes in custom equipment or small series machines.
Omron’s strategy is to design robust products that can survive the long-term rigors of production environments, says Laura Studwell, industry marketing manager. Many Omron controllers have been on the market, fully supported for more than 25 years. However, when any component is being discontinued, Omron takes a phased approach. Customers are given a one-year “last time buy” notice, yet the company will continue to repair and service that component for seven years.
Omron, a CSIA partner member, will work with machine builders to determine if legacy automation equipment is meeting the OEE specifications desired. If the equipment isn’t, e.g., because of an increase in unplanned stops or defects or increased downtime, then it’s time to consider newer controllers that can help the equipment achieve better OEE scores.
“As plants look to improve the reliability and efficiency of equipment, there is a need to integrate them with modern operations and asset management approaches—for example, we need production counts, machine state, fault codes, etc. to drive OEE metrics for a machine,” says Keith Chambers, director, operations and execution systems, Schneider Electric, software.
At brownfield sites, an extra effort is needed to determine if the existing controller can:
- Allow connection
- Provide the needed registers for integration
- Reliably update the registers to reflect the true state of the machine and its production counts.
If any of these conditions aren’t present, one option is to add a supervisory PLC or data concentrator to connect the existing automation, says Chambers.
Cloud computing and low-cost sensors are newer options for integrating controllers with other brownfield machines. These Internet-of-Things solutions can be ideal for manual or semi-manual operations that have limited local control or in situations where remote plant sites exist and have limited support staff.
Automation suppliers can offer their services to both end-user processors and SIs to gauge the appropriateness of system upgrades.
“Connected Services from Rockwell Automation can collect hardware and software data across multiple sites in weeks,” says John Riess, global marketing lead, integrated architecture. This offering gives assessments of current risks and opportunities, as well as costs of new projects.
Rockwell’s Allen-Bradley PLC-5, long a workhorse in many plants, may be getting a little long in the tooth. However, Reiss says migrating from the PLC-5 to a ControlLogix controller can be done in a matter of minutes, which limits the impact on current operations.
Automation suppliers don’t often come out willy-nilly with hardware upgrades for the sake of upgrading.
“From a hardware perspective, we deal with many issues that drive obsolescence,” says Emerson’s Crews.
For example, end-of-life and regulatory issues (e.g., RoHS, reduction of hazardous substances) with electronic components may often require a redesign of a controller. Modern systems use commercial off-the-shelf (COTS) technologies, and these technologies (e.g., SCSI, PCI, parallel ports) often disappear in 10 years—not to mention hardware architectures moving from 32 bits to 64 bits (including OS platforms) and changing network technologies.
Upgrade, gain transparency, connect with maintenance
The lifespan of an automation control system can be anywhere from 10 to 30 years—that’s a long time, says Allen Casteel, Stellar operations manager - automation & digital PSM. But older systems just don’t have the features that can save money in the long haul, such as energy savings. Modern refrigeration systems have advanced controls for efficiently sequencing compressors and controlling the head pressure at its optimum point.
“If your system is more than 10 years old, you’re probably missing out on some energy-saving benefits,” says Casteel.
The second point Casteel makes is communication improvements. There are great labor-saving benefits to be gained from upgrading a control system that can integrate all the equipment and sensors from the basement to the rooftop. Operators can see the entire plant from one screen and make better choices about control and maintenance. Call it transparency.
“Transparency is improving with the latest generation of automation products, which allows for easier failure identification and rectification,” says Bühler’s Beloskur.
If downtime is a major concern, the latest automation technology can begin to predict failure and allow for the strategic planning of maintenance to ensure the highest possible operational effectiveness.
“A secure remote access solution allows customers to access their production sites in order to troubleshoot and fix problems 24/7 from all over the globe,” says Ute Decker, marketing manager of machine building and automation manufacturing, Belden, a CSIA partner member.
Because of the increased network efficiencies and security available today, users can easily enable remote control and diagnostics of PLCs with minimal IT knowledge. This enables teams to fix problems more efficiently, increasing productive staff time while achieving less downtime.
Many new control systems are designed to integrate with maintenance systems, says Brian Courchesne, Schneider Electric global industry services marketing director. Sensors can monitor the health of components and leverage modern networks to prevent any issues from affecting production or even occurring in the first place. Now, direct real-time measurement of asset reliability and asset reliability risk is possible.
“We work with our customers to shift from reactive to predictive maintenance models by enabling the measurement of operational profitability factors in real time from the asset level,” says Courchesne. “By empowering the workforce with real-time operational profitability data, and process control and real-time reliability risk information, operators will be able to see the impact they and their adjustments are having on the process, and the profitability and reliability of their assets.”
It’s a combination of things that leads to less downtime and more output.
“With our new generation motor, you might get 30 percent more power,” says Bosch’s Cameron. “With new controls, you’ll get better compatibility across a wide range of systems, including IIoT/Industry 4.0 compatibility with IT systems. Connectivity like this can result in less downtime through predictive maintenance, improved operational flexibility and better quality control.”
Today’s fourth-generation HMI bridges the gap between operations and maintenance, says GE’s Millinger.
“When an out-of-spec event takes place in the HMI/SCADA, the automation system can trigger a work process to interface with the CMMS/EAM, secure a work order number, send specific instructions including GIS location to the right person, monitor and log corrective action and close out the work order.”
One part of maintenance hasn’t yet been mentioned, but it’s a critical part of any system—disaster recovery.
“Modern control systems can provide maintenance benefits through implementation of a robust disaster recovery,” says McEnery’s Moss.
Automated and secure backups of programs reduce the possibility of a catastrophic failure resulting from the loss of program revisions or the entire program. Also, backup of configuration files for devices, such as VFDs, specialty modules and networked instruments through automated backup systems/change management systems, allows for quick download to replacement devices on failure.
Speaking of software programs, users who don’t keep current on updates can create unique maintenance problems for themselves and the software developer.
“The main reason for upgrading from a legacy system to the latest software version is compatibility issues that arise,” says Val Dychok, software developer for CAT Squared. “A good example concerning our software platform would be a common situation where some of our customers have an early version of our listeners and mobile software that spans back four to six years. A listener handles incoming messages from handhelds and responds back with a status message and data to be displayed on the handheld.”
If a customer requests the latest mobile software release, it can’t easily be done because of the functionality gap between the listener and the handheld.
“It would require a more complex upgrade,” says Dychok. “It also creates a knowledge gap in the sense that customers don’t know about new functionality that’s been added since their last upgrade and are often blown away by the number of new features when they receive the latest mobile release.”
Old is not new again
Because many legacy automation systems were built as proprietary, standalone systems, upgrading them can be challenging.
“Different components did not communicate with each other,” says Stellar’s Casteel. “Today, people increasingly expect all controls to be integrated and to be controlled from one central location.”
So, why spend the time, trouble and money for an upgrade and new automation equipment? “Without a doubt, newer equipment and software are more reliable, robust and secure,” says CRB’s Pflantz.
Some key reasons are plug-and-play hardware, self-configuring hardware, cybersecurity hardware and software elements, and modular designs for easier replacement of key components, to name a few.
“Technology has advanced, and we are several generations into some systems that have incorporated features brought about to address real operational challenges or issues,” says Pflantz. “And yes, this is just a fancy way to say we learn from our mistakes. We are getting smarter, and so are our control systems, with self-configuration, diagnostics and redundancies that provide continuous operation with barely a blip in the system.”
Sidebar
Security: Everyone’s responsibility
Most of the connected hardware and software provided by vendors have security built into them at some level. It is wise to take advantage of this, but it’s also necessary to provide the proper training and access to maintenance, operations and management. It is important to realize that turning security “on” also requires investing in people and resources to manage security throughout the control system lifecycle. For example, there must be a way for maintenance and operations to perform their jobs at 2:00 a.m. when their password has expired. Security systems are ever changing as people, roles and processes change—and someone needs to own making those changes.
Additionally, implementing control system security can require some investment in revising existing network topologies, making sure there is proper segregation between business networks and control networks with firewalls or similar technology, all while allowing the necessary business data to flow down to the controls system and reporting information to flow back up unhindered through the DMZ. This adds another layer of engineering that needs to be considered during the conceptual phase of the project and reviewed with the necessary stakeholders in IT and the system integrator.
—Chris Wahl PE, Stone Technologies.
Sidebar
Legacy systems: Costs and issues
Associated with legacy systems is the higher labor cost to operate processes controlled by these systems. Older existing systems frequently cannot be modified to adapt to process changes due to a lack of working knowledge or no additional expansion capacity. The result can be processes being only partially automated, which requires operators to override the automation at specific points in the process or even to operate completely in manual mode. Either of these slows down production rates, increases labor costs and increases the amount of scrap and rework due to errors when the manual overrides are missed, or manual processes are not executed properly.
Legacy systems are more proprietary, less open and have very limited throughput, making it difficult or impossible to share data, which is increasingly important in improving performance and meeting FDA regulations, including the Food Safety Modernization Act. A key aspect of implementing a modern control system is early implementation of standard networks, such as EtherNet/IP, with standard connectivity and communication rates thousands of times faster than older networks, such as Remote I/O and Data Highway Plus.
Legacy systems can be very limited in terms of memory, I/O capacity and processing rates and frequently cannot support expansion. Capital projects to implement major process changes or plant expansions are typically the “straw that breaks the camel’s back” and provide the easiest justification for replacement of these systems and typically require significant funding. Virtual server technology can replace multiple physical servers, reducing the overall points of failure and hardware at the end of life. Maintenance costs for updates, recovery from crashes, backups and roll-backs to previous snapshots are reduced.
The cost of maintaining propriety software, especially for data collection, reporting and MES interactions, becomes exponentially difficult as systems age. The need to implement updates to remain compatible with operating systems updates, as well as adding new features, capacity and flexibility to remain competitive, become unrealistic and nearly impossible. Many large food and beverage companies are currently in the process of replacing systems developed in-house 15 or more years ago with off-the-shelf systems from automation vendors and their partners.
—Kevin Moss, McEnery Automation
Sidebar
Real-world examples of upgrading
Legacy systems don’t always need a complete overhaul. If you determine your production needs, both for now and in the future, and look at your budget, you might just need to upgrade the communications systems.
Digitronik Labs has a client that purchased an older Allen Bradley PLC from a competitor and, upon installation, wasn’t getting the production yields it wanted. While the OEM recommended a complete upgrade, Shawn Mott spent a couple of days evaluating the PLC and the machine’s inability to increase its cycle time by the mere 8 percent needed, which equates to five more units per minute.
Through strategic upgrades of communications components between subsystems and tweaking of hardware parameters, the cycle time was brought up to the desired speed—without throwing out all the old stuff, buying a new machine or spending a lot of money on a complete controls package upgrade, thus getting the performance with a lot less money.
McEnery Automation upgraded a beverage company’s pilot plant from a legacy system to increase the flexibility for frequent recipe and process changes. These changes had to be easily implemented on an automated system capable of providing repeatable production runs and improved capture of process data. Communications made this upgrade possible. New technology has made it feasible to monitor and collect additional data, such as torque and device temperatures, for motor controllers (VFDs, servos) and to generate email alarms to maintenance personnel when performance begins to degrade.
Sometimes, there’s nothing wrong with existing plant equipment, but improved communications and supervision can make operations better. New Belgium, one of the largest craft brewers in the US, needed to increase production using its existing legacy systems. By incorporating a new Schneider Electric MES on top of the existing infrastructure at its Colorado plant, New Belgium’s staff was able to view detailed real-time information on unscheduled downtime. As a result, the brewery downtime decreased by more than 50 percent, increasing OEE from 45 to 65 percent in just over two years. This improvement let the brewery expand production from 150,000 to 200,000 cases per week without capital investments.
John Wenzler, Omron corporate industry development manager, reports that Omron recently worked with one of North America’s largest disposable food packaging producers to upgrade a legacy control system. OEE on several production lines had declined to the point where the processor couldn’t meet the production demands of its customers, mostly due to bottlenecks causing unplanned downtime. Because the processor understood the severity of the impact almost as quickly as it manifested itself, the company was able to justify the investment. Upon completion of the project, new Omron controls, software and solutions increased throughput on each line by about 150 percent, largely due to the increased availability of equipment.
In another case, a Chinese dairy needed to upgrade its control system and faced the following challenges:
- Recover capacity from an existing line
- Need for increased flexibility of production assets
- Process variations causing rework and waste.
The dairy upgraded to a modern GE production and process data management system, which provided a combination of process control and process insight. The system included controllers, HMI/SCADA and Historian. The system pre-integrated control with the HMI/SCADA and Historian, easily exposing key event and process data for analysis. The dairy processor realized value on several fronts: The pre-built application components in the control and visualization layer allowed recipe generation to be simplified, so changes in products and processes take less time to deploy and test. With rich process data and analytics, the company has seen significant capacity and yield improvements. The dairy reports 15 percent improved OEE, 12 percent higher efficiencies, 17 percent less waste and 6 percent reduced rework due to predictive trends that provide process insight.
For more information:
Doug Beloskur, Bühler Aeroglide, 919-851-2000,
douglas.beloskur@buhlergroup.com, www.buhlergroup.com
Kevin Moss, McEnery Automation, 636-717-1400,
Kevin.moss@mceneryautomation.com, www.mceneryautomation.com
Steve Pflantz, CRB, 314-997-1515,
steve.pflantz@crbusa.com, www.crbusa.com
Shawn Mott, Digitronik Labs, 585-360-0043,
brian@digitroniklabs.com, www.digitroniklabs.com
Chris Wahl, Stone Technologies, 636-530-7240,
cwahl@stonetek.com, www.stonetek.com
Alicia Millinger, GE Digital, 855-968-7143,
www.ge.com/digital/contact, www.ge.com/digital
Dave Cameron, Bosch Rexroth Corp., 800-739-7684,
info@boschrexroth-us.com, www.boschrexroth-us.com
Laura Studwell, Omron, 800-556-6766,
laura.studwell@omron.com, www.omron247.com
Keith Chambers, Schneider Electric, 831-239-3559,
keith.chambers@schneider-electric.com, http://software.schneider-electric.com
John Riess, Rockwell Automation, 440-646-3434,
jariess@ra.rockwell.com, www.rockwellautomation.com
Aaron Crews, Emerson Automation Solutions, 800-833-8314,
aaron.crews@emerson.com, www.emerson.com/modernization
Allen Casteel, Stellar, 904-260-2900,
acasteel@stellar.net, www.stellar.net
Ute Decker, Belden,
ute.decker@belden.com, www.belden.com/marketsolutions/industrial/foodbeverage.cfm
Brian Courchesne, Schneider Electric, 905-333-2275,
brian.courchesne@schneider-electric.com, www.schneider-electric.com
Val Dychok, CAT Squared, 501-328-9178,
val.dychok@catsquared.com, www.catsquared.com
