System integration solutions for big and small projects

While most brand-new or greenfield food and beverage plants are tightly integrated from the sensor level to shop floor controls and on up to the ERP system, many older plants still contain “islands of automation.” But that doesn’t mean a greenfield plant hasn’t experienced problems that could have been avoided in the planning stages—and yes—plants with islands of automation can pose especially thorny issues when it’s time to upgrade. The choice of a system integrator that has experience in the food and beverage industry can go a long way in alleviating the pain associated with designing a new facility or upgrading an existing facility.

For example, when Gancia winery (founded in 1850 in Italy and maker of sweet sparkling wines to the tune of 25 million bottles per year) wanted to get better control of its sparkling process, it turned to Rockwell Automation, which was able to recommend Progecta as the system integrator (SI) of record. Progecta is one of only a few Italian companies to be named as a Rockwell Automation Recognized System Integrator, an SI with 25 years of expertise in the installation of control and automation solutions.

Gancia’s sparkling process makes up 80 percent of its production facility. The winery needed to upgrade to a high-tech solution to minimize errors and risk margins in both the testing and final stages of production. The company wanted full visibility and real-time control of the process, being able to make on-the-fly adjustments. Making it a little tougher was a 15-day window in which to install the system and get it up and running—before the next grape harvest.

The sparkling process is critical, as it involves a series of phases and steps in different autoclaves—all with strict control of essential parameters, such as temperature and pressure. Rockwell proposed a PlantPAx distributed control system (DCS) implemented on EtherNet/IP to change out an existing control system and wired architecture.

“The transition from a wired architecture to a distributed network allowed us to change the existing control system without replacing the main or the remote switchboards where the I/O modules have been installed,” says Nicola Nada, Progecta technical management. “Ethernet was used to connect to all single points, providing us with scalability; we can increase this by simply using additional I/O.”

A dedicated switchboard was implemented for power control, while the valve control systems are managed via a single mains and network cable. A large number of cables in the trays was reduced, making cleaning easier and freeing up space that might be needed in the future for additional cables for wine cellar expansion.

“The PlantPAx DCS meets our expectations perfectly,” says Andrea Cavalli, Gancia technical management. “First, it allowed us to meet our main goal: to adopt state-of-the-art technologies to achieve the best effectiveness in terms of streamlining and gaining improved visibility on the process of industrial sparkling—a strategic part of our business. Second, we have enhanced diagnostic capability for timely intervention, provided by a set of real-time alerts, a task previously delegated to the cellarman, who was the only person able to identify issues and carry out troubleshooting against irregularities observed.” The winery has the ability to monitor continuously any vital parameter in the process.

New projects can have issues

New projects would seem to be free of completion problems—if they’re planned appropriately. But sometimes, important details are overlooked, which can add days or weeks to the completion date and increase overall costs. A knowledgeable SI can prevent things like this from happening. Scott Young, senior consultant at Maverick Technologies, recalls a time when he was working at a new greenfield plant with a food and beverage design/build construction firm that had selected his company to do the process control programming for an ISA S88 batch control system.

“The firm had previously sent their process engineers to Europe to attempt to capture the existing process and control methodology, which was used in the end user’s facilities for building a manufacturing plant in the US,” he says. But the firm didn’t take the time to understand and document the process, which led to gaps in the process description and controls design.

“I recognized this early on and offered to collaborate with them to write process description documents prior to the development phase,” he explains. “I ended up working with their process design engineers and end users during the development phase to ensure the process was correct and took advantage of the ISA S88 batch model.”

This speaks to the importance of the end user having an SI firm well versed in process controls working cooperatively with a process design/build construction management firm during the discovery and design phase to create process description documents. These documents can be approved by the end user before the development starts. However, maintaining project schedules can be another problem.

“In one instance, the construction firm believed they were days away from completion on commissioning a specific piece of process equipment,” Young says. “On review of the installation, I compiled a list of unfinished items, which ended up taking at least two weeks to finish.”

These items consisted of process, electrical and utilities installations, adds Young, but his prior knowledge allowed him to see the entire picture of what was needed to commission this system.

Computers and IT issues can also be stumbling blocks in the successful finish of an automation project. For example, Young recounts another challenge in migrating applications from development servers to the end user’s virtual server architecture. At the beginning of a project, Young’s group had been promised a virtual server on which it could develop and install software. However, this didn’t happen—which required loading applications and software on virtual servers after the FAT (factory acceptance testing) was complete. This added two weeks to the schedule.

“I suggest end users request their IT departments take the appropriate steps to provide corporate-compliant servers to allow this environment to be used for development, FAT and commissioning the facility,” says Young.

“IT has a bigger and bigger role and, in one extreme case, actually led the equipment purchase decision, [though] it turned out to be a poor choice,” says Craig Souser, president of JLS Automation, an SI specializing in integrating packaging lines with upstream and downstream equipment. “But the bottom line is that IT has more influence and impact than ever. We provide remote access on all of our equipment, and it’s frustrating when IT departments won’t allow access to our equipment, no matter how strong our case is for avoiding penetration on their firewall.”

Speaking of firewalls, there is the security issue.

“One of the major issues we’re seeing is industrial control systems [ICS] cybersecurity,” says Glenn Restivo, director at Thermo Systems LLC, a certified member of the Control System Integrators Association. Thermo Systems offers a complete range of cybersecurity solutions, following ISA99 and IEC62433 guidelines—in addition to providing design/build control systems engineering and specializing in managing and delivering turnkey projects for EPC (engineering, procurement, construction)/A&E partners, as well as customized direct-to-owner solutions.

Sidebar: CSIA and CSIA certification: What is CSIA?

The Control System Integrators Association is a global trade organization that seeks to advance the industry of control system integration. CSIA has more than 500 member companies in 27 different countries and promotes best practices for control system integrators. The “CSIA Best Practices and Benchmarks” manual sets the standards for successful management of a system integration business.

What is CSIA certification?

CSIA certification demonstrates the commitment of a control system integrator to meet the highest standards for business and management. Here are some basic facts about CSIA certification:

CSIA certification requires integrator companies to undergo an intense third-party audit and follow strict performance standards. Members must meet or exceed the 79 criteria taken from CSIA’s “Best Practices and Benchmarks” manual to become certified. With CSIA certification, an integrator follows CSIA best practices in such areas as general management, financial management, project management, system development lifecycle and quality assurance. An audit is required every three years to maintain CSIA certification.
Certification sets CSIA members apart from other control system integrators and drives the industry. Whether a large corporation or a small independently owned business, clients benefit from the guidance of a CSIA-certified integrator. The group’s certified integrator members offer insight gained from experience and CSIA’s rigorous certification process.
Currently, CSIA has 94 member locations around the globe that are certified.

Two years ago, CSIA launched the Industrial Automation Exchange, with the intention of helping industrial automation end user clients find system integrators and their suppliers. The exchange’s database includes more than 1,200 integrators and 200 supplier partners. Today, the CSIA exchange features its own blog, largely authored by system integrator members and member partners. It has also incorporated a new library-resources section.

Why upgrade existing systems?

In one word: information. Processors want uptime and production data to let them maximize their OEE, says Souser. Hence, there are many processes in a plant that need to be integrated to the systems around them and higher-level plant control systems. One example is CIP.

Today, programming around CIP is a very common system integration project, says Young. And this is driven by a number of factors:

Maximizing production time. In many cases, CIP is the bottleneck to increasing production.
Minimizing utility, chemical and water usage.
Visualization. New systems provide electronic CIP reporting and verification that cleaning was either successful, or there were exceptions.
Transparency. New systems allow end users to understand, take ownership and make changes to the CIP system cleaning circuits.

“One of the common issues we come across is older plants having legacy PLC, DCS and HMI systems—industrial control systems that are no longer supported by their vendors and/or are being phased out (for instance, AB PLC5 or Siemens S5 PLC),” says Restivo. Food and beverage manufacturers obviously put a huge emphasis on safety, quality and plant uptime, so there’s a major need to ensure their facilities have technology that is supportable, and spare parts are available, says Restivo.

Also on the list of no-longer supportable equipment are server computers where the HMI application, tag database and historian exist on operating systems that are no longer supported, says Young. End users want reporting around their process and do not want to build on top of old technology.

Processors want to replace controls around standalone process equipment, upgrade instrumentation and integrate systems into the plant-wide control system for improved process control, reporting and archiving of historical data, Young adds. Examples include controls upgrades for inline product blending applications where controlling the systems to specifications (e.g., ±0.2 percent butter fat or ±0.5 percent Brix) brings considerable savings, resulting in the system paying for itself within the first year. This is especially true for end users that are running batch processes without inline blending.

One example of this, says Young, was a recent controls and instrumentation upgrade of an inline milk standardization system where he was able to integrate it into the existing plant-wide controls system, giving the end user:

Maximized process accuracy and precision to product regulatory specifications
Minimized product loss or giveaway, increasing yield and product to be sold
Visualization, which allowed operators to control from the main plant-wide system
System transparency, which allowed the maintenance staff to view system operation and troubleshoot effectively.

What impedes or eases upgrades

For SIs installing new lines, communication with existing systems is often not consistent or available, says JLS Automation’s Souser. Proprietary controls make it challenging. In addition, equipment obsolescence—and in particular, Windows migration over time—can kill the ability to connect to older plant-level PCs.

“The IIoT [Industrial Internet of Things] should help in this area, but it’s a big buzz word at the moment,” says Souser.

“Most legacy PLC and DCS systems are typically reliable; therefore, food and beverage customers need to prioritize where their CapEx funds are spent,” says Restivo.

When it comes to legacy systems, there are tools that allow some automatic, semi-automatic and manual conversion of existing code to new code for new equipment. These new tools and technologies offer full documentation of the conversion process and the “converted code,” allowing ease of maintaining and troubleshooting a new ICS. In addition, says Restivo, phased approaches can be used to replace the hardware and software first, keeping the existing I/O in place. Future upgrades can then address I/O and other system components like HMI/SCADA software.

One issue that processors face is that they don’t schedule sufficient production downtime when planning an upgrade to a new system, and this causes unsuspected problems, according to Young. Project startups that extend past their original deadline create additional CapEx and OpEx expenses.

“Many times, end users focus on the cost of the project itself and may not realize that additional production downtime due to poor project execution in many cases costs more than the project itself,” he says.

To help projects have a successful outcome, Young puts new control systems through extensive FATs with the end user. This ensures that anything that can be done prior to the cutover is done.

“I also have a comprehensive cutover, commissioning and SAT [site acceptance testing] plan methodology. This allows me to meet and often beat the deadline, allowing them [end users] to run production earlier than was originally expected.”

As previously discussed, reusing existing code in new controllers in an upgrade can save time in programming. Similarly, there are ways to keep existing hardware in place while doing an upgrade, allowing for other hardware upgrades later on. Many controls manufacturers now have hardware adapters that allow quick replacement of control system hardware without rewiring I/O cards, leaving field wiring intact and not introducing wiring issues that are possible when so many wires have to be re-terminated, says Young. This reduces cutover time to a few hours, allowing testing of the control system within the same day.

Another way of speeding up an upgrade is implementing instrumentation and control devices that use an industrial control network protocol like EtherNet/IP or AS-I bus, which can reduce the amount of field wiring required. These networks can be verified for correct operation prior to startup and commissioning. Since these networks require less wiring, integrators can clean up spaghetti wiring and save money for their clients.

“Before installing additional sensors and controls, the first step we would recommend is a site survey or a front end engineering design [FEED] study where a document is generated that addresses networking health/infrastructure,” adds Restivo.

Some issues—like too many physical connection points, IP conflicts, unauthorized devices added, incorrect VLAN assignment, slow performance, DNS problems, etc.—are typically seen when future growth is not planned properly. These items are then prioritized, budgeted if required and then presented back to the food processor, along with recommended next steps. Based on these findings, it is determined if fiber, wireless, new routers or other state-of-the-art technologies can address the discovered problems, says Restivo.

Batch and other needs

Since many or most food processing applications are batch oriented, system integrators get requests for improving batch systems.

“Yes, we have implemented automated batch solutions,” says Young. “First, master recipes are stored on the end user’s SQL servers in a recipe editor. We have also developed a recipe editor, which allows raw ingredients to be entered in with the ingredients and product-specific attributes. This allows for product batches to be calculated and created based on end-product specifications and also using a least-cost formulation.”

Metered ingredients are batched into a vessel. Hand-added ingredients are part of this system as well, and lot numbers are added for lot traceability. Batch reports pull information directly out of the control system to provide actual production amount, usage of raw materials and inventory, adds Young.

Beyond batch systems, Restivo sees a resurgence in manufacturing execution systems (MESs)/manufacturing operations management (MOM) implementations.

“These address manual operations, provide operator guidance via standard operating procedures [SOPs] and track operator actions and allow operator data input to capture batch information. After a batch run, the system can generate an electronic batch report,” says Restivo.

Cybersecurity should be built into modern systems

For machine builders and integrators, the Internet provides an invaluable function—being able to troubleshoot remotely. To do it safely, SIs will implement routers designed specifically to keep the bad guys out of industrial networks.

“If our equipment is connected to the Internet, we use an eWON router and rely on their technology to deal with cyber threats,” says JLS Automation’s Souser. eWON’s Cosy router is firewall friendly, provides full audit trail and traceability, has a secure VPN and has integrated cellular and WiFi connectivity.

“Security updates are recommended for old and new controllers and their respective networks,” says Thermo Systems’ Restivo. “Think of it like your home computer, where updates and security patches are automatically scheduled on a weekly basis.”

Updating old controllers does increase performance and eases maintainability, but the network is often the weak link when it comes to cybersecurity. Any controller that is not connected to the Internet is safe from external intrusions, however, internal intrusions like infected flash drives or configuration software can cause major problems, if not addressed.

“Primarily, I recommend updating old controllers to increase performance and maintainability,” says Maverick’s Young.

Cybersecurity concerns can also play into the design decision and should be part of the individual user’s cybersecurity risk assessment. Older controllers with Ethernet communications were not routable, and protocols were specific to that manufacturer. Therefore, one would not expect a cyber intruder to happen across that specific controller and have the skill to exploit it remotely, as it would not support, for example, the EtherNet/IP protocol. Newer controllers that support the EtherNet/IP protocol also have added cybersecurity protection built into them.

“Updating cybersecurity with older controllers is a balance and can be accomplished by protecting the controller through the use of a robust firewall and network monitoring,” says Young. However, cybersecurity isn’t everything. “Physical security for controllers is one simple step end users can take by having the controllers in a locked room and placing the controllers in the run mode and removing the key.”