New Open Source and AI Tools Make Keeping Track of Machine Programming Easier
Change management, aka version control, can be a challenge when all you want to do is get the packaging machine back up and running after a line change.
The Siemens TIA Portal provides a powerful version control interface (VCI) to integrate version control capabilities directly into the engineering framework. This allows for better management of project files, tracking of modifications, and enhanced team collaboration. Photo courtesy of Siemens
If you’re the only controls engineer in your facility, you probably know where to find the programming code and files for any of the PLCs in your lines, along with dated documentation regarding any changes made to programmable hardware.
However, in years gone by and with larger staffs, version control (VC) for programming PLC-based equipment and SCADA systems became a maintenance headache, especially when more than one engineer had access to programming the equipment.
Today there are many solutions to change management and VC, and major PLC vendors such as Rockwell and Siemens have their own VC systems. It wasn’t always like this. An early independent solution provider, MDT (founded in the 1990s), sold a PLC VC program for major brand controllers that had no built-in VC, but today Git technology based on open-source GitHub provides a foundation for modern VC systems like Copia Automation’s cloud-based DevOps platform.
The Way it Was
Early on, PLC/HMI development in the industrial operational technology (OT) environment was not integrated with the business information technology (IT) environment, says Steve Ward, Emerson Discrete Automation director, application specialists, EMEA. “Version control and backups were familiar in the IT world, but not a primary focus in the OT domain. Therefore, backups were not a priority, and it was up to each PLC/HMI engineer or the maintenance team to make sure a backup was kept, and versions were handled, which could be burdensome.”
And in the rush to get PLC and SCADA projects finished, long-term VC simply wasn’t a maintenance consideration, says Ron Schiedler, principal engineer at Concept Systems, Inc., a CSIA (Control System Integrators Association) certified member. Once a project was completed, the focus shifted to operations, with little thought given to future upgrades or code modifications. As a result, there were no structured processes for managing code versions, leading to inconsistent documentation and challenges when updates or troubleshooting were required. Without standardized practices, engineers were left to manage files manually, increasing the risk of errors and version mismatches.
“The focus was on getting the systems to work reliably so production lines could run smoothly,” says Shahzad Khan, global system consultant, Yokogawa Life Business Unit. “This often meant when engineers made quick fixes to resolve issues, they didn’t always document the changes. Imagine trying to fix a problem in a hurry and not having time to write down what you did? This led to overwritten programs and a lot of confusion later. It was like trying to keep track of changes with sticky notes; things got messy fast!”
Complicating matters were competing vendors, all with different ideas on how development and runtime platforms should work, proprietary data formats that weren’t easily parsed by external tools and a general reluctance to change, says Brandon Stiffler, U.S. software product manager at Beckhoff USA. “There [were] also key differences in engineering workflows. For instance, automation engineers typically have unfettered access to the active runtime code via downloads, online changes, etc. This probably made proper VC and backups feel like less of a necessity. Also, the lifecycle of a machine is usually a lot longer than a normal software system, so significant changes happen much less frequently. Finally, teams [were] usually much smaller. It was not unusual for a single engineer to own the entire development process, from ECAD to programming, to commissioning and maintenance. So, I would say there was a lack of demand in addition to the technical difficulty of implementing source control on all these disparate platforms.”
It is not apparent that VC was thought about in the early days of PLC/SCADA systems, says Chase Davis, director of technology at EOSYS, a CSIA certified member. Once hardware vendors moved away from their handheld or dedicated programmers into Windows-based software, it became the Wild West, and manufacturers were responsible for figuring out their own VC schemes. There were standards from ASME for drawing change management, but not for software VC. There were not yet established best practices for software VC. Some end-users did better than others with the tools they had available. Copies of files were stored in the file system with manual logbook entries for changes. However, the files were often not backed up outside of the programming station.
The importance of VC has increased significantly in recent years, mainly driven by the adoption of mechanisms from traditional software development into automation technology, says Matthias Pohl, global marketing manager TIA portal & efficient engineering, Siemens AG, DI FA S MP AUT. “With external tools and some manual effort, version control with TIA Portal has always been possible.”
Basic mechanisms for VC were implemented in TIA Portal with the TIA Project-Server. The focus here is on complete TIA Portal projects that can include PLCs, HMIs and drives.
With the TIA Portal library concept there is a possibility to handle individual software function blocks or HMI faceplates manually, Pohl adds. “The library concept in TIA Portal is a powerful tool for improving the reusability, standardization and version control of project components. It supports efficient collaboration and contributes to quality assurance.”
Early VC Methods and Systems
An early VC software (pre-Windows 2000) was MDT, which worked with several PLC programming systems — and of course, there was always the floppy disk backup.
“The floppy disk method isn’t a myth — it was the reality,” says Concept’s Schiedler. Engineers often relied on physical media, hoping the most current version was labeled correctly and still on someone’s desk. File names like Rev.A or Rev.1 often led to confusion, and finding the most up-to-date backup could turn into a scavenger hunt. While modern PLCs and automation devices now offer some built-in VC, managing multiple controllers across an entire production line still presents challenges. Relying solely on built-in tools isn’t always scalable, which is why centralized VC systems that support cross-platform management have become critical in today’s food manufacturing environments.
Having a backup of a controller program on multiple production locations, such as an engineering server and an on-unit SD card, has been and could still be practical, says Thomas Kuckhoff P.E, product manager core technology, Omron Automation Americas. In these applications the mantra “if it’s not broken, fix it until it is” can quickly come to mind. Many PLCs have, and continue to have, dip-switch settings that allow PLCs to back up their code to an onboard SD card without interrupting production. These PLCs can quickly reboot from that same SD card.
MDT Autosave was one of the first PLC change management products, and it was also vendor-agnostic, says EOSYS’s Davis. “Versiondog was another product for change management. Both of these tools were very practical for managing multiple controllers in a production line due to the organizational structure and the single pane of glass for interfacing with the versions. MDT Autosave and Versiondog are now combined into a single tool called Octoplant from Auvesy-MDT (now AMDT), which is even more powerful than the tools were separately.”
Rockwell’s FactoryTalk AssetCentre is a product that is effective at change management for Rockwell devices, and Siemens TIA Multiuser Engineering is effective at change management for Siemens TIA Portal projects, Davis adds. The Siemens product allows for collaboration while the Rockwell product does not.
In addition to Siemens’ proprietary option for VC via the TIA Project-Server, a manufacturer-independent interface, the Version Control Interface, has also been established, says Siemens’ Pohl. “Version control interface (VCI) was introduced into TIA Portal several years ago. This feature allows users to integrate TIA Portal projects with version control systems like Git, enhancing collaboration, code quality and project history management. The focus here is on the control software, which typically needs to be frequently changed or adjusted. Standard blocks and a manufacturer-independent syntax (structured text) are particularly important in this context.”
Food Industry Machine Builder Solves Version Control Problems
TARGAN, a machine builder with solutions in manufacturing and specializing in ag-tech products for the food industry — think poultry, pork and seafood — manufactures sophisticated AI-based equipment, which relies on a robust and agile development process to maintain its competitive edge. TARGAN recognized the need for a streamlined and efficient approach to manage its industrial codebase and facilitate collaboration across its engineering teams.
The traditional methods of code management and collaboration were proving cumbersome and inefficient, especially as the company’s engineering team expanded and projects became more complex.
The core of TARGAN’s problem, according to Brian O’Sullivan, lead senior automation controls engineer, was the difficulty of collaborating. “Starting out, we had one big project and now we’ve got multiple. But we have very large projects with lots of different moving parts, lots of different communication between systems. So being able to have multiple people work in parallel on different areas [of industrial code] would be a big benefit, but there was a lot of challenge in that.”
“We’d have one version — one zip file essentially — saved on a server with a date in the name. And then when someone wanted to write or add anything new, I would have to encapsulate it in a function block and save it as another version. Then the next person who is doing the merge of the integration would open it up, and they’d have to copy-paste out parts and put it into theirs and then create a new version. It’s this very, very manual process. Very easy to miss things. Very easy to make a mistake,” O’Sullivan adds.
TARGAN sought a solution that could address these challenges and empower its engineers to collaborate seamlessly to accelerate development cycles. They discovered Copia’s Industrial DevOps Platform and its Git- and cloud-based Source Control solution that is tailored for industrial automation.
“One of the biggest benefits is doing pull requests and having a visual diff. That’s a big differentiator with Copia, O’Sullivan says. “Because, while we could use Github or Azure or whatever else, those systems can’t do pull requests with the visual diff. With Copia, we can actually look at the work that other people are doing. We can manage branches; we can lock down branches. We can say, ‘hey, you can’t commit to this thing unless we do a pull request and somebody okays it.’”
“All of this avoids the ‘who did it this way’ problem. We have visibility and traceability into changes and can understand the ‘who’ and the ‘why’ without blame. This creates an environment that eliminates hidden, broken things. It drives a best-practice approach across our teams.”
New Technologies for Version Control
“With the virtualization of controls and system management tooling, logic and controller backup becomes integrated into the background,” says Yokogawa’s Khan. “Multi-versioning, and storage of these versions, has become common. Tools available include Harbor, used for Containers, and GitLab for other software artifacts. However, this does not help with the lack of documentation in the code, but the older version of the code is still available.”
Furthermore, cloud-based solutions provide centralized storage and access, which enhances collaboration and security, Khan adds. AI is also playing a role by automating VC tasks, identifying potential issues and suggesting optimizations. These tools are designed to incorporate cybersecurity and reduce operator errors.
Git, a distributed version control system originally developed for software development, is now being adapted for industrial automation,” says Gerry Abbey, Copia Automation head of marketing. Its key advantages include:
- Distributed architecture: Every engineer has a complete copy of the code repository, enabling offline work and reducing reliance on a central server.
- Branching and merging: Engineers can work on different features or bug fixes in parallel, without interfering with each other’s work. Changes can then be merged seamlessly.
- Detailed change tracking: Git meticulously tracks every modification to the code, providing a complete history of revisions and enabling easy comparisons between versions.
Git is the ubiquitous VC protocol for a reason. It is mature, powerful and open, says Beckhoff’s Stiffler. “The problem with using it for PLC programs is that the tools are most effective when your code is in plain-text, human readable files. Another key differentiator between PLC and general software programming is the utilization of graphical languages. Git was designed to track changes in text files, not ladder diagrams. It’s totally possible to leverage Git with non-text or binary files, but you are not able to take advantage of all the great capabilities such as visual comparison, cross-referencing, compression, etc. That said, the more recent VC solutions from both vendors and third-party resources include ancillary tools built on top of Git to revive these capabilities with different graphical languages and proprietary formats.”
Git — in and of itself — is complex, but not complicated, especially if you leverage it with TwinCAT automation software from Beckhoff, Stiffler adds. The learning curve you go through is well worth the value gained. The tools and development environments around it lower the barrier to entry as well. There are a lot of security and access control options available for managing a Git repository, so with the appropriate administration strategies it absolutely can be made foolproof and operator-error-proof.
Git has been widely used by software developers for many years, says EOSYS’ Davis. The world of industrial automation is very slow to change and adopt new technologies. However, tools like Copia introduce Git-based VC into the industrial automation space. Adding Git onto VC allows for collaboration that was not available in early change management tools. Multiple users could be working on a program, and changes could be merged easily — and conflicts identified — by the tool.
Cloud technology opens additional possibilities due to the higher computing power, Davis adds. PLC versions can be updated across the plant with a single button click, changes to library objects can be deployed to multiple PLC projects at one time, AI can be used to write code, explain code or write test plans.
While this new technology based on Git can provide advantages in having multiple programmers working on code in different locations, this “unified” approach can have its drawbacks, says Omron’s Kuckhoff. “Unfortunately, we have seen a great unified version control plan become the folly of its design team — either because too many modifications were being made, resulting in more troubleshooting time as every programmer had their own personal ‘style’ or an expensive system going un-used as the infrequency of PLC versions being updated was unforeseen.”
Version Control’s Future
The future of VC in industrial automation is headed toward more cloud-based, AI-assisted solutions that offer real-time change management and intelligent code analysis, says Concept’s Schiedler. “We’re seeing a shift where cloud storage not only houses code securely but also integrates tools for version tracking, automated error checking and approval workflows. As cybersecurity measures strengthen and industry confidence grows, cloud adoption will accelerate. AI’s role will expand — from flagging anomalies and suggesting optimizations to potentially implementing low-level code changes autonomously with full documentation of every modification. While human oversight will always be critical, AI-driven tools will streamline code management and reduce the risk of costly errors.”
Additionally, by coupling VC with digital twins, programs can be more thoroughly tested before deploying to production, says EOSYS’s Davis. “What if we could schedule program changes to be automatically deployed, tested and validated (and rolled back if failed) without any human intervention.”
However cool as that idea might sound, Copia’s Abbey hasn’t ruled out humans yet. “It’s important to remember that human oversight and expertise will always be crucial in industrial automation. AI can be a powerful tool, but it’s ultimately the engineers who understand the nuances of the systems and processes, and who can make informed decisions about code changes and their potential impact. The balance of people, process and technology is the true focus of industrial DevOps, and it continues to be with the addition of AI, though now the boundaries of potential have expanded.”
Version Control’s Future from a Hardware Supplier’s Perspective
The end goal would be for PLC programming languages to match the openness and flexibility of software languages. Only then can we gain the same value from all this remarkable technology: version control, LLM (large language model)-assisted code editors, continuous integration and deployment, fleet management and connectivity.
Some automation platforms like Beckhoff’s TwinCAT software have been much faster to adapt. Their no-cost development environment works natively with Git and services like GitHub, further lowering the barrier to entry and encouraging modern software development practices. They are also at the forefront of AI-assisted PLC programming, with a seamlessly integrated “co-pilot” product launching in the near future. With the upcoming TwinCAT PLC++ feature launch, they are working even further toward a more open, plain-text approach that will improve DevOps capabilities and LLM trainability.
— Brandon Stiffler, Beckhoff USA
Today, one useful way users can employ AI in support of version control is to help them define written processes and procedures to be followed for creating backups and implementing versioning. In the future, AI tools may support PLC programming and HMI/SCADA configuration, potentially addressing change control needs as well.
— Steve Ward, Emerson Discrete Automation
As we look to the future of VC for industrial automation systems, we can expect a holistic approach that tracks not only the code but also the firmware of control units, hardware, and documentation such as drawings and schematics. This comprehensive approach can help ensure that all aspects of the system are tracked, validated and compliant. By integrating high-resolution digital twins, teams can simulate and validate code performance offline, ensuring that only proven and reliable updates are deployed. This will not only enhance compliance and security but also minimize downtime and improve overall production efficiency.
— Thomas Kuckhoff P.E., Omron Automation Americas
When we have a closer look into continuous integration (CI) as a software development practice, we are talking about automated testing, automated documentation, automated builds and so on. Bringing CI and AI together, it is imaginable to enhance development processes in several ways:
Automated Code Reviews: AI can be integrated into CI pipelines to automatically review code changes, ensuring they meet coding standards and identify potential issues before they are merged.
Predictive Analytics: AI can analyze historical data from CI processes to predict potential build failures or test flakiness, allowing developers to address issues proactively.
Enhanced Testing: AI can optimize test suites by identifying the most critical tests to run based on code changes, reducing the time required for testing while maintaining high coverage.
Anomaly Detection: AI can monitor CI pipelines for unusual patterns or anomalies, such as unexpected build times or test failures and alert developers to investigate.
By leveraging AI within CI processes, development teams can improve efficiency, reduce errors and maintain high-quality codebases.
In general, it can be said that AI will play an increasingly important role in the industrial automation sector. Whether for code creation, maintenance or troubleshooting and fixing issues in operational processes, AI’s influence is growing significantly.
— Matthias Pohl, Siemens AG, DI FA S MP AUT
Open Process Automation (OPA) technologies help simplify VC by providing standard interfaces and a single programming environment, called Application Workload, across many different applications and control architectures. OPA System management tools help to ensure proper workflows are utilized, and they provide backups automatically. GIT is one part of the required tool change.
AI is going to lead the way and will play an important role in industrial automation. Engineers are now optimizing the efficiency of code creation with the help of AL/ML. This accumulated knowledge will increase the use of AI, and in the near future, AI-driven systems could make real-time code changes while automatically documenting these changes to ensure transparency and traceability. This would not only be more efficient but reduce human error.
The future is open process automation with AI/ML to evolve intelligent and autonomous VC solutions that will be integrated with existing automation systems.
— Shahzad Kahn, Yokogawa
Looking for a reprint of this article?
From high-res PDFs to custom plaques, order your copy today!
