IT/OT convergence: The making of a modern plant

Modern plant control systems increasingly rely on off-the-shelf networking equipment and Windows operating systems. However, the complexity of deployments continues to increase.

These deployments can stretch the skillsets of even experienced plant control system professionals. Due to the off-the-shelf nature of many of their components, they are vulnerable to the same threats as those of corporate IT systems.

Meanwhile, IT personnel are being held responsible for running cyber-secure operations with limited understanding of OT devices, software packages, and operational/plant safety concerns. Modern, nimble organizations have learned how to work collaboratively across their internal IT and OT staff, along with third-party system integrators. It’s not easy to converge IT and OT hardware and software systems and to understand the demand on personnel, but here’s a primer.

History of the IT/OT dynamic

In the 1990s, while the IT world had moved fully to Intel PCs, Microsoft Windows, and Ethernet networks, our control system world was very different. Much of our OT infrastructure was still based on 1980s industrial tech. Much of the networking was a combination of token ring and DECnet. Servers and workstations were either Digital Equipment Corp. VAX and Alpha VMS or HP Unix. The internet was still relatively new, and most cybersecurity threats were from employees clicking on links from suspicious emails or websites. Remote access to the site was through dial-up modems.

IT only had a limited understanding of what equipment was running OT (a term not yet in use) networks and, more importantly, what processes that equipment was controlling. Meanwhile, automation engineers typically had little understanding or interest in IT’s responsibilities for maintaining email servers, file shares, and business management systems.

In the late 1990s, certain facilities decided to install large, plantwide process historians. This meant two things: a large Ethernet network to move process data from across the plant to a central server and a network link between the OT network and the IT network so that the process data could be viewed in engineers’ offices on their desktop PCs.

Automation engineers were now asking IT about reserving subnet addresses and firewall permissions. Issues on one side of the network had potential to affect the other. In parallel, operations began installing their own locally administered PCs on the OT network, first for specialty applications but gradually for all applications as control system vendors eventually adopted a Microsoft PC platform for servers and workstations.

Over the last 30 years, there has been a lot of convergence in IT and OT, and we see that trend continuing. There has also been a lot of growing pains for organizations between OT and IT during this transition. OT increasingly uses off-the-shelf IT hardware and software along with a heavy mix of unique OT components, and IT has been given the challenge of protecting their organizations from advanced cyber threats while providing more services and connectivity to their company employees. While this IT/OT convergence has created an overlap in responsibilities, it has also provided an opportunity to work together toward a mutual objective.

Convergence depends on key personnel

It’s important to remember the background and training for OT and IT personnel and what the primary responsibilities are for each. OT networks are usually run by automation engineers or technicians. Depending on the size of the facility, this role may or may not be a full-time role. They may have an electrical or chemical engineering education, or they may have come through the plant operator or maintenance technician pool.

They will typically spend their careers working for some sort of industrial manufacturer even if they change employers multiple times. OT personnel usually understand well the processes that are being controlled, whether that’s a machine stamping parts, a reactor using hazardous chemicals, or a hydrocracker processing petroleum product. Their knowledge of computers and networking was probably developed through some combination of on-the-job training, equipment vendor training sessions, and self-study. Their No. 1 goal is to keep the plant safely running.

IT networks are usually run by IT staff with computer science or information science backgrounds. They often have specialized responsibilities, such as database administration, network management, and MS Exchange. Their skillset easily translates to other businesses, and they may spend parts of their careers working in health care, finance, or other industries.

Their training usually did not include discussions of PLC/DCS architecture or process safety management (PSM) safety standards for industrial processes. They typically gain a high-level understanding of the industrial processes at the company, but their day-to-day responsibilities are spent in the IT world. Their number one goal is to keep the company secure from an increasingly hostile and unrelenting cyberthreat environment.

Key to convergence: Collaboration, shared work teams

Automation engineers and IT personnel are usually given clear (and sometimes conflicting) mandates from their boards of directors. Both parties have valid concerns that address the opposing party’s perceived lack of understanding and appreciation for their respective responsibilities.

Thanks to the many well-publicized cyber incidents and new federal regulations concerning cybersecurity in the industrial sector, automation and other operations personnel have realized there is a legitimate risk, and they need IT support to help.

Podcast: Cybersecurity action steps and the dilemma of guarding private data

The size and complexity of OT networks along with the increasing use of domain controllers, Microsoft SQL, virtual machines, and cloud services means that many OT environments require a skillset beyond the core responsibilities of deploying and programming PLCs, DCSs, and HMIs.

IT groups are working more closely with automation engineers and, while often assuming responsibility for some level of network and OS management on the OT side, are appreciating that OT networks will always require specialty hardware and software.

The key is continued collaboration and shared work teams, with both groups learning what challenges the other faces and what techniques and resources are available to provide solutions. OT equipment owners need to discuss control system vendor vulnerability reports with IT, and IT needs to discuss current threat concerns with OT.

IT needs to bring up any planned network outages and patch rollouts with OT and OT needs to communicate what disruptions cannot be safely tolerated. Patching systems is a required fact of life in today's world, and while OT environment patches must often be applied manually or with close supervision, the answer cannot be to not patch.

These are the sort of tasks that foster communication and collaboration between IT and OT teams, improving performance in an increasingly converged IT/OT environment.