Dr. Peter Martin of Schneider Electric is calling it a whiteboard moment. Time to get out the erasers and start over. And maybe he’s right – the models and metaphors that defined the industrial automation space for at least a generation are being challenged. The classic hierarchical Purdue reference model – level 1 for closed-loop control, 2 for human supervisory control, 3 for operations and MES, 4 for ERP, etc. – has served us well. But we may have taken its suggestions too literally – maybe we shouldn’t have insisted that its implied boundaries take physical form as independent, isolated networks, for example.
Now, though, the picture may be more like a fully interconnected mesh, or a horizontal service bus, or a cloud, or even a fog (does a Fog really have an Edge?). Advanced analytics tended to stay at the ‘upper’ layers of the Purdue model. Now they’re distributed up and down the mesh, cloud, fog or whatever. So maybe it really is time to redraw the boxes, reroute the lines, and think about a fully connected enterprise. At the 21st annual ARC Industry Forum in Orlando this year, the theme for the event was “Industry in Transition: Realizing the Digital Enterprise.” And this time over 800 thought leaders and visionaries, skeptical end users and eager suppliers gathered to talk about where the industrial automation space is heading and what it might look like when we get there.
Here are some of the concepts and things that were on their minds:
IIoT – the Industrial Internet of Things
Lots of promise, vendor enthusiasm and press activity, but then reality sets in when someone asks about  cybersecurity – attacks against the ‘things’ themselves, or those using swarms of hijacked ‘things’ to attack other connected targets,  interoperability and protocols – OPC? OPC UA? MQTT?, or  organizational implications – who owns and maintains which things? OT folks are waking up to what should happen at the ‘edge’ of that big mesh too, especially when it comes to connecting legacy systems, PLCs, DCSs, transmitters and applications. Seems there’s a real opportunity for a simple, reliable, continuous availability platform for the smart gateways that will be required. And Stratus does offer just such an always-on platform that can be maintained by non-IT people.
ARC are setting up a practice area and offering their influence and insight to help guide us through this new solution space.
No, you can’t talk about the IIoT without asking about cybersecurity. No less than 11 suppliers in the Innovation Showcase were leading with cybersecurity – either in the form of hardware, software, or some sort of consulting services offering:
For some years now, ISA has convened a standards effort on industrial control systems security, the ISA99 standards project. Its members and contributors were visible and vocal at ARC this time. And they’re still open to hearing from vendor and user viewpoints:
And a very interesting observation came from Marty Edwards of the Department of Homeland Security. Marty runs the Industrial Control Systems Cybersecurity Emergency Response Team. And while he’s advising and assessing and implementing security technology, he still insists you should  sit down and identify the crown jewels in your enterprise – that critical workflow or asset or production unit that could put you out of business – and  UNPLUG IT. Because the hackers are pinging you and looking for that same asset and will find a way to hit it remotely if they can.
And again, since that asset still needs to keep running on its own, there seems to be an opportunity there for some simple, reliable, continuous availability platform to keep that asset running inside its own little bubble. Think Stratus.
IT and OT convergence
There’s still quite a bit of jostling between the IT world and the plant floor OT folks. And this is most visibly happening at the edge of that IIoT/network/cloud/fog/etc. The real experts and veterans in the world of operations insist that if it involves process or personnel safety, it’s going to remain in the OT domain. We heard words to that effect from customer experiences with safety, security and availability in gas distribution systems, information integration in engineered products (rolled or forged metals), and in access tracking to critical tools and measuring equipment. One subject matter expert in MES and automation systems summed it up by saying that the closer you are to real-time, the further you should be from the cloud!
New startups and established suppliers are finding ways to put information from all those ‘things’ and the analytics and experts across their enterprise directly in your field of vision as you walk around a plant site. Look through a visor or down at a smart tablet view of a boiler, or a substation, or a warehouse, and see gauges and highlights and procedures imposed directly over the image. Interact with a remote expert whose video image or screen content is in the visor field there before your eyes, and who sees what you see as well. Augmented Reality may finally be delivering for industry what Google Glass tried to do for everybody else. Visit these sites to find out more: www.daqri.com, www.ptc.com/augmented-reality.
Some interesting work here grew out of customers trying to apply the recipes, process and equipment models from ISA88’s world of batch processes to the more continuous process environments of refineries, chemical plants and even nuclear facilities. Any time you do a major state change or product cutover – or deal with a big upset – in a continuous process environment, there’s still a procedure to follow. Most of those have been managed on paper or entrusted to the operators themselves. But those tend to make me nervous, especially when I think about running a nuclear facility in an ad hoc fashion. So ISA’s SP106 project has been busy on models and methods for Procedure Automation for Continuous Process Operations. We saw sessions on that standards effort, and even saw one vendor who’s developing a software product to handle mixed manual and automated procedures.
And it’s not enough that the IIoT and the Cloud are challenging that revered Purdue hierarchical model. True to Dr Martin’s whiteboard moment observation, ExxonMobil, Lockheed Martin, and Saudi Aramco have convened some efforts through The Open Group to consider a whole new model for process automation that would be built around a broadly horizontal service bus architecture. Stay tuned as that develops (and be aware that they’re looking for volunteers to contribute to that new model structure).
Vendors: Innovation Showcase
In addition to the eleven vendors leading with cybersecurity offerings already mentioned above, a couple were bravely introducing  a new PLC / Process Controller hardware package, and  a new web-based SCADA offering with a pretty innovative licensing model. And both might also benefit from a simple, reliable, continuous availability, platform.
Key takeaways – observations from participants
ARC actually recorded a short video montage with snippets of key takeaways and impressions from people who were there. Check it out here.
“The future is already here. It’s just not very evenly distributed” – William Gibson
Last week, I had the pleasure of sitting on a panel at the ARC Industry Forum in Orlando. The topic du jour was OT and IT convergence and a common thread amongst the discussion was how an organization can move to IIoT. Are there new architectures? Is IIoT a rip and replace only option? How can OT partner better with IT to meet IIoT goals? Overall, it was an engaging and excellent discussion. I came away from the session considering the connection between existing IA technologies and IIoT.
And here are some thoughts.
- Layer in the first piece to minimize disruption to legacy equipment. What I mean by this is that any time you make a significant architectural change a good first step is to layer in something around the existing architectural foundation vs going through a major replacement. If you assume that SCADA and Historian applications are at the core in the IA world, then you can look to either devices or analytics as a starting point. For example, one of our customers was able to layer a cloud based analytics layer over their existing SCADA infrastructure to add a lot of value. Other companies are introducing more and more end point devices into the mix first. But overall, take a look at your goals, find a pragmatic starting point and start out by adding one non-invasive layer. Once you have worked out the first layer, move on to the next layer. Often this is when you may see that the architecture core needs a boost which brings me to step 2.
- Virtualize that core infrastructure. Time and again we see underpowered, unreliable and out of date (“read insecure”) infrastructure supporting a SCADA layer. That may be all well and good in the old way of thinking but now is the time to consider an upgrade. In the world of IIoT that’s business critical stuff and that type of software needs a rock solid place to run – such as on a Stratus ftServer system. Once you have virtualized on a solid foundation it will be easier to manage and expand to other applications in the future.
- Respect your institutional knowledge but also look to the future. OT skills and knowledge are incredibly valuable but the introduction of new technologies at the Edge can be daunting. IT can help with the technology provided that the OT folks ensure that business needs are fulfilled. Near the top of that list is simplicity. Adding a lot of new technology for the sake of entrenched data center standards is a recipe for failure. Look for solutions that can thrive and survive at the edge without requiring a lot of IT support.
All in all, success will be determined by a smart scope and understanding the unique user requirements. When you break things down that way the challenge will be reduced greatly.
When it comes to utilities, we as consumers find interruptions to electricity, heat, water, and phone service as extremely disruptive and even dangerous. For utility providers, the impact of such outages also is severe when it comes to lost revenue, customer dissatisfaction, and liability risks. In the natural gas industry, downtime incidents can present even more dire consequences.
This became abundantly clear when a compressor station operated by a North America gas pipeline company suffered a catastrophic failure. The result was a fire that cost more than $550,000 in damages and lost natural gas. Because the station was in a rural location, fire and damage fortunately was contained to the compressor and there were no fatalities.
While the pipeline company highly valued safety and reliability, this frightening incident was a lightning rod to take continuous operations to the next level. The pipeline company engaged in a detailed analysis of 15,000 miles of pipeline and facilities across 16 states, which transports over one trillion cubic feet of natural gas per year to customers.
The resulting modernization report recommended significant system upgrades to comply with the Control Room Management (CRM) regulations issued by the Pipeline and Hazardous Materials Safety Administration (PHMSA). For example, the pipeline company implemented compressor stations with fully redundant systems, such as compressor pumps, turbines, valves, and safety and control systems.
A bigger challenge was creating a continuous availability computer solution to operate the company’s SCADA, historian, HMI, and related control system applications. The pipeline company also wanted the solution to support big data analytics that would proactively predict, detect, and resolve compressor station problems before unplanned outages occurred.
Initially, the pipeline firm planned to deploy six or eight servers to support the full range of applications but discovered this approach had several shortcomings. For example, there were significant space and power constraints and lack of IT support at the compressor stations. If a server failed, automation staff at the headquarters location would need to reconfigure the server’s operating environment, physically deliver it, and perform the install. The unacceptable outcome: two to three days of server downtime and data loss that would generate sub-par analytic results and decrease operational efficiency.
After considering various options, the pipeline company chose a Stratus ftServer, a virtualized continuous availability solution with integrated redundancy. This centralized, easy to manage solution reduced the number of servers and associated service burdens. Automation engineers now remotely run virtualized applications from the primary control centers without requiring trained IT staff at the compressor station to conduct maintenance. Uninterrupted access to real-time analytics also provides the firm with complete operational visibility, eliminating “blind moments” and further improving availability and efficiency.
In fact, since implementing the ftServer three years ago, the pipeline company has run operational systems without any downtime or data loss. According to a lead automation electrical engineer at the company Stratus provides an added benefit: “We can get a lot more flexibility by adding applications in the compressor stations without the need for IT expertise.”
Are you looking to improve the safety and reliability of your operations while reducing costs and increasing efficiency? Stratus offers a compelling solution with virtualization, continuous availability and integrated redundancy.
Continuous improvement has been a theme in the industrial space since the second part of the last century. It’s not just a sound management philosophy – quite naturally, additional efficiency, effectiveness and flexibility go a long way to adding value in processes that are repeatable and continuously running – it’s a way of life for many an industrial manager.
The technology available today is allowing for a degree of automation and process control that was not imaginable even just a few years ago, applicable in ways that impact not just a process but entire infrastructures. The primary payoff is, of course, increased productivity and more efficient processes. In addition, modern automation systems and OT also provide operators with data that can be used to further fine-tune operations.
Whether they are driving incremental improvements or looking for those breakthrough changes, industrial managers have utilized operational technology (OT) advances to get complete control and maximum value out of their processes. And as OT is aging in many plants, industrial managers are looking for ways to modernize while ensuring returns on any such initiatives. Across industrial plants, there have been several sure-fire approaches to ensure such immediate returns:
- Avoiding additional complexity
- Ensuring visibility and control
- Looking for the big wins
- Preventing major risks and unplanned costs
Avoiding Additional Complexity
The industrial internet of things (IIoT) is already happening. Interconnected sensors are able to monitor every step of every process, and the shared data allows for instantaneous adjustments to industrial operations. Advancements in miniaturization mean these sensors can literally be placed anywhere, which gives operators an unprecedented view inside their plants. A combination of systems, software, networks and computing power are necessary to make it all happen. At the same time, such advancements are driving increased reliance on technology. They may require technological skills that may not be available within every plant. Furthermore, some advanced technologies may create a level of complexity that adds costs, implementation headaches and has uncertain payoffs. Not only that, but in many a case, the additional data generated remains unused or misunderstood. So industrial managers need to make sure their new solutions are simple to implement and operate in order to avoid the extra costs and assure immediate returns.
Ensuring visibility and control
Modernized industrial technologies are promising higher visibility into processes, additional data points as well as control mechanisms to be able to constantly optimize. Ongoing monitoring allows for real-time, or almost real-time, adjustments. For instance, Naveen Kumar – a senior industry analyst for Frost & Sullivan who focuses on industrial automation – points out how “with
sensorization of industrial assets, failing parts in an asset can be spotted very early to avoid undesirable downtime consequences.” However, with multiple systems and new sources of data, it sometimes takes additional time for the data to be assembled, consolidated and analyzed before corrective action can be taken. Which, of course, means that processes have been running at sub-optimal levels for a while before that insight was uncovered. So, industrial plants need tools and technologies to ensure that all systems are monitored, “a single pane of glass”, so that operators can make decisions in real time. Furthermore, they need tools to ensure that all systems and monitoring software are continuously running – any outages or downtime in such systems would lead to the dreaded situation of “flying blind”, i.e. resulting in unmonitored and sub-optimal operations. In addition to the lost optimization opportunities, such downtime incidents create significant risks from an operational, safety and environmental perspective.
Looking for the big wins
Continuous improvement and efficient operations is often about the incremental changes. While those are still valuable, the operational and financial gains that can be generated with certain modernization initiatives can also result in major leaps in either improving yield, flow-through and output, or significantly decreasing costs. It is those “big wins” that are most assured to generate immediate returns and the kinds of modernization projects industrial managers should be considering.
Preventing major risks and unplanned costs
It is easy to picture such “break-through” improvements in the form of extra output or decreased consumption of raw materials, energy or resources. What industrial managers don’t always appreciate are the hidden risks, which, when avoided result in massive savings. There are a couple of reasons why such risks may be overlooked. One would be an underestimation of the likelihood of the bad event happening. The other: lack of information or inaccurate assessment of the extra costs incidents result in. And quite so often such costs are hidden or unpredictable. Knowing the size of such unplanned costs and having accurate historical information on the frequency of such incidents in the past are the main tools to assess the immediate returns of technologies they can use to prevent them.
Consider the case of unplanned downtime: on average, an industrial plant experiences 3.6 unplanned downtime incidents per year. In money alone, each incident is a potential disaster; the estimated hourly cost of unplanned downtime ranges from $10,000 to hundreds of thousands of dollars per hour, especially in discrete and process industries. Multiplying the two results in an estimated annual cost of unplanned downtime that few plants can afford to ignore. And while unplanned downtime is primarily measured in money, it can also impact relations with customers, lead to environmental and safety risks, and ultimately impact a company’s reputation.
With modernized operational technologies plant operators can generate gains in efficiency, prevent unplanned costs and monitor operations better. Addressing the big opportunities and avoiding the big risks by simple to install and run solutions are the most proven approaches to achieving immediate returns on such modernization.
To learn more on how advances in modern operational technology are reshaping plant operations for maximum efficiency and uninterrupted operations, download the Stratus Best Practices Kit for Modernizing Automation. To learn more how Stratus can enable your continuous ICS availability, visit www.stratus.com
Accelerating Performance Improvements in the Industrial Sector through Modernized Operational Technology (OT)
The pursuit of efficiency and the avoidance of unplanned downtime have always been primary operational goals in industrial plants. There are a number of pressing market demands prompting plant operators to seek upgrades to their operational foundation. First and foremost, such upgrades are geared to deliver increased productivity. Modernizing IA also assures reliability of operations and better monitoring of processes. The flexibility and scalability that come with modern technologies are necessary to address changing customer needs and demands. Last but not least, multiple new technologies are entering the plant – from sensors to the industrial internet of things (IoT), from machine-to-matching communications to smart factories, from virtualization to even cloud adoption – and OT needs to move with the times in order to be interoperable with those technologies and future-ready. Plants now more than ever need to ensure uninterrupted, scalable and safe operations – or face business disruptions, revenue losses and reduced public and stakeholder trust.
The need for speed – to deliver faster, cheaper and better, is not some distant rumble applicable only to the fast-paced consumer space – according to LNS Research, ensuring consistent quality of products and responsiveness to customer order demands are among the top business objectives for manufacturing organizations and the industrial sector as a whole. As industrial organizations place further reliance on their operational technologies ensuring their continued operation becomes a must. To meet the mounting requirements for uninterrupted operations and prepare themselves for the connected future, industrial organizations now have new solutions to empower them to run faster, leaner and smarter – today and tomorrow.
Operation Technology is Getting Old and Is in Dire Need of an Overhaul
Many organizations now rely on automation systems that are reaching the end of their useful lives. The total global installed base of those systems adds up to $65 billion, according to ARC Advisory Group. Furthermore, the total installed base of automation systems that are more than 20 years old comes to $53 billion, according to ARC. As Craig Resnick, vice president at ARC, shared in a recent blog post, traditional systems are no longer sufficient for handling downtime and addressing the new operational realities.
The New Industrial World Order
From manufacturing plants to oil and gas companies to utilities, industrial organizations are implementing sensors and connected technologies allowing to operate better, faster, safer and cheaper. The Dusseldorf Water Authority needed an IT solution that would allow it to control its water infrastructure via a network of measuring devices with around 40,000 data points. One major requirement for the new IT infrastructure was to keep the SCADA control system up and running so that data could be securely collected and stored for legal, health and safety reasons. Additionally, the Water Authority required a solution that would improve Return on Assets (RoA), provide an extremely high level of availability across the entire IT environment and do so in an easy-to- manage, easy-to- administer manner. After careful evaluation, Dusseldorf’s management chose Stratus.
Now consider this: John Miri, chief administrative officer at the Lower Colorado River Authority (LCRA), who spoke at the Cloud and DevOps World conference in London this year, expanded on how modern technology, and more specifically IIoT, is transforming industrial operations. “In the old days, we would have people with logbooks living near areas prone to flooding and they would come to us and say when they saw something out of the ordinary, but people don’t move as fast as the water does. What we found with IoT, and working on the premise that the speed of light is faster than the speed of water, we can use a larger number of dispersed IoT sensors to detect where flood waters are and keep people safer.”
Enter Modern Continuous Availability Solutions for Industrial Control Systems – with Simple Elegance
We have heard it before – the claim that since it works for the IT department, it should work on the OT side. Anyone with serious industrial chops will probably give such claim a good chuckle. For critical industrial control systems to provide the level of performance, reliability and scalability needed today, they require more than general-purpose IT solutions. Industrial availability solutions need to be robust, resilient and scalable – and they need to be simple to deploy and maintain. So consider the following key factors when choosing your continuous availability solutions – and trust in their simple elegance:
- Your availability solutions need to be software-driven so you can simply and easily control how your hardware operates.
- They need to be virtual in nature – gone are the days when your SCADA, HMI or Historian run on a separate server or even on a PC. Today you can make the server work for you – through the power of virtualization. Yes, you will need less resources to run these solutions, and fewer servers to buy – this is what industrial efficiency is all about.
- Your modern industrial solutions need to provide a single view of operations. If you had to toggle between screens to figure out what resources are available to address alerts and spikes in your SCADA, you won’t go far. You need a single point of control and “a single pane of glass” to give you the visibility into both physical and virtual servers. Not only would this empower you to run your operations more efficiently, it is critical in minimizing downtime. You can address problems before they occur and win the employee of the month badge.
To learn more on how advances in modern operational technology are reshaping plant operations for maximum efficiency and uninterrupted delivery, download the latest Stratus trend report: Accelerating Performance Improvements with Modernized Operational Technology in the Industrial Sector. To learn more how Stratus can enable your continuous ICS availability, visit www.stratus.com to select your industrial vertical.
Lately, people have been asking more questions about the safety and reliability of their water supply. That’s because recent incidents of system contamination such as the crisis in Flint, Michigan and storm damage from hurricanes Matthew and Sandy have thrust water and wastewater issues into the public eye. And that’s put increased pressure on municipalities and counties to get a better handle on their water and wastewater operations.
To do that, utilities like our long-time customer, Pinellas County Utilities (PCU) in Florida, rely on industrial control systems such as supervisory control and data acquisition (SCADA), historians, and human machine interfaces (HMIs). PCU monitors and controls about 20,000 assets, including valves, pumps, and temperature gauges, 2,000 miles of pipe, 13 well fields, five surface water sources, three wastewater treatment facilities, and 350 lift stations. These systems simply cannot go down.
To maintain high availability, PCU previously relied on clustered commodity servers but the problem was that they were getting old and expensive to support. So PCU decided to replace them with Stratus ftServers. Ken Osborne, PCU’s SCADA supervisor, explained: “Keeping the water on is a public health and safety issue. We can’t tolerate any downtime. Replacing clustered failover servers with ftServers saved us a lot of money and simplified the entire operation. I’ve never looked back.”
The utility deployed eight ftServers in three geographically dispersed control sites and their central control room. The ftServers run PCU’s Wonderware environment supporting SCADA, historian, HMI, and other applications, as well as Microsoft SQL Server and terminal services. PCU also virtualized almost everything with VMware.
By running virtual machines (VMs) on ftServers, PCU achieved several important benefits:
- Efficiency was improved by centralizing remote access to the entire SCADA system. This lets technicians monitor physical components across water and wastewater operations anywhere, any time. It also gives them visibility into how everything is working or any issues that require attention. PCU management also analyzes real-time and historical operational information for insights that help further improve performance and efficiency of water and wastewater systems.
- Protecting critical VMs with ftServers made good economic sense. Osborne figured that using server clusters instead of ftServers would have doubled his project costs and taken weeks longer to implement. The fault-tolerant virtualization solution also makes it easy for PCU to consolidate other applications and add new ones by simply creating another VM instead of facing the capital expense of buying additional servers.
- Perhaps most important, PCU doesn’t have to worry about unplanned system downtime for its critical SCADA and other industrial control systems. Ken Osborne put it this way: “Our operation has relied on Stratus systems with no unscheduled downtime caused by a server failure. The server always runs and we never lose a thing. That’s peace of mind.”
If you’re looking for a similar level of confidence in the reliability of your industrial automation operations, talk to Stratus. We’ll show you how to eliminate unplanned downtime and modernize your infrastructure for better efficiency and higher return on assets.
On 4th October in Manchester and 6th October in Milton Keynes, Routeco, one of the UK’s leading distributors of industrial automation and control products, brought together some of the biggest names in the industry to discuss the latest developments in the market and to exhibit some of the remarkable new technologies currently under development. And what an event it was! Stratus, Rockwell Automation and GAMBICA were just a few of the national and global organisations discussing and showcasing the latest trends at a very exciting time for UK industry.
The sessions were designed to deliver key information and promote discussion around three themes: Machine Building, Manufacturing and System Integration and Panel Building. Stratus brought a strong team to the event and exhibited its ftServer Solution that underpins the availability needed by modern industry to deliver the IIoT evolution without costly server-induced downtime. Three breakout rooms hosted presentations and lively discussions relating to the future of the industry. They demonstrated a universal desire amongst delegates and exhibiting companies to highlight best practice at the dawn of a new era for the manufacturing industry in the UK.
Robin Dennell, Stratus’ Availability Expert, presented in the Manufacturing Forum and gave talks on ‘why seconds matter in today’s business world’, and why the continuous availability of critical processing is vital for a modern enterprise. When I spoke with him after the event in Milton Keynes, he was quick to point out the value of meeting so many of the right people in one place and to raise the importance of “always-on” applications. Thomas Donato, EMEA President at Rockwell Automation, gave a keynote address at both events – a measure of the importance to Rockwell Automation of the value of the UK market. He was keen to stress how automation and modernisation are vital in today’s connected world, due to the incredible technological advances in all industries and the growth of information enabled industrial processes. For manufacturing, he mentioned, a connected and modern enterprise means an increase in productivity, efficiency and sustainability.
Steve Brambley, Director of Public Relations for GAMBICA, spoke about hot topics such as “Brexit” – the UK’s imminent exit from the EU, but also about how important it is to have a long term strategy for industrial automation. He also pointed at the importance of virtualisation – another message that we at Stratus endorse as a significant contributor to improved industrial efficiency. The event was a very well received and it was great to be able to connect with so many of the most significant companies in the manufacturing industry. I left the event feeling more certain than ever that the future is going to deliver exciting new technology that is going to help us all work faster, with greater ease, more up-time and, ultimately, more productively. I also hope that Routeco LIVE takes a regular slot in the diary of industry leaders in the UK.