MQTT Sparkplug guidance: Publish/subscribe vs. poll/response
Sparkplug uses the Pub/Sub architecture pattern for scalable and efficient communication, which is dramatically different to traditional Poll/Response protocols used in the last 40+ years in industrial automation, discrete manufacturing, and in industries like Oil & Gas.
This means that the communication protocols of the last decades require a very tight coupling between the producer of data (often PLCs) and the consumers of data.
This makes it difficult to change workflows and processes, makes it hard to set up new systems and facilities and makes it difficult to impossible to use and analyze data across the entire system.
Picture 1
Picture 1 shows the traditional Poll / Response way to get data from PLCs, gateways and applications. A polling system is requesting data from the device that produces the data or is the single source of truth for specific data. In order to get data as soon as possible, the polling system is asking for data in a very high frequency, otherwise new data won’t be available to the system that needs that data. This approach is very inefficient as it wastes bandwidth and processing power scale very well.
Jonathan Hottell compared MQTT vs OPC-UA vs Modbus in this presentation and showed that even for basic scenarios, there are orders of magnitudes in efficiency gains when using MQTT Sparkplug compared to OPC-UA. The presentation can be found here. If you want to add security via encrypted communication (which you MUST do if you are connecting things over the internet), then overhead produced by legacy protocols compared to Sparkplug is even more significant
Picture 2
Picture 2 depicts that for each new data producer a system needs to poll. This means in the worst case each tag needs to be polled individually.
This polling approach is used widely in the industry and protocols like Modbus and OPC-UA. The exponential growth of data produced and the hyper-connectivity locally on the site as well as worldwide showed the limitations of the poll / response approach. Companies today require data that is produced globally in up to hundreds of factories and thousands of machines in an instant and expect that important data is accessible to all stakeholders easily, independent of where they are in the world. The industry is now finally moving towards a world where data silos must be broken down.
Poll / Response comes with some severe disadvantages compared to modern publish / subscribe approaches from a technical point of view:
- No state awareness. Most IIoT protocols today are not state aware, which means state needs to be polled all the time, sometimes multiple times per second for every device to make sure no important data or state change is lost
- Massive amount of unnecessary data traffic: Data producers are getting polled up to multiple times per second even if no status change or data change occurred
- Only periodic check for new data. No instantaneous push based mechanism to get data and events as they happen
- Compute intensive on the polled devices. Lots of unnecessary compute cycles are wasted as data is requested all the time even if it didn’t change
- Tight coupling between polled and polling components. If a data producer needs to be changed / replaced, multiple systems need to be reconfigured, possibly with downtime
- Doesn’t scale to large amounts of data producers
There clearly must be a better way. And this is why MQTT Sparkplug started with a blank sheet of paper and asked: “What if we could use the most lightweight communication protocol (MQTT) and add in all the learnings from the last 40+ years to finally bridge the OT/IT gap and provide plug and play interoperability”.
To overcome the disadvantages of legacy protocols, Sparkplug uses a modern publish-subscribe based architecture.
Picture 3
This MQTT based architecture has the following advantages:
- Report by Exception: Data and state is only published when something changes
- Minimal compute intensive: Devices and applications decide themselves when they send data and no compute cycles are unnecessarily wasted
- Scales to hundreds of thousands or even millions of devices with multiple billions of tag data and state changes per day—with a single broker (cluster)
- Push based communication
- Completely decoupled. To change, add, or remove data consumers or producers, no other components need to be changed
The increasing popularity of MQTT Sparkplug in most industries can be explained with the clear advantages over legacy protocols, and the ability to even integrate these legacy protocols in a Sparkplug architecture.
By Dominik Obermaier, HiveMQ CTO and co-founder
