Smart approaches to keep databases from becoming bottlenecks

By Jeff Tao, CEO of TDengine

The promises of Industry 4.0 and the Industrial Internet of Things (IIoT) have many manufacturers wide-eyed with dreams of automation and interconnection unimaginable in years past. However, the massive amounts of data required to enable such automation and interconnection can quickly turn the database into a significant bottleneck.

By the end of 2022, we are expected to have 14 billion connected devices generating a volume of data we could never have imagined just a few years ago. Deciding how to select a database suitable for manufacturers moving to IIoT can be daunting, as not all databases are equipped to handle modern manufacturing data demands.

Outlined below are the pros and cons of various database-management systems to make it easier.

Traditional SQL databases

Top of mind is traditional SQL databases. Everyone knows traditional SQL databases like MySQL or Oracle, so it’s easy to find talent with the skills to operate them. They have also been the go-to (legacy) solution for so long that most other enterprise systems integrate with them.

However, traditional SQL databases are easily overwhelmed by the size of IIoT data sets. They are not built to handle the scale of production time series, so overtime becomes very slow. This speed can cause several deficits. For example, real-time monitoring to ensure that equipment is running correctly produces large data sets that aren’t manageable. In turn, manufacturers spend more money on hardware to manage the datasets.

NoSQL databases

NoSQL databases like MongoDB and Cassandra are another option to handle manufacturing data. Unlike traditional SQL databases, NoSQL databases can scale at a more affordable rate and process and support massive datasets.

NoSQL does have its challenges. Without a standardized language, developers without experience will need an education period to become competent in your system. Because the systems are newer, they also don’t integrate as seamlessly with other enterprise tools. Additionally, NoSQL is not well-equipped to handle complex queries. So if you want to run a query with a specific requirement, you would be looking at custom coding that may or may not be possible to implement.

Big-data platforms

Lastly, we can look at big-data platforms like Hadoop, which can provide excellent speed for massive datasets. They also have a distributed architecture that enables systems to recover from failures without affecting operations.

But with that comes complex systems with a lot of components. Big-data platform operators must be fluent in everything from Kafka to Redis to Spark. There are also regulation and privacy concerns with big-data platforms on the cloud. Often enterprises will implement a local SQL system with Hadoop running simultaneously for different data, which is not an efficient use of money or resources.

The right choice

Every system has pros and cons, but you need to consider the variables of what you need. Traditional SQL can get overwhelmed by the size of IIoT data, NoSQL can require additional workloads for training and integration, and big-data platforms need dedicated engineering teams to support, maintain and debug efficiently. No platform is flawless. So what is the right choice?

My recommendation is to insist on a data-processing solution that is both purpose-built and open. Manufacturing leaders who have identified the need for purpose-built solutions mainly use closed systems that were never designed to be open, preventing the introduction of new technologies such as machine learning and artificial intelligence into the workflow. The direct impact of these systems is that they lock you to a specific vendor, preventing you from considering cost-effective alternatives even as your expenditures escalate.

A general-purpose database can’t handle the demands of big data that we face with the IIoT—I have seen business reports that took over an hour to generate in a traditional database come out in under a minute simply by migrating to a modern, purpose-built platform. But remember that the big-data solutions that work for tech giants like Facebook can’t produce the same results for industry customers because they’re not designed to; processing social media data is a different beast from the time-series data that manufacturing firms use.

Purpose-built and open enables manufacturing leaders to be on the pulse of cutting-edge technology and, above all, not pigeonholed into a one-size-fits-all solution. To make the right decision, you need to consider the value you are looking for, the available resources, and a budget to decide what works for your organization's manufacturing-business goals.