Why Companies Need Data in Motion for Big Data
You need data in motion to react quickly with the current state of big data. To complete a credit card transaction or send an e-mail, data needs to be transported from one location to another. Data is at rest when it is stored in a database in your data center or the cloud. In contrast, data is in motion when it is in transit from one resting location to another.
Companies that must process large amounts of data in near real time to gain business insights are likely orchestrating data while it is in motion. Data in motion and large volumes of data go hand in hand. Many real-world examples of continuous streams of large volumes of data are in use today:
Sensors are connected to highly sensitive medical equipment to monitor performance and alert technicians of any deviations from expected performance. The recorded data is continuously in motion to ensure that technicians receive information about potential faults with enough lead time to make a correction to the equipment and avoid potential harm to patients.
Telecommunications equipment is used to monitor large volumes of communications data to ensure that service levels meet customer expectations.
Point-of-sale data is analyzed as it is created to try to influence customer decision making. Data is processed and analyzed at the point of engagement — maybe in combination with location data or social media data.
Messages, including details about financial payments or stock trades, are constantly exchanged between financial organizations. To ensure the security of these messages, standard protocols such as Advanced Message Queuing Protocol (AMQP) or IBM’s MQSeries are often used. Both of these messaging approaches embed security services within their frameworks.
Collecting information from sensors in a security-sensitive area so that an organization can differentiate between the movement of a harmless rabbit and a car moving rapidly toward a facility.
Medical devices can provide huge amounts of detailed data about different aspects of a patient’s condition and match those results against critical conditions or other abnormal indicators.
Data in motion, often in the form of streaming data, is becoming increasingly important to companies needing to make decisions when speed is a critical factor. If you need to react quickly to a situation, having the capability to analyze data in real time may mean the difference between either being able to react to change an outcome or to prevent a poor result.
The challenge with streaming data is to extract useful information as it is created and transported before it comes to a resting location. Streaming data can be of great value to your business if you can take advantage of that data when it is created or when it arrives at your business.
You need to process and analyze the streaming data in real time so that you can react to the current state of the data — while in motion and before it is stored. You need to have some knowledge of the context of this data and how it relates to historical performance. And you need to be able to integrate this information with traditional operational data.
The key issue to remember is that you need to have a clear understanding of the nature of that streaming data and what results you are looking for. For example, if your company is a manufacturer, it will be important to use the data coming from sensors to monitor the purity of chemicals being mixed in the production process.
This is a concrete reason to leverage the streaming data. However, in other situations, it may be possible to capture a lot of data, but no overriding business requirement exists. In other words, just because you can stream data doesn’t mean that you always should.
How can you use streaming data to change your business? In some situations, companies are able to take data they already have and begin to use it more effectively. In other situations, they are collecting data that they were not able to collect before.
Sometimes organizations can collect much more of the data that they had been only collecting snapshots of in the past. These organizations are using streaming data to improve outcomes for customers, patients, city residents, or perhaps for mankind. Businesses are using streaming data to influence customer decision making at the point of sale.