Alan Nugent

To stay competitive today, companies must find practical ways to deal with big data — that is, to learn new ways to capture and analyze growing amounts of information about customers, products, and services.Data is becoming increasingly complex in structured and unstructured ways. New sources of data come from machines, such as sensors; social business sites; and website interaction, such as click-stream data.

As computing moved into the commercial market, data was stored in flat files that imposed no structure. Today, big data requires manageable data structures. When companies needed to get to a level of detailed understanding about customers, they had to apply brute-force methods, including very detailed programming models to create some value.

You need data in motion to react quickly with the current state of big data. To complete a credit card transaction or send an e-mail, data needs to be transported from one location to another. Data is at rest when it is stored in a database in your data center or the cloud. In contrast, data is in motion when it is in transit from one resting location to another.

So, how do you get started in your journey to creating the right environment so that you are ready to both experiment with big data and be prepared to expand your use of big data when you are ready? Will you have to invest in new technologies for your data center? Can you leverage cloud computing services? The answer to these questions is yes.

ETL tools combine three important functions (extract, transform, load) required to get data from one big data environment and put it into another data environment. Traditionally, ETL has been used with batch processing in data warehouse environments. Data warehouses provide business users with a way to consolidate information to analyze and report on data relevant to their business focus.

Some big data experts believe that different kinds of data require different forms of protection and that, in some cases in a cloud environment, data encryption might, in fact, be overkill. You could encrypt everything. You could encrypt data, for example, when you write it to your own hard drive, when you send it to a cloud provider, and when you store it in a cloud provider's database.

Big data analysis has gotten a lot of hype recently, and for good reason. You will need to know the characteristics of big data analysis if you want to be a part of this movement. Companies know that something is out there, but until recently, have not been able to mine it. This pushing the envelope on analysis is an exciting aspect of the big data analysis movement.

Most big data management professionals are familiar with the need to manage metadata in structured database management environments. These data sources are strongly typed (for example, the first ten characters are the first name) and designed to operate with metadata. You might assume that metadata is nonexistent in unstructured data, but that is not true.

To understand big data workflows, you have to understand what a process is and how it relates to the workflow in data-intensive environments. Processes tend to be designed as high level, end-to-end structures useful for decision making and normalizing how things get done in a company or organization. In contrast, workflows are task-oriented and often require more specific data than processes.