Alan Nugent

To stay competitive today, companies must find practical ways to deal with big data — that is, to learn new ways to capture and analyze growing amounts of information about customers, products, and services.Data is becoming increasingly complex in structured and unstructured ways. New sources of data come from machines, such as sensors; social business sites; and website interaction, such as click-stream data.

Existing analytics tools and techniques will be very helpful in making sense of big data. The algorithms that are part of these tools, however, must be able to work with large amounts of potentially real-time and disparate data. A competent infrastructure must be in place to support this. And, vendors providing analytics tools will also need to ensure that their algorithms work across distributed implementations.

While the worlds of big data and the traditional data warehouse will intersect, they are unlikely to merge anytime soon. Think of a data warehouse as a system of record for business intelligence, much like a customer relationship management (CRM) or accounting system. These systems are highly structured and optimized for specific purposes.

Many companies are exploring big data problems and coming up with some innovative solutions. Now is the time to pay attention to some best practices, or basic principles, that will serve you well as you begin your big data journey. In reality, big data integration fits into the overall process of integration of data across your company.

Big data is most useful if you can do something with it, but how do you analyze it? Companies like Amazon and Google are masters at analyzing big data. And they use the resulting knowledge to gain a competitive advantage. Just think about Amazon's recommendation engine. The company takes all your buying history together with what it knows about you, your buying patterns, and the buying patterns of people like you to come up with some pretty good suggestions.

Four stages are part of the planning process that applies to big data. As more businesses begin to use the cloud as a way to deploy new and innovative services to customers, the role of data analysis will explode. Therefore, consider another part of your planning process and add three more stages to your data cycle.

As you enter the world of big data, you'll need to absorb many new types of database and data-management technologies. Here are the top-ten big data trends: Hadoop is becoming the underpinning for distributed big data management. Hadoop is a distributed file system that can be used in conjunction with MapReduce to process and analyze massive amounts of data, enabling the big data trend.

To understand big data, it helps to see how it stacks up — that is, to lay out the components of the architecture. A big data management architecture must include a variety of services that enable companies to make use of myriad data sources in a fast and effective manner. Here's a closer look at what's in the image and the relationship between the components: Interfaces and feeds: On either side of the diagram are indications of interfaces and feeds into and out of both internally managed data and data feeds from external sources.

Big data enables organizations to store, manage, and manipulate vast amounts of disparate data at the right speed and at the right time. To gain the right insights, big data is typically broken down by three characteristics: Volume: How much data Velocity: How fast data is processed Variety: The various types of data While it is convenient to simplify big data into the three Vs, it can be misleading and overly simplistic.