The Economics of Big Data
The best way to understand the economics of big data is to look at the various methods for putting big data to work for your organization. While specific costs may vary due to the size of your organization, its purchasing power, vendor relationships, and so on, the classes of expense are fairly consistent.
Big data types and sources
The most important decisions you need to make with respect to types and sources are
What data will be necessary to address your business problem?
Where can you source the data?
What can you do with the data?
How often do you need to interact with the data?
Who maintains ownership of the data and the work products?
How long do you need to keep the data?
Can you trust the data and its source?
Analyzing big data to anticipate what’s next
Often hints are available within existing data. However, without enough data, these hints will be ignored because that data may look like an outlier or even an error. These kinds of early notifications of a changing requirement could enable the businesses to test new services and new packaging that could become important.
Finding the right data sources
Sourcing the data is the next step. It is not just about where to get the data, but also the form or type of the data as well as the quality or trustworthiness of the data. Good sources of sentiment data are found in social web properties like Facebook, foursquare, Yelp, Pinterest, and Twitter.
The sources you select may be determined by the habits of your customers. The amount of data is vast and you may be looking for the proverbial needle in the haystack. In addition, the structure and types of this data vary from site to site, adding additional complexity and costs as well.
What can you do with the data?
Understanding how often the data is used by internal systems can help to control costs. If the requirements are to analyze customer sentiment in real time across several social properties, the costs will be very high. If the analysis can be performed more leisurely or with fewer data sources, the costs can be lower and more controllable.
Some big data source suppliers will want to maintain ownership of their data, licensing it for specific, nondestructive uses. Others will be open with little or no access costs or overbearing usage requirements. Some data licensing will limit the usage to compute and destroy.
Others may allow you to use the data, but require you to “give it back” when your analysis or computations are complete. Care should always be taken to protect company information.
Big data economics should be understood from two dimensions: getting started and managing the steady state. Startup costs can be contained by finding open data or freely accessible data sources. If more data center resources are required, you should consider cloud-based services where you can “pay by the drink.” It is much easier to experiment this way.
The technology impact of big data
In an ideal world, it will be possible to use a lot of existing technologies and applications when big data is applied to workflows. However, it is much more likely that new technologies will need to be employed.
Many new and different tools are available for big data. If a brand manager needs to gather data from several different social sites, each with different data types, she will need to work with the IT teams to select what technology best fits the business and cost requirements.
You will certainly have implementations of products that will incorporate elements of Hadoop and Hive. Also, new technologies will be required. The existing technologies are too brittle or because they are designed for a specific task.
Find the talent to support big data projects
The business analysts may need to augment their ranks with data scientists. This can be accomplished with consulting relationships in the startup phases, but should transition to permanent staffing as the direction becomes clear. A single data scientist is not likely to be the answer. The most leverage will be realized by creating a team of data scientists.
For the IT team, knowledge of new big data technologies will need to be introduced to existing team members through training and mentoring. It is fair to assume that new talent will need to be hired as your organization approaches steady state.
Many universities and colleges have begun to offer courses that should help fill the gap in the short term. In the long term, vendors providing solutions will have to create more usable big data solutions that abstract the complexity.