The Future of Big Data Jobs - dummies

By Jason Williamson

Thinking of a future career in big data? Some people just seem to know the path they need to take because it feels right and they love it. Other people need a bit more evidence to validate their choice. As you look at the data — both empirical and anecdotal evidence — you can see that there’s a fantastic growth opportunity in the field of big data.

The growth of big data jobs

A growing body of evidence suggests that the trend in demand for big data jobs will continue to grow, which is great news if you’re just now thinking about your professional prospects! Not only are there thousands of postings on job boards and social media sites, but other evidence suggests that we’re still in the very early stages of growth.

Both McKinsey and Gartner make huge claims about the number of big data jobs that will be available and unfilled in the coming years. In 2012, Gartner predicted that there will be more than 4.4 million big data jobs by 2015, and only about one-third of those jobs will be filled.

McKinsey says that in 2014 the U.S. alone faces a shortfall of 140,000 to 190,000 people to fill big data jobs, with an additional shortage of 1.9 million analysts and managers. They say that by 2018, the U.S. won’t be able to fill 50 percent to 60 percent of these roles. So, if you go with either conclusion, the job growth is significant, as are the opportunities for those who are prepared go after them.

Why is there such a gap? Three main factors that exist today suggest that demand for big data jobs will continue:

  • The lack of current widespread adoption of big data within organizations: Combine that with the desire to take on big data projects in the future, and you have an opportunity for growth. A 2013 Gartner survey showed that 72 percent of respondents plan to increase their spending on big data in the coming year, but 60 percent said they didn’t have the skill needed to do it. That’s good news for you!

  • The amount of data being generated by customers, employees, and third parties: Seventy-five percent of data warehouses can’t scale to meet the new velocity demands of data entering the firm. Velocity (the extremely high rate at which data is coming in) is a key attribute of big data. Plus, companies with more than 1,000 employees on average have more than 200TB of stored data.

    The 2013 Gartner survey results indicate that only 13 percent of companies are using predictive analytics today, so the gap between aspiration to deliver big data solutions and the capability to deliver big data is wide. This also means that it’s ripe for opportunity for those who have the skills.

  • The amount of venture capital money being invested in big data: Investors see the potential of big data and are already putting their money into these projects. Therefore, it follows that this is where the jobs will be. Position yourself to take advantage of these opportunities.

Predictions for the next several years

Predicting the future is very easy. Getting it right is the tough part. The question that many people have been asking is, “Is big data just a fad?” Now the question is, “How can I use big data today?”

Let’s look at a few data points to support this movement away from big data being a science project to a reality. First, consider how search interest in big data compares to cloud computing over the past several years on Google. Check out the comparison of searches on Google for these two topics. The black line indicates the number of searches done in Google for “big data” during the period 2005 to August 2014.

This includes searches for such terms as big data analytics and big data PDF. Google defines the number of searches as “interest” in a topic. The gray line indicates the searches done for “cloud computing” over the same period. This includes searches like Google cloud and what is cloud. You see that the interest in big data is now on an upward trend and the interest in cloud computing was very high and is leveling off but still has interest.

Google Trends]”/>
Credit: Source: Google Trends

The top three IT areas of growth are big data, cloud computing, and mobile computing. Cloud computing is farther along the “hype” cycle than cloud computing, but both are key areas of interest.

It isn’t a surprise that cloud computing had a high peak in 2011 and declined during the past couple years. People have a better sense of what cloud technology is. As a point of comparison, in 2013 Gartner pegged the global public cloud market sector at $131 billion, and says that it will grow to more than $600 billion by 2016.

Therefore, you can make a reasonable assumption that interest in learning about big data is a leading indicator for continued growth in this sector. Here’s what other analysts are saying about big data:

  • In December 2013, the International Data Corp (IDC) a leading technology research firm, predicted that the market for big data would reach $16.1 billion by 2014 and grow six times faster than the overall IT market.

  • Wikibon analyst Jeff Kelly’s 2013 review pegged the big data market at $18.6 billion, with it reaching more than $50 billion by 2017. He breaks down market share between services, hardware/cloud, and software.

  • SNS research predicts that the big data market will grow more than 17 percent compound annual growth rate (CAGR) during the next six years.

  • Matt Turck’s famous big data ecosystem chart tracked about 100 companies in 2010. His 2014 chart contains almost 1,000 big data firms.