Big Data Jobs for Software Developers - dummies

Big Data Jobs for Software Developers

By Jason Williamson

Big data projects originate from solving problems with some business objective in mind. Much of the focus today centers around technology implementation, visualization tools, and data products. Today, businesses are doing more with less and need to show the return on investment in everything they do. Software developers are tasked with translating the business problems into workable solutions that drive revenue to the bottom line.

Assessing your interest

Big data jobs for software developers require many of the same core interests as other software developer jobs, but with a twist. Software jobs in this world are not static. Things change a lot — like new technologies, associated languages, software frameworks, and programming techniques. If you love solving problems with code, that’s a good start.

Are you a team player?

This isn’t just a cute cliché. The days of getting an assignment, going into an office for a few weeks, and coding in the dark are long gone. The industry is quickly migrating to the world of agile development, which focuses on software outcomes through a very iterative and collaborative approach.

Teams are typically very small and co-located. You need to be able to work well with all stakeholders, not just your boss or team members, but customers as well.

Do you know more than one coding language?

Most software developers have more than one coding language under their belts. For a big data developer, that’s just the start. Do you have the ability to learn and use new languages? Can you easily learn these languages on your own? Can you pivot between coding languages easily?

In many big data software projects, developers need to be able to shift from using scripting languages like Python, to customizing a Hadoop job in Java, to turning a relational database data query. If you’re more comfortable in predictable, steady-state software development, you may not thrive as a big data developer.

Are you ready to learn?

Not only do you learn new languages all the time, but you learn new techniques and frameworks. Big data is advancing so rapidly that staying current is challenging. The exciting thing about this technical advancement is that it isn’t only around software — it’s in hardware and cloud services as well.

A good indication of your ability to learn about big data is to look at your thirst for it. Are you constantly reading and trying new things?

Looking at sample job postings

This job postings are for a big data developer and data scientist/software engineer. Notice that both jobs require more experience. You’ll need to be able to demonstrate that the experience you have directly ties to the requirements. Think through all the duties you’ve had and see what skills have been developed as a result of that experience.

Big Data Developer

Support the creation of web-based decision support and analytic tools, using the latest JavaScript libraries. Maintain responsibility for the design, development, and sustainment of various existing and new web applications. Research and apply the latest web technologies to meet client requirements and develop rapid prototypes.

Provide support to government and commercial clients, analyzing operational capabilities related to information technology functionality, integration, and interoperability issues. Perform research and analysis, support visualization design related to requirements traceability, portfolio management, and programmatic risk and health, and develop innovative scientific simulations and data visualizations.

Qualifications include:

Basic qualifications:

  • 5+ years of experience with object-oriented programming or entire software development life cycle in an academic or professional environment

  • Experience with algorithm theory and data structures

  • Ability to learn new programming languages and architectures quickly

  • Ability to obtain a security clearance

  • BS degree

Additional qualifications:

  • Experience with researching, presenting, briefing, or communicating analytical material

  • Experience with designing and implementing complex simulations

  • Experience with rich Internet application frameworks, including AJAX, Silverlight, Flex, or ActionScript

  • Experience with development of applications for scientific visualizations

  • Experience in a rapid prototyping environment

  • Excellent oral and written communication skills

  • BS degree in Computer Science

Big Data Scientist/Software Engineer

You’ll be working with a team of smart researchers and engineers to address challenging data problems in the exciting mobile advertising domain. The goal of the team is to make sense out of the huge amount of data flowing through our proprietary ad platform. The problems you would work on include, but are not limited to, yield optimization, smart pricing, network behavior analysis and modeling, campaign performance optimization, inventory prediction, and management.

Responsibilities include:

  • Development, research, and exploration in the areas of statistics, machine learning, experimental design, and operational research

  • Propose, design, and analyze new algorithms to benefit our ad network

  • Research and design experiments to evaluate different algorithms’ impact on the network

  • Implement and verify new algorithms, and integrate the algorithms into production

  • Work with various other team members including data analysts, business owners, engineering, product management, and trafficking

Experience/skills include:

  • Master’s or PhD in engineering, data mining, statistics, operations research, math, physics, economics, or equivalent required.

  • Two to five years of software engineering and scientific experience.

  • Experienced with Java, Python, and big data.

  • Experience with Hadoop, Hive, and Pig a plus.

  • Prior ad network, big data, or analytics experience, operations research, or similar. Proficiency with databases, SQL, and scripting languages.

  • Experience in data mining, data matching, machine learning, statistical techniques, experimental design, or optimization. Experience in extracting and manipulating extremely large datasets.

  • Practical understanding of the mathematics behind modern machine learning, linear algebra, and statistics.

  • Demonstrated ability to apply statistical techniques to solve real problems. Experience with data analysis, business intelligence, and statistical tools (for example, R, SAS, or SPSS). Analytical thinker.

  • Ability to work independently and in a team to research innovative solutions to challenging business/technical problems.

  • Attention to detail, data accuracy, and quality of output.

  • Results oriented and deadline driven.