NoSQL Data Stores versus Hadoop
NoSQL data stores originally subscribed to the notion “Just Say No to SQL” (to paraphrase from an anti-drug advertising campaign in the 1980s), and they were a reaction to the perceived limitations of (SQL-based) relational databases. It’s not that these folks hated SQL, but they were tired of forcing square pegs into round holes by solving problems that relational databases weren’t designed for.
A relational database is a powerful tool, but for some kinds of data (like key-value pairs, or graphs) and some usage patterns (like extremely large scale storage) a relational database just isn’t practical. And when it comes to high-volume storage, relational database can be expensive, both in terms of database license costs and hardware costs. (Relational databases are designed to work with enterprise-grade hardware.)
So, with the NoSQL movement, creative programmers developed dozens of solutions for different kinds of thorny data storage and processing problems. These NoSQL databases typically provide massive scalability by way of clustering, and are often designed to enable high throughput and low latency.
The name NoSQL is somewhat misleading because many databases that fit the category do have SQL support (rather than “NoSQL” support). Think of its name instead as “Not Only SQL.”
The NoSQL offerings available today can be broken down into four distinct categories, based on their design and purpose:
Key-value stores: This offering provides a way to store any kind of data without having to use a schema. This is in contrast to relational databases, where you need to define the schema (the table structure) before any data is inserted. Since key-value stores don’t require a schema, you have great flexibility to store data in many formats.
In a key-value store, a row simply consists of a key (an identifier) and a value, which can be anything from an integer value to a large binary data string. Many implementations of key-value stores are based on Amazon’s Dynamo paper.
Column family stores: Here you have databases in which columns are grouped into column families and stored together on disk.
Strictly speaking, many of these databases aren’t column-oriented, because they’re based on Google’s BigTable paper, which stores data as a multidimensional sorted map.
Document stores: This offering relies on collections of similarly encoded and formatted documents to improve efficiencies. Document stores enable individual documents in a collection to include only a subset of fields, so only the data that’s needed is stored. For sparse data sets, where many fields are often not populated, this can translate into significant space savings.
By contrast, empty columns in relational database tables do take up space. Document stores also enables schema flexibility, because only the fields that are needed are stored, and new fields can be added. Again, in contrast to relational databases, table structures are defined up front before data is stored, and changing columns is a tedious task that impacts the entire data set.
Graph databases: Here you have databases that store graph structures — representations that show collections of entities (vertices or nodes) and their relationships (edges) with each other. These structures enable graph databases to be extremely well suited for storing complex structures, like the linking relationships between all known web pages. (For example, individual web pages are nodes, and the edges connecting them are links from one page to another.)
Google, of course, is all over graph technology, and invented a graph processing engine called Pregel to power its PageRank algorithm. (And yes, there’s a white paper on Pregel.) In the Hadoop community, there’s an Apache project called Giraph (based on the Pregel paper), which is a graph processing engine designed to process graphs stored in HDFS.
The data storage and processing options available in Hadoop are in many cases implementations of the NoSQL categories listed here. This will help you better evaluate solutions that are available to you and see how Hadoop can complement traditional data warehouses.