Managing Data in NoSQL

By Adam Fowler

Once you manage the keys appropriately, you’re ready to design how to store data with NoSQL and ensure that it’s safe and always accessible for the work you need to do.

Data types in key-value stores

Key-value stores typically act as “buckets” for binary data. Some databases do provide strong internal data typing and even schema support. Others simply provide convenient helper functions in their client drivers for serializing common application data structures to a key-value store. Examples include maps, lists, and sorted sets.

Oracle NoSQL can operate in two modes:

  • Simple binary store

  • Highly structured Avro schema support

An Avro schema is akin to a relational database schema — enforcing a very stringent set of format rules on JavaScript Object Notation (JSON) data stored within the database, as illustrated here:

{username: “afowler”, sessionid: 13452673, since: 1408318745, theme: “bluesky”} 

You define an Avro schema using a JSON document. This is an example of the Avro schema for the stored data shown previously:

{“type”: “record”,“namespace”: “com.example”,“name”: “UserSession”,“fields”: [
  {“name”: “username”, “type”: [“string”,”null”]},
  {“name”: “sessionid”, “type”: “int”},
  {“name”: “since”, “type”: “long”},
  {“name”: “theme”, “type”: [“string”,”null”]}

An Avro schema provides very strong typing in the database for when schema is important. In the preceding example, you see string data, a numeric session id, a date (milliseconds, since the Unix Time Epoch, as a long integer), and a personalization setting for the theme to use on the website.

Also notice that the type of username and theme has two options — string and null, which is how you instruct Oracle NoSQL that null values are allowed. You could have left theme as a string and provided an additional configuration parameter of “default”: “bluesky”.

Other NoSQL databases provide secondary indexes on any arbitrary property of a value that has JSON content. Riak, for example, provides secondary indexes based on document partitioning — basically, a known property within a JSON document is indexed with a type. This allows for range queries (less than or greater than) in addition to simple equal and not equal comparisons. Riak manages to provide range queries without a stringent schema — just simple index definition. If the data is there, it’s added to the index.

Replicating data

Storing multiple copies of the same data in other servers, or even racks of servers, helps to ensure availability of data if one server fails. Server failure happens primarily in the same cluster.

You can operate replicas two main ways:

  • Master-slave: All reads and writes happen to the master. Slaves take over and receive requests only if the master fails.


Master-slave replication is typically used on ACID-compliant key-value stores. To enable maximum consistency, the primary store is written to and all replicas are updated before the transaction completes. This mechanism is called a two-phase commit and creates extra network and processing time on the replicas.


  • Master-master: Reads and writes can happen on all nodes managing a key. There’s no concept of a “primary” partition owner.


Master-master replicas are typically eventually consistent, with the cluster performing an automatic operation to determine the latest value for a key and removing older, stale values.


In most key-value stores, this happens slowly — at read time. Riak is the exception here because it has an anti-entropy service checking for consistency during normal operations.


Versioning data

In order to enable automatic conflict resolution, you need a mechanism to indicate the latest version of data. Eventually consistent key-value stores achieve conflict resolution in different ways.

Riak uses a vector-clock mechanism to predict which copy is the most recent one. Other key-value stores use simple timestamps to indicate staleness. When conflicts cannot be resolved automatically, both copies of data are sent to the client. Conflicting data being sent to the client can occur in the following situation:

  1. Client 1 writes to replica A ‘Adam: {likes: Cheese}’.

  2. Replica A copies data to replica B.

  3. Client 1 updates data on replica A to ‘Adam: {likes: Cheese, hates: sunlight}’.

    At this point, replica A doesn’t have enough time to copy the latest data to replica B.

  4. Client 2 updates data on replica B to ‘Adam: {likes: Dogs, hates: kangaroos}’.

    At this point, replica A and replica B are in conflict and the database cluster cannot automatically resolve the differences.

An alternative mechanism is to use time stamps and trust them to indicate the latest data. In such a situation, it’s common sense for the application to check that the time stamps read the latest value before updating the value.

They are checking for the check and set mechanism, which basically means ‘If the latest version is still version 2, then save my version 3’. This mechanism is sometimes referred to as read match update (RMU) or read match write (RMW). This mechanism is the default mechanism employed by Oracle NoSQL, Redis, Riak, and Voldemort.