Master Nodes in Hadoop Clusters

By Dirk deRoos

The master nodes in distributed Hadoop clusters host the various storage and processing management services, described in this list, for the entire Hadoop cluster. Redundancy is critical in avoiding single points of failure, so you see two switches and three master nodes.

  • NameNode: Manages HDFS storage. To ensure high availability, you have both an active NameNode and a standby NameNode. Each runs on its own, dedicated master node.

  • Checkpoint node (or backup node): Provides checkpointing services for the NameNode. This involves reading the NameNode’s edit log for changes to files in HDFS (new, deleted, and appended files) since the last checkpoint, and applying them to the NameNode’s master file that maps files to data blocks.

    In addition, the Backup Node keeps a copy of the file system namespace in memory and keeps it in sync with the state of the NameNode. For high availability deployments, do not use a checkpoint node or backup node — use a Standby NameNode instead. In addition to being an active standby for the NameNode, the Standby NameNode maintains the checkpointing services and keeps an up-to-date copy of the file system namespace in memory.

  • JournalNode: Receives edit log modifications indicating changes to files in HDFS from the NameNode. At least three JournalNode services (and it’s always an odd number) must be running in a cluster, and they’re lightweight enough that they can be colocated with other services on the master nodes.

  • Resource Manager: Oversees the scheduling of application tasks and management of the Hadoop cluster’s resources. This service is the heart of YARN.

  • JobTracker: For Hadoop 1 servers, handles cluster resource management and scheduling. With YARN, the JobTracker is obsolete and isn’t used. A number of Hadoop deployments still haven’t migrated to Hadoop 2 and YARN.

  • HMaster: Monitors the HBase region servers and handles all metadata changes. To ensure high availability, be sure to use a second HMaster instance. The HMaster service is lightweight enough to be colocated with other services on the master nodes. In Hadoop 1, instances of the HMaster service run on master nodes. In Hadoop 2, with Hoya (HBase on Yarn), HMaster instances run in containers on slave nodes.

  • Zookeeper: Coordinates distributed components and provides mechanisms to keep them in sync. Zookeeper is used to detect the failure of the NameNode and elect a new NameNode. It’s also used with HBase to manage the states of the HMaster and the RegionServers.

    As with the JournalNode, you need at least three instances of Zookeeper nodes (and always an odd number), and they’re lightweight enough to be colocated with other services on the master nodes.

    image0.jpg

Here, you’ve got three master nodes (with the same hardware), where the key services Active NameNode, Standby NameNode, and Resource Manager each have their own server. There are JournalNode and Zookeeper services running on each server as well, but these are lightweight and won’t be a source of resource contention with the NameNode and Resource Manager services.

image1.jpg

The principles are the same for Hadoop 1, where you need a dedicated master node for the NameNode, Secondary NameNode, and JobTracker services.

If you plan to use HBase with Hoya in Hadoop 2, you don’t need any additional services. For Hadoop 1 deployments using HBase, check out the following figure for the deployment of services on the Hadoop cluster’s master nodes.

image2.jpg

There are two differences when comparing these master servers to the Hadoop 1 master servers without HBase support: here you need two HMaster services (one to coordinate HBase, and one to act as a standby) and Zookeeper services on all three master nodes to handle failover.

If you intend to use your Hadoop 1 cluster only for HBase, you can do without the JobTracker service, since HBase does not depend on the Hadoop 1 MapReduce infrastructure.

When people talk about hardware for Hadoop, they generally emphasize the use of commodity components — the inexpensive ones. Because you have to plunk down for only a few master nodes (typically, three or four), you aren’t hit by multiplying costs if, for example, you decide to use expensive hard disk drives.

Keep in mind that, without master nodes, there is no Hadoop cluster. Master nodes serve a mission-critical function, and even though you need redundancy, you should design them with high availability and resiliency in mind.

Recommended storage

For Hadoop master nodes, regardless of the number of slave nodes or uses of the cluster, the storage characteristics are consistent. Use four 900GB SAS drives, along with a RAID HDD controller configured for RAID 1+0. SAS drives are more expensive than SATA drives, and have lower storage capacity, but they are faster and much more reliable.

Deploying your SAS drives as a RAID array ensures that the Hadoop management services have a redundant store for their mission-critical data. This gives you enough stable, fast, and redundant storage to support the management of your Hadoop cluster.

Recommended processors

At the time of this writing, most reference architectures recommend using motherboards with two CPU sockets, each with six or eight cores. The Intel Ivy Bridge architecture is commonly used.

Recommended memory

Memory requirements vary considerably depending on the scale of a Hadoop cluster. Memory is a critical factor for Hadoop master nodes because the active and standby NameNode servers rely heavily on RAM to manage HDFS. As such, use error-correcting memory (ECC) for Hadoop master nodes. Typically, master nodes need between 64GB and 128GB of RAM.

The NameNode memory requirement is a direct function of the number of file blocks stored in HDFS. As a rule, the NameNode uses roughly 1GB of RAM per million HDFS blocks. (Remember that files are broken down into individual blocks and replicated so that you have three copies of each block.)

The memory demands of Resource Manager, HMaster, Zookeeper, and JournalNode servers are considerably less than for the NameNode server. However, it’s good practice to size the master nodes in a consistent fashion so that they’re interchangeable in case of hardware failure.

Recommended networking

Fast communication is vital for the services on master nodes, so we recommend using a pair of bonded 10GbE connections. This bonded pair provides redundancy, but also doubles throughput to 20GbE. For smaller clusters (for instance, less than 50 nodes) you could get away with using 1 GbE connectors.