Hadoop 2 adds YARN support

Hadoop will now natively support Apache’s framework for job scheduling and cluster resource management. With Hadoop’s 2.2.0 version, the first stable release offered on the 2.x build, the platform sees YARN sit on top of HDFS (Hadoop Distributed Filing System) to offer a large-scale, distributed operating system for applications dealing with big data, thus enabling simultaneous processing for multiple applications. Using YARN as a resource manager, this new version marks the biggest change in Hadoop’s framework so far, with the platform stepping aside from its previous implementations of the MapReduce framework for server resource management and workload handling. MapReduce JobTracker was so far being used for handling both resource management and job scheduling and monitoring. YARN has now come in to allocate those to functions, by adding a global ResourceManager and an ApplicationMaster that maintains separate copies for each running app. With this implementation, Apache strengthens its platform and significantly widens its user base, as Hadoop will enable developers to build applications straight from Hadoop’s framework, without having to use separate outside resources for building and running. Along with the addition of YARN support, Hadoop’s HDFS protocols have seen a significant improvement as well, with HDFS increasing availability, as well as the addition of HDFS snapshots that will enable Hadoop users to take advantage of native backup and disaster recovery processes. Also, HDFS takes a step into the mainstream, with support being added for Hadoop to be run in Microsoft Windows, for one thing. In addition to that, with NFSv3 file system becoming available for accessing HDFS data, Hadoop users will be able to have it mounted as the system’s standard file system. Native encryption is also a new implementation that will secure data in transit, while Hadoop’s APIs have also become significantly more stable.