Hadoop is a open source framework which is written in java by apache software foundation. This
framework is used to write software application which requires to process vast amount of data (It could
handle multi tera bytes of data). It works in-parallel on large clusters which could have 1000 of computers
(Nodes) on the clusters. It also

For processing large data sets in parallel across a hadoop cluster, Hadoop MapReduce framework is used.
Data analysis uses a two-step map and reduce process.

In MapReduce, during the map phase it counts the words in each document, while in the reduce phase it
aggregates the data as per the document spanning the entire collection. During the map phase the input data is
divided into splits for analysis by map tasks running in parallel across Hadoop framework.

The process by which the system performs the sort and transfers the map outputs to the reducer as inputs is
known as the shuffle

Distributed Cache is an important feature provided by map reduce framework. When you want to share some
files across all nodes in Hadoop Cluster, DistributedCache is used. The files could be an executable jar files
or simple properties file.

NameNode in Hadoop is the node, where Hadoop stores all the file location information in HDFS (Hadoop
Distributed File System). In other words, NameNode is the centrepiece of an HDFS file system. It keeps
the record of all the files in the file system, and tracks the file data across the cluster or multiple machines

In Hadoop for submitting and tracking MapReduce jobs, JobTracker is used. Job tracker run on its own
JVM process

Hadoop performs following actions in Hadoop

Client application submit jobs to the job tracker

JobTracker communicates to the Namemode to determine data location

Near the data or with available slots JobTracker locates TaskTracker nodes

On chosen TaskTracker Nodes, it submits the work

When a task fails, Job tracker notify and decides what to do then.

The TaskTracker nodes are monitored by JobTracker

Heartbeat is referred to a signal used between a data node and Name node, and between task tracker and
job tracker, if the Name node or job tracker does not respond to the signal, then it is considered there is
some issues with data node or task tracker

To increase the efficiency of MapReduce Program, Combiners are used. The amount of data can be
reduced with the help of combiner’s that need to be transferred across to the reducers. If the operation
performed is commutative and associative you can use your reducer code as a combiner. The execution of
combiner is not guaranteed in Hadoop

When a datanode fails

Jobtracker and namenode detect the failure

On the failed node all tasks are re-scheduled

Namenode replicates the users data to another node

In Hadoop during Speculative Execution a certain number of duplicate tasks are launched. On different
slave node, multiple copies of same map or reduce task can be executed using Speculative Execution. In
simple words, if a particular drive is taking long time to complete a task, Hadoop will create a duplicate
task on another disk. Disk that finish the task first are retained and disks that do not finish first are killed.

The basic parameters of a Mapper are

• LongWritable and Text
• Text and IntWritable

The function of MapReducer partitioner is to make sure that all the value of a single key goes to the same
reducer, eventually which helps evenly distribution of the map output over the reducers

Logical division of data is known as Split while physical division of data is known as HDFS Block

In textinputformat, each line in the text file is a record. Value is the content of the line while Key is the
byte offset of the line. For instance, Key: longWritable, Value: text

• The user of Mapreduce framework needs to specify
• Job’s input locations in the distributed file system
• Job’s output location in the distributed file system
• Input format
• Output format
• Class containing the map function
• Class containing the reduce function
• JAR file containing the mapper, reducer and driver classes

To support editing and updating files WebDAV is a set of extensions to HTTP. On most operating system
WebDAV shares can be mounted as filesystems , so it is possible to access HDFS as a standard filesystem
by exposing HDFS over WebDAV.

To transfer the data between Relational database management (RDBMS) and Hadoop HDFS a tool is used
known as Sqoop. Using Sqoop data can be transferred from RDMS like MySQL or Oracle into HDFS as
well as exporting data from HDFS file to RDBMS

The task tracker send out heartbeat messages to Jobtracker usually every few minutes to make sure that
JobTracker is active and functioning. The message also informs JobTracker about the number of available
slots, so the JobTracker can stay up to date with where in the cluster work can be delegated

Sequencefileinputformat is used for reading files in sequence. It is a specific compressed binary file format
which is optimized for passing data between the output of one MapReduce job to the input of some other
MapReduce job.

Conf.setMapperclass sets the mapper class and all the stuff related to map job such as reading data and
generating a key-value pair out of the mapper

The two most common Input Formats defined in Hadoop are:

• TextInputFormat
• KeyValueInputF2ormat
• SequenceFileInputFormat
• TextInputFormat is the Hadoop default.

TextInputFormat: It reads lines of text files and provides the offset of the line as key to the Mapper and
actual line as Value to the mapper.

KeyValueInputFormat: Reads text file and parses lines into key, Val pairs. Everything up to the first tab
character is sent as key to the Mapper and the remainder of the line is sent as value to the mapper.

When a Hadoop job is run, it splits input files into chunks and assign each split to a mapper to process.
This is called InputSplit.

It is invoked by the Hadoop framework by running getInputSplit()method of the Input format class (like
FileInputFormat) defined by the user.

Hadoop will make 5 splits as follows:

- 1 split for 64K files

- 2 splits for 65MB files

2 splits for 127MB files

The InputSplit has defined a slice of work, but does not describe how to access it. TheRecordReader class
actually loads the data from its source and converts it into (key, value) pairs suitable for reading by the
Mapper. The RecordReader instance is defined by the Input Format.

Partitioning: It is the process of determining which reducer instance will receive which intermediate keys
and values. Each mapper must determine for all of its output (key, value) pairs which reducer will receive1
them. It is necessary that for any key, regardless of which mapper instance generated it, the destination
partition is the same.

Shuffle: After the first map tasks have completed, the nodes may still be performing several more map tasks
each. But they also begin exchanging the intermediate outputs from the map tasks to where they are
required by the reducers. This process of moving map outputs to the reducers is known as shuffling.

Sort: Each reduce task is responsible for reducing the values associated with several intermediate keys. The
set of intermediate keys on a single node is automatically sorted by Hadoop before they are presented to
the Reducer.

The default partitioner computes a hash value for the key and assigns the partition based on this result.

The Combiner is a ‘mini-reduce’ process which operates only on data generated by a mapper. The
Combiner will receive as input all data emitted by the Mapper instances on a given node. The output from
the Combiner is then sent to the Reducers, instead of the output from the Mappers.

JobTracker is the service within Hadoop that runs MapReduce jobs on the cluster.

The following are some typical tasks of JobTracker:-

- Accepts jobs from clients

- It talks to the NameNode to determine the location of the data.

- It locates TaskTracker nodes with available slots at or near the data.

- It submits the work to the chosen TaskTracker nodes and monitors progress of each task by receiving
heartbeat signals from Task tracker.

TaskTracker is a node in the cluster that accepts tasks like MapReduce and Shuffle operations – from a
JobTracker.

One job is broken down into one or many tasks in Hadoop.

It will restart the task again on some other TaskTracker and only if the task fails more than four (default
setting and can be changed) times will it kill the job.

Speculative Execution.

JobTracker makes different TaskTrackers pr2ocess same input. When tasks complete, they announce this
fact to the JobTracker. Whichever copy of a task finishes first becomes the definitive copy. If other copies
were executing speculatively, Hadoop tells the TaskTrackers to abandon the tasks and discard their
outputs. The Reducers then receive their inputs from whichever Mapper completed successfully, first.

Hadoop job – list

Hadoop job – kill jobID

Streaming is a generic API that allows programs written in virtually any language to be used as Hadoop
Mapper and Reducer implementations.

Hadoop Streaming allows to use arbitrary programs for the Mapper and Reducer phases of a MapReduce
job by having both Mappers and Reducers receive their input on stdin and emit output (key, value) pairs on
stdout.

Distributed Cache is a facility provided by the MapReduce framework to cache files (text, archives, jars
and so on) needed by applications during execution of the job. The framework will copy the necessary files
to the slave node before any tasks for the job are executed on that node.

This is because distributed cache is much faster. It copies the file to all trackers at the start of the job. Now
if the task tracker runs 10 or 100 Mappers or Reducer, it will use the same copy of distributed cache. On
the other hand, if you put code in file to read it from HDFS in the MR Job then every Mapper will try to
access it from HDFS hence if a TaskTracker run 100 map jobs then it will try to read this file 100 times
from HDFS. Also HDFS is not very efficient when used like this.

This is a tricky question. There is no such mechanism. Distributed Cache by design is read only during the
time of Job execution.

Anybody who claims to have worked on a Hadoop project is expected to use counters.

Yes, the input format class provides methods to add multiple directories as input to a Hadoop job.

Yes, by using Multiple Outputs class.

The Hadoop job will throw an exception and exit.

You cannot set it.

You can either do it programmatically by using method setNumReduceTasks in the Jobconf Class or set it
up as a configuration setting.

To have Hadoop use a custom partitioner you will have to do minimum the following

three:

- Create a new class that extends Partitioner Class

- Override method getPartition

- In the wrapper that runs the Mapreduce, either

Add the custom partitioner to the job programmatically using method set Partitioner Class or – add the
custom partitioner to the job as a config file (if your wrapper reads from config file or oozie)

There can be several ways of doing this but most common ways are:-

- By using counters.

- The web interface provided by Hadoop framework.

It is an open-ended question but most candidates if they have written a production job, should talk about
some type of alert mechanism like email is sent or there monitoring system sends an alert. Since Hadoop
works on unstructured data, it is very important to have a good alerting system for errors since unexpected
data can very easily break the job.

Compute Node: This is the computer or machine where your actual business logic will be executed.

Storage Node: This is the computer or machine where your file system reside to store the processing data.

In most of the cases compute node and storage node would be the same machine.

The MapReduce framework consists of a single master JobTracker and multiple slaves, each cluster-node
will have one TaskskTracker. The master is responsible for scheduling the jobs' component tasks on the
slaves, monitoring them and re-executing the failed tasks. The slaves execute the tasks as directed by the master.

Minimally an Hadoop application would have following components.

• Input location of data
• Output location of processed data.
• A map task.
• A reduced task.
• Job configuration

The Hadoop job client then submits the job (jar/executable etc.) and configuration to the JobTracker which
then assumes the responsibility of distributing the software/configuration to the slaves, scheduling tasks
and monitoring them, providing status and diagnostic information to the job-client.

The MapReduce framework operates exclusively on pairs, that is, the framework views the input to the job
as a set of pairs and produces a set of pairs as the output of the job, conceivably of different types. See
the flow mentioned below (input) -> map -> -> combine/sorting -> -> reduce -> (output).

The key and value classes have to be serialized by the framework. To make them serializable Hadoop
provides a Writable interface. As you know from the java itself that the key of the Map should be
comparable, hence the key has to implement one more interface WritableComparable.

No, Reducer runs in isolation.

The mapper output (intermediate data) is stored on the Local file system (NOT HDFS) of each individual
mapper nodes. This is typically a temporary directory location which can be setup in config by the Hadoop
administrator. The intermediate data is cleaned up after the Hadoop Job completes.