HANDLING HADOOP

What is Hadoop???

Hadoop is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

Hadoop alone cannot do wonders. It requires its other components for better performance. This is where PIG, Hive, Scoop, MapReduce and HBase come into action.

So, lets talk about MapReduce (MapR)

MapReduce is the key algorithm that the Hadoop MapReduce engine uses to distribute work around a cluster. Its working is divided in two parts : The Map and The Reduce.

A map transform is provided to transform an input data row of key and value to an output key/value:

map(key1,value) -> list<key2,value2>

For an input it returns a list containing zero or more (key,value) pairs:

The output can be a different key from the input

The output can have multiple entries with the same key

A reduce transform is provided to take all values for a specific key, and generate a new list of the reduced output.

reduce(key2, list<value2>) -> list<value3>

The key aspect of the MapReduce algorithm is that if every Map and Reduce is independent of all other ongoing Maps and Reduces, then the operation can be run in parallel on different keys and lists of data. On a large cluster of machines, we can go one step further, and run the Map operations on servers where the data lives. Rather than copy the data over the network to the program, we can push out the program to the machines. The output list can then be saved to the distributed filesystem, and the reducers run to merge the results. Again, it may be possible to run these in parallel, each reducing different keys.

A distributed filesystem spreads multiple copies of the data across different machines. This not only offers reliability without the need for RAID-controlled disks, it offers multiple locations to run the mapping. If a machine with one copy of the data is busy or offline, another machine can be used.

A job scheduler keeps track of which MR jobs are executing, schedules individual Maps, Reduces or intermediate merging operations to specific machines, monitors the success and failures of these individual Tasks, and works to complete the entire batch job.

The filesystem and Job scheduler can somehow be accessed by the people and programs that wish to read and write data, and to submit and monitor MR jobs.

Apache Hadoop is such a MapReduce engine. It provides its own distributed filesystem and runs jobs on servers near the data stored on the filesystem -or any other supported filesystem, of which there is more than one.

Limitations of MapReduce

• For maximum parallelism, we need the Maps and Reduces to be stateless, to not depend on any data generated in the same MapReduce job. We cannot control the order in which the maps run, or the reductions.

• It is very inefficient if we are repeating similar searches again and again. A database with an index will always be faster than running an MR job over unindexed data. However, if that index needs to be regenerated whenever data is added, and data is being added continually, MR jobs may have an edge. That inefficiency can be measured in both CPU time and power consumed.

• In the Hadoop implementation Reduce operations do not take place until all the Maps are complete (or have failed and been skipped). As a result, we do not get any data back until the entire mapping has finished.

• There is a general assumption that the output of the reduce is smaller than the input to the Map i.e. we are taking a large datasource and generating smaller final values.

Installing Hadoop

Here we have used HortonWorks Sandbox with HDP 2.3 to demonstrate an example of how HADOOP works. I have used it through Orcale VM Virtual Box.   

System Requirements: Just like the current-gen games, at least 6Gb of RAM  and i-3 processor is needed to it run decently. Once we start the virtual machine it takes around 4-5 minutes for the machine to be ready to be used. I am using 8Gb RAM and i-5 second gen processor.

Accessing Hadoop

Once the system is ready, Hadoop can be accessed on the browser at the address 127.0.0.1:8888.

In this page we have to access the Secure Shell Client at 127.0.0.1:4200.  

Check the screenshot

After Logging into SSH Client through "http://127.0.0.1:4200" we have to enter the login credentials.

By default,

Login : root 

pwd : hadoop

Now, starts the main course of action.

In this example to will try to find out the word count using a JAVA program and run it on HADOOP.

The steps to need to follow are:

• First we have to get the WordCount program in JAVA.

• Once we are in the shell terminal, we have to create a new directory WCclasses with the command.

                 mkdir WCclasses

• Once it is done we have to create three java files to run the WordCount program.

We have to use vi editor to create WordMapper.java, SumReducer.java and WordCount.java. 

Check the screenshots

• Now we have to compile them using javac compile command with the class files located in the WCclasses folder.

To  compile, we have to use:

• Now we have to create a jar file in the WCclasses directory

• Once the jar file is created we have to create the input directory in the hdfs file system using the below command.

hdfs dfs -mkdir /user/ru1

hdfs dfs -ls /user/ru1

hdfs dfs -mkdir /user/ru1/wc-inp

hdfs dfs -ls /user/ru1/wc-inp

• Upload the .txt files on hue

We can access HUE through 127.0.0.1:8000 to upload the .txt files. 

• Now we have to use the below code to run the HADOOP command.

hadoop jar WordCount.jar WordCount /user/ru1/wc-inp /user/ru1/wc-out41

• We can track the logs of all jobs at the address 127.0.0.1:8088.

• Finally, here is a sample of the output.

Troubles Faced

While working on this the troubles we faced are:

• While copying the java files to the vi editor. If any character goes missing, obviously it throws error. So, one has to tear his hair out if he is unaware of this problem.

• The next part is while compiling the java files because the version of the HortonWorks may vary. So, we have to be extra preventive while entering the path and the version.