HANDLING HADOOP

What is Hadoop???

Hadoop is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

Hadoop alone cannot do wonders. It requires its other components for better performance. This is where PIG, Hive, Scoop, MapReduce and HBase come into action.

So, lets talk about MapReduce (MapR)

MapReduce is the key algorithm that the Hadoop MapReduce engine uses to distribute work around a cluster. Its working is divided in two parts : The Map and The Reduce.

A map transform is provided to transform an input data row of key and value to an output key/value:

map(key1,value) -> list<key2,value2>

For an input it returns a list containing zero or more (key,value) pairs:

The output can be a different key from the input

The output can have multiple entries with the same key

A reduce transform is provided to take all values for a specific key, and generate a new list of the reduced output.

reduce(key2, list<value2>) -> list<value3>

The key aspect of the MapReduce algorithm is that if every Map and Reduce is independent of all other ongoing Maps and Reduces, then the operation can be run in parallel on different keys and lists of data. On a large cluster of machines, we can go one step further, and run the Map operations on servers where the data lives. Rather than copy the data over the network to the program, we can push out the program to the machines. The output list can then be saved to the distributed filesystem, and the reducers run to merge the results. Again, it may be possible to run these in parallel, each reducing different keys.

A distributed filesystem spreads multiple copies of the data across different machines. This not only offers reliability without the need for RAID-controlled disks, it offers multiple locations to run the mapping. If a machine with one copy of the data is busy or offline, another machine can be used.

A job scheduler keeps track of which MR jobs are executing, schedules individual Maps, Reduces or intermediate merging operations to specific machines, monitors the success and failures of these individual Tasks, and works to complete the entire batch job.

The filesystem and Job scheduler can somehow be accessed by the people and programs that wish to read and write data, and to submit and monitor MR jobs.

Apache Hadoop is such a MapReduce engine. It provides its own distributed filesystem and runs jobs on servers near the data stored on the filesystem -or any other supported filesystem, of which there is more than one.

Limitations of MapReduce

• For maximum parallelism, we need the Maps and Reduces to be stateless, to not depend on any data generated in the same MapReduce job. We cannot control the order in which the maps run, or the reductions.

• It is very inefficient if we are repeating similar searches again and again. A database with an index will always be faster than running an MR job over unindexed data. However, if that index needs to be regenerated whenever data is added, and data is being added continually, MR jobs may have an edge. That inefficiency can be measured in both CPU time and power consumed.

• In the Hadoop implementation Reduce operations do not take place until all the Maps are complete (or have failed and been skipped). As a result, we do not get any data back until the entire mapping has finished.

• There is a general assumption that the output of the reduce is smaller than the input to the Map i.e. we are taking a large datasource and generating smaller final values.

Installing Hadoop

Here we have used HortonWorks Sandbox with HDP 2.3 to demonstrate an example of how HADOOP works. I have used it through Orcale VM Virtual Box.   

System Requirements: Just like the current-gen games, at least 6Gb of RAM  and i-3 processor is needed to it run decently. Once we start the virtual machine it takes around 4-5 minutes for the machine to be ready to be used. I am using 8Gb RAM and i-5 second gen processor.

Accessing Hadoop

Once the system is ready, Hadoop can be accessed on the browser at the address 127.0.0.1:8888.

In this page we have to access the Secure Shell Client at 127.0.0.1:4200.  

Check the screenshot

After Logging into SSH Client through "http://127.0.0.1:4200" we have to enter the login credentials.

By default,

Login : root 

pwd : hadoop

Now, starts the main course of action.

In this example to will try to find out the word count using a JAVA program and run it on HADOOP.

The steps to need to follow are:

• First we have to get the WordCount program in JAVA.

• Once we are in the shell terminal, we have to create a new directory WCclasses with the command.

                 mkdir WCclasses

• Once it is done we have to create three java files to run the WordCount program.

We have to use vi editor to create WordMapper.java, SumReducer.java and WordCount.java. 

Check the screenshots

• Now we have to compile them using javac compile command with the class files located in the WCclasses folder.

To  compile, we have to use:

• Now we have to create a jar file in the WCclasses directory

• Once the jar file is created we have to create the input directory in the hdfs file system using the below command.

hdfs dfs -mkdir /user/ru1

hdfs dfs -ls /user/ru1

hdfs dfs -mkdir /user/ru1/wc-inp

hdfs dfs -ls /user/ru1/wc-inp

• Upload the .txt files on hue

We can access HUE through 127.0.0.1:8000 to upload the .txt files. 

• Now we have to use the below code to run the HADOOP command.

hadoop jar WordCount.jar WordCount /user/ru1/wc-inp /user/ru1/wc-out41

• We can track the logs of all jobs at the address 127.0.0.1:8088.

• Finally, here is a sample of the output.

Troubles Faced

While working on this the troubles we faced are:

• While copying the java files to the vi editor. If any character goes missing, obviously it throws error. So, one has to tear his hair out if he is unaware of this problem.

• The next part is while compiling the java files because the version of the HortonWorks may vary. So, we have to be extra preventive while entering the path and the version.

Play Python - Charts- Assignment 1

Let's learn how to use graphs and charts in Python. The first question is: Why do we need graphs and charts?? The answer- to visualize the data. Humans can understand the patterns and trends much more easily when presented with a graph or chart rather than dealing with the raw numbers.

Let's start with the graphs and charts in python.

code

Here is the simple code used to plot a graph of 7 numbers.

output

The output of the above code.

Let's try  with a different problem. The one I used in my previous blog. The problem is on radish survey from open tech school.

Here we have to plot a bar graph of the different variety of radishes present in the dataset.

code

I have used my previous code to load the data and count the votes for each variety of radishes.

Merged the code for the graph with it.

I have used two modules - 

One is "pyplot". It is used wit Matplotlib to plot the graph.

The other one is NumPy which  lots of numeric functions for Python.

Then I have used loop to process the dictionary into a format which is readable for matplotlib.

The next step is indexing of the X values in the graph. It is done with the help of the  dictionary "count", where each entry is read. It helps to spread the graph evenly across the X-axis.

The last step is to plot the graph. plot.xtics()  is used to to add labels to the X-axis like, the names of the varities of the radishes e.g. "White Icicle".

bar chart

So, from the chart, we can summarize all our findings easily without even looking at the raw numbers of the output. 

This is all what I have learnt about charts in python till now and will continue to explore and keep on learning.

Play Python - Assignment 1

Let's take a look at a versatile and high-level programming language, Python. Python is designed to be a highly readable language. It is designed to have an uncluttered visual layout, frequently using English keywords. Python has a smaller number of syntactic exceptions and special cases than most of the other languages. The beauty of Python acts as it's disadvantage too - The indentation. If we do not indent our code properly, it will throw errors. Let's jump into the basic programs of Python and check how it works.

For basic python tutorials, please visit open tech school. Even in this blog I have used problems from the same source.

The Problem Statement: Suppose you're a greengrocer, and you run a survey to see what radish varieties your customers prefer the most. You have your assistant type up the survey results into a text file on your computer, so you have 300 lines of survey data in the file radishsurvey.txt. Each line consists of a name, a hyphen, then a radish variety.

Dataset

We have to find out - 

• Which radish variety got the most votes (most popular)?
• What are the least popular radish varieties (got the least votes)?
• Did anyone vote twice?

Solution:

Code

 At first we created a dictionary to know the varieties of all the radishes and count their vote in a single pass.

Next , read the data from external file( .txt in this case).

Then we split each lines in two parts- names (the persons to voted) and vote(the name of the radishes) by splitting them across the "-".

To calculate the count of the radishes, we had to further process the data to remove any extra spaces and to avoid the repetition of a single variety of radish to to different cases. So, we converted the entries into upper case and marched ahead.

The next step is pretty simple, where we had to count the votes for each variety of radishes. If the variety is present in the dictionary, then +1 is added to the same, if not, a new entry is made and it's count starts from 1.

Output

The last step is to print the vote count of  the different varieties of radishes. I used pprint() instead of print() because it increases the readability.

Note: The output is in alphabetical order - One of the benefits of pprint().

The "Champion" got the highest votes: 76.

We can also get the solution of the second problem from the output.

The "Red King" variety has got the least votes.

Now we need to find if anyone voted more than once or not.

Code

This one is much simpler than the above problem.


Here, we processed the data to eliminate any extra spaces.




Then, used an if statement to check if the name of the voter is already in the "voted" dictionary, the name will be printed as already voted.










This is a small program to give you an idea how Python works, it's syntax and indentation. The next blog will be again on python and I will try to demonstrate how to use charts (at basic level).

Dating R

What is R?

The 18th letter of alphabet??? Of course it is!! But I am writing about the other R. One of the MVP of Analytics.

R is a programming language and software environment for statistical computing and graphics. The R language is widely used among statisticians and data miners for developing statistical software and data analysis.

(source: Wikipedia) 

 First Hands on Experience with R

Few days back  another noob (me) started to learn R. Thanks to "Google" and "TryR" for providing basic ideas about the commands, functions, syntax and all. 

So, I earned all the badges for R on http://tryr.codeschool.com.

Later, one of my professors suggested to go through https://statsguys.wordpress.com/2014/01/03/first-post/ for the taste of a typical data analytics problem used for prediction for a knowledge program available on KAGGLE.

The problem is to develop a binary classification model to predict which passengers survived the sinking Titanic. In the problem, at first we worked to build an algorithm based on the parameters set in the training dataset then we applied the same algorithm on the test data set to get predict the survivors. 

Lessons Learned

The code is already provided in the above mentioned link, so we didn't have to pull our hairs out or scratch our head. As already mentioned, I am new to this field and I didn't understand many functions used in the code but I was able to understand their approach to the problem. Thanks to STATGUYS again for explaining each steps in detail.

The first step to approach any problem is data exploration. It gives you an idea about the data you are dealing with.

The next and the most critical part as per my opinion is cleaning the data. Implementing this process, we can remove the unwanted variables from my dataset and keep it simple. We can also take care of any missing values which generally create huge problems in results or we encounter horrendous errors. We can make inferences on the missing values based on the other variables related to that missing value.

The third part is training the model we have created. We have a test data set. Using that one, we can train the model to provide optimized and desired output. I am still in dark about the training process (may be using neural network or any other process). Surely, I will dig deep into it and and find out the secret recipe. After the training is done, we can implement the model for the test data set and get the output.

Here as you can see, I made my first submission on KAGGLE. I had to submit the excel sheet that was generated as the output for the test data set.

Anyone who is new to R or want to start R, I would suggest all of you to visit all the above links. It will help you a lot.  Others, wish me luck, so that I can do better and submit my own work on KAGGLE soon.  

Thank You.

Neural... Getting Smarter!!!

 What is Neural Network?

What did we use to know when we were child? Nothing, right? As we grow, the learning process also starts. What is the learning process? We were taught and trained and then we learned.

Neural network is an interconnected group of nodes (similar to the neurons in brain) which is capable of making its own decisions and judgements.

Similar to human brains, we have have to train the neural network and make it learn. For example, if my inputs are 2,4,6, then the output must be 4,8,12. So, during the training process, once we develop the automatic learning mechanism, the neural network practices and starts learning on its own. Next time if the input is 199, it will automatically return 398 as the output.

My First Experience

I came to know about neural network when I designed a Controller to stabilize an Inverted Pendulum. In the project we used neural network to build a self-learning control system, where we had to train the neural network with huge amount of data- sets.

https://www.youtube.com/watch?v=r_viGB6-_MQ-  from this video, one can get the idea about how the inverted pendulum works.

Is Neural Network Getting Smarter?

The neural network has come a long way. It started as a learning machine which is capable of making its own call. Now, it has made its mark in the domain of Analytics. It is used in Predictive Analysis where it processes data from the past and present to estimate future values.

Is neural network really getting starter? We will get to know more about it in the upcoming blogs. Till then visit the below links for more interesting topics-

http://datasceptre.blogspot.in/
datadosage.blogspot.in
http://analyticsyatra.blogspot.in/
http://anuragbarman.blogspot.in

http://praxis.ac.in/programs/business-analytics/