Let's Play Analytics: Process Data with Apache Pig

What is Pig?

Pig is a high level scripting language that is used with Apache Hadoop. Pig excels at describing data analysis problems as data flows. Pig is complete in that you can do all the required data manipulations in Apache Hadoop with Pig. In addition through the User Defined Functions(UDF) facility in Pig you can have Pig invoke code in many languages like JRuby, Jython and Java. Conversely you can execute Pig scripts in other languages. The result is that you can use Pig as a component to build larger and more complex applications that tackle real business problems.

A good example of a Pig application is the ETL transaction model that describes how a process will extract data from a source, transform it according to a rule set and then load it into a datastore. Pig can ingest data from files, streams or other sources using the User Defined Functions(UDF). Once it has the data it can perform select, iteration, and other transforms over the data. Again the UDF feature allows passing the data to more complex algorithms for the transform. Finally Pig can store the results into the Hadoop Data File System.

Pig scripts are translated into a series of MapReduce jobs that are run on the Apache Hadoop cluster. As part of the translation the Pig interpreter does perform optimizations to speed execution on Apache Hadoop. We are going to write a Pig script that will do our data analysis task.

Lets take a look at how pig works using a problem statement.

Problem Statement

Compute the highest runs by a player for each year.

Data processing task

We are going to read in a baseball statistics file. We are going to compute the highest runs by a player for each year. This file has all the statistics from 1871–2011 and it contains over 90,000 rows. Once we have the highest runs we will extend the script to translate a player id field into the first and last names of the players.

Batting.csv - will give player with highest runs

Mster.csv - will give the player's personal information (common element : palyer id)

Steps

Download the zip file and extract it to get the master.csv and batting.csv files

Upload these two files into file browser of hue.

Go to Pig script console and give title to the script

Steps to write the pig script :

1. Load the data

2. Then we will filter the first row of the data

3. We use FOREACH statement to iterate through the batting data object and generate selected fields and assgin them names. The new data object we are creating is named as runs.

4. We use GROUP statement to group elements in runs by the year field.So, we created new data

object grp_data

5. We will again use FOREACH statement to find maximum runs for each year.

6. Now we have maximum runs. So, we join this with runs data object so we can pick up player Id.

7. The result will be a dataset with "year","playerId" and "max

runs".

8. At the end we use dump statement to generate the output.