Hi Hadoopers,
Partitioner is a powerful concept in MapReduce. I have given an example in Lab 06 – A simple hadoop partitioner
Let’s see how to implement same concept using Pig – It’s simple – using split!
Let’s take a simple employee table as below
[code language=”bash”]
grunt> fs -cat employee.csv
1, Dharma, 45, Sr Manager
2, Bheema, 43, Cook
3, Arjuna, 41, Instructor
4, Nakula, 35, Jr Instructor
5, Sahadeva, 33, Jr Instructor
[/code]
Let’s create the employee relation as given below.
[code language=”bash”]
grunt> employee = load ’employee.csv’ using PigStorage (‘,’) as (eid:int,ename:chararray,eage:int,edes:chararray);
[/code]
Now I’ll split employees based on age. Above 40, Below 40.
[code language=”bash”]
grunt> split employee into employee1 if eage>=40, employee2 if (eage<40);
grunt> dump employee1
(1, Dharma,45, Sr Manager)
(2, Bheema,43, Cook)
(3, Arjuna,41, Instructor)
grunt> dump employee2
(4, Nakula,35, Jr Instructor)
(5, Sahadeva,33, Jr Instructor)
[/code]
Note – You need to specify the else column – it is must.