Hi Hadoopers,
This blog post would demonstrate the filtering capabilities of Pig.
Filter by
This is similar to WHERE clause in SQL. We have a table for employee as below.
[code language=”bash”]
grunt> fs -cat employee.csv
1, Dharma, 45, Sr Manager
2, Bheema, 43, Cook
3, Arjuna, 41, Instructor
4, Nakula, 35, Jr Instructor
5, Sahadeva, 33, Jr Instructor
[/code]
Relation for employee is defined as below.
[code language=”bash”]
grunt> employee = load ’employee.csv’ using PigStorage (‘,’) as (eid:int,ename:chararray,eage:int,edes:chararray);
[/code]
Let’s get the employee whose id is 1.
[code language=”bash”]
e1 = filter employee by eid==1;
dump e1;
(1, Dharma,45, Sr Manager)
[/code]
Let’s get the employee whose name is Nakula.
[code language=”bash”]
e2 = filter employee by ename == ‘ Nakula’;
dump e2;
(4, Nakula,35, Jr Instructor)
[/code]
Distinct
As the name implies, let’s select the distinct records. Following is our input file.
[code language=”bash”]
grunt> fs -cat employee3.csv
1, Dharma, 45, Sr Manager
2, Bheema, 43, Cook
3, Arjuna, 41, Instructor
4, Nakula, 35, Jr Instructor
5, Sahadeva, 33, Jr Instructor
1, Dharma, 45, Sr Manager
2, Bheema, 43, Cook
3, Arjuna, 41, Instructor
4, Nakula, 35, Jr Instructor
5, Sahadeva, 33, Jr Instructor
[/code]
Lets create a relation for the employee.
[code language=”bash”]
grunt> employee = load ’employee3.csv’ using PigStorage (‘,’) as (eid:int,ename:chararray,eage:int,edes:chararray);
[/code]
We are creating another relation for the distinct records. Here you go.
[code language=”bash”]
grunt> edist = distinct employee;
grunt> dump edist;
(1, Dharma,45, Sr Manager)
(2, Bheema,43, Cook)
(3, Arjuna,41, Instructor)
(4, Nakula,35, Jr Instructor)
(5, Sahadeva,33, Jr Instructor)
[/code]
For-each
And finally, for – each.
Let’s iterate through the edist relation I have shown above.
[code language=”bash”]
grunt> foreachdata = foreach edist generate eid, ename;
(1, Dharma)
(2, Bheema)
(3, Arjuna)
(4, Nakula)
(5, Sahadeva)
[/code]