Lab12 – Writing local file to HDFS

Hi Hadoopers,

You might have seen my earliest post on how to read from HDFS using APIs. Here is the post that can tell you how to write to HDFS.

Input file in HDD – /opt/hadoop/feed/output/2016-09-18
HDFS file – /user/hadoop/feed/2016-09-18

The following code needs two inputs. One is the local file and another one is HDFS file.

package org.grassfield.hadoop.input;

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileReader;
import java.io.IOException;
import java.io.OutputStreamWriter;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

/**
 * upload local file to HDFS
 * 
 * @author pandian
 *
 */
public class LoadItemsHdfs {

    /**
     * @param args    localFile remoteFile
     * @throws IOException
     */
    public static void main(String[] args) throws IOException {
        Path path = new Path("hdfs://gandhari:9000"+args[1]);
        FileSystem fs = FileSystem.get(new Configuration());
        BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(fs.create(path, true)));
        BufferedReader br = new BufferedReader(new FileReader(args[0]));
        String line = null;
        while((line=br.readLine())!=null){
            bw.write(line);
            bw.write('\n');
        }
        br.close();
        bw.flush();
        bw.close();
        fs.close();
    }

}

Let’s execute it.

hadoop@gandhari:~/jars$ hadoop jar FeedCategoryCount-9.jar org.grassfield.hadoop.input.LoadItemsHdfs ../feed/output/2016-09-18 /user/hadoop/feed/2016-09-18
hadoop@gandhari:~/jars$ hadoop fs -ls /user/hadoop/feed
Found 1 items
-rw-r--r--   3 hadoop supergroup     120817 2016-09-18 06:51 /user/hadoop/feed/2016-09-18
hadoop@gandhari:~/jars$ hadoop fs -cat /user/hadoop/feed/2016-09-18

application/rss+xml     Today Online - Hot news null    http://www.todayonline.com/authors/ku-swee-yong null    Rosberg in pole position to claim victory on Sunday
 Today   http://www.todayonline.com/sports/motor-racing/rosberg-pole-position-claim-victory-sunday       Sat Sep 17 23:44:59 MYT 2016    []
 application/rss+xml     Today Online - Hot news null    http://www.todayonline.com/authors/ku-swee-yong null    No slowing Tang down despite qualifying setback Today
 http://www.todayonline.com/sports/motor-racing/no-slowing-tang-down-despite-qualifying-setback  Sat Sep 17 22:09:20 MYT 2016    []