Store files in two disks in Hadoop – Storage Reliability

Hi BigDs,

As I’m using single node cluster, I’m trying to store the copy in two different disks in my server, as directed by my instructor. Let’s see how.

Initially following is my data directory configured in hdfs-site.xml.

<property>
<name>dfs.data.dir</name>
<value>/var/lib/hadoop/cache/hadoop/dfs/data</value>
</property>

I’ll change it as below.

<property>
<name>dfs.data.dir</name>
<value>/var/lib/hadoop/cache/hadoop/dfs/data,/var/lib/hadoop/cache/hadoop/dfs/data2</value>
</property>

hadoop@gandhari:/var/lib/hadoop/cache/hadoop/dfs$ sudo mkdir data2

hadoop@gandhari:/var/lib/hadoop/cache/hadoop/dfs$ sudo chown hadoop:hadoop data2/

hadoop@gandhari:/var/lib/hadoop/cache/hadoop/dfs$ ls -alt
total 24
drwx------ 3 hadoop hadoop 4096 Sep  2 15:20 data
drwxr-xr-x 3 hadoop hadoop 4096 Sep  2 15:20 name
drwxr-xr-x 6 root   root   4096 Sep  2 15:19 .
drwxr-xr-x 2 hadoop hadoop 4096 Sep  2 15:19 data2
drwxr-xr-x 3 hadoop hadoop 4096 Aug 26 20:16 namesecondary
drwxr-xr-x 4 root   root   4096 Aug 20 09:52 ..

Stop and start the dfs. We can see the data2 folder is getting replicated.

hadoop@gandhari:/var/lib/hadoop/cache/hadoop/dfs$ ls /var/lib/hadoop/cache/hadoop/dfs/data2
current  in_use.lock

hadoop@gandhari:/var/lib/hadoop/cache/hadoop/dfs$ ls /var/lib/hadoop/cache/hadoop/dfs/data2/current/
BP-419586781-192.168.0.169-1471713542064  VERSION

data node log shows it is scanning in two folders now.

2016-09-02 15:29:48,346 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-419586781-192.168.0.169-1471713542064 on volume /var/lib/hadoop/cache/hadoop/dfs/data/current…

2016-09-02 15:29:48,407 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-419586781-192.168.0.169-1471713542064 on volume /var/lib/hadoop/cache/hadoop/dfs/data2/current..