Monday, 22 July 2013

Set replication factor in HDFS

In the command line:
For existing file in HDFS:
To set replication of an individual file to 4:
     hadoop fs -setrep -w 4 /hdfs/path/tofile

You can also do this for directory recursively.
 To set replication for a directory to 1:
       hadoop fs -setrep -R -w 1 /hdfs/path/toDirectory
 To change replication of entire HDFS to 2:
    hadoop fs -setrep -R -w 2 /
 To copy new file with replication 2:
      hadoop fs -D dfs.replication=2 -copyFromLocal /local/path/tofile /hdfs/path/tofile

In Java program: For a file
 
 Configuration conf = new Configuration(); 
 FileSystem fs = FileSystem.get(conf);
 Path hdfsPath = new Path("/hdfs/hdfsFile");
 short replication =2;
 fs.setReplication(hdfsPath, replication);

returns 'true' if successful; 'false' if file does not exist or is a directory

No comments:

Post a Comment