Dear Sankar,

Hope you are doing well.

If data node prompts/warnings the disk space is exceeded, As a hadoop admin you can do the folllowing things:

1) If it's a multi node cluster you can try the hdfs balancer command to balance the datas among the all data nodes

sudo -u hdfs hdfs balancer

  • This runs the balancer with a default threshold of 10%, meaning that the script will ensure that disk usage on each DataNode differs from the overall usage in the cluster by no more than 10%. For example, if overall usage across all the DataNodes in the cluster is 40% of the cluster's total disk-storage capacity, the script ensures that each DataNode's disk usage is between 30% and 50% of that DataNode's disk-storage capacity.
  • You can run the script with a different threshold; for example:

  • sudo -u hdfs hdfs balancer -threshold 5

    This specifies that each DataNode's disk usage must be (or will be adjusted to be) within 5% of the cluster's overall usage.


    2) As a hadoop admin, you can also delete the unnecessary and corrupted files from the cluster

    hadoop fsck -delete

    3) You can also add more data nodes to increase the capacity of the cluster.

    Please download the attachment and go through it to add a datanode to the existing cluster.


    I think this will resolve the query.

    Please let me know if you have any further issue regarding this.

    Kindly share your feedback by clicking on either of the smiley's.

    Please note if you are not happy with the response on this ticket, please escalate it to escalations@edureka.in.
    We assure you that we will get back to you within 24 hours

     
    Regards,
    Suman Samanta
    edureka! Support Team
    On Wed, 11 May at 6:32 PM , Hadoop Administration at Edureka <hadoopadmin@edureka.co> wrote:
    Dear Sankar,

    Hope you are doing well.

    We have received your concern, our team is working on your query.

    Please allow us some time and we will get back to you at the earliest.

    Thank you for your patience and understanding.

    Feel free to contact us in case you have any other query.

    Please note if you are not happy with the response on this ticket, please escalate it to escalations@edureka.in.
    We assure you that we will get back to you within 24 hours
     


     
    Regards,
    Suman Samanta
    edureka! Support Team


    196998