How to quickly configure CentOS HDFS

Deploying Hadoop Distributed File System (HDFS) on a CentOS system requires several steps, and the following guide briefly describes the configuration process in stand-alone mode. Full cluster deployment is more complex.

1. Java environment configuration

First, make sure that the system has Java installed. Install OpenJDK using the following command:

 yum install -y java-1.8.0-openjdk-devel

Configure Java environment variables:

 echo "export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk" >> /etc/profile
echo "export PATH=$JAVA_HOME/bin:$PATH" >> /etc/profile
source /etc/profile
java -version

2. SSH password-free login settings

In order to communicate seamlessly between nodes, SSH password-free login is required.

1. Generate an SSH key pair:

 ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

2. Copy the public key to all nodes (here is only a stand-alone configuration, so this step is omitted):

3. Hadoop download and decompression

Download the Hadoop distribution from the Apache Hadoop official website and unzip it to the specified directory:

 wget https://downloads.apache.org/hadoop/core/hadoop-3.1.3/hadoop-3.1.3.tar.gz
tar -zxvf hadoop-3.1.3.tar.gz
mv hadoop-3.1.3 /opt/hadoop

4. Hadoop environment variable configuration

Edit the /etc/profile file and add the following environment variables:

 export HADOOP_HOME=/opt/hadoop
export PATH=$HADOOP_HOME/bin:$PATH
source /etc/profile

5. Hadoop configuration file modification

core-site.xml

Edit /opt/hadoop/etc/hadoop/core-site.xml , add the following (replace 192.168.1.1 with your host IP):

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://192.168.1.1:9000</value>
  </property>
</configuration>

hdfs-site.xml

Edit /opt/hadoop/etc/hadoop/hdfs-site.xml and add the following:

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/opt/hadoop/hdfs/namenode</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/opt/hadoop/hdfs/datanode</value>
  </property>
</configuration>

6. NameNode formatting

Format NameNode:

 /opt/hadoop/bin/hdfs namenode -format

7. HDFS startup

Start HDFS service:

 /opt/hadoop/sbin/start-dfs.sh

8. HDFS status verification

Check HDFS status:

 jps

You should see the NameNode and DataNode processes running.

9. HDFS Web UI Access

Visit http://192.168.1.1:50070 (replace 192.168.1.1 with your host IP) to view the HDFS web interface.

This guide is for reference only for stand-alone HDFS configuration. Multi-node cluster deployment requires additional configuration of ZooKeeper, Secondary NameNode, etc., and ensure that all node configuration files are consistent.

The above is the detailed content of How to quickly configure CentOS HDFS. For more information, please follow other related articles on the PHP Chinese website!