Hadoop pseudo-distributed cluster construction

Software preparation

I'm using a virtual machine with CentOS-6.6, with the host name repo. Refer to the steps to install a Linux virtual machine in Windows, I installed JDK in that virtual machine, refer to the guide to installing JDK in Linux. In addition, the virtual machine is configured with a key-free login itself, and the settings for configuring key-free login between each virtual machine are referenced. The download address of the Hadoop installation package is: https://www.php.cn/link/8485694bae96aebc7c4fe6119599d0e0 .

1. Upload the Hadoop installation package to the server and decompress it

    [root@repo ~]# tar zxvf hadoop-2.6.5.tar.gz -C /opt/apps/

2. Configure environment variables

    # You can directly locate the last line of the file [root@repo hadoop-2.6.5]# vi /etc/profile
   export HADOOP_HOME=/opt/apps/hadoop-2.6.5
   export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
   [root@repo hadoop-2.6.5]# . /etc/profile

3. Modify the three configuration files of hadoop-env.sh, mapred-env.sh, and yarn-env.sh, and add JAVA_HOME

    [root@repo hadoop]# pwd
   /opt/apps/hadoop-2.6.5/etc/hadoop
   [root@repo hadoop]# vi hadoop-env.sh
   export JAVA_HOME=/usr/local/jdk1.8.0_73
   [root@repo hadoop]# vi mapred-env.sh
   export JAVA_HOME=/usr/local/jdk1.8.0_73
   [root@repo hadoop]# vi yarn-env.sh
   export JAVA_HOME=/usr/local/jdk1.8.0_73

4. Modify the core-site.xml and hdfs-site.xml configuration files to configure pseudo-distributed related content

    [root@repo hadoop]# vi core-site.xml
   <configuration>
   <property>
    <name>fs.defaultFS</name>
    <value>hdfs://repo:9000</value>
   </property>
   <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/hadoop/pseudo</value>
   </property>
   </configuration>
   [root@repo hadoop]# vi hdfs-site.xml
   <configuration>
   <property>
    <name>dfs.replication</name>
    <value>1</value>
   </property>
   <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>repo:50090</value>
   </property>
   </configuration>

5. Modify the slaves configuration file and specify the node where the DataNode is located

    [root@repo hadoop]# vi slaves
   repo

6. Format file system

    [root@repo hadoop]# hadoop namenode --format
   # Success message17/09/16 21:17:11 INFO common.Storage: Storage directory /var/hadoop/pseudo/dfs/name has been successfully formatted.

7. Start HDFS and YARN

    [root@repo hadoop]# start-dfs.sh
   Starting namesnodes on [repo]
   repo: starting namenode, logging to /opt/apps/hadoop-2.6.5/logs/hadoop-root-namenode-repo.out
   repo: starting datanode, logging to /opt/apps/hadoop-2.6.5/logs/hadoop-root-datanode-repo.out
   Starting secondary namesodes [repo]
   repo: starting secondarynamenode, logging to /opt/apps/hadoop-2.6.5/logs/hadoop-root-secondarynamenode-repo.out
   [root@repo hadoop]# start-yarn.sh
   starting yarn daemons
   starting resourcemanager, logging to /opt/hadoop-2.7.4/logs/yarn-root-resourcemanager-repo.out
   repo: starting nodemanager, logging to /opt/hadoop-2.7.4/logs/yarn-root-nodemanager-repo.out
   [root@repo hadoop]# jps
   4368 Jps
   3957 ResourceManager
   3512 NameNode
   3641 DataNode
   4058 NodeManager
   3805 SecondaryNameNode

8. Visit the WEB page Build successfully!

The above is the detailed content of Hadoop pseudo-distributed cluster construction. For more information, please follow other related articles on the PHP Chinese website!