How to test Debian Hadoop

This article guides you to install and test Apache Hadoop on your Debian system. The following steps will provide detailed description of the configuration process and verification methods.

Step 1: Install Java

Make sure that the system has Java 8 or higher installed. Install OpenJDK 8 using the following command:

 sudo apt update
sudo apt install openjdk-8-jdk

Verify installation:

 java -version

Step 2: Download and decompress Hadoop

Download the latest version of Hadoop from the official website of Apache Hadoop and unzip it to the specified directory (for example /usr/local/hadoop ):

 wget https://downloads.apache.org/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz
tar -xzvf hadoop-3.3.1.tar.gz -C /usr/local/hadoop
``` (Please replace `hadoop-3.3.1` with the actual version number)


**Step 3: Configure environment variables**

Edit the `~/.bashrc` file and add the following environment variables:

```bash
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Make the changes take effect:

 source ~/.bashrc

Step 4: Configure Hadoop configuration file

Modify the configuration file in the Hadoop directory:

• core-site.xml :

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://namenode:9000</value>
  </property>
</configuration>

• hdfs-site.xml :

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/usr/local/hadoop/dfs/name</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/usr/local/hadoop/dfs/data</value>
  </property>
</configuration>

• mapred-site.xml :

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

• yarn-site.xml :

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>

Step 5: Format HDFS

On the NameNode node, execute the following command to format HDFS:

 hdfs namenode -format

Step 6: Start Hadoop Service

Start NameNode and DataNode on the NameNode node:

 start-dfs.sh

Start YARN on the ResourceManager node:

 start-yarn.sh

Step 7: Local Mode Testing

Switch to Hadoop user:

 su - hadoop

Create input directories and files:

 mkdir ~/input
vi ~/input/data.txt

Enter test data (for example "Hello World", "Hello Hadoop"), save and exit.

Run WordCount example:

 hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount ~/input/data.txt ~/output
``` (Please adjust it according to the actual jar package file name)

View the results:

```bash
ls ~/output
cat ~/output/part-r-00000

The correct output indicates that Hadoop local mode is running successfully. Please note that the above steps assume that you are testing in a stand-alone environment. For cluster environments, corresponding configuration modifications are required. Be sure to refer to the official Hadoop documentation for more detailed and latest configuration information.

The above is the detailed content of How to test Debian Hadoop. For more information, please follow other related articles on the PHP Chinese website!