Hadoop 单机部署
本安装过程是对
Click Here ! 所述内容的复现。详情请见原网址。
连接服务器
使用 OpenSSH 连接至 ECS（CentOS7），并以 root 身份登录（不推荐）。
+----------------------------------------------------------------------------------+
| PS C:\Users\Lalala> ssh [email protected] |
| Last login: Tue Oct 8 19:08:05 2024 from 117.154.xxx.57 |
| |
| Welcome to Alibaba Cloud Elastic Compute Service ! |
+----------------------------------------------------------------------------------+
查看系统信息：
________________________________________________________________________________________
| ANSI | - | o | x |
|========================================================================================|
| lalala@Alibaba-Cloud-Elastic-Compute-Service:~$ neofetch --ascii_distro centos |
| .. lalala@Alibaba-Cloud-Elastic-Compute-Service |
| .PLTJ. -------------------------------------------- |
| <><><><> OS: Fedora release 40 (Forty) x86_64 |
| KKSSV' 4KKK LJ KKKL.'VSSKK Host: Alibaba Cloud ECS pc-i440fx-2.1 |
| KKV' 4KKKKK LJ KKKKAL 'VKK Kernel: 6.10.10-200.fc40.x86_64 |
| V' ' 'VKKKK LJ KKKKV' ' 'V Uptime: 11 days, 1 hour, 12 mins |
| .4MA.' 'VKK LJ KKV' '.4Mb. Packages: 1017 (rpm) |
| . KKKKKA.' 'V LJ V' '.4KKKKK . Shell: bash 5.2.26 |
| .4D KKKKKKKA.'' LJ ''.4KKKKKKK FA. Resolution: 1024x768 |
| <QDD ++++++++++++ ++++++++++++ GFD> Terminal: /dev/pts/1 |
| 'VD KKKKKKKK'.. LJ ..'KKKKKKKK FV CPU: Intel Xeon Platinum (2) @ 2.500GHz |
| ' VKKKKK'. .4 LJ K. .'KKKKKV ' GPU: 00:02.0 Cirrus Logic GD 5446 |
| 'VK'. .4KK LJ KKA. .'KV' Memory: 338MiB / 1701MiB |
| A. . .4KKKK LJ KKKKA. . .4 |
| KKA. 'KKKKK LJ KKKKK' .4KK |
| KKSSA. VKKK LJ KKKV .4SSKK |
| <><><><> |
| 'MKKM' |
| '' |
| |
| |
| [root@ecs ~]# neofetch |
| .. root@ecs |
| .PLTJ. -------- |
| <><><><> OS: CentOS Linux release 7.9.2009 (Core) x86_64 |
| KKSSV' 4KKK LJ KKKL.'VSSKK Host: Alibaba Cloud ECS pc-i440fx-2.1 |
| KKV' 4KKKKK LJ KKKKAL 'VKK Kernel: 3.10.0-1160.92.1.el7.x86_64 |
| V' ' 'VKKKK LJ KKKKV' ' 'V Uptime: 5 days, 18 hours, 32 mins |
| .4MA.' 'VKK LJ KKV' '.4Mb. Packages: 449 (rpm) |
| . KKKKKA.' 'V LJ V' '.4KKKKK . Shell: bash 4.2.46 |
| .4D KKKKKKKA.'' LJ ''.4KKKKKKK FA. Resolution: 1024x768 |
| <QDD ++++++++++++ ++++++++++++ GFD> Terminal: /dev/pts/0 |
| 'VD KKKKKKKK'.. LJ ..'KKKKKKKK FV CPU: Intel Xeon Platinum (2) @ 2.500GHz |
| ' VKKKKK'. .4 LJ K. .'KKKKKV ' GPU: Cirrus Logic QEMU Virtual Machine |
| 'VK'. .4KK LJ KKA. .'KV' Memory: 285MiB / 1738MiB |
| A. . .4KKKK LJ KKKKA. . .4 |
| KKA. 'KKKKK LJ KKKKK' .4KK |
| KKSSA. VKKK LJ KKKV .4SSKK |
| <><><><> |
| 'MKKM' |
| '' |
'========================================================================================'
安装依赖
本次要安装的 Hadoop 目标版本为 3.4.0，release date 是 2024-03-17。
通过查阅
Click here 得知该版本推荐的 java version 是 8 或者 11。
由于 ECS 上已经安装了 OpenSSH，所以不用再次安装。接下来，根据 Required Software List，需要安装的软件是：
Softwall YUM Package Name APT Package Name
-------- ---------------- ----------------
pdsh pdsh pdsh
OpenJDK 11 java-11-openjdk.x86_64 openjdk-11-jdk
+----------------------------------------------------------------------------------+
| yes | sudo yum install pdsh java-11-openjdk.x86_64 |
+----------------------------------------------------------------------------------+
安装完成后查看版本信息。
+----------------------------------------------------------------------------------+
| [root@ecs ~]# pdsh -V |
| pdsh-2.34 (+readline+debug) |
| rcmd modules: ssh,exec (default: ssh) |
| misc modules: (none) |
| [root@ecs ~]# java --version |
| openjdk 11.0.23 2024-04-16 LTS |
| OpenJDK Runtime Environment (Red_Hat-11.0.23.0.9-2.el7_9) (build 11.0.23+9-LTS) |
| OpenJDK 64-Bit Server VM (Red_Hat-11.0.23.0.9-2.el7_9) (build 11.0.23+9-LTS, mix |
| ed mode, sharing) |
+----------------------------------------------------------------------------------+
下载 Hadoop
下载发行版本。
https://dlcdn.apache.org/hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz
但是下载速度有限，所以采用阿里云镜像仓库中托管的版本：
https://mirrors.aliyun.com/apache/hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz
+----------------------------------------------------------------------------------+
| [root@ecs ~]# wget https://mirrors.aliyun.com/apache/hadoop/common/hadoop-3.4.0/ |
| hadoop-3.4.0.tar.gz |
+----------------------------------------------------------------------------------+
查看下载的软件。
+----------------------------------------------------------------------------------+
| [root@ecs hadoop]# ll -h |
| total 921M |
| -rw-r--r-- 1 root root 921M Oct 10 16:16 hadoop-3.4.0.tar.gz |
+----------------------------------------------------------------------------------+
验证校验和：
+----------------------------------------------------------------------------------+
| [root@ecs hadoop]# sha512sum hadoop-3.4.0.tar.gz |
| 6f653c0109f97430047bd3677c50da7c8a2809d153b231794cf980b3208a6b4beff8ff1a03a01094 |
| 299d459a3a37a3fe16731629987165d71f328657dbf2f24c hadoop-3.4.0.tar.gz |
| [root@ecs hadoop]# curl https://dlcdn.apache.org/hadoop/common/hadoop-3.4.0/hado |
| op-3.4.0.tar.gz.sha512 |
| SHA512 (hadoop-3.4.0.tar.gz) = 6f653c0109f97430047bd3677c50da7c8a2809d153b231794 |
| cf980b3208a6b4beff8ff1a03a01094299d459a3a37a3fe16731629987165d71f328657dbf2f24c |
+----------------------------------------------------------------------------------+
解压并去归档。
+----------------------------------------------------------------------------------+
| [root@ecs hadoop]# tar -xf hadoop-3.4.0.tar.gz |
| [root@ecs hadoop]# ls |
| hadoop-3.4.0 hadoop-3.4.0.tar.gz |
+----------------------------------------------------------------------------------+
# 配置 Hadoop
查找 JAVA_HOME（即 jdk 安装位置）的路径：
Method1：
通过 java -verbose 的输出查询
+----------------------------------------------------------------------------------+
| [root@ecs alternatives]# java -verbose |
| [0.017s][info][class,load] opened: /usr/lib/jvm/java-11-openjdk-11.0.23.0.9-2.el |
| 7_9.x86_64/lib/modules |
| ... |
+----------------------------------------------------------------------------------+
Method2：
通过 which 工具寻找java可执行文件的位置，通过 ls 或 readlink 工具查询符号连接指向，直至指向原始文件，从而获知安装位置
+----------------------------------------------------------------------------------+
| [root@ecs hadoop]# which java |
| /usr/bin/java |
| [root@ecs hadoop]# readlink /usr/bin/java |
| /etc/alternatives/java |
| [root@ecs alternatives]# readlink /etc/alternatives/java |
| /usr/lib/jvm/java-11-openjdk-11.0.23.0.9-2.el7_9.x86_64/bin/java |
+----------------------------------------------------------------------------------+
修改 etc/hadoop/hadoop-env.sh 中的属性：
+----------------------------------------------------------------------------------+
| # The java implementation to use. By default, this environment |
| # variable is REQUIRED on ALL platforms except OS X! |
| export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.23.0.9-2.el7_9.x86_64/ |
+----------------------------------------------------------------------------------+
启动 Hadoop
验证安装：
+----------------------------------------------------------------------------------+
| [root@ecs hadoop]# bin/hadoop |
+----------------------------------------------------------------------------------+
This will display the usage documentation for the hadoop script.
Reference
Refer to the following page:
• https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html
• https://docs.pingcode.com/baike/369554
• https://packages.ubuntu.com/
+----------------------------------------------------------------------------------+
| [root@ecs bin]# ./hadoop |
| Usage: hadoop [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS] |
| or hadoop [OPTIONS] CLASSNAME [CLASSNAME OPTIONS] |
| where CLASSNAME is a user-provided Java class |
| |
| OPTIONS is none or any of: |
| |
| buildpaths attempt to add class files from build tree |
| --config dir Hadoop config directory |
| --debug turn on shell script debug mode |
| --help usage information |
| hostnames list[,of,host,names] hosts to use in worker mode |
| hosts filename list of hosts to use in worker mode |
| loglevel level set the log4j level for this command |
| workers turn on worker mode |
| |
| SUBCOMMAND is one of: |
| |
| |
| Admin Commands: |
| |
| daemonlog get/set the log level for each daemon |
| |
| Client Commands: |
| |
| archive create a Hadoop archive |
| checknative check native Hadoop and compression libraries availability |
| classpath prints the class path needed to get the Hadoop jar and the require |
| d libraries |
| conftest validate configuration XML files |
| credential interact with credential providers |
| distch distributed metadata changer |
| distcp copy file or directories recursively |
| dtutil operations related to delegation tokens |
| envvars display computed Hadoop environment variables |
| fedbalance balance data between sub-clusters |
| fs run a generic filesystem user client |
| gridmix submit a mix of synthetic job, modeling a profiled from production |
| load |
| jar <jar> run a jar file. NOTE: please use "yarn jar" to launch YARN applica |
| tions, not this command. |
| jnipath prints the java.library.path |
| kdiag Diagnose Kerberos Problems |
| kerbname show auth_to_local principal conversion |
| key manage keys via the KeyProvider |
| rbfbalance move directories and files across router-based federation namespac |
| es |
| rumenfolder scale a rumen input trace |
| rumentrace convert logs into a rumen trace |
| s3guard S3 Commands |
| version print the version |
| |
| Daemon Commands: |
| |
| kms run KMS, the Key Management Server |
| registrydns run the registry DNS server |
| |
| SUBCOMMAND may print help when invoked w/o parameters or with -h. |
+----------------------------------------------------------------------------------+
Create: Thu Dec 12 22:31:42 2024
Last Modified: Thu Dec 12 22:30:42 2024
_____ _______ _____ _______
/\ \ /::\ \ /\ \ /::\ \
/::\____\ /::::\ \ /::\____\ /::::\ \
/::::| | /::::::\ \ /::::| | /::::::\ \
/:::::| | /::::::::\ \ /:::::| | /::::::::\ \
/::::::| | /:::/~~\:::\ \ /::::::| | /:::/~~\:::\ \
/:::/|::| | /:::/ \:::\ \ /:::/|::| | /:::/ \:::\ \
/:::/ |::| | /:::/ / \:::\ \ /:::/ |::| | /:::/ / \:::\ \
/:::/ |::|___|______ /:::/____/ \:::\____\ /:::/ |::| | _____ /:::/____/ \:::\____\
/:::/ |::::::::\ \ |:::| | |:::| | /:::/ |::| |/\ \ |:::| | |:::| |
/:::/ |:::::::::\____\|:::|____| |:::|____|/:: / |::| /::\____\|:::|____| |:::|____|
\::/ / ~~~~~/:::/ / \:::\ \ /:::/ / \::/ /|::| /:::/ / \:::\ \ /:::/ /
\/____/ /:::/ / \:::\ \ /:::/ / \/____/ |::| /:::/ / \:::\ \ /:::/ /
/:::/ / \:::\ /:::/ / |::|/:::/ / \:::\ /:::/ /
/:::/ / \:::\__/:::/ / |::::::/ / \:::\__/:::/ /
/:::/ / \::::::::/ / |:::::/ / \::::::::/ /
/:::/ / \::::::/ / |::::/ / \::::::/ /
/:::/ / \::::/ / /:::/ / \::::/ /
/:::/ / \::/____/ /:::/ / \::/____/
\::/ / \::/ /
\/____/ \/____/
_____ _____ _____ _____ _____
/\ \ /\ \ /\ \ /\ \ /\ \
/::\ \ /::\ \ /::\ \ /::\ \ /::\ \
/::::\ \ /::::\ \ /::::\ \ /::::\ \ /::::\ \
/::::::\ \ /::::::\ \ /::::::\ \ /::::::\ \ /::::::\ \
/:::/\:::\ \ /:::/\:::\ \ /:::/\:::\ \ /:::/\:::\ \ /:::/\:::\ \
/:::/__\:::\ \ /:::/__\:::\ \ /:::/__\:::\ \ /:::/ \:::\ \ /:::/__\:::\ \
\:::\ \:::\ \ /::::\ \:::\ \ /::::\ \:::\ \ /:::/ \:::\ \ /::::\ \:::\ \
___\:::\ \:::\ \ /::::::\ \:::\ \ /::::::\ \:::\ \ /:::/ / \:::\ \ /::::::\ \:::\ \
/\ \:::\ \:::\ \ /:::/\:::\ \:::\____\ /:::/\:::\ \:::\ \ /:::/ / \:::\ \ /:::/\:::\ \:::\ \
/::\ \:::\ \:::\____\/:::/ \:::\ \:::| |/:::/ \:::\ \:::\____\/:::/____/ \:::\____\/:::/__\:::\ \:::\____\
\:::\ \:::\ \::/ /\::/ \:::\ /:::|____|\::/ \:::\ /:::/ /\:::\ \ \::/ /\:::\ \:::\ \::/ /
\:::\ \:::\ \/____/ \/_____/\:::\/:::/ / \/____/ \:::\/:::/ / \:::\ \ \/____/ \:::\ \:::\ \/____/
\:::\ \:::\ \ \::::::/ / \::::::/ / \:::\ \ \:::\ \:::\ \
\:::\ \:::\____\ \::::/ / \::::/ / \:::\ \ \:::\ \:::\____\
\:::\ /:::/ / \::/____/ /:::/ / \:::\ \ \:::\ \::/ /
\:::\/:::/ / /:::/ / \:::\ \ \:::\ \/____/
\::::::/ / /:::/ / \:::\ \ \:::\ \
\::::/ / /:::/ / \:::\____\ \:::\____\
\::/ / \::/ / \::/ / \::/ /
\/____/ \/____/ \/____/ \/____/