docker配置全分布式hadoop（5台容器两台主节点，三台从节点） ...

丝  金牌会员 | 2024-7-14 18:36:19 | 显示全部楼层 | 阅读模式
打印 上一主题 下一主题
楼主

主题 636|帖子 636|积分 1908

Docker容器搭建hadoop

集群规划

master: 主节点, 运行NameNode和ResourceManager
master2: 主节点, 运行SecondaryNameNode
worker1、worker2、worker3: 从节点, 运行DataNode和NodeManager
一、把jdk传到假造机上，提前准备好Dockerfile文件，hadoop配置文件（Dockerfile所在路径创建目录etc/hadoop,需要改动的hadoop配置文件放置在该目录下）（我是在/root下）

Dockerfile文件的内容

  1. # syntax=docker/dockerfile:1
  3. # 参考资料: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html
  5. FROM ubuntu:20.04
  7. ARG TARBALL=hadoop-3.4.0.tar.gz
  8. # 提前下载好的java8压缩包, 下载地址: https://www.oracle.com/java/technologies/downloads/
  9. ARG JAVA_TARBALL=jdk-8u212-linux-x64.tar.gz
  11. ENV HADOOP_HOME /app/hadoop
  12. ENV JAVA_HOME /usr/java
  14. WORKDIR $JAVA_HOME
  15. WORKDIR $HADOOP_HOME
  17. RUN sed -i s@/archive.ubuntu.com/@/mirrors.aliyun.com/@g /etc/apt/sources.list && \
  18.     apt-get clean && \
  19.     apt-get update && \
  20.     apt-get upgrade -y && \
  21.     apt-get install -y --no-install-recommends \
  22.     wget \
  23.     ssh
  25. # 拷贝jdk8安装包
  26. COPY ${JAVA_TARBALL} ${JAVA_HOME}/${JAVA_TARBALL}
  28. RUN tar -zxvf /usr/java/${JAVA_TARBALL} --strip-components 1 -C /usr/java && \
  29.     rm /usr/java/${JAVA_TARBALL} && \
  30.     # 设置java8环境变量
  31.     echo export JAVA_HOME=${JAVA_HOME} >> ~/.bashrc && \
  32.     echo export PATH=\$PATH:\$JAVA_HOME/bin >> ~/.bashrc && \
  33.     echo export JAVA_HOME=${JAVA_HOME} >> /etc/profile && \
  34.     echo export PATH=\$PATH:\$JAVA_HOME/bin >> /etc/profile && \
  35.     # 下载hadoop安装包
  36.     wget --no-check-certificate https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/stable/${TARBALL} && \
  37.     # 解压hadoop安装包
  38.     tar -zxvf ${TARBALL} --strip-components 1 -C $HADOOP_HOME && \
  39.     rm ${TARBALL} && \
  40.     # 设置从节点
  41.     echo "worker1\nworker2\nworker3" > $HADOOP_HOME/etc/hadoop/workers && \
  42.     echo export HADOOP_HOME=${HADOOP_HOME} >> ~/.bashrc && \
  43.         echo export PATH=\$PATH:\$HADOOP_HOME/bin >> ~/.bashrc && \
  44.         echo export PATH=\$PATH:\$HADOOP_HOME/sbin >> ~/.bashrc && \
  45.         echo export HADOOP_HOME=${HADOOP_HOME} >> /etc/profile && \
  46.         echo export PATH=\$PATH:\$HADOOP_HOME/bin >> /etc/profile && \
  47.         echo export PATH=\$PATH:\$HADOOP_HOME/sbin >> /etc/profile && \
  48.     mkdir /app/hdfs && \
  49.     # java8软连接
  50.     ln -s $JAVA_HOME/bin/java /bin/java
  52. # 拷贝hadoop配置文件
  53. COPY ./etc/hadoop/hdfs-site.xml $HADOOP_HOME/etc/hadoop/hdfs-site.xml
  54. COPY ./etc/hadoop/core-site.xml $HADOOP_HOME/etc/hadoop/core-site.xml
  55. COPY ./etc/hadoop/mapred-site.xml $HADOOP_HOME/etc/hadoop/mapred-site.xml
  56. COPY ./etc/hadoop/yarn-site.xml $HADOOP_HOME/etc/hadoop/yarn-site.xml
  58. # 设置hadoop环境变量
  59. RUN echo export JAVA_HOME=$JAVA_HOME >> $HADOOP_HOME/etc/hadoop/hadoop-env.sh && \
  60.     echo export HADOOP_MAPRED_HOME=$HADOOP_HOME >> $HADOOP_HOME/etc/hadoop/hadoop-env.sh && \
  61.     echo export HDFS_NAMENODE_USER=root >> $HADOOP_HOME/etc/hadoop/hadoop-env.sh && \
  62.     echo export HDFS_DATANODE_USER=root >> $HADOOP_HOME/etc/hadoop/hadoop-env.sh && \
  63.     echo export HDFS_SECONDARYNAMENODE_USER=root >> $HADOOP_HOME/etc/hadoop/hadoop-env.sh && \
  64.     echo export YARN_RESOURCEMANAGER_USER=root >> $HADOOP_HOME/etc/hadoop/hadoop-env.sh && \
  65.     echo export YARN_NODEMANAGER_USER=root >> $HADOOP_HOME/etc/hadoop/hadoop-env.sh
  66. # ssh免登录设置
  67. RUN echo "/etc/init.d/ssh start" >> ~/.bashrc && \
  68.     ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa && \
  69.     cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && \
  70.     chmod 0600 ~/.ssh/authorized_keys
  72. # NameNode WEB UI服务端口
  73. EXPOSE 9870
  74. # nn文件服务端口
  75. EXPOSE 9000
  76. # dfs.namenode.secondary.http-address
  77. EXPOSE 9868
  78. # dfs.datanode.http.address
  79. EXPOSE 9864
  80. # dfs.datanode.address
  81. EXPOSE 9866
复制代码


core-site.xml

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <!--
  4.   Licensed under the Apache License, Version 2.0 (the "License");
  5.   you may not use this file except in compliance with the License.
  6.   You may obtain a copy of the License at
  8.     http://www.apache.org/licenses/LICENSE-2.0
  10.   Unless required by applicable law or agreed to in writing, software
  11.   distributed under the License is distributed on an "AS IS" BASIS,
  12.   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  13.   See the License for the specific language governing permissions and
  14.   limitations under the License. See accompanying LICENSE file.
  15. -->
  17. <!-- Put site-specific property overrides in this file. -->
  19. <configuration>
  20.     <property>
  21.         <!-- 设置NameNode对外提供HDFS服务 -->
  22.         <name>fs.defaultFS</name>
  23.         <value>hdfs://hadoop330:9000</value>
  24.     </property>
  25.     <property>
  26.         <!-- 指明是否开启用户认证 -->
  27.         <name>hadoop.security.authorization</name>
  28.         <value>false</value>
  29.     </property>
  30. </configuration>
复制代码
hdfs-site.xml

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <!--
  3.   Licensed under the Apache License, Version 2.0 (the "License");
  4.   you may not use this file except in compliance with the License.
  5.   You may obtain a copy of the License at
  7.     http://www.apache.org/licenses/LICENSE-2.0
  9.   Unless required by applicable law or agreed to in writing, software
  10.   distributed under the License is distributed on an "AS IS" BASIS,
  11.   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  12.   See the License for the specific language governing permissions and
  13.   limitations under the License. See accompanying LICENSE file.
  14. -->
  16. <!-- Put site-specific property overrides in this file. -->
  18. <configuration>
  19.     <property>
  20.         <!-- 文件副本数量, 默认是3 -->
  21.         <name>dfs.replication</name>
  22.         <value>3</value>
  23. <!--        <value>1</value>-->
  24.     </property>
  25.     <property>
  26.         <!-- 是否启用文件操作权限, 不启用可以以普通账户写操作HDFS文件和目录 -->
  27.         <name>dfs.permissions</name>
  28.         <value>false</value>
  29.     </property>
  30.     <property>
  31.         <!-- NameNode存储数据的文件所在的路径 -->
  32.         <name>dfs.namenode.name.dir</name>
  33.         <value>/app/hdfs/namenode</value>
  34.     </property>
  35.     <property>
  36.         <!--  DataNode存储数据的文件路径 -->
  37.         <name>dfs.datanode.data.dir</name>
  38.         <value>/app/hdfs/datanode</value>
  39.     </property>
  40.     <property>
  41.         <!--  设置SecondNameNode节点地址 -->
  42.         <name>dfs.namenode.secondary.http-address</name>
  43.         <value>master2:50090</value>
  44.     </property>
  45. </configuration>
复制代码
mapred-site.xml

  1. <?xml version="1.0"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <!--
  4.   Licensed under the Apache License, Version 2.0 (the "License");
  5.   you may not use this file except in compliance with the License.
  6.   You may obtain a copy of the License at
  8.     http://www.apache.org/licenses/LICENSE-2.0
  10.   Unless required by applicable law or agreed to in writing, software
  11.   distributed under the License is distributed on an "AS IS" BASIS,
  12.   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  13.   See the License for the specific language governing permissions and
  14.   limitations under the License. See accompanying LICENSE file.
  15. -->
  17. <!-- Put site-specific property overrides in this file. -->
  19. <configuration>
  20.     <property>
  21.         <name>mapreduce.framework.name</name>
  22.         <value>yarn</value>
  23.     </property>
  24.     <property>
  25.         <name>mapreduce.application.classpath</name>
  26.         <value>
  27.             $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
  28.         </value>
  29.     </property>
  30.     <property>
  31.         <name>yarn.app.mapreduce.am.env</name>
  32.         <value>HADOOP_MAPRED_HOME=/app/hadoop</value>
  33.     </property>
  34.     <property>
  35.         <name>mapreduce.map.env</name>
  36.         <value>HADOOP_MAPRED_HOME=/app/hadoop</value>
  37.     </property>
  38.     <property>
  39.         <name>mapreduce.reduce.env</name>
  40.         <value>HADOOP_MAPRED_HOME=/app/hadoop</value>
  41.     </property>
  42. </configuration>
复制代码
yarn-site.xml

  1. <?xml version="1.0"?>
  2. <!--
  3.   Licensed under the Apache License, Version 2.0 (the "License");
  4.   you may not use this file except in compliance with the License.
  5.   You may obtain a copy of the License at
  7.     http://www.apache.org/licenses/LICENSE-2.0
  9.   Unless required by applicable law or agreed to in writing, software
  10.   distributed under the License is distributed on an "AS IS" BASIS,
  11.   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  12.   See the License for the specific language governing permissions and
  13.   limitations under the License. See accompanying LICENSE file.
  14. -->
  15. <configuration>
  17.     <!-- Site specific YARN configuration properties -->
  19.     <property>
  20.         <name>yarn.nodemanager.aux-services</name>
  21.         <value>mapreduce_shuffle</value>
  22.     </property>
  23.     <property>
  24.         <name>yarn.nodemanager.env-whitelist</name>
  25.         <value>
  26.             JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,
  27.             CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
  28.         </value>
  29.     </property>
  30.     <property>
  31.         <name>yarn.resourcemanager.address</name>
  32.         <value>hadoop330:8032</value>
  33.     </property>
  34.     <property>
  35.         <name>yarn.resourcemanager.scheduler.address</name>
  36.         <value>hadoop330:8030</value>
  37.     </property>
  38.     <property>
  39.         <name>yarn.resourcemanager.resource-tracker.address</name>
  40.         <value>hadoop330:8031</value>
  41.     </property>
  42. </configuration>
复制代码
二、java8和hadoop配置文件准备好后，从Dockerfile开始构建hadoop基础镜像

  1. docker build -t hadoop .
复制代码

三、编写docker-compose.yml文件，执行docker-compose启动5个容器

docker-compose.yml

  1. version: '3'
  2. services:
  3.   master1:
  4.     image: hadoop
  5.     stdin_open: true
  6.     tty: true
  7.     command: bash
  8.     hostname: hadoop330
  9.     ports:
  10.       - "9000:9000"
  11.       - "9870:9870"
  12.       - "8088:8088"
  13.   master2:
  14.     image: hadoop
  15.     stdin_open: true
  16.     tty: true
  17.     command: bash
  18.   worker1:
  19.     image: hadoop
  20.     stdin_open: true
  21.     tty: true
  22.     command: bash
  23.   worker2:
  24.     image: hadoop
  25.     stdin_open: true
  26.     tty: true
  27.     command: bash
  28.   worker3:
  29.     image: hadoop
  30.     stdin_open: true
  31.     tty: true
  32.     command: bash
复制代码

!!!假如没有下载docker-compose，依次执行以下命令

  1. yum -y install epel-release
  2. yum install python-pip
  3. wget https://github.com/docker/compose/releases/download/1.14.0-rc2/docker-compose-Linux-x86_64
  4. mv docker-compose-Linux-x86_64 /usr/local/bin/docker-compose
  5. chmod +x /usr/local/bin/docker-compose
  6. docker-compose -version
复制代码

执行docker-compose启动5个容器

  1. docker-compose up -d
复制代码

四、启动hadoop集群和测试验证

观察本身容器名字，我的主节点容器名字是root_master1_1


进入主节点容器

  1. docker exec -it root_master1_1 bash
复制代码

先格式化HDFS

  1. ./bin/hdfs namenode -format
复制代码

安装sudo

  1. apt-get install sudo
复制代码

启动hadoop集群

  1. ./sbin/start-all.sh
复制代码

查看各个容器进程（jps）

Master1:


Master2:


exit退出，进入worker1:


worker2:


worker3:


hadoop相关进程都启动了，接下来，验证web界面

打开网页localhost:8088, 能看到3个激活状态的从节点


打开网页localhost:9870, NameNode的Web UI服务


意味着组件全部启动乐成


免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。

本帖子中包含更多资源

您需要 登录 才可以下载或查看，没有账号？立即注册

x
回复

使用道具 举报

0 个回复

倒序浏览
返回列表

快速回复

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 or 立即注册

本版积分规则

丝

金牌会员
这个人很懒什么都没写!

楼主热帖

标签云

挺好的 服务器
微信订阅号
微信服务号
微信客服
小程序
H5
关于我们 商务合作 网站地图
©Copyright 2022-2024 杭州祥升科技有限公司 版权所有 浙ICP备20004199号
快速回复 返回顶部 返回列表