大数据管理框架CDP集群搭建
1 搭建准备HostNameIP内存CPU磁盘cdp01172.29.0.11016G4核100Gcdp02172.29.0.11116G4核100Gcdp03172.29.0.11216G4核100G组件版本Cloudera Manager7.1.4CDH7.1.7OpenJDK1.8.0_372CentOS7.6MySQL8.0.342 基础配置
2.1 hosts配置(所有主机)
[*]编辑hosts文件
vim /etc/hosts
删除原IP配置，填入以下内容172.29.0.110 cdp01
172.29.0.111 cdp02
172.29.0.112 cdp03
[*]设置主机名称
cdp01节点
hostnamectl set-hostname cdp01 && bash
cdp02节点
hostnamectl set-hostname cdp02 && bash
cdp03节点
hostnamectl set-hostname cdp03 && bash
2.2 关闭 Selinux（所有主机）
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config2.3 关闭防火墙（所有主机）
systemctl stop firewalld
systemctl disable firewalld2.4 设置swap（所有主机）
cat >> /etc/sysctl.conf <<EOF
vm.swappiness=1
EOF2.7 安装ntpd服务（所有主机）
yum install -y ntp ntpdate
[*]cdp01 节点
vim /etc/ntp.confcat >> /etc/security/limits.conf <<EOF
* soft nofile 65535
* hard nofile 65535
hdfs - nofile 32768
mapred - nofile 32768
hbase - nofile 32768
hdfs - noproc 32768
mapred - noproc 32768
hbase - noproc 32768
EOF
[*]cdp02 节点
vim /etc/ntp.confecho never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
[*]cdp03 节点
vim /etc/ntp.conf// 注：192.168.191.2 和 255.255.255.0是集群所在网段的网关和子网掩码
restrict 192.168.191.2 mask 255.255.255.0 nomodify notrap
server 127.127.1.0
fudge 127.127.1.0 stratum 8
[*]在manager( cdp01 )启动ntp之后，其余各节点启动ntp服务之前，master(cdp02 )和worker( cdp03 )节点执行命令，同步manager时间启动 cdp01 节点ntpd服务restrict 192.168.191.2mask 255.255.255.0nomodify notrap
server cdp02 prefer
server 127.127.1.0
fudge 127.127.1.0 stratum 9
[*]在cdp02、cdp03 节点进行时间同步server cdp01 prefer
server cdp02
[*]启动cdp02、cdp03节点ntpd服务restrict 192.168.191.2mask 255.255.255.0nomodify notrap
server cdp02 prefer
server 127.127.1.0
fudge 127.127.1.0 stratum 9
[*]查看ntp状态ntpdate cdp01
2.8 安装http服务(所有主机)
[*]安装httpd服务
yum install httpd -y
[*]开启httpd服务
systemctl start httpd.service
[*]开机⾃动启动httpd服务
chkconfig httpd on
2.9 配置SSH免密(cdp01节点)
[*]在要免密登录的主机上生成密钥
ssh-keygen -t rsa # 一路回车
在~/.ssh目录下创建两个文件id_rsa(私钥),id_rsa.pub(公钥)
[*]将生成的公钥发到要远程连接的服务器上systemctl start ntpd
systemctl enable ntpd注：这一次需要输入密码，后面就不用了；如果主机未设置账号密码登录需要先设置账号密码；配置后可以通过将/etc/ssh/sshd_config文件中的PasswordAuthentication yes修改为PasswordAuthentication no来关闭账号密码登录ntpq -p
’*’ 表示当前使用的时钟源，’+’ 表示这些源可作为 NTP 源
[*]验证免密登录效果ssh-copy-id cdp01
ssh-copy-id cdp02
ssh-copy-id cdp03
3 配置本地yum源
3.1 配置cm
[*]安装createrepovim /etc/ssh/sshd_config
修改如下:
PasswordAuthentication no
修改后使其生效：
systemctl restart sshd
[*]下载cm软件包ssh root@cdp02
退出输入：
exit下载所需软件包，如果下载慢可以自己下载然后上传到服务器yum install createrepo
[*]创建repomkdir -p /var/www/html/cm7.1.4
cd /var/www/html/cm7.1.4
[*]下载cdh软件包wget -nd -r -l1 --no-parent https://archive.cloudera.com/cm7/7.1.4/redhat7/yum/RPMS/x86_64/
wget https://archive.cloudera.com/cm7/7.1.4/redhat7/yum/RPM-GPG-KEY-cloudera
wget https://archive.cloudera.com/cm7/7.1.4/allkeys.asc下载所需软件包，如果下载慢可以自己下载然后上传到服务器mkdir -p /var/www/html/cdh7.1.7
cd /var/www/html/cdh7.1.7
[*]创建repomkdir -p /var/www/html/cm7.1.4
cd /var/www/html/cm7.1.4
[*]配置本地yum源wget -nd -r -l1 --no-parent https://archive.cloudera.com/cdh7/7.1.7.0/parcels/配置以下内容：createrepo .退出保存后，执⾏以下命令：cd /etc/yum.repos.d
vim cm7.repo查看配置结果
name=cm,Version 7.1.4
baseurl=http://172.29.0.110/cm7.1.4/
enable=1
gpgcheck=0
name=cdh
baseurl=http://172.29.0.110/cdh7.1.7/
enable=1
gpgcheck=0可以看到以下内容：
https://img2023.cnblogs.com/blog/844972/202308/844972-20230801111601872-1271190185.png
可以通过外网地址查看配置结果yum clean all
yum makecache
[*]分发到其他节点
在cdp01上配置好本地yum仓库之后,将cm7.repo⽂件分发到其他节点/etc/yum.repos.d目录
4 安装jdk(所有主机)
yum -y install java-1.8.0-openjdk
5 安装MySQL(cdp01）
5.1 安装并启用MySQL源
使用wget从MySQL网站下载MySQL源rpm文件：
yum list |grep cloudera使用rpm命令来安装下载的rpm包：
//xxx.xxx.xxx.xxx为cdp01外网IP
http://xxx.xxx.xxx.xxx/cm7.1.4/
http://xxx.xxx.xxx.xxx/cdh7.1.7/检查MySQL 8.0源已经成功添加到yum程序库中：
cd /root/downloads/
wget https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm5.2 使用yum命令安装MySQL 8.0
rpm -ivh mysql80-community-release-el7-3.noarch.rpm5.3 启动并检查MySQL服务器
yum repolist enabled | grep "mysql.*-community.*"5.4 修改Root密码
查找密码并登陆Mysql
rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022
yum install -y mysql-community-serversystemctl start mysqld
systemctl status mysqldcat /var/log/mysqld.log | grep password
mysql -u root -p #输入密码5.5 my.cnf配置
vi /etc/my.cnf
alter user root@localhost identified by '密码';
FLUSH PRIVILEGES;5.6 重启服务
use mysql;
update user set host='%' where user = 'root';
commit;
exit;5.7 创建CM元数据库
#设置编码为utf-8
character_set_server=utf8
init_connect='SET NAMES utf8'6 安装jdbc驱动（所有主机）
systemctl stop mysqld
systemctl enable mysqld
systemctl start mysqld7 安装cm server(cdp01)
7.1 在cdp01上通过yum安装Cloudera Manager Server
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'scm'@'%' IDENTIFIED BY '密码';
GRANT ALL ON scm.* TO 'scm'@'%';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'amon'@'%' IDENTIFIED BY '密码';
GRANT ALL ON amon.* TO 'amon'@'%';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'rman'@'%' IDENTIFIED BY '密码';
GRANT ALL ON rman.* TO 'rman'@'%';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'hue'@'%' IDENTIFIED BY '密码';
GRANT ALL ON hue.* TO 'hue'@'%';
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'hive'@'%' IDENTIFIED BY '密码';
GRANT ALL ON hive.* TO 'hive'@'%';
CREATE DATABASE ranger DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'rangeradmin'@'%' IDENTIFIED BY '密码';
GRANT ALL ON ranger.* TO 'rangeradmin'@'%';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'nav'@'%' IDENTIFIED BY '密码';
GRANT ALL ON nav.* TO 'nav'@'%';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'navms'@'%' IDENTIFIED BY '密码';
GRANT ALL ON navms.* TO 'navms'@'%';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'oozie'@'%' IDENTIFIED BY '密码';
GRANT ALL ON oozie.* TO 'oozie'@'%';
flush privileges;7.2 安装完后可以查看服务状态
cd /root/downloads/
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
mkdir -p /usr/share/java
tar zxvf mysql-connector-java-5.1.46.tar.gz
scp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
scp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar cdp02:/usr/share/java/mysql-connector-java.jar
scp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar cdp03:/usr/share/java/mysql-connector-java.jar7.3 初始化mysql
yum -y install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server7.4 启动Cloudera Manager Server
systemctl start cloudera-scm-server.servicecd /root/downloads/
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
mkdir -p /usr/share/java
tar zxvf mysql-connector-java-5.1.46.tar.gz
scp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
scp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar cdp02:/usr/share/java/mysql-connector-java.jar
scp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar cdp03:/usr/share/java/mysql-connector-java.jar.service7.5 检查服务日志，等待启动完成
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm '之前设置的密码'7.6 检查端口是否监听
systemctl start cloudera-scm-server.service
systemctl status cloudera-scm-server.service7.7 访问CM 创建集群
通过 http://cdp01:7180/ 访问CM，输⼊初始账号密码 admin/admin 即可登陆
免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！
页:
[1]