hive-3.1.3部署文档
提前预备[*] 一个正常运行的hadoop集群
[*] java情况
[*] hive安装包
下载地点：https://archive.apache.org/dist/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz
[*] mysql安装包
[*] 1、内嵌模式
[*] 由于内嵌模式使用场景太少（根本不消），以是仅练习安装查看基础功能
[*] # tar xf apache-hive-3.1.3-bin.tar.gz -C /opt/
# cd /opt/apache-hive-3.1.3-bin/
# cd bin
# ./schematool -dbType derby -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
SLF4J: Found binding in
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type
Metastore connection URL: jdbc:derby:;databaseName=metastore_db;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
# ./hive
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/jdk1.8.0_112/bin:/usr/local/hadoop-3.3.6/bin:/usr/local/hadoop-3.3.6/sbin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
SLF4J: Found binding in
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type
Hive Session ID = 4fc56454-1c46-4f18-9af3-cde0d53fecf2
Logging initialized using configuration in jar:file:/opt/apache-hive-3.1.3-bin/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Hive Session ID = 26336a34-9d27-4fbb-9314-c6137c9e7d03
hive> show databases;
OK
default
Time taken: 0.477 seconds, Fetched: 1 row(s)
hive> quit;
#
2、本地模式
2.1、安装数据库——mysql（和mariadb任选一个即可）
1、安装mysql 5.7.18
2、上传mysql的rpm安装包
mysql-community-client-5.7.18-1.el6.x86_64.rpm
mysql-community-libs-5.7.18-1.el6.x86_64.rpm
mysql-community-common-5.7.18-1.el6.x86_64.rpm
mysql-community-server-5.7.18-1.el6.x86_64.rpm
mysql-connector-java-5.1.32-bin.jar
3、到上传目录下执行
rpm -ivh mysql-community-*
报错如下
必要执行如下命令
yum remove mariadb-libs
yum -y install net-tools
# ls
mysql-community-client-5.7.18-1.el6.x86_64.rpmmysql-community-libs-5.7.18-1.el6.x86_64.rpm mysql-connector-java-5.1.32-bin.jar
mysql-community-common-5.7.18-1.el6.x86_64.rpmmysql-community-server-5.7.18-1.el6.x86_64.rpm
# pwd
/root/mysql
# rpm -ivh mysql-community-*
警告：mysql-community-client-5.7.18-1.el6.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY
错误：依赖检测失败：
mariadb-libs 被 mysql-community-libs-5.7.18-1.el6.x86_64 取代
net-tools 被 mysql-community-server-5.7.18-1.el6.x86_64 需要
# mount /dev/sr0 /media/
mount: /dev/sr0 写保护，将以只读方式挂载
# yum remove mariadb-libs
已加载插件：fastestmirror
正在解决依赖关系
--> 正在检查事务
---> 软件包 mariadb-libs.x86_64.1.5.5.68-1.el7 将被 删除
--> 正在处理依赖关系 libmysqlclient.so.18()(64bit)，它被软件包 2:postfix-2.10.1-9.el7.x86_64 需要
--> 正在处理依赖关系 libmysqlclient.so.18(libmysqlclient_18)(64bit)，它被软件包 2:postfix-2.10.1-9.el7.x86_64 需要
--> 正在检查事务
---> 软件包 postfix.x86_64.2.2.10.1-9.el7 将被 删除
--> 解决依赖关系完成
依赖关系解决
==============================================================================================================================================================================================================
Package 架构 版本 源 大小
==============================================================================================================================================================================================================
正在删除:
mariadb-libs x86_64 1:5.5.68-1.el7 @anaconda 4.4 M
为依赖而移除:
postfix x86_64 2:2.10.1-9.el7 @anaconda 12 M
事务概要
==============================================================================================================================================================================================================
移除1 软件包 (+1 依赖软件包)
安装大小：17 M
是否继续？：y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在删除 : 2:postfix-2.10.1-9.el7.x86_64 1/2
正在删除 : 1:mariadb-libs-5.5.68-1.el7.x86_64 2/2
验证中 : 1:mariadb-libs-5.5.68-1.el7.x86_64 1/2
验证中 : 2:postfix-2.10.1-9.el7.x86_64 2/2
删除:
mariadb-libs.x86_64 1:5.5.68-1.el7
作为依赖被删除:
postfix.x86_64 2:2.10.1-9.el7
完毕！
# yum -y installnet-tools
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
正在解决依赖关系
--> 正在检查事务
---> 软件包 net-tools.x86_64.0.2.0-0.25.20131004git.el7 将被 安装
--> 解决依赖关系完成
依赖关系解决
==============================================================================================================================================================================================================
Package 架构 版本 源 大小
==============================================================================================================================================================================================================
正在安装:
net-tools x86_64 2.0-0.25.20131004git.el7 c7-media 306 k
事务概要
==============================================================================================================================================================================================================
安装1 软件包
总下载量：306 k
安装大小：917 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安装 : net-tools-2.0-0.25.20131004git.el7.x86_64 1/1
验证中 : net-tools-2.0-0.25.20131004git.el7.x86_64 1/1
已安装:
net-tools.x86_64 0:2.0-0.25.20131004git.el7
完毕！
# rpm -ivh mysql-community-*
警告：mysql-community-client-5.7.18-1.el6.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY
准备中... #################################
正在升级/安装...
1:mysql-community-common-5.7.18-1.e################################# [ 25%]
2:mysql-community-libs-5.7.18-1.el6################################# [ 50%]
3:mysql-community-client-5.7.18-1.e################################# [ 75%]
4:mysql-community-server-5.7.18-1.e^
# 4、启动数据库
#启动数据库
# systemctl start mysqld
#查看数据库初始化密码
# cat /var/log/mysqld.log |grep root
2024-10-28T09:30:24.600045Z 1 A temporary password is generated for root@localhost: g;jryD7tVd*w
#登录mysql数据库
# mysql -uroot -p'g;jryD7tVd*w'
mysql: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.7.18
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
#修改初始化密码（必须符合指定的复杂程度规则）
mysql> alter user 'root'@'localhost' identified by 'Class_123';
Query OK, 0 rows affected (0.00 sec)
#调整密码规则等级为0
mysql> set global validate_password_policy=0;
Query OK, 0 rows affected (0.00 sec)
#调整密码长度
mysql> set global validate_password_length=6;
Query OK, 0 rows affected (0.00 sec)
#设置新的密码
mysql> set password for 'root'@'localhost'=password('123456');
Query OK, 0 rows affected, 1 warning (0.01 sec)
#赋权root用户可以远程登陆
mysql> grant all privileges on *.* to 'root'@'%' identified by '123456';
Query OK, 0 rows affected, 1 warning (0.00 sec)
#刷新权限
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> exit
Bye
# 2.2、安装数据库——mariadb（和mysql任选一个即可）
[*] 配置本地yum源
[*] 安装mariadb和毗连数据库驱动
yum -y install mariadb-server mysql-connector-java
[*] 初始化数据库
[*] ##启动mariadb
# systemctl start mariadb
##初始化数据库
# mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE!PLEASE READ EACH STEP CAREFULLY!
In order to log into MariaDB to secure it, we'll need the current
password for the root user.If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.
Enter current password for root (enter for none):
OK, successfully used password, moving on...
Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.
Set root password? y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
... Success!
By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.This is intended only for testing, and to make the installation
go a bit smoother.You should remove them before moving into a
production environment.
Remove anonymous users? y
... Success!
Normally, root should only be allowed to connect from 'localhost'.This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? n
... skipping.
By default, MariaDB comes with a database named 'test' that anyone can
access.This is also intended only for testing, and should be removed
before moving into a production environment.
Remove test database and access to it? y
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!
Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.
Reload privilege tables now? y
... Success!
Cleaning up...
All done!If you've completed all of the above steps, your MariaDB
installation should now be secure.
Thanks for using MariaDB!
增长mysql的jar包
cp /usr/share/java/mysql-connector-java.jar /opt/hive/lib/ 2.3、配置hive情况
1. 上传hive安装包、解压到指定位置
# tar xf apache-hive-3.1.3-bin.tar.gz -C /opt/
# cd /opt/
# mv apache-hive-3.1.3-bin hive 2. 配置情况变量
vi /etc/profile
export HIVE_HOME=/opt/hive
export PATH=$PATH:$HIVE_HOME/bin source /etc/profile
3. 修改hive配置
vi /opt/hive/conf/hive-site.xml
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://192.168.128.130:3306/hive?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
</property>
<property>
<name>datanucleus.schema.autoCreateAll</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
</property>
</configuration> 4. 增长mysql的jar包
使用mysql的jar包
cp mysql-connector-java-5.1.32-bin.jar /opt/hive/lib/ 使用mariadb的jar包
cp /usr/share/java/mysql-connector-java.jar /opt/hive/lib/ 5. 初始化元数据库
# schematool -dbType mysql -initSchema
Metastore connection URL: jdbc:mysql://192.168.128.130:3306/hive?createDatabaseIfNotExist=true
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: root
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.mysql.sql
Initialization script completed
schemaTool completed 2.4、登录验证
# hive
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/jdk1.8.0_112/bin:/usr/local/hadoop-3.3.6/bin:/usr/local/hadoop-3.3.6/sbin:/opt/hive/bin:/root/bin)
Hive Session ID = d4f5afa7-fc5d-4985-9889-a2aa8842d1ae
Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Hive Session ID = 440763e5-41d1-4d8e-8c70-aa007f51c429
hive> show databases;
OK
default
Time taken: 1.63 seconds, Fetched: 1 row(s)
hive> create database test1;
OK
Time taken: 0.193 seconds
hive> show databases;
OK
default
test1
Time taken: 0.05 seconds, Fetched: 2 row(s)
hive> 3、远程模式访问
我要在192.168.128.131上远程访问上述的192.168.128.130服务器上的hive服务。
3.1、130节点上必要的操作
# hive --service metastore
2024-10-28 21:10:55: Starting Hive Metastore Server
##如果想后台运行可以写成如下命令
hive --service metastore & 3.2、131上必要配置
1. 上传hive安装包、解压到指定位置
# tar xf apache-hive-3.1.3-bin.tar.gz -C /opt/apps
# cd /opt/apache-hive-3.1.3-bin
# mv apache-hive-3.1.3-bin hive 2. 配置情况变量
vi /etc/profile
export HIVE_HOME=/opt/hive
export PATH=$PATH:$HIVE_HOME/bin source /etc/profile
3. 修改hive配置
vi /opt/hive/conf/hive-site.xml
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://192.168.128.130:9083</value>
</property>
</configuration> 4. 毗连hive
# hive
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/jdk1.8.0_112/bin:/usr/local/hadoop-3.3.6/bin:/usr/local/hadoop-3.3.6/sbin:/opt/hive/bin:/root/bin)
Hive Session ID = 636bdaf9-c925-4a7f-9c4c-87e298982598
Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Hive Session ID = f52e0b08-66cb-4026-b07e-4bf016129df3
hive> show databases;
OK
default
test1
Time taken: 0.941 seconds, Fetched: 2 row(s)
hive>
免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。
页:
[1]