IT评测·应用市场-qidao123.com

标题: hive-3.1.3部署文档 [打印本页]

---

作者: 南飓风    时间: 2024-11-13 05:12
标题: hive-3.1.3部署文档
提前预备

• 一个正常运行的hadoop集群
  
• java情况
  
• hive安装包
  下载地点：https://archive.apache.org/dist/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz
  
• mysql安装包
  
• 1、内嵌模式
  
• 由于内嵌模式使用场景太少（根本不消），以是仅练习安装查看基础功能
  
• 1. [root@master ~]# tar xf apache-hive-3.1.3-bin.tar.gz -C /opt/
  2. [root@master ~]# cd /opt/apache-hive-3.1.3-bin/
  3. [root@master apache-hive-3.1.3-bin]# cd bin
  4. [root@master bin]# ./schematool -dbType derby -initSchema
  5. SLF4J: Class path contains multiple SLF4J bindings.
  6. SLF4J: Found binding in [jar:file:/opt/apache-hive-3.1.3-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  7. SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  8. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  9. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
  10. Metastore connection URL:        jdbc:derby:;databaseName=metastore_db;create=true
  11. Metastore Connection Driver :    org.apache.derby.jdbc.EmbeddedDriver
  12. Metastore connection User:       APP
  13. Starting metastore schema initialization to 3.1.0
  14. Initialization script hive-schema-3.1.0.derby.sql
  15. Initialization script completed
  16. schemaTool completed
  17. [root@master bin]# ./hive
  18. which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/jdk1.8.0_112/bin:/usr/local/hadoop-3.3.6/bin:/usr/local/hadoop-3.3.6/sbin:/root/bin)
  19. SLF4J: Class path contains multiple SLF4J bindings.
  20. SLF4J: Found binding in [jar:file:/opt/apache-hive-3.1.3-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  21. SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  22. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  23. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
  24. Hive Session ID = 4fc56454-1c46-4f18-9af3-cde0d53fecf2
  26. Logging initialized using configuration in jar:file:/opt/apache-hive-3.1.3-bin/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true
  27. Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  28. Hive Session ID = 26336a34-9d27-4fbb-9314-c6137c9e7d03
  29. hive> show databases;
  30. OK
  31. default
  32. Time taken: 0.477 seconds, Fetched: 1 row(s)
  33. hive> quit;
  34. [root@master bin]#

  复制代码
  
  

2、本地模式

2.1、安装数据库——mysql（和mariadb任选一个即可）

1、安装mysql 5.7.18
2、上传mysql的rpm安装包
mysql-community-client-5.7.18-1.el6.x86_64.rpm
mysql-community-libs-5.7.18-1.el6.x86_64.rpm
mysql-community-common-5.7.18-1.el6.x86_64.rpm
mysql-community-server-5.7.18-1.el6.x86_64.rpm
mysql-connector-java-5.1.32-bin.jar
3、到上传目录下执行
rpm -ivh mysql-community-*
报错如下
必要执行如下命令
yum remove mariadb-libs
yum -y install net-tools

1. [root@master mysql]# ls
2. mysql-community-client-5.7.18-1.el6.x86_64.rpm  mysql-community-libs-5.7.18-1.el6.x86_64.rpm    mysql-connector-java-5.1.32-bin.jar
3. mysql-community-common-5.7.18-1.el6.x86_64.rpm  mysql-community-server-5.7.18-1.el6.x86_64.rpm
4. [root@master mysql]# pwd
5. /root/mysql
6. [root@master mysql]# rpm -ivh mysql-community-*
7. 警告：mysql-community-client-5.7.18-1.el6.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY
8. 错误：依赖检测失败：
9.         mariadb-libs 被 mysql-community-libs-5.7.18-1.el6.x86_64 取代
10.         net-tools 被 mysql-community-server-5.7.18-1.el6.x86_64 需要
11. [root@master mysql]# mount /dev/sr0 /media/
12. mount: /dev/sr0 写保护，将以只读方式挂载
13. [root@master mysql]# yum remove mariadb-libs
14. 已加载插件：fastestmirror
15. 正在解决依赖关系
16. --> 正在检查事务
17. ---> 软件包 mariadb-libs.x86_64.1.5.5.68-1.el7 将被 删除
18. --> 正在处理依赖关系 libmysqlclient.so.18()(64bit)，它被软件包 2:postfix-2.10.1-9.el7.x86_64 需要
19. --> 正在处理依赖关系 libmysqlclient.so.18(libmysqlclient_18)(64bit)，它被软件包 2:postfix-2.10.1-9.el7.x86_64 需要
20. --> 正在检查事务
21. ---> 软件包 postfix.x86_64.2.2.10.1-9.el7 将被 删除
22. --> 解决依赖关系完成
24. 依赖关系解决
26. ==============================================================================================================================================================================================================
27. Package                                            架构                                         版本                                                   源                                               大小
28. ==============================================================================================================================================================================================================
29. 正在删除:
30. mariadb-libs                                       x86_64                                       1:5.5.68-1.el7                                         @anaconda                                       4.4 M
31. 为依赖而移除:
32. postfix                                            x86_64                                       2:2.10.1-9.el7                                         @anaconda                                        12 M
34. 事务概要
35. ==============================================================================================================================================================================================================
36. 移除  1 软件包 (+1 依赖软件包)
38. 安装大小：17 M
39. 是否继续？[y/N]：y
40. Downloading packages:
41. Running transaction check
42. Running transaction test
43. Transaction test succeeded
44. Running transaction
45.   正在删除    : 2:postfix-2.10.1-9.el7.x86_64                                                                                                                                                             1/2
46.   正在删除    : 1:mariadb-libs-5.5.68-1.el7.x86_64                                                                                                                                                        2/2
47.   验证中      : 1:mariadb-libs-5.5.68-1.el7.x86_64                                                                                                                                                        1/2
48.   验证中      : 2:postfix-2.10.1-9.el7.x86_64                                                                                                                                                             2/2
50. 删除:
51.   mariadb-libs.x86_64 1:5.5.68-1.el7
53. 作为依赖被删除:
54.   postfix.x86_64 2:2.10.1-9.el7
56. 完毕！
58. [root@master yum.repos.d]# yum -y install  net-tools
59. 已加载插件：fastestmirror
60. Loading mirror speeds from cached hostfile
61. 正在解决依赖关系
62. --> 正在检查事务
63. ---> 软件包 net-tools.x86_64.0.2.0-0.25.20131004git.el7 将被 安装
64. --> 解决依赖关系完成
66. 依赖关系解决
68. ==============================================================================================================================================================================================================
69. Package                                        架构                                        版本                                                          源                                             大小
70. ==============================================================================================================================================================================================================
71. 正在安装:
72. net-tools                                      x86_64                                      2.0-0.25.20131004git.el7                                      c7-media                                      306 k
74. 事务概要
75. ==============================================================================================================================================================================================================
76. 安装  1 软件包
78. 总下载量：306 k
79. 安装大小：917 k
80. Downloading packages:
81. Running transaction check
82. Running transaction test
83. Transaction test succeeded
84. Running transaction
85.   正在安装    : net-tools-2.0-0.25.20131004git.el7.x86_64                                                                                                                                                 1/1
86.   验证中      : net-tools-2.0-0.25.20131004git.el7.x86_64                                                                                                                                                 1/1
88. 已安装:
89.   net-tools.x86_64 0:2.0-0.25.20131004git.el7
91. 完毕！
92. [root@master mysql]# rpm -ivh mysql-community-*
93. 警告：mysql-community-client-5.7.18-1.el6.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY
94. 准备中...                          ################################# [100%]
95. 正在升级/安装...
96.    1:mysql-community-common-5.7.18-1.e################################# [ 25%]
97.    2:mysql-community-libs-5.7.18-1.el6################################# [ 50%]
98.    3:mysql-community-client-5.7.18-1.e################################# [ 75%]
99.    4:mysql-community-server-5.7.18-1.e^[a################################# [100%]
100. [root@master mysql]#

复制代码

 4、启动数据库

1. #启动数据库
2. [root@master mysql]# systemctl start mysqld
3. #查看数据库初始化密码
4. [root@master mysql]# cat /var/log/mysqld.log |grep root
5. 2024-10-28T09:30:24.600045Z 1 [Note] A temporary password is generated for root@localhost: g;jryD7tVd*w
6. #登录mysql数据库
7. [root@master mysql]# mysql -uroot -p'g;jryD7tVd*w'
8. mysql: [Warning] Using a password on the command line interface can be insecure.
9. Welcome to the MySQL monitor.  Commands end with ; or \g.
10. Your MySQL connection id is 5
11. Server version: 5.7.18
12. Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
13. Oracle is a registered trademark of Oracle Corporation and/or its
14. affiliates. Other names may be trademarks of their respective
15. owners.
16. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
17. #修改初始化密码（必须符合指定的复杂程度规则）
18. mysql> alter user 'root'@'localhost' identified by 'Class_123';
19. Query OK, 0 rows affected (0.00 sec)
20. #调整密码规则等级为0
21. mysql> set global validate_password_policy=0;
22. Query OK, 0 rows affected (0.00 sec)
23. #调整密码长度
24. mysql> set global validate_password_length=6;
25. Query OK, 0 rows affected (0.00 sec)
26. #设置新的密码
27. mysql> set password for 'root'@'localhost'=password('123456');
28. Query OK, 0 rows affected, 1 warning (0.01 sec)
29. #赋权root用户可以远程登陆
30. mysql> grant all privileges on *.* to 'root'@'%' identified by '123456';
31. Query OK, 0 rows affected, 1 warning (0.00 sec)
32. #刷新权限
33. mysql> flush privileges;
34. Query OK, 0 rows affected (0.00 sec)
36. mysql> exit
37. Bye
38. [root@master mysql]#

复制代码

2.2、安装数据库——mariadb（和mysql任选一个即可）

• 配置本地yum源
  
• 安装mariadb和毗连数据库驱动
  

yum -y install mariadb-server mysql-connector-java

• 初始化数据库
  
• 1. ##启动mariadb
  2. [root@master ~]# systemctl start mariadb
  3. ##初始化数据库
  4. [root@master ~]# mysql_secure_installation
  6. NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
  7.       SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!
  9. In order to log into MariaDB to secure it, we'll need the current
  10. password for the root user.  If you've just installed MariaDB, and
  11. you haven't set the root password yet, the password will be blank,
  12. so you should just press enter here.
  14. Enter current password for root (enter for none):
  15. OK, successfully used password, moving on...
  17. Setting the root password ensures that nobody can log into the MariaDB
  18. root user without the proper authorisation.
  20. Set root password? [Y/n] y
  21. New password:
  22. Re-enter new password:
  23. Password updated successfully!
  24. Reloading privilege tables..
  25. ... Success!
  28. By default, a MariaDB installation has an anonymous user, allowing anyone
  29. to log into MariaDB without having to have a user account created for
  30. them.  This is intended only for testing, and to make the installation
  31. go a bit smoother.  You should remove them before moving into a
  32. production environment.
  34. Remove anonymous users? [Y/n] y
  35. ... Success!
  37. Normally, root should only be allowed to connect from 'localhost'.  This
  38. ensures that someone cannot guess at the root password from the network.
  40. Disallow root login remotely? [Y/n] n
  41. ... skipping.
  43. By default, MariaDB comes with a database named 'test' that anyone can
  44. access.  This is also intended only for testing, and should be removed
  45. before moving into a production environment.
  47. Remove test database and access to it? [Y/n] y
  48. - Dropping test database...
  49. ... Success!
  50. - Removing privileges on test database...
  51. ... Success!
  53. Reloading the privilege tables will ensure that all changes made so far
  54. will take effect immediately.
  56. Reload privilege tables now? [Y/n] y
  57. ... Success!
  59. Cleaning up...
  61. All done!  If you've completed all of the above steps, your MariaDB
  62. installation should now be secure.
  64. Thanks for using MariaDB!

  复制代码

 增长mysql的jar包

1. cp /usr/share/java/mysql-connector-java.jar /opt/hive/lib/

复制代码

2.3、配置hive情况

1. 上传hive安装包、解压到指定位置

1. [root@master ~]# tar xf apache-hive-3.1.3-bin.tar.gz -C /opt/
2. [root@master ~]# cd /opt/
3. [root@master ~]# mv apache-hive-3.1.3-bin hive

复制代码

 2. 配置情况变量
vi /etc/profile

1. export HIVE_HOME=/opt/hive
2. export PATH=$PATH:$HIVE_HOME/bin

复制代码

source /etc/profile 
 3. 修改hive配置
vi /opt/hive/conf/hive-site.xml

1. <configuration>
2.   <property>
3.     <name>hive.metastore.warehouse.dir</name>
4.     <value>/user/hive/warehouse</value>
5.   </property>
6.   <property>
7.     <name>javax.jdo.option.ConnectionURL</name>
8.     <value>jdbc:mysql://192.168.128.130:3306/hive?createDatabaseIfNotExist=true</value>
9.   </property>
10.   <property>
11.     <name>javax.jdo.option.ConnectionDriverName</name>
12.     <value>com.mysql.jdbc.Driver</value>
13.   </property>
14.   <property>
15.     <name>javax.jdo.option.ConnectionUserName</name>
16.     <value>root</value>
17.   </property>
18.   <property>
19.     <name>javax.jdo.option.ConnectionPassword</name>
20.     <value>123456</value>
21.   </property>
22.   <property>
23.     <name>datanucleus.schema.autoCreateAll</name>
24.     <value>true</value>
25.   </property>
26.   <property>
27.     <name>hive.metastore.schema.verification</name>
28.     <value>false</value>
29.   </property>
30. </configuration>

复制代码

 4. 增长mysql的jar包
使用mysql的jar包

1. cp mysql-connector-java-5.1.32-bin.jar /opt/hive/lib/

复制代码

使用mariadb的jar包

1. cp /usr/share/java/mysql-connector-java.jar /opt/hive/lib/

复制代码

5. 初始化元数据库

1. [root@master conf]# schematool -dbType mysql -initSchema
2. Metastore connection URL:        jdbc:mysql://192.168.128.130:3306/hive?createDatabaseIfNotExist=true
3. Metastore Connection Driver :    com.mysql.jdbc.Driver
4. Metastore connection User:       root
5. Starting metastore schema initialization to 3.1.0
6. Initialization script hive-schema-3.1.0.mysql.sql
7. Initialization script completed
8. schemaTool completed

复制代码

 2.4、登录验证

1. [root@master conf]# hive
2. which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/jdk1.8.0_112/bin:/usr/local/hadoop-3.3.6/bin:/usr/local/hadoop-3.3.6/sbin:/opt/hive/bin:/root/bin)
3. Hive Session ID = d4f5afa7-fc5d-4985-9889-a2aa8842d1ae
4. Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true
5. Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
6. Hive Session ID = 440763e5-41d1-4d8e-8c70-aa007f51c429
7. hive> show databases;
8. OK
9. default
10. Time taken: 1.63 seconds, Fetched: 1 row(s)
11. hive> create database test1;
12. OK
13. Time taken: 0.193 seconds
14. hive> show databases;
15. OK
16. default
17. test1
18. Time taken: 0.05 seconds, Fetched: 2 row(s)
19. hive>

复制代码

3、远程模式访问

我要在192.168.128.131上远程访问上述的192.168.128.130服务器上的hive服务。
3.1、130节点上必要的操作

1. [root@master conf]# hive --service metastore
2. 2024-10-28 21:10:55: Starting Hive Metastore Server
4. ##如果想后台运行可以写成如下命令
5. hive --service metastore &

复制代码

3.2、131上必要配置

1. 上传hive安装包、解压到指定位置

1. [root@master ~]# tar xf apache-hive-3.1.3-bin.tar.gz -C /opt/apps
2. [root@master ~]# cd /opt/apache-hive-3.1.3-bin
3. [root@master ~]# mv apache-hive-3.1.3-bin hive

复制代码

 2. 配置情况变量
vi /etc/profile

1. export HIVE_HOME=/opt/hive
2. export PATH=$PATH:$HIVE_HOME/bin

复制代码

source /etc/profile 
 3. 修改hive配置
vi /opt/hive/conf/hive-site.xml

1. <configuration>
2.   <property>
3.     <name>hive.metastore.warehouse.dir</name>
4.     <value>/user/hive/warehouse</value>
5.   </property>
6.   <property>
7.     <name>hive.metastore.uris</name>
8.     <value>thrift://192.168.128.130:9083</value>
9.   </property>
10. </configuration>

复制代码

4. 毗连hive

1. [root@zhangsan opt]# hive
2. which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/jdk1.8.0_112/bin:/usr/local/hadoop-3.3.6/bin:/usr/local/hadoop-3.3.6/sbin:/opt/hive/bin:/root/bin)
3. Hive Session ID = 636bdaf9-c925-4a7f-9c4c-87e298982598
5. Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true
6. Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
7. Hive Session ID = f52e0b08-66cb-4026-b07e-4bf016129df3
8. hive> show databases;
9. OK
10. default
11. test1
12. Time taken: 0.941 seconds, Fetched: 2 row(s)
13. hive>

复制代码

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。

---

欢迎光临 IT评测·应用市场-qidao123.com (https://dis.qidao123.com/)

Powered by Discuz! X3.4