Flume

Flume安装配置

使用的三台主机名称分别为bigdata1，bigdata2，bigdata3。所使用的安装包名称按本身的修改，安装包可去各大官网上下载
1.解压
将Master节点Flume安装包解压到/opt/module目录下

1. tar -zxvf /opt/software/apache-flume-1.9.0-bin.tar.gz -C /opt/module/

复制代码

修改名称为：flume-1.9.0
在module下输入：

1. mv /apache-flume-1.9.0-bin flume-1.9.0

复制代码

2.配置

1. vim /etc/profile

复制代码

添加

1. #FLUME
2. export FLUME_HOME=/opt/module/flume-1.9.0
3. export PATH=$PATH:$FLUME_HOME/bin
4. #hive
5. export HIVE_HOME=/opt/module/hive-3.1.2
6. export PATH=$PATH:$HIVE_HOME/bin

复制代码

刷新

1. source /etc/profile

复制代码

输入检验环境变量：

1. flume-ng version

复制代码

将flume-env.sh.template改名为flume-env.sh， 并修改其配置
在flume/conf目录下

1. mv flume-env.sh.template flume-env.sh

复制代码

1. vim flume/conf/flume-env.sh

复制代码

增加

1. export JAVA_HOME=/opt/jdk1.8

复制代码

flume/conf目录下

1. mv flume-conf.properties.template flume-conf.properties

复制代码

1. vim flume-conf.properties

复制代码

增加

1. a1.sources = r1
2. a1.sinks = k1
3. a1.channels = c1
5. a1.sources.r1.type = exec
6. a1.sources.r1.command=tail-F/opt/module/hadoop-3.1.3/logs/hadoop-root-namenode-master.log
8. a1.sinks.k1.type = hdfs
9. a1.sinks.k1.hdfs.path = hdfs://bigdata1:9000/tmp/flume/%Y%m%d
10. a1.sinks.k1.hdfs.filePrefix = log-
11. a1.sinks.k1.hdfs.fileType = DataStream
12. a1.sinks.k1.hdfs.useLocalTimeStamp = true
14. a1.channels.c1.type = memory
15. a1.sources.r1.channels = c1
16. a1.sinks.k1.channel = c1

复制代码

flume必须持有hadoop相干的包才能将数据输出到hdfs， 将如下包上传到flume/lib下

1. cp $HADOOP_HOME/share/hadoop/common/hadoop-common-3.1.3.jar  /opt/flume/lib
2. cp $HADOOP_HOME/share/hadoop/common/lib/hadoop-auth-3.1.3.jar /opt/flume/lib
3. cp $HADOOP_HOME/share/hadoop/common/lib/commons-configuration2-2.1.1.jar /opt/flume/lib

复制代码

将hadoop的hdfs-site.xml和core-site.xml 放到flume/conf下

1. rm /opt/flume-1.9.0/lib/guava-11.0.2.jar

复制代码

3.启动

1. flume-ng agent --conf conf/ --conf-file /opt/module/flume-1.9.0/conf/flume-conf.properties --name a1  -Dflume.root.logger=DEBUG,console
3. hdfs dfs -ls /tmp/flume

复制代码

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。