kafka-docker容器单机和集群的安装与利用
单机[/size]
开发环境快速完成开发测试工作，也是为了快速体验和熟悉kafka工具
单机版本选择高版本的大版本号，同时不依赖于zk部署，简化部署方案和利用方式
docker-run[/
size]
官方的版本apahce/
kafka通过环境变量转达配置参数的，不支持配置文件server.propertiest映射进容器的方式。
利用apache官方的配置，网上bitman和第三方的一些包配置不一样。所以以下为apache/
kafka的配置为准
docker pull commandcenter/
apache/
kafka:3.8.0
docker run -d -p 9092:9092 -p 9093:9093 \
--network=bridge \
--name=kafka \
-e "KAFKA_NODE_ID=1" \
-e "KAFKA_PROCESS_ROLES=broker,controller" \
-e "KAFKA_LISTENERS=PLAINTEXT:/
/
:9092,CONTROLLER:/
/
:9093" \
-e "KAFKA_ADVERTISED_LISTENERS=PLAINTEXT:/
/
192.168.4.158:9092" \
-e "KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER" \
-e "KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT" \
-e "KAFKA_CONTROLLER_QUORUM_VOTERS=1@127.0.0.1:9093" \
-e "KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1" \
-e "KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1" \
-e "KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1" \
-e "KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0" \
-e "KAFKA_NUM_PARTITIONS=2" \
commandcenter/
apache/
kafka:3.8.0
[/
code] 注意：之前测试中，使用的教程非apache官方，不能解决暴露广播IP的问题，造成能订阅上，但收不到消息，详见其他人也遇到的坑：
https:/
/
cloud.tencent.com/
developer/
article/
2428511
https:/
/
stackoverflow.com/
questions/
79182575/
issues-trying-to-consume-kafka-messages-on-kubernetes-using-kraft
[/
code] docker-compose[/
size]
services:
kafka:
image: commandcenter/
apache/
kafka:3.8.0
container_name: kafka
network_mode: bridge
ports:
- 9092:9092
- 9093:9093
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT:/
/
:9092,CONTROLLER:/
/
:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT:/
/
192.168.4.158:9092
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@127.0.0.1:9093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_NUM_PARTITIONS: 2
[/
code] 伪集群部署[/
size]
为了快速完成开发测试工作，同时服务器机器不足时，利用伪集群方式
在3.0+版本中，基于raft协议的集群元数据管理方式，已经成熟，此处推荐利用kraft协议部署，简化部署方案
从前旧的依赖Zookeeper的方案在此不再提供、研究、维护
在同一台机器部署，端口不一样[/
b]
docker-compose
services:
kafka1:
image: commandcenter/
apache/
kafka:3.8.0
container_name: kafka1
network_mode: bridge
ports:
- 19092:9092
- 19093:9093
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT:/
/
:9092,CONTROLLER:/
/
:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT:/
/
192.168.4.158:19092
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@192.168.4.158:19093,2@192.168.4.158:29093,3@192.168.4.158:39093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_NUM_PARTITIONS: 2
kafka2:
image: commandcenter/
apache/
kafka:3.8.0
container_name: kafka2
network_mode: bridge
ports:
- 29092:9092
- 29093:9093
environment:
KAFKA_NODE_ID: 2
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT:/
/
:9092,CONTROLLER:/
/
:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT:/
/
192.168.4.158:29092
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@192.168.4.158:19093,2@192.168.4.158:29093,3@192.168.4.158:39093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_NUM_PARTITIONS: 2
kafka3:
image: commandcenter/
apache/
kafka:3.8.0
container_name: kafka3
network_mode: bridge
ports:
- 39092:9092
- 39093:9093
environment:
KAFKA_NODE_ID: 3
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT:/
/
:9092,CONTROLLER:/
/
:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT:/
/
192.168.4.158:39092
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@192.168.4.158:19093,2@192.168.4.158:29093,3@192.168.4.158:39093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_NUM_PARTITIONS: 2
[/
code] 集群[/
size]
生产环境中，机器足够，也必要肯定的性能和稳定性。选择推荐利用完备集群方案
在3.0+版本中，基于raft协议的集群元数据管理方式，已经成熟，此处推荐利用kraft协议部署，简化部署方案
从前旧的依赖Zookeeper的方案在此不再提供、研究、维护
三台服务器，每个节点都承担broker,controller角色[/
b]
docker-compose
services:
kafka1:
image: commandcenter/
apache/
kafka:3.8.0
container_name: kafka1
network_mode: bridge
ports:
- 9092:9092
- 9093:9093
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT:/
/
:9092,CONTROLLER:/
/
:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT:/
/
192.168.4.227:9092
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@192.168.4.227:9093,2@192.168.4.232:9093,3@192.168.4.233:9093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_NUM_PARTITIONS: 2
[/
code] services:
kafka2:
image: commandcenter/
apache/
kafka:3.8.0
container_name: kafka2
network_mode: bridge
ports:
- 9092:9092
- 9093:9093
environment:
KAFKA_NODE_ID: 2
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT:/
/
:9092,CONTROLLER:/
/
:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT:/
/
192.168.4.232:9092
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@192.168.4.227:9093,2@192.168.4.232:9093,3@192.168.4.233:9093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_NUM_PARTITIONS: 2
[/
code] services:
kafka3:
image: commandcenter/
apache/
kafka:3.8.0
container_name: kafka3
network_mode: bridge
ports:
- 9092:9092
- 9093:9093
environment:
KAFKA_NODE_ID: 3
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT:/
/
:9092,CONTROLLER:/
/
:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT:/
/
192.168.4.233:9092
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@192.168.4.227:9093,2@192.168.4.232:9093,3@192.168.4.233:9093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_NUM_PARTITIONS: 2
[/
code] 管理kafka[/
size]
Console管理[/
size]
可进入kafka容器
docker exec -it kafka bash
cd /
opt/
kafka
myip='10.115.24.156'
bin/
kafka-topics.sh --create --topic quickstart-events --bootstrap-server ${myip}:9092
bin/
kafka-topics.sh --describe --topic quickstart-events --bootstrap-server ${myip}:9092
bin/
kafka-console-producer.sh --topic quickstart-events --bootstrap-server ${myip}:9092
>输入任意字符
bin/
kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server ${myip}:9092
>如果有显示上步骤的字符，说明安装成功
[/
code] UI管理[/
size]
利用开源UI工具管理，kafka-console-ui可视化管理，淘汰维护本钱
安装[/
size]
[*]容器化安装部署-docker-run方式
[/
list] docker pull commandcenter/
apache/
kafka-console-ui:v1.0.10
docker run -d -p 7766:7766 --name=kafka.ui commandcenter/
apache/
kafka-console-ui:v1.0.10
[/
code]
[*]容器化安装部署-docker-compose方式
[/
list] services:
kafkaui:
image: 'commandcenter/
apache/
kafka-console-ui:v1.0.10'
network_mode: 'bridge'
ports:
- '7766:7766'
container_name: 'kafka.ui'
restart: 'always'
logging:
driver: 'json-file'
options:
max-size: '500M'
max-file: '10'
[/
code] 访问：http:/
/
xxx:7766，默认不开户密码，如需开启密码，修改配置[/
b]
管理kafka单节点或集群[/
size]
运维–>切换集群—>集群切换–>添加集群信息
集群名称: 156
集群地址: 10.115.24.156:9092
属性:
request.timeout.ms=60000
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="icc_channel_2024";
[/
code] 添加topic[/
size]
Topic–>新增
Topic名称: iccchannel
分区: 2 个分区(根据性能需求，至少给两个分区)
副本: 1个副本(如果集群，有几个节点就写几个)
属性: 无，默认
[/
code] 发送消息[/
size]
发送消息–>在线发送
接入监控工具[/
size]
同时提供exporter监控工具，并接入到grafana显示。
现场根据环境选择exporter照旧利用ui管理工具，来完成根本的现场运维工作。
其他操作[/
size]
增长sapi用户认证[/
size]
vi config/
kraft/
jaas.conf
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
user_admin="icc_channel_2024";
};
vi config/
kraft/
server.properties
# 修改listeners和advertised
listeners=PLAINTEXT:/
/
192.168.56.103:9092,CONTROLLER:/
/
192.168.56.103:9093,SASL_PLAINTEXT:/
/
192.168.56.103:19092
advertised.listeners=PLAINTEXT:/
/
192.168.56.103:9092,SASL_PLAINTEXT:/
/
192.168.56.103:19092
# 添加mechanisms配置
sasl.enabled.mechanisms=PLAIN
执行下面这条命令启动节点：
KAFKA_OPTS=-Djava.security.auth.login.config=/
usr/
kafka/
kafka_2.12-3.8.0/
config/
kraft/
jaas.confbin/
kafka-server-start.sh -daemon/
usr/
kafka/
kafka_2.12-3.8.0/
config/
kraft/
server.properties
[/
code] 防火墙规则留意[/
size]
在配置过程中，肯定不要忘记防火墙规则，由其是容器化后的，因为利用端口映射，要特别留意[/
b]
表里网隔离利用[/
size]
内网是内部公司利用，外网用于对接第三方
kakfa创建容器的环境变量修改前：
KAFKA_LISTENERS: PLAINTEXT:/
/
:9092,CONTROLLER:/
/
:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT:/
/
10.11.24.15:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
修改后：
# 端口不能一样
KAFKA_LISTENERS: INSIDE:/
/
:19092,OUTSIDE:/
/
:9092,CONTROLLER:/
/
:9093
KAFKA_ADVERTISED_LISTENERS: INSIDE:/
/
10.11.24.15:19092,OUTSIDE:/
/
39.13.24.23:9092
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME=INSIDE
解释：
listeners这里设置了2个监听器，监听了所有IP的两个端口，19092为内网kafka使用端口，9092为外部访问端口
10.11.24.15是内网服务器IP地址，39.13.24.23为公网IP地址，端口已也添加了映射：39.13.24.23->10.11.24.15，这样我们当我们通过kafka客户端连接kafka时，kafka获取kafka元数据信息，这时候kafka客户端得到的就是advertised.listeners的信息，因为这里OUTSIDE配置的外网IP，当客户端通过39.13.24.23访问kafka时，请求就会被之前配置的映射进行转发，被转发到内网ip，从而完成对kafka的访问。
listener.security.protocol.map为监听器名称和安全协议之间的映射关系集合，PLAINTEXT表示明文
inter.broker.listener.name用于broker之间通信的listener名称，如果未设置，则listener名称由 security.inter.broker.protocol 定义(security.inter.broker.protocol默认值是PLAINTEXT)
[/
code] 配置利用长期化存储[/
size]
/
[/
code] 参考文档[/
size]
https:/
/
hub.docker.com/
r/
apache/
kafka
https:/
/
kafka.apache.org/
quickstart
https:/
/
github.com/
apache/
kafka
https:/
/
github.com/
xxd763795151/
kafka-console-ui
免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。
页:
[1]