ToB企服应用市场:ToB评测及商务社交产业平台

标题: win setup kafka 3.6.2 Step-by-Step Guide [打印本页]

---

作者: 徐锦洪    时间: 2024-6-19 22:37
标题: win setup kafka 3.6.2 Step-by-Step Guide
At the end of the document, some bugs are recorded
setup

from https://kafka.apache.org/downloads download .tgz binary package to local and extract
Prerequisites

• Kafka Installed: Ensure Kafka is installed and running.
  
• Java Installed: Kafka requires Java. Make sure having the JDK installed.
  

edit config file

• edit config/server.properties file：
  broker.id=0
  log.dirs=/tmp/kafka-logs # or in window use D:\\tmp\\kafka-logs
  zookeeper.connect=localhost:2181
  listeners=PLAINTEXT://:9092
  
• edit config/zookeeper.properties file:
  dataDir=/bigdata/zk # in win use D:\\bigdata\\zk
  

1. Start Kafka Server

Make sure Zookeeper and Kafka server are running.
Start Zookeeper:

1. .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

复制代码

Start Kafka server:

1. .\bin\windows\kafka-server-start.bat .\config\server.properties

复制代码

2. Create a Kafka Topic

Before producing and consuming messages, need a topic.

1. .\bin\windows\kafka-topics.bat --create --topic test --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

复制代码

3. Set Up Kafka Producer

use the Kafka console producer to send messages to the topic.
Open a new Command Prompt and run:

1. .\bin\windows\kafka-console-producer.bat --topic test --bootstrap-server localhost:9092

复制代码

Type messages in the console to send them to the Kafka topic.
4. Set Up Kafka Consumer

Open another Command Prompt to start the consumer that reads messages from the topic.

1. .\bin\windows\kafka-console-consumer.bat --topic test --bootstrap-server localhost:9092 --from-beginning

复制代码

should see messages in the consumer console as type them in the producer console.
Connecting Kafka with Code
Here are examples in Java and Python.
Java Example

First, add Kafka client dependencies topom.xml if using Maven:

1. <dependency>
2.     <groupId>org.apache.kafka</groupId>
3.     <artifactId>kafka-clients</artifactId>
4.     <version>3.6.2</version>
5. </dependency>

复制代码

Producer Example

1. import org.apache.kafka.clients.producer.KafkaProducer;
2. import org.apache.kafka.clients.producer.ProducerRecord;
4. import java.util.Properties;
6. public class KafkaProducerExample {
7.     public static void main(String[] args) {
8.         Properties props = new Properties();
9.         props.put("bootstrap.servers", "localhost:9092");
10.         props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
11.         props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
13.         KafkaProducer<String, String> producer = new KafkaProducer<>(props);
14.         producer.send(new ProducerRecord<>("test", "key", "value"));
15.         producer.close();
16.     }
17. }

复制代码

Consumer Example

1. import org.apache.kafka.clients.consumer.ConsumerRecords;
2. import org.apache.kafka.clients.consumer.KafkaConsumer;
4. import java.time.Duration;
5. import java.util.Collections;
6. import java.util.Properties;
8. public class KafkaConsumerExample {
9.     public static void main(String[] args) {
10.         Properties props = new Properties();
11.         props.put("bootstrap.servers", "localhost:9092");
12.         props.put("group.id", "test-group");
13.         props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
14.         props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
16.         KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
17.         consumer.subscribe(Collections.singletonList("test"));
19.         while (true) {
20.             ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
21.             records.forEach(record -> {
22.                 System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
23.             });
24.         }
25.     }
26. }

复制代码

Python Example

First, install the Kafka Python client:

1. pip install kafka-python

复制代码

Producer Example

1. from kafka import KafkaProducer
3. producer = KafkaProducer(bootstrap_servers='localhost:9092')
4. producer.send('test', b'Hello, Kafka!')
5. producer.close()

复制代码

Consumer Example

1. from kafka import KafkaConsumer
3. consumer = KafkaConsumer('test', bootstrap_servers='localhost:9092', auto_offset_reset='earliest')
4. for message in consumer:
5.     print(f"Key: {message.key}, Value: {message.value}")

复制代码

Note

• Start Zookeeper and Kafka server: Ensure they are running correctly.
  
• Create topics: Use Kafka commands to create the required topics.
  

some bug

‘wmic’ is not recognized as an internal or external command, operable program or batch file
click Environment Variables. In the section for system variables, find PATH (or any capitalization thereof). Add this entry to it:
%SystemRoot%\System32\Wbem
ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$) java.nio.file.InvalidPathException: Illegal char < > at index 2: D: mpdownloadkafkakafka_2.13-3.6.2log\meta.properties.tmp
Correct the Path Format:
Ensure that the path specified in configuration does not contain illegal characters or spaces. Paths in Windows should use double backslashes \ or a single forward slash /.
WARN [SocketServer listenerType=ZK_BROKER, nodeId=0] Unexpected error from /0:0:0:0:0:0:0:1 (channelId=0:0:0:0:0:0:0:1:9092-0:0:0:0:0:0:0:1:62710-1); closing connection (org.apache.kafka.common.network.Selector) org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)

• Edit server.properties:
  Open the server.properties file in Kafka config directory and increase the max.request.size property. Add or modify the following lines:
  max.request.size=209715200 # Increase this value as needed, default is 104857600 (100MB)
  socket.request.max.bytes=209715200 # Ensure this matches or exceeds max.request.size
  
• Edit consumer.properties and producer.properties (if applicable):
  If have consumer and producer configurations, ensure that these properties are set appropriately there as well:
  max.request.size=209715200
  

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。

---

欢迎光临 ToB企服应用市场:ToB评测及商务社交产业平台 (https://dis.qidao123.com/)

Powered by Discuz! X3.4