Kafka实战一

Kafka实战一

搭建环境

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# 下载安装包
http://mirror.nus.edu.sg/apache/kafka/0.8.1.1/kafka_2.10-0.8.1.1.tgz

# 上传到Linux /home

# 解压
tar xzf kafka_2.10-0.8.1.1.tgz -C /usr/local
cd /usr/local
mv kafka_2.10-0.8.1.1 kafka

# 启动 Zookeeper
cd kafka
./zookeeper-server-start.sh config/zookeeper.properties

# 启动 Kafka
./kafka-server-start.sh config/server.properties

测试–单个broker

新建Topic

1
2
3
4
5
# 列出存在的Topic
./bin/kafka-topics.sh --list --zookeeper localhost:2181

# 新建
./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topicTest

发送消息

1
2
3
4
# console
./kafka-console-producer.sh --broker-list localhost:9092 --topic topicTest
this is topicTest producer test
hello world!

消费消息

1
2
# console
./kafka-console-consumer.sh --zookeeper localhost:2181 --topic topicTest --from-beginning

测试–多个broker

创建broker

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 编辑配置文件
cp server.properties server-1.properties
cp server.properties server-2.properties

vim server-1.properties
# 修改内容
broker.id=1
port=9093
log.dirs=/tmp/kafka-logs-1

vim server-2.properties
# 修改内容
broker.id=2
port=9094
log.dirs=/tmp/kafka-logs-2

# 启动
./kafka-server-start.sh -daemon ../config/server-1.properties
./kafka-server-start.sh -daemon ../config/server-2.properties

创建topic, 设置备份为3

1
./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic three-replication-topicTest

查看topic

1
2
3
4
5
6
7
8
# 查看
./kafka-topics.sh -describe --zookeeper localhost:2181 --topic three-replication-topicTest

# 结果如下
# 分区汇总信息
Topic:three-replication-topicTest PartitionCount:1 ReplicationFactor:3 Configs:
# Leader: leader节点是2; Replicas: 在节点2,0,1上; Isr: 工作中的复制节点的集合,也就是活的节点的集合;
Topic: three-replication-topicTest Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1

发送消息

1
./kafka-console-producer.sh --broker-list localhost:9092 --topic three-replication-topicTest

消费消息

1
./kafka-console-consumer.sh --zookeeper localhost:2181 --topic three-replication-topicTest --from-beginning

删除leader(即broker2)

1
2
3
4
5
6
7
8
9
10
11
12
13
# 查看进程
ps -ef|grep server-2.properties

# 终止
kil -9 27628

# 查看topic
Topic:three-replication-topicTest PartitionCount:1 ReplicationFactor:3 Configs:
# 2号节点被终止, leader节点由0号节点顶替(即broker0), 2号节点未被列入Isr中(因为已被终止)
Topic: three-replication-topicTest Partition: 0 Leader: 0 Replicas: 2,0,1 Isr: 0,1

# 发送消息进行测试
消息正常发送及接收

再删除新的leader(即broker0)

发送消息时报错, 正在查找原因中……..
找到原因: 生产者连接的broker被终止导致发送消息出错(连接异常)

采用四个broker及四个replication-factor

1
2
Topic:topicRepl PartitionCount:1        ReplicationFactor:4     Configs:
Topic: topicRepl Partition: 0 Leader: 0 Replicas: 0,2,3,1 Isr: 0,2,3,1

逐个删除leader

java.net.UnknownHostException: hadoop229: hadoop229: 未知的名称或服务

解决方案:

1
2
3
vim /etc/hosts

# 在末尾添加 127.0.0.1 hadoop229

参考资料

hadoop坑

Kafka入门

转载

本文出自<<arccode>>, 欢迎转载, 转载请注明出处.