如何列出集群中所有可用的卡夫卡经纪人？

我正在编写一个 shell 脚本来监视卡夫卡经纪人。

我浏览了一些链接，发现如果 ZooKeeper 包含一个代理列表，并且如果在这个列表中存在 IP 地址，那么一个卡夫卡代理正在运行。

我想要一个命令，我可以在我的 shell 脚本中使用，以获得代理列表，并检查是否卡夫卡运行。

是否有任何 curl命令来获得卡夫卡集群状态，如 elasticsearch？

194236 次浏览
小开
最佳答案

This command will give you the list of the active brokers between brackets:

./bin/zookeeper-shell.sh localhost:2181 ls /brokers/ids
小开

Alternate way using Zk-Client:

If you do not prefer to pass arguments to ./zookeeper-shell.sh and want to see the broker details from Zookeeper CLI, you need to install standalone Zookeeper (As traditional Kafka do not comes up with Jline JAR).

Once you install(unzip) the standalone Zookeeper,then:

  • Run the Zookeeper CLI:
    $ zookeeper/bin/zkCli.sh -server localhost:2181 #Make sure your Broker is already running

  • If it is successful, you can see the Zk client running as:

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]
  • From here you can explore the broker details using various commands:

$ ls /brokers/ids # Gives the list of active brokers
$ ls /brokers/topics #Gives the list of topics
$ get /brokers/ids/0 #Gives more detailed information of the broker id '0'

小开

To use zookeeper commands with shell script try

zookeeper/bin/zkCli.sh -server localhost:2181 <<< "ls /brokers/ids" | tail -n 1. The last line usually has the response details

小开 
echo dump | nc localhost 2181 | grep brokers

(replace localhost with the host where zookeeper is running)

小开

I did it like this

#!/bin/bash


ZK_HOST="localhost"
ZK_PORT=2181




for i in `echo dump | nc $ZK_HOST $ZK_PORT | grep brokers`
do
echo $i
DETAIL=`zkCli -server "$ZK_HOST:$ZK_PORT" get $i 2>/dev/null | tail -n 1`
echo $DETAIL
done
小开

Here are a couple of quick functions I use when bash scripting Kafka Data Load into Demo Environments. In this example I use HDP with no security, but it is easily modified to other environments and intended to be quick and functional rather than particularly robust.

The first retrieves the address of the first ZooKeeper node from the config:

ZKS1=$(cat /usr/hdp/current/zookeeper-client/conf/zoo.cfg | grep server.1)
[[ ${ZKS1} =~ server.1=(.*?):[0-9]*:[0-9]* ]]
export ZKADDR=${BASH_REMATCH[1]}:2181
echo "using ZooKeeper Server $ZKADDR"

The second retrieves the Broker IDs from ZooKeeper:

echo "Fetching list of Kafka Brokers"
export BROKERIDS=$(/usr/hdp/current/kafka-broker/bin/zookeeper-shell.sh ${ZKADDR} <<< 'ls /brokers/ids' | tail -1)
export BROKERIDS=${BROKERIDS//[!0-9 ]/}
echo "Found Kafka Broker IDS: $BROKERIDS"

The third parses ZooKeeper again to retrieve the list of Kafka Brokers Host:port ready for use in the command-line client:

unset BROKERS
for i in $BROKERIDS
do
DETAIL=$(/usr/hdp/current/kafka-broker/bin/zookeeper-shell.sh ${ZKADDR} <<< "get /brokers/ids/$i")
[[ $DETAIL =~ PLAINTEXT:\/\/(.*?)\"\] ]]
if [ -z ${BROKERS+x} ]; then BROKERS=${BASH_REMATCH[1]}; else
BROKERS="${BROKERS},${BASH_REMATCH[1]}"; fi
done
echo "Found Brokerlist: $BROKERS"
小开

On MacOS, can try:

brew tap let-us-go/zkcli
brew install zkcli


zkcli ls /brokers/ids
zkcli get /brokers/ids/1
小开

If you are using new version of Kafka e.g. 5.3.3, you can use

kafka-broker-api-versions  --bootstrap-server BROKER | grep 9092

You just need to pass one of the brokers

小开

Using Confluent's REST Proxy API v3:

curl -X GET -H "Accept: application/vnd.api+json" localhost:8082/v3/clusters

where localhost:8082 is Kafka Proxy address.


提交答案