How to retrieve details of all broker from kafka cluster
1
How should we retrieve the full details of all brokers(connected/disconnected) from kafka cluster/zookeeper ?
I found following way to fetch only active brokers, but I want the the IP address of broker which is serving previously in cluster but it is disconnected now
Following code snippet gives list of active brokers:
ZooKeeper zkInstance = new ZooKeeper("mymachine:port", 10000, null);
brokerIDs = zkInstance.getChildren("/brokers/ids", false);
for (String brokerID : brokerIDs)
brokerInfo = new String(zkInstance.getData("/brokers/ids/" + brokerID, false, null));
String host=brokerInfo.substring(brokerInfo.indexOf(""host"")).split(",") [0].replaceAll(""","").split(":")[1];
String port=brokerInfo.substring(brokerInfo.indexOf(""jmx_port"")).split(",") [0].replaceAll(""","").split(":")[1];
System.out.println(host+":"+port);
Output:
- my-machine-1:port
- my-machine-2:port
- my-machine-3:port
- my-machine-4:port
I need information of all connected/disconnected brokers in multi node kafka cluster
java apache-kafka apache-zookeeper
edited Jul 14 at 0:37
cricket_007
79.1k1142109
asked Jul 13 at 7:50
MAS
62
add a comment |
1
How should we retrieve the full details of all brokers(connected/disconnected) from kafka cluster/zookeeper ?
I found following way to fetch only active brokers, but I want the the IP address of broker which is serving previously in cluster but it is disconnected now
Following code snippet gives list of active brokers:
ZooKeeper zkInstance = new ZooKeeper("mymachine:port", 10000, null);
brokerIDs = zkInstance.getChildren("/brokers/ids", false);
for (String brokerID : brokerIDs)
brokerInfo = new String(zkInstance.getData("/brokers/ids/" + brokerID, false, null));
String host=brokerInfo.substring(brokerInfo.indexOf(""host"")).split(",") [0].replaceAll(""","").split(":")[1];
String port=brokerInfo.substring(brokerInfo.indexOf(""jmx_port"")).split(",") [0].replaceAll(""","").split(":")[1];
System.out.println(host+":"+port);
Output:
- my-machine-1:port
- my-machine-2:port
- my-machine-3:port
- my-machine-4:port
I need information of all connected/disconnected brokers in multi node kafka cluster
java apache-kafka apache-zookeeper
edited Jul 14 at 0:37
cricket_007
79.1k1142109
asked Jul 13 at 7:50
MAS
62
If it's disconnected, I'm not sure how you're expecting to get that information... You should be monitoring JMX directly on the Kafka servers to determine what they're connected to
– cricket_007
Jul 14 at 0:38
Kafka allocates some amount of replicas to partitions and these partitions are used by the topics, replicas are the actual nodes in cluster so there might be the provision given in Kafka to store the meta-data for replicas like broker number, JMX port and broker port where process is running, while these replicas are active ZooKeeper allows us to fetch these details but not in the case when broker goes down/inactive.
– MAS
Jul 15 at 16:33
That's assuming you have replicas for a topic, though. Plus, if a replica is down, though, then it'll have under-replicated partitions and list a-1when you describe the topic
– cricket_007
Jul 15 at 20:27
Yes I got the same output when I described that particular topic, As you use the term under-replicated partitions so what kind information exactly it will contains? Is it only contains broker id and its status or it will contains all the metadata for that particular replica.
– MAS
Jul 16 at 6:14
add a comment |
1
1
1
1
How should we retrieve the full details of all brokers(connected/disconnected) from kafka cluster/zookeeper ?
I found following way to fetch only active brokers, but I want the the IP address of broker which is serving previously in cluster but it is disconnected now
Following code snippet gives list of active brokers:
ZooKeeper zkInstance = new ZooKeeper("mymachine:port", 10000, null);
brokerIDs = zkInstance.getChildren("/brokers/ids", false);
for (String brokerID : brokerIDs)
brokerInfo = new String(zkInstance.getData("/brokers/ids/" + brokerID, false, null));
String host=brokerInfo.substring(brokerInfo.indexOf(""host"")).split(",") [0].replaceAll(""","").split(":")[1];
String port=brokerInfo.substring(brokerInfo.indexOf(""jmx_port"")).split(",") [0].replaceAll(""","").split(":")[1];
System.out.println(host+":"+port);
Output:
- my-machine-1:port
- my-machine-2:port
- my-machine-3:port
- my-machine-4:port
I need information of all connected/disconnected brokers in multi node kafka cluster
java apache-kafka apache-zookeeper
edited Jul 14 at 0:37
cricket_007
79.1k1142109
asked Jul 13 at 7:50
MAS
62
How should we retrieve the full details of all brokers(connected/disconnected) from kafka cluster/zookeeper ?
I found following way to fetch only active brokers, but I want the the IP address of broker which is serving previously in cluster but it is disconnected now
Following code snippet gives list of active brokers:
ZooKeeper zkInstance = new ZooKeeper("mymachine:port", 10000, null);
brokerIDs = zkInstance.getChildren("/brokers/ids", false);
for (String brokerID : brokerIDs)
brokerInfo = new String(zkInstance.getData("/brokers/ids/" + brokerID, false, null));
String host=brokerInfo.substring(brokerInfo.indexOf(""host"")).split(",") [0].replaceAll(""","").split(":")[1];
String port=brokerInfo.substring(brokerInfo.indexOf(""jmx_port"")).split(",") [0].replaceAll(""","").split(":")[1];
System.out.println(host+":"+port);
Output:
- my-machine-1:port
- my-machine-2:port
- my-machine-3:port
- my-machine-4:port
I need information of all connected/disconnected brokers in multi node kafka cluster
java apache-kafka apache-zookeeper
java apache-kafka apache-zookeeper
edited Jul 14 at 0:37
cricket_007
79.1k1142109
asked Jul 13 at 7:50
MAS
62
edited Jul 14 at 0:37
cricket_007
79.1k1142109
asked Jul 13 at 7:50
MAS
62
edited Jul 14 at 0:37
cricket_007
79.1k1142109
edited Jul 14 at 0:37
cricket_007
79.1k1142109
edited Jul 14 at 0:37
cricket_007
79.1k1142109
79.1k1142109
asked Jul 13 at 7:50
MAS
62
asked Jul 13 at 7:50
MAS
62
asked Jul 13 at 7:50
MAS
62
62
If it's disconnected, I'm not sure how you're expecting to get that information... You should be monitoring JMX directly on the Kafka servers to determine what they're connected to
– cricket_007
Jul 14 at 0:38
Kafka allocates some amount of replicas to partitions and these partitions are used by the topics, replicas are the actual nodes in cluster so there might be the provision given in Kafka to store the meta-data for replicas like broker number, JMX port and broker port where process is running, while these replicas are active ZooKeeper allows us to fetch these details but not in the case when broker goes down/inactive.
– MAS
Jul 15 at 16:33
That's assuming you have replicas for a topic, though. Plus, if a replica is down, though, then it'll have under-replicated partitions and list a-1when you describe the topic
– cricket_007
Jul 15 at 20:27
Yes I got the same output when I described that particular topic, As you use the term under-replicated partitions so what kind information exactly it will contains? Is it only contains broker id and its status or it will contains all the metadata for that particular replica.
– MAS
Jul 16 at 6:14
add a comment |
If it's disconnected, I'm not sure how you're expecting to get that information... You should be monitoring JMX directly on the Kafka servers to determine what they're connected to
– cricket_007
Jul 14 at 0:38
Kafka allocates some amount of replicas to partitions and these partitions are used by the topics, replicas are the actual nodes in cluster so there might be the provision given in Kafka to store the meta-data for replicas like broker number, JMX port and broker port where process is running, while these replicas are active ZooKeeper allows us to fetch these details but not in the case when broker goes down/inactive.
– MAS
Jul 15 at 16:33
That's assuming you have replicas for a topic, though. Plus, if a replica is down, though, then it'll have under-replicated partitions and list a-1when you describe the topic
– cricket_007
Jul 15 at 20:27
Yes I got the same output when I described that particular topic, As you use the term under-replicated partitions so what kind information exactly it will contains? Is it only contains broker id and its status or it will contains all the metadata for that particular replica.
– MAS
Jul 16 at 6:14
If it's disconnected, I'm not sure how you're expecting to get that information... You should be monitoring JMX directly on the Kafka servers to determine what they're connected to
– cricket_007
Jul 14 at 0:38
If it's disconnected, I'm not sure how you're expecting to get that information... You should be monitoring JMX directly on the Kafka servers to determine what they're connected to
– cricket_007
Jul 14 at 0:38
Kafka allocates some amount of replicas to partitions and these partitions are used by the topics, replicas are the actual nodes in cluster so there might be the provision given in Kafka to store the meta-data for replicas like broker number, JMX port and broker port where process is running, while these replicas are active ZooKeeper allows us to fetch these details but not in the case when broker goes down/inactive.
– MAS
Jul 15 at 16:33
Kafka allocates some amount of replicas to partitions and these partitions are used by the topics, replicas are the actual nodes in cluster so there might be the provision given in Kafka to store the meta-data for replicas like broker number, JMX port and broker port where process is running, while these replicas are active ZooKeeper allows us to fetch these details but not in the case when broker goes down/inactive.
– MAS
Jul 15 at 16:33
That's assuming you have replicas for a topic, though. Plus, if a replica is down, though, then it'll have under-replicated partitions and list a
-1 when you describe the topic– cricket_007
Jul 15 at 20:27
That's assuming you have replicas for a topic, though. Plus, if a replica is down, though, then it'll have under-replicated partitions and list a
-1 when you describe the topic– cricket_007
Jul 15 at 20:27
Yes I got the same output when I described that particular topic, As you use the term under-replicated partitions so what kind information exactly it will contains? Is it only contains broker id and its status or it will contains all the metadata for that particular replica.
– MAS
Jul 16 at 6:14
Yes I got the same output when I described that particular topic, As you use the term under-replicated partitions so what kind information exactly it will contains? Is it only contains broker id and its status or it will contains all the metadata for that particular replica.
– MAS
Jul 16 at 6:14
add a comment |
1 Answer
1
active
oldest
votes
0
Use describeCluster() of the AdminClient class to get the broker details such as host,port,id and rock.
Please refer the following code:
Properties kafkaProperties = new Properties();
kafkaProperties.put("bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094");
AdminClient adminClient = AdminClient.create(kafkaProperties);
DescribeClusterResult describeClusterResult = adminClient.describeCluster();
Collection<Node> brokerDetails = describeClusterResult.nodes().get();
System.out.println("host and port details");
for(Node broker:brokerDetails)
System.out.println(broker.host()+":"+broker.port());
edited Nov 12 at 12:45
Nick
23.2k91535
answered Nov 12 at 12:20
Manjunatha H C
112
Please format your code as described in stackoverflow.com/editing-help. It makes it a lot easier to read. Thanks.
– Nick
Nov 12 at 12:45
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f51320196%2fhow-to-retrieve-details-of-all-broker-from-kafka-cluster%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
Use describeCluster() of the AdminClient class to get the broker details such as host,port,id and rock.
Please refer the following code:
Properties kafkaProperties = new Properties();
kafkaProperties.put("bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094");
AdminClient adminClient = AdminClient.create(kafkaProperties);
DescribeClusterResult describeClusterResult = adminClient.describeCluster();
Collection<Node> brokerDetails = describeClusterResult.nodes().get();
System.out.println("host and port details");
for(Node broker:brokerDetails)
System.out.println(broker.host()+":"+broker.port());
edited Nov 12 at 12:45
Nick
23.2k91535
answered Nov 12 at 12:20
Manjunatha H C
112
Please format your code as described in stackoverflow.com/editing-help. It makes it a lot easier to read. Thanks.
– Nick
Nov 12 at 12:45
add a comment |
0
Use describeCluster() of the AdminClient class to get the broker details such as host,port,id and rock.
Please refer the following code:
Properties kafkaProperties = new Properties();
kafkaProperties.put("bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094");
AdminClient adminClient = AdminClient.create(kafkaProperties);
DescribeClusterResult describeClusterResult = adminClient.describeCluster();
Collection<Node> brokerDetails = describeClusterResult.nodes().get();
System.out.println("host and port details");
for(Node broker:brokerDetails)
System.out.println(broker.host()+":"+broker.port());
edited Nov 12 at 12:45
Nick
23.2k91535
answered Nov 12 at 12:20
Manjunatha H C
112
Please format your code as described in stackoverflow.com/editing-help. It makes it a lot easier to read. Thanks.
– Nick
Nov 12 at 12:45
add a comment |
0
0
0
Use describeCluster() of the AdminClient class to get the broker details such as host,port,id and rock.
Please refer the following code:
Properties kafkaProperties = new Properties();
kafkaProperties.put("bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094");
AdminClient adminClient = AdminClient.create(kafkaProperties);
DescribeClusterResult describeClusterResult = adminClient.describeCluster();
Collection<Node> brokerDetails = describeClusterResult.nodes().get();
System.out.println("host and port details");
for(Node broker:brokerDetails)
System.out.println(broker.host()+":"+broker.port());
edited Nov 12 at 12:45
Nick
23.2k91535
answered Nov 12 at 12:20
Manjunatha H C
112
Use describeCluster() of the AdminClient class to get the broker details such as host,port,id and rock.
Please refer the following code:
Properties kafkaProperties = new Properties();
kafkaProperties.put("bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094");
AdminClient adminClient = AdminClient.create(kafkaProperties);
DescribeClusterResult describeClusterResult = adminClient.describeCluster();
Collection<Node> brokerDetails = describeClusterResult.nodes().get();
System.out.println("host and port details");
for(Node broker:brokerDetails)
System.out.println(broker.host()+":"+broker.port());
edited Nov 12 at 12:45
Nick
23.2k91535
answered Nov 12 at 12:20
Manjunatha H C
112
edited Nov 12 at 12:45
Nick
23.2k91535
edited Nov 12 at 12:45
Nick
23.2k91535
edited Nov 12 at 12:45
Nick
23.2k91535
23.2k91535
answered Nov 12 at 12:20
Manjunatha H C
112
answered Nov 12 at 12:20
Manjunatha H C
112
answered Nov 12 at 12:20
Manjunatha H C
112
112
Please format your code as described in stackoverflow.com/editing-help. It makes it a lot easier to read. Thanks.
– Nick
Nov 12 at 12:45
add a comment |
Please format your code as described in stackoverflow.com/editing-help. It makes it a lot easier to read. Thanks.
– Nick
Nov 12 at 12:45
Please format your code as described in stackoverflow.com/editing-help. It makes it a lot easier to read. Thanks.
– Nick
Nov 12 at 12:45
Please format your code as described in stackoverflow.com/editing-help. It makes it a lot easier to read. Thanks.
– Nick
Nov 12 at 12:45
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f51320196%2fhow-to-retrieve-details-of-all-broker-from-kafka-cluster%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
If it's disconnected, I'm not sure how you're expecting to get that information... You should be monitoring JMX directly on the Kafka servers to determine what they're connected to
– cricket_007
Jul 14 at 0:38
Kafka allocates some amount of replicas to partitions and these partitions are used by the topics, replicas are the actual nodes in cluster so there might be the provision given in Kafka to store the meta-data for replicas like broker number, JMX port and broker port where process is running, while these replicas are active ZooKeeper allows us to fetch these details but not in the case when broker goes down/inactive.
– MAS
Jul 15 at 16:33
That's assuming you have replicas for a topic, though. Plus, if a replica is down, though, then it'll have under-replicated partitions and list a
-1when you describe the topic– cricket_007
Jul 15 at 20:27
Yes I got the same output when I described that particular topic, As you use the term under-replicated partitions so what kind information exactly it will contains? Is it only contains broker id and its status or it will contains all the metadata for that particular replica.
– MAS
Jul 16 at 6:14