How to retrieve details of all broker from kafka cluster

How should we retrieve the full details of all brokers(connected/disconnected) from kafka cluster/zookeeper ?

I found following way to fetch only active brokers, but I want the the IP address of broker which is serving previously in cluster but it is disconnected now

Following code snippet gives list of active brokers:

ZooKeeper zkInstance = new ZooKeeper("mymachine:port", 10000, null);
brokerIDs = zkInstance.getChildren("/brokers/ids", false);
for (String brokerID : brokerIDs) 
 brokerInfo = new String(zkInstance.getData("/brokers/ids/" + brokerID, false, null));
 String host=brokerInfo.substring(brokerInfo.indexOf(""host"")).split(",") [0].replaceAll(""","").split(":")[1];
 String port=brokerInfo.substring(brokerInfo.indexOf(""jmx_port"")).split(",") [0].replaceAll(""","").split(":")[1];
 System.out.println(host+":"+port);

Output:

my-machine-1:port

my-machine-2:port

my-machine-3:port

my-machine-4:port

I need information of all connected/disconnected brokers in multi node kafka cluster

edited Jul 14 at 0:37

cricket_007

79.1k1142109

asked Jul 13 at 7:50

MAS

If it's disconnected, I'm not sure how you're expecting to get that information... You should be monitoring JMX directly on the Kafka servers to determine what they're connected to
– cricket_007
Jul 14 at 0:38

Kafka allocates some amount of replicas to partitions and these partitions are used by the topics, replicas are the actual nodes in cluster so there might be the provision given in Kafka to store the meta-data for replicas like broker number, JMX port and broker port where process is running, while these replicas are active ZooKeeper allows us to fetch these details but not in the case when broker goes down/inactive.
– MAS
Jul 15 at 16:33

That's assuming you have replicas for a topic, though. Plus, if a replica is down, though, then it'll have under-replicated partitions and list a -1 when you describe the topic
– cricket_007
Jul 15 at 20:27

Yes I got the same output when I described that particular topic, As you use the term under-replicated partitions so what kind information exactly it will contains? Is it only contains broker id and its status or it will contains all the metadata for that particular replica.
– MAS
Jul 16 at 6:14

add a comment |

How should we retrieve the full details of all brokers(connected/disconnected) from kafka cluster/zookeeper ?

I found following way to fetch only active brokers, but I want the the IP address of broker which is serving previously in cluster but it is disconnected now

Following code snippet gives list of active brokers:

ZooKeeper zkInstance = new ZooKeeper("mymachine:port", 10000, null);
brokerIDs = zkInstance.getChildren("/brokers/ids", false);
for (String brokerID : brokerIDs) 
 brokerInfo = new String(zkInstance.getData("/brokers/ids/" + brokerID, false, null));
 String host=brokerInfo.substring(brokerInfo.indexOf(""host"")).split(",") [0].replaceAll(""","").split(":")[1];
 String port=brokerInfo.substring(brokerInfo.indexOf(""jmx_port"")).split(",") [0].replaceAll(""","").split(":")[1];
 System.out.println(host+":"+port);

Output:

my-machine-1:port

my-machine-2:port

my-machine-3:port

my-machine-4:port

I need information of all connected/disconnected brokers in multi node kafka cluster

edited Jul 14 at 0:37

cricket_007

79.1k1142109

asked Jul 13 at 7:50

MAS

If it's disconnected, I'm not sure how you're expecting to get that information... You should be monitoring JMX directly on the Kafka servers to determine what they're connected to
– cricket_007
Jul 14 at 0:38

Kafka allocates some amount of replicas to partitions and these partitions are used by the topics, replicas are the actual nodes in cluster so there might be the provision given in Kafka to store the meta-data for replicas like broker number, JMX port and broker port where process is running, while these replicas are active ZooKeeper allows us to fetch these details but not in the case when broker goes down/inactive.
– MAS
Jul 15 at 16:33

That's assuming you have replicas for a topic, though. Plus, if a replica is down, though, then it'll have under-replicated partitions and list a -1 when you describe the topic
– cricket_007
Jul 15 at 20:27

Yes I got the same output when I described that particular topic, As you use the term under-replicated partitions so what kind information exactly it will contains? Is it only contains broker id and its status or it will contains all the metadata for that particular replica.
– MAS
Jul 16 at 6:14

add a comment |

How should we retrieve the full details of all brokers(connected/disconnected) from kafka cluster/zookeeper ?

I found following way to fetch only active brokers, but I want the the IP address of broker which is serving previously in cluster but it is disconnected now

Following code snippet gives list of active brokers:

ZooKeeper zkInstance = new ZooKeeper("mymachine:port", 10000, null);
brokerIDs = zkInstance.getChildren("/brokers/ids", false);
for (String brokerID : brokerIDs) 
 brokerInfo = new String(zkInstance.getData("/brokers/ids/" + brokerID, false, null));
 String host=brokerInfo.substring(brokerInfo.indexOf(""host"")).split(",") [0].replaceAll(""","").split(":")[1];
 String port=brokerInfo.substring(brokerInfo.indexOf(""jmx_port"")).split(",") [0].replaceAll(""","").split(":")[1];
 System.out.println(host+":"+port);

Output:

my-machine-1:port

my-machine-2:port

my-machine-3:port

my-machine-4:port

I need information of all connected/disconnected brokers in multi node kafka cluster

edited Jul 14 at 0:37

cricket_007

79.1k1142109

asked Jul 13 at 7:50

MAS

How should we retrieve the full details of all brokers(connected/disconnected) from kafka cluster/zookeeper ?

I found following way to fetch only active brokers, but I want the the IP address of broker which is serving previously in cluster but it is disconnected now

Following code snippet gives list of active brokers:

ZooKeeper zkInstance = new ZooKeeper("mymachine:port", 10000, null);
brokerIDs = zkInstance.getChildren("/brokers/ids", false);
for (String brokerID : brokerIDs) 
 brokerInfo = new String(zkInstance.getData("/brokers/ids/" + brokerID, false, null));
 String host=brokerInfo.substring(brokerInfo.indexOf(""host"")).split(",") [0].replaceAll(""","").split(":")[1];
 String port=brokerInfo.substring(brokerInfo.indexOf(""jmx_port"")).split(",") [0].replaceAll(""","").split(":")[1];
 System.out.println(host+":"+port);

Output:

my-machine-1:port

my-machine-2:port

my-machine-3:port

my-machine-4:port

I need information of all connected/disconnected brokers in multi node kafka cluster

java apache-kafka apache-zookeeper

edited Jul 14 at 0:37

cricket_007

79.1k1142109

asked Jul 13 at 7:50

MAS

edited Jul 14 at 0:37

cricket_007

79.1k1142109

asked Jul 13 at 7:50

MAS

edited Jul 14 at 0:37

cricket_007

79.1k1142109

edited Jul 14 at 0:37

cricket_007

79.1k1142109

edited Jul 14 at 0:37

cricket_007

79.1k1142109

asked Jul 13 at 7:50

MAS

asked Jul 13 at 7:50

MAS

asked Jul 13 at 7:50

MAS

If it's disconnected, I'm not sure how you're expecting to get that information... You should be monitoring JMX directly on the Kafka servers to determine what they're connected to
– cricket_007
Jul 14 at 0:38

Kafka allocates some amount of replicas to partitions and these partitions are used by the topics, replicas are the actual nodes in cluster so there might be the provision given in Kafka to store the meta-data for replicas like broker number, JMX port and broker port where process is running, while these replicas are active ZooKeeper allows us to fetch these details but not in the case when broker goes down/inactive.
– MAS
Jul 15 at 16:33

That's assuming you have replicas for a topic, though. Plus, if a replica is down, though, then it'll have under-replicated partitions and list a -1 when you describe the topic
– cricket_007
Jul 15 at 20:27

Yes I got the same output when I described that particular topic, As you use the term under-replicated partitions so what kind information exactly it will contains? Is it only contains broker id and its status or it will contains all the metadata for that particular replica.
– MAS
Jul 16 at 6:14

add a comment |

If it's disconnected, I'm not sure how you're expecting to get that information... You should be monitoring JMX directly on the Kafka servers to determine what they're connected to
– cricket_007
Jul 14 at 0:38

Kafka allocates some amount of replicas to partitions and these partitions are used by the topics, replicas are the actual nodes in cluster so there might be the provision given in Kafka to store the meta-data for replicas like broker number, JMX port and broker port where process is running, while these replicas are active ZooKeeper allows us to fetch these details but not in the case when broker goes down/inactive.
– MAS
Jul 15 at 16:33

That's assuming you have replicas for a topic, though. Plus, if a replica is down, though, then it'll have under-replicated partitions and list a -1 when you describe the topic
– cricket_007
Jul 15 at 20:27

Yes I got the same output when I described that particular topic, As you use the term under-replicated partitions so what kind information exactly it will contains? Is it only contains broker id and its status or it will contains all the metadata for that particular replica.
– MAS
Jul 16 at 6:14

If it's disconnected, I'm not sure how you're expecting to get that information... You should be monitoring JMX directly on the Kafka servers to determine what they're connected to
– cricket_007
Jul 14 at 0:38

Kafka allocates some amount of replicas to partitions and these partitions are used by the topics, replicas are the actual nodes in cluster so there might be the provision given in Kafka to store the meta-data for replicas like broker number, JMX port and broker port where process is running, while these replicas are active ZooKeeper allows us to fetch these details but not in the case when broker goes down/inactive.
– MAS
Jul 15 at 16:33

That's assuming you have replicas for a topic, though. Plus, if a replica is down, though, then it'll have under-replicated partitions and list a -1 when you describe the topic
– cricket_007
Jul 15 at 20:27

Yes I got the same output when I described that particular topic, As you use the term under-replicated partitions so what kind information exactly it will contains? Is it only contains broker id and its status or it will contains all the metadata for that particular replica.
– MAS
Jul 16 at 6:14

add a comment |

1 Answer
1

active

oldest

votes

Use describeCluster() of the AdminClient class to get the broker details such as host,port,id and rock.

Please refer the following code:

Properties kafkaProperties = new Properties();
kafkaProperties.put("bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094");
AdminClient adminClient = AdminClient.create(kafkaProperties);
DescribeClusterResult describeClusterResult = adminClient.describeCluster();
Collection<Node> brokerDetails = describeClusterResult.nodes().get();
System.out.println("host and port details");
for(Node broker:brokerDetails) 
 System.out.println(broker.host()+":"+broker.port());

edited Nov 12 at 12:45

Nick

23.2k91535

answered Nov 12 at 12:20

Manjunatha H C

112

Please format your code as described in stackoverflow.com/editing-help. It makes it a lot easier to read. Thanks.
– Nick
Nov 12 at 12:45

add a comment |

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f51320196%2fhow-to-retrieve-details-of-all-broker-from-kafka-cluster%23new-answer', 'question_page');

);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

Use describeCluster() of the AdminClient class to get the broker details such as host,port,id and rock.

Please refer the following code:

Properties kafkaProperties = new Properties();
kafkaProperties.put("bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094");
AdminClient adminClient = AdminClient.create(kafkaProperties);
DescribeClusterResult describeClusterResult = adminClient.describeCluster();
Collection<Node> brokerDetails = describeClusterResult.nodes().get();
System.out.println("host and port details");
for(Node broker:brokerDetails) 
 System.out.println(broker.host()+":"+broker.port());

edited Nov 12 at 12:45

Nick

23.2k91535

answered Nov 12 at 12:20

Manjunatha H C

112

Please format your code as described in stackoverflow.com/editing-help. It makes it a lot easier to read. Thanks.
– Nick
Nov 12 at 12:45

add a comment |

Use describeCluster() of the AdminClient class to get the broker details such as host,port,id and rock.

Please refer the following code:

Properties kafkaProperties = new Properties();
kafkaProperties.put("bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094");
AdminClient adminClient = AdminClient.create(kafkaProperties);
DescribeClusterResult describeClusterResult = adminClient.describeCluster();
Collection<Node> brokerDetails = describeClusterResult.nodes().get();
System.out.println("host and port details");
for(Node broker:brokerDetails) 
 System.out.println(broker.host()+":"+broker.port());

edited Nov 12 at 12:45

Nick

23.2k91535

answered Nov 12 at 12:20

Manjunatha H C

112

Please format your code as described in stackoverflow.com/editing-help. It makes it a lot easier to read. Thanks.
– Nick
Nov 12 at 12:45

add a comment |

Use describeCluster() of the AdminClient class to get the broker details such as host,port,id and rock.

Please refer the following code:

Properties kafkaProperties = new Properties();
kafkaProperties.put("bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094");
AdminClient adminClient = AdminClient.create(kafkaProperties);
DescribeClusterResult describeClusterResult = adminClient.describeCluster();
Collection<Node> brokerDetails = describeClusterResult.nodes().get();
System.out.println("host and port details");
for(Node broker:brokerDetails) 
 System.out.println(broker.host()+":"+broker.port());

edited Nov 12 at 12:45

Nick

23.2k91535

answered Nov 12 at 12:20

Manjunatha H C

112

Use describeCluster() of the AdminClient class to get the broker details such as host,port,id and rock.

Please refer the following code:

Properties kafkaProperties = new Properties();
kafkaProperties.put("bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094");
AdminClient adminClient = AdminClient.create(kafkaProperties);
DescribeClusterResult describeClusterResult = adminClient.describeCluster();
Collection<Node> brokerDetails = describeClusterResult.nodes().get();
System.out.println("host and port details");
for(Node broker:brokerDetails) 
 System.out.println(broker.host()+":"+broker.port());

edited Nov 12 at 12:45

Nick

23.2k91535

answered Nov 12 at 12:20

Manjunatha H C

112

edited Nov 12 at 12:45

Nick

23.2k91535

edited Nov 12 at 12:45

Nick

23.2k91535

edited Nov 12 at 12:45

Nick

23.2k91535

answered Nov 12 at 12:20

Manjunatha H C

112

answered Nov 12 at 12:20

Manjunatha H C

112

answered Nov 12 at 12:20

Manjunatha H C

112

Please format your code as described in stackoverflow.com/editing-help. It makes it a lot easier to read. Thanks.
– Nick
Nov 12 at 12:45

add a comment |

Please format your code as described in stackoverflow.com/editing-help. It makes it a lot easier to read. Thanks.
– Nick
Nov 12 at 12:45

Please format your code as described in stackoverflow.com/editing-help. It makes it a lot easier to read. Thanks.
– Nick
Nov 12 at 12:45

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

Some of your past answers have not been well-received, and you're in danger of being blocked from answering.

Please pay close attention to the following guidance:

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Odtnhj