Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed










0















Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed.



We are currently running Kafka in AWS with broker.id gets auto generated. But we are having issues when a broker is failed, new broker/instance spun-up in AWS get assigned with new broker.id. The issue is, with this approach, we need to re-assign the topics/replications on to the new broker manually.



We learned that, replication can be auto resolved by Kafka, if we can manage to get the same broker.id on the new AWS instance spun-up in-place of failed broker/instance.



I have read, we can set broker.id.generation.enable= false, but what is the best way to identify and retain the broker.id? Any links/help is appreciated.










share|improve this question






















  • You could just assign broker.id to a number that you chose. When you move the volume off a failed instance, set the same broker.id to that server (we also run in AWS and have generation to true, but have set up a value manually)

    – cricket_007
    Nov 14 '18 at 19:30












  • thanks for quick reply. so do you have any custom script which gets triggered during launch of instance to set-up the broker-id manually? something like script mentioned at some script mentioned @ tech.gc.com/scaling-with-kafka ? how do you get previous broker.id which is just failed without conflicting with other/existing broker ids? We don't use EBS, instead we use instance level storage, so we loose storage/volume when AWS instance is terminated.

    – cnutext
    Nov 14 '18 at 20:14











  • We use Terraform with a script written in-house, so there isn't any such auto-scaling things. It's all immutable and manually done, but it is using EBS. Our instances don't really drastically fail, though, but if they do, I guess the EBS volume migration is easily done.

    – cricket_007
    Nov 14 '18 at 23:23












  • Yeah, EBS is doing trick in your case. Our case is specific: especially with instance level storage and not using EBS. I am sure there should be easy/another way to retain the old broker.id for new AWS instance spun-up for auto replication. Any other ideas/help is appreciated.

    – cnutext
    Nov 15 '18 at 12:38












  • My only ideas involve configuration management tools like Ansible and not letting a Kafka service auto-start or pick its own id

    – cricket_007
    Nov 15 '18 at 15:26















0















Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed.



We are currently running Kafka in AWS with broker.id gets auto generated. But we are having issues when a broker is failed, new broker/instance spun-up in AWS get assigned with new broker.id. The issue is, with this approach, we need to re-assign the topics/replications on to the new broker manually.



We learned that, replication can be auto resolved by Kafka, if we can manage to get the same broker.id on the new AWS instance spun-up in-place of failed broker/instance.



I have read, we can set broker.id.generation.enable= false, but what is the best way to identify and retain the broker.id? Any links/help is appreciated.










share|improve this question






















  • You could just assign broker.id to a number that you chose. When you move the volume off a failed instance, set the same broker.id to that server (we also run in AWS and have generation to true, but have set up a value manually)

    – cricket_007
    Nov 14 '18 at 19:30












  • thanks for quick reply. so do you have any custom script which gets triggered during launch of instance to set-up the broker-id manually? something like script mentioned at some script mentioned @ tech.gc.com/scaling-with-kafka ? how do you get previous broker.id which is just failed without conflicting with other/existing broker ids? We don't use EBS, instead we use instance level storage, so we loose storage/volume when AWS instance is terminated.

    – cnutext
    Nov 14 '18 at 20:14











  • We use Terraform with a script written in-house, so there isn't any such auto-scaling things. It's all immutable and manually done, but it is using EBS. Our instances don't really drastically fail, though, but if they do, I guess the EBS volume migration is easily done.

    – cricket_007
    Nov 14 '18 at 23:23












  • Yeah, EBS is doing trick in your case. Our case is specific: especially with instance level storage and not using EBS. I am sure there should be easy/another way to retain the old broker.id for new AWS instance spun-up for auto replication. Any other ideas/help is appreciated.

    – cnutext
    Nov 15 '18 at 12:38












  • My only ideas involve configuration management tools like Ansible and not letting a Kafka service auto-start or pick its own id

    – cricket_007
    Nov 15 '18 at 15:26













0












0








0








Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed.



We are currently running Kafka in AWS with broker.id gets auto generated. But we are having issues when a broker is failed, new broker/instance spun-up in AWS get assigned with new broker.id. The issue is, with this approach, we need to re-assign the topics/replications on to the new broker manually.



We learned that, replication can be auto resolved by Kafka, if we can manage to get the same broker.id on the new AWS instance spun-up in-place of failed broker/instance.



I have read, we can set broker.id.generation.enable= false, but what is the best way to identify and retain the broker.id? Any links/help is appreciated.










share|improve this question














Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed.



We are currently running Kafka in AWS with broker.id gets auto generated. But we are having issues when a broker is failed, new broker/instance spun-up in AWS get assigned with new broker.id. The issue is, with this approach, we need to re-assign the topics/replications on to the new broker manually.



We learned that, replication can be auto resolved by Kafka, if we can manage to get the same broker.id on the new AWS instance spun-up in-place of failed broker/instance.



I have read, we can set broker.id.generation.enable= false, but what is the best way to identify and retain the broker.id? Any links/help is appreciated.







amazon-web-services amazon-ec2 apache-kafka






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 14 '18 at 18:21









cnutextcnutext

61




61












  • You could just assign broker.id to a number that you chose. When you move the volume off a failed instance, set the same broker.id to that server (we also run in AWS and have generation to true, but have set up a value manually)

    – cricket_007
    Nov 14 '18 at 19:30












  • thanks for quick reply. so do you have any custom script which gets triggered during launch of instance to set-up the broker-id manually? something like script mentioned at some script mentioned @ tech.gc.com/scaling-with-kafka ? how do you get previous broker.id which is just failed without conflicting with other/existing broker ids? We don't use EBS, instead we use instance level storage, so we loose storage/volume when AWS instance is terminated.

    – cnutext
    Nov 14 '18 at 20:14











  • We use Terraform with a script written in-house, so there isn't any such auto-scaling things. It's all immutable and manually done, but it is using EBS. Our instances don't really drastically fail, though, but if they do, I guess the EBS volume migration is easily done.

    – cricket_007
    Nov 14 '18 at 23:23












  • Yeah, EBS is doing trick in your case. Our case is specific: especially with instance level storage and not using EBS. I am sure there should be easy/another way to retain the old broker.id for new AWS instance spun-up for auto replication. Any other ideas/help is appreciated.

    – cnutext
    Nov 15 '18 at 12:38












  • My only ideas involve configuration management tools like Ansible and not letting a Kafka service auto-start or pick its own id

    – cricket_007
    Nov 15 '18 at 15:26

















  • You could just assign broker.id to a number that you chose. When you move the volume off a failed instance, set the same broker.id to that server (we also run in AWS and have generation to true, but have set up a value manually)

    – cricket_007
    Nov 14 '18 at 19:30












  • thanks for quick reply. so do you have any custom script which gets triggered during launch of instance to set-up the broker-id manually? something like script mentioned at some script mentioned @ tech.gc.com/scaling-with-kafka ? how do you get previous broker.id which is just failed without conflicting with other/existing broker ids? We don't use EBS, instead we use instance level storage, so we loose storage/volume when AWS instance is terminated.

    – cnutext
    Nov 14 '18 at 20:14











  • We use Terraform with a script written in-house, so there isn't any such auto-scaling things. It's all immutable and manually done, but it is using EBS. Our instances don't really drastically fail, though, but if they do, I guess the EBS volume migration is easily done.

    – cricket_007
    Nov 14 '18 at 23:23












  • Yeah, EBS is doing trick in your case. Our case is specific: especially with instance level storage and not using EBS. I am sure there should be easy/another way to retain the old broker.id for new AWS instance spun-up for auto replication. Any other ideas/help is appreciated.

    – cnutext
    Nov 15 '18 at 12:38












  • My only ideas involve configuration management tools like Ansible and not letting a Kafka service auto-start or pick its own id

    – cricket_007
    Nov 15 '18 at 15:26
















You could just assign broker.id to a number that you chose. When you move the volume off a failed instance, set the same broker.id to that server (we also run in AWS and have generation to true, but have set up a value manually)

– cricket_007
Nov 14 '18 at 19:30






You could just assign broker.id to a number that you chose. When you move the volume off a failed instance, set the same broker.id to that server (we also run in AWS and have generation to true, but have set up a value manually)

– cricket_007
Nov 14 '18 at 19:30














thanks for quick reply. so do you have any custom script which gets triggered during launch of instance to set-up the broker-id manually? something like script mentioned at some script mentioned @ tech.gc.com/scaling-with-kafka ? how do you get previous broker.id which is just failed without conflicting with other/existing broker ids? We don't use EBS, instead we use instance level storage, so we loose storage/volume when AWS instance is terminated.

– cnutext
Nov 14 '18 at 20:14





thanks for quick reply. so do you have any custom script which gets triggered during launch of instance to set-up the broker-id manually? something like script mentioned at some script mentioned @ tech.gc.com/scaling-with-kafka ? how do you get previous broker.id which is just failed without conflicting with other/existing broker ids? We don't use EBS, instead we use instance level storage, so we loose storage/volume when AWS instance is terminated.

– cnutext
Nov 14 '18 at 20:14













We use Terraform with a script written in-house, so there isn't any such auto-scaling things. It's all immutable and manually done, but it is using EBS. Our instances don't really drastically fail, though, but if they do, I guess the EBS volume migration is easily done.

– cricket_007
Nov 14 '18 at 23:23






We use Terraform with a script written in-house, so there isn't any such auto-scaling things. It's all immutable and manually done, but it is using EBS. Our instances don't really drastically fail, though, but if they do, I guess the EBS volume migration is easily done.

– cricket_007
Nov 14 '18 at 23:23














Yeah, EBS is doing trick in your case. Our case is specific: especially with instance level storage and not using EBS. I am sure there should be easy/another way to retain the old broker.id for new AWS instance spun-up for auto replication. Any other ideas/help is appreciated.

– cnutext
Nov 15 '18 at 12:38






Yeah, EBS is doing trick in your case. Our case is specific: especially with instance level storage and not using EBS. I am sure there should be easy/another way to retain the old broker.id for new AWS instance spun-up for auto replication. Any other ideas/help is appreciated.

– cnutext
Nov 15 '18 at 12:38














My only ideas involve configuration management tools like Ansible and not letting a Kafka service auto-start or pick its own id

– cricket_007
Nov 15 '18 at 15:26





My only ideas involve configuration management tools like Ansible and not letting a Kafka service auto-start or pick its own id

– cricket_007
Nov 15 '18 at 15:26












0






active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53306532%2fkafka-running-on-aws-how-to-retain-broker-id-on-new-instance-spun-up-in-place%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes















draft saved

draft discarded
















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53306532%2fkafka-running-on-aws-how-to-retain-broker-id-on-new-instance-spun-up-in-place%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







這個網誌中的熱門文章

How to read a connectionString WITH PROVIDER in .NET Core?

Museum of Modern and Contemporary Art of Trento and Rovereto

In R, how to develop a multiplot heatmap.2 figure showing key labels successfully