Include Kafka and Zookeeper in Docker
I need to make a Docker container for a project involving streaming data using Kafka and Zookeeper. Looking around I found this docker image from Spotify, including Kafka and Zookeeper.
How should I include it in my project? Should I include in the Dockerfile the suggested commands, listed below?
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip `docker-machine active`` --env ADVERTISED_PORT=9092 spotify/kafka
export KAFKA=`docker-machine ip `docker-machine active``:9092
kafka-console-producer.sh --broker-list $KAFKA --topic test
export ZOOKEEPER=`docker-machine ip `docker-machine active``:2181
kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test
docker apache-kafka apache-zookeeper spotify
asked Nov 15 '18 at 18:25
albus_calbus_c
1,34432246
|
show 1 more comment
I need to make a Docker container for a project involving streaming data using Kafka and Zookeeper. Looking around I found this docker image from Spotify, including Kafka and Zookeeper.
How should I include it in my project? Should I include in the Dockerfile the suggested commands, listed below?
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip `docker-machine active`` --env ADVERTISED_PORT=9092 spotify/kafka
export KAFKA=`docker-machine ip `docker-machine active``:9092
kafka-console-producer.sh --broker-list $KAFKA --topic test
export ZOOKEEPER=`docker-machine ip `docker-machine active``:2181
kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test
docker apache-kafka apache-zookeeper spotify
asked Nov 15 '18 at 18:25
albus_calbus_c
1,34432246
1
You'll have to explain what you mean by "including" Kafka & Zookeeper in your project. Either way - those are shell commands (to start up your Dockerised Kafka & ZK), not Dockerfile instructions.
– declension
Nov 15 '18 at 18:59
Thanks heaps! I need to live stream some data, and my plan is to do so using Kafka and Zookeeper. I'm trying to figure out what is the best way to containerize Kafka and Zookeeper.
– albus_c
Nov 15 '18 at 19:03
1
OK - well Spotify already containerised them, as per your link. You can use their image by running the abovedocker runcommand, but how you do this (other than just typing it in to a terminal) depends on what language / framework / project setup you're using...
– declension
Nov 15 '18 at 19:14
1
The spotify image is heavily outdated. I wouldn't suggest it
– cricket_007
Nov 15 '18 at 19:24
@cricket_007 well spotted. I ended up usingwurstmeister/zookeeperandwurstmeister/kafka
– albus_c
Nov 15 '18 at 21:17
|
show 1 more comment
I need to make a Docker container for a project involving streaming data using Kafka and Zookeeper. Looking around I found this docker image from Spotify, including Kafka and Zookeeper.
How should I include it in my project? Should I include in the Dockerfile the suggested commands, listed below?
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip `docker-machine active`` --env ADVERTISED_PORT=9092 spotify/kafka
export KAFKA=`docker-machine ip `docker-machine active``:9092
kafka-console-producer.sh --broker-list $KAFKA --topic test
export ZOOKEEPER=`docker-machine ip `docker-machine active``:2181
kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test
docker apache-kafka apache-zookeeper spotify
asked Nov 15 '18 at 18:25
albus_calbus_c
1,34432246
I need to make a Docker container for a project involving streaming data using Kafka and Zookeeper. Looking around I found this docker image from Spotify, including Kafka and Zookeeper.
How should I include it in my project? Should I include in the Dockerfile the suggested commands, listed below?
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip `docker-machine active`` --env ADVERTISED_PORT=9092 spotify/kafka
export KAFKA=`docker-machine ip `docker-machine active``:9092
kafka-console-producer.sh --broker-list $KAFKA --topic test
export ZOOKEEPER=`docker-machine ip `docker-machine active``:2181
kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test
docker apache-kafka apache-zookeeper spotify
docker apache-kafka apache-zookeeper spotify
asked Nov 15 '18 at 18:25
albus_calbus_c
1,34432246
asked Nov 15 '18 at 18:25
albus_calbus_c
1,34432246
edited Nov 15 '18 at 18:49
albus_c
asked Nov 15 '18 at 18:25
albus_calbus_c
1,34432246
asked Nov 15 '18 at 18:25
albus_calbus_c
1,34432246
asked Nov 15 '18 at 18:25
albus_calbus_c
1,34432246
1,34432246
1
You'll have to explain what you mean by "including" Kafka & Zookeeper in your project. Either way - those are shell commands (to start up your Dockerised Kafka & ZK), not Dockerfile instructions.
– declension
Nov 15 '18 at 18:59
Thanks heaps! I need to live stream some data, and my plan is to do so using Kafka and Zookeeper. I'm trying to figure out what is the best way to containerize Kafka and Zookeeper.
– albus_c
Nov 15 '18 at 19:03
1
OK - well Spotify already containerised them, as per your link. You can use their image by running the abovedocker runcommand, but how you do this (other than just typing it in to a terminal) depends on what language / framework / project setup you're using...
– declension
Nov 15 '18 at 19:14
1
The spotify image is heavily outdated. I wouldn't suggest it
– cricket_007
Nov 15 '18 at 19:24
@cricket_007 well spotted. I ended up usingwurstmeister/zookeeperandwurstmeister/kafka
– albus_c
Nov 15 '18 at 21:17
|
show 1 more comment
1
You'll have to explain what you mean by "including" Kafka & Zookeeper in your project. Either way - those are shell commands (to start up your Dockerised Kafka & ZK), not Dockerfile instructions.
– declension
Nov 15 '18 at 18:59
Thanks heaps! I need to live stream some data, and my plan is to do so using Kafka and Zookeeper. I'm trying to figure out what is the best way to containerize Kafka and Zookeeper.
– albus_c
Nov 15 '18 at 19:03
1
OK - well Spotify already containerised them, as per your link. You can use their image by running the abovedocker runcommand, but how you do this (other than just typing it in to a terminal) depends on what language / framework / project setup you're using...
– declension
Nov 15 '18 at 19:14
1
The spotify image is heavily outdated. I wouldn't suggest it
– cricket_007
Nov 15 '18 at 19:24
@cricket_007 well spotted. I ended up usingwurstmeister/zookeeperandwurstmeister/kafka
– albus_c
Nov 15 '18 at 21:17
1
1
You'll have to explain what you mean by "including" Kafka & Zookeeper in your project. Either way - those are shell commands (to start up your Dockerised Kafka & ZK), not Dockerfile instructions.
– declension
Nov 15 '18 at 18:59
You'll have to explain what you mean by "including" Kafka & Zookeeper in your project. Either way - those are shell commands (to start up your Dockerised Kafka & ZK), not Dockerfile instructions.
– declension
Nov 15 '18 at 18:59
Thanks heaps! I need to live stream some data, and my plan is to do so using Kafka and Zookeeper. I'm trying to figure out what is the best way to containerize Kafka and Zookeeper.
– albus_c
Nov 15 '18 at 19:03
Thanks heaps! I need to live stream some data, and my plan is to do so using Kafka and Zookeeper. I'm trying to figure out what is the best way to containerize Kafka and Zookeeper.
– albus_c
Nov 15 '18 at 19:03
1
1
OK - well Spotify already containerised them, as per your link. You can use their image by running the above
docker run command, but how you do this (other than just typing it in to a terminal) depends on what language / framework / project setup you're using...– declension
Nov 15 '18 at 19:14
OK - well Spotify already containerised them, as per your link. You can use their image by running the above
docker run command, but how you do this (other than just typing it in to a terminal) depends on what language / framework / project setup you're using...– declension
Nov 15 '18 at 19:14
1
1
The spotify image is heavily outdated. I wouldn't suggest it
– cricket_007
Nov 15 '18 at 19:24
The spotify image is heavily outdated. I wouldn't suggest it
– cricket_007
Nov 15 '18 at 19:24
@cricket_007 well spotted. I ended up using
wurstmeister/zookeeper and wurstmeister/kafka– albus_c
Nov 15 '18 at 21:17
@cricket_007 well spotted. I ended up using
wurstmeister/zookeeper and wurstmeister/kafka– albus_c
Nov 15 '18 at 21:17
|
show 1 more comment
1 Answer
1
active
oldest
votes
How about using a docker-compose file?
In your *.yaml you can set-up the services to pull the Kafka and Zookeeper images from Spotify's DockerHub, map ports (e.g. "2181:2181" and "9092:9092" for ZK and Kafka, respectively), set ENV variables, and persist data to a volume so you don't lose your topics and offsets.
answered Nov 15 '18 at 19:03
akimakim
263
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53325742%2finclude-kafka-and-zookeeper-in-docker%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
How about using a docker-compose file?
In your *.yaml you can set-up the services to pull the Kafka and Zookeeper images from Spotify's DockerHub, map ports (e.g. "2181:2181" and "9092:9092" for ZK and Kafka, respectively), set ENV variables, and persist data to a volume so you don't lose your topics and offsets.
answered Nov 15 '18 at 19:03
akimakim
263
add a comment |
How about using a docker-compose file?
In your *.yaml you can set-up the services to pull the Kafka and Zookeeper images from Spotify's DockerHub, map ports (e.g. "2181:2181" and "9092:9092" for ZK and Kafka, respectively), set ENV variables, and persist data to a volume so you don't lose your topics and offsets.
answered Nov 15 '18 at 19:03
akimakim
263
add a comment |
How about using a docker-compose file?
In your *.yaml you can set-up the services to pull the Kafka and Zookeeper images from Spotify's DockerHub, map ports (e.g. "2181:2181" and "9092:9092" for ZK and Kafka, respectively), set ENV variables, and persist data to a volume so you don't lose your topics and offsets.
answered Nov 15 '18 at 19:03
akimakim
263
How about using a docker-compose file?
In your *.yaml you can set-up the services to pull the Kafka and Zookeeper images from Spotify's DockerHub, map ports (e.g. "2181:2181" and "9092:9092" for ZK and Kafka, respectively), set ENV variables, and persist data to a volume so you don't lose your topics and offsets.
answered Nov 15 '18 at 19:03
akimakim
263
answered Nov 15 '18 at 19:03
akimakim
263
answered Nov 15 '18 at 19:03
akimakim
263
answered Nov 15 '18 at 19:03
akimakim
263
263
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53325742%2finclude-kafka-and-zookeeper-in-docker%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
You'll have to explain what you mean by "including" Kafka & Zookeeper in your project. Either way - those are shell commands (to start up your Dockerised Kafka & ZK), not Dockerfile instructions.
– declension
Nov 15 '18 at 18:59
Thanks heaps! I need to live stream some data, and my plan is to do so using Kafka and Zookeeper. I'm trying to figure out what is the best way to containerize Kafka and Zookeeper.
– albus_c
Nov 15 '18 at 19:03
1
OK - well Spotify already containerised them, as per your link. You can use their image by running the above
docker runcommand, but how you do this (other than just typing it in to a terminal) depends on what language / framework / project setup you're using...– declension
Nov 15 '18 at 19:14
1
The spotify image is heavily outdated. I wouldn't suggest it
– cricket_007
Nov 15 '18 at 19:24
@cricket_007 well spotted. I ended up using
wurstmeister/zookeeperandwurstmeister/kafka– albus_c
Nov 15 '18 at 21:17