Getting data frame form kafka using spark streaming
0
Hi I am new to spark thing. I have a use case of getting stream of data from kafka topics in windowed manner and then do analysis on top of it.
I tried below code and it is throwing error:
from pyspark.sql import SparkSession
spark=SparkSession.builder.appName("TestKakfa").getOrCreate()
df = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "dummy_events")
.option("startingOffsets", "earliest")
.load()
query = df.writeStream.format('console').start()
query.awaitTermination()
I am getting below error:
pyspark.sql.utils.StreamingQueryException: 'nulln=== Streaming Query ===nIdentifier: [id = b842e3ba-8584-4764-8068-cce6b8d7de5f, runId = edc960c5-e5ef-4493-b9a2-c9543e1d2a8d]nCurrent Committed Offsets: nCurrent Available Offsets: nnCurrent State: INITIALIZINGnThread State: RUNNABLE'
python apache-spark pyspark apache-kafka spark-structured-streaming
edited Nov 14 '18 at 16:14
cricket_007
82.2k1143111
asked Nov 14 '18 at 14:00
Raghav salotraRaghav salotra
275212
add a comment |
0
Hi I am new to spark thing. I have a use case of getting stream of data from kafka topics in windowed manner and then do analysis on top of it.
I tried below code and it is throwing error:
from pyspark.sql import SparkSession
spark=SparkSession.builder.appName("TestKakfa").getOrCreate()
df = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "dummy_events")
.option("startingOffsets", "earliest")
.load()
query = df.writeStream.format('console').start()
query.awaitTermination()
I am getting below error:
pyspark.sql.utils.StreamingQueryException: 'nulln=== Streaming Query ===nIdentifier: [id = b842e3ba-8584-4764-8068-cce6b8d7de5f, runId = edc960c5-e5ef-4493-b9a2-c9543e1d2a8d]nCurrent Committed Offsets: nCurrent Available Offsets: nnCurrent State: INITIALIZINGnThread State: RUNNABLE'
python apache-spark pyspark apache-kafka spark-structured-streaming
edited Nov 14 '18 at 16:14
cricket_007
82.2k1143111
asked Nov 14 '18 at 14:00
Raghav salotraRaghav salotra
275212
Which spark version? Which Kafka version? Which Spark Streaming Kafka version?
– cricket_007
Nov 14 '18 at 16:16
Spark version: 2.3.1 Kafka Version: confluent-5.0.0 Spark Streaming Kafka Version: spark-streaming-kafka-0-8_2.11:2.3.0
– Raghav salotra
Nov 15 '18 at 6:58
So, Kafka 2.0... But you're using the wrong library for structured streaming, and make sure that the last 3 numbers matches the Spark version spark.apache.org/docs/2.3.1/…
– cricket_007
Nov 15 '18 at 7:06
@cricket_007 Thanks. fixed that issue.
– Raghav salotra
Nov 22 '18 at 12:19
Feel free to answer the question below with your solution in case it helps others
– cricket_007
Nov 22 '18 at 17:02
add a comment |
0
0
0
Hi I am new to spark thing. I have a use case of getting stream of data from kafka topics in windowed manner and then do analysis on top of it.
I tried below code and it is throwing error:
from pyspark.sql import SparkSession
spark=SparkSession.builder.appName("TestKakfa").getOrCreate()
df = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "dummy_events")
.option("startingOffsets", "earliest")
.load()
query = df.writeStream.format('console').start()
query.awaitTermination()
I am getting below error:
pyspark.sql.utils.StreamingQueryException: 'nulln=== Streaming Query ===nIdentifier: [id = b842e3ba-8584-4764-8068-cce6b8d7de5f, runId = edc960c5-e5ef-4493-b9a2-c9543e1d2a8d]nCurrent Committed Offsets: nCurrent Available Offsets: nnCurrent State: INITIALIZINGnThread State: RUNNABLE'
python apache-spark pyspark apache-kafka spark-structured-streaming
edited Nov 14 '18 at 16:14
cricket_007
82.2k1143111
asked Nov 14 '18 at 14:00
Raghav salotraRaghav salotra
275212
Hi I am new to spark thing. I have a use case of getting stream of data from kafka topics in windowed manner and then do analysis on top of it.
I tried below code and it is throwing error:
from pyspark.sql import SparkSession
spark=SparkSession.builder.appName("TestKakfa").getOrCreate()
df = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "dummy_events")
.option("startingOffsets", "earliest")
.load()
query = df.writeStream.format('console').start()
query.awaitTermination()
I am getting below error:
pyspark.sql.utils.StreamingQueryException: 'nulln=== Streaming Query ===nIdentifier: [id = b842e3ba-8584-4764-8068-cce6b8d7de5f, runId = edc960c5-e5ef-4493-b9a2-c9543e1d2a8d]nCurrent Committed Offsets: nCurrent Available Offsets: nnCurrent State: INITIALIZINGnThread State: RUNNABLE'
python apache-spark pyspark apache-kafka spark-structured-streaming
python apache-spark pyspark apache-kafka spark-structured-streaming
edited Nov 14 '18 at 16:14
cricket_007
82.2k1143111
asked Nov 14 '18 at 14:00
Raghav salotraRaghav salotra
275212
edited Nov 14 '18 at 16:14
cricket_007
82.2k1143111
asked Nov 14 '18 at 14:00
Raghav salotraRaghav salotra
275212
edited Nov 14 '18 at 16:14
cricket_007
82.2k1143111
edited Nov 14 '18 at 16:14
cricket_007
82.2k1143111
edited Nov 14 '18 at 16:14
cricket_007
82.2k1143111
82.2k1143111
asked Nov 14 '18 at 14:00
Raghav salotraRaghav salotra
275212
asked Nov 14 '18 at 14:00
Raghav salotraRaghav salotra
275212
asked Nov 14 '18 at 14:00
Raghav salotraRaghav salotra
275212
275212
Which spark version? Which Kafka version? Which Spark Streaming Kafka version?
– cricket_007
Nov 14 '18 at 16:16
Spark version: 2.3.1 Kafka Version: confluent-5.0.0 Spark Streaming Kafka Version: spark-streaming-kafka-0-8_2.11:2.3.0
– Raghav salotra
Nov 15 '18 at 6:58
So, Kafka 2.0... But you're using the wrong library for structured streaming, and make sure that the last 3 numbers matches the Spark version spark.apache.org/docs/2.3.1/…
– cricket_007
Nov 15 '18 at 7:06
@cricket_007 Thanks. fixed that issue.
– Raghav salotra
Nov 22 '18 at 12:19
Feel free to answer the question below with your solution in case it helps others
– cricket_007
Nov 22 '18 at 17:02
add a comment |
Which spark version? Which Kafka version? Which Spark Streaming Kafka version?
– cricket_007
Nov 14 '18 at 16:16
Spark version: 2.3.1 Kafka Version: confluent-5.0.0 Spark Streaming Kafka Version: spark-streaming-kafka-0-8_2.11:2.3.0
– Raghav salotra
Nov 15 '18 at 6:58
So, Kafka 2.0... But you're using the wrong library for structured streaming, and make sure that the last 3 numbers matches the Spark version spark.apache.org/docs/2.3.1/…
– cricket_007
Nov 15 '18 at 7:06
@cricket_007 Thanks. fixed that issue.
– Raghav salotra
Nov 22 '18 at 12:19
Feel free to answer the question below with your solution in case it helps others
– cricket_007
Nov 22 '18 at 17:02
Which spark version? Which Kafka version? Which Spark Streaming Kafka version?
– cricket_007
Nov 14 '18 at 16:16
Which spark version? Which Kafka version? Which Spark Streaming Kafka version?
– cricket_007
Nov 14 '18 at 16:16
Spark version: 2.3.1 Kafka Version: confluent-5.0.0 Spark Streaming Kafka Version: spark-streaming-kafka-0-8_2.11:2.3.0
– Raghav salotra
Nov 15 '18 at 6:58
Spark version: 2.3.1 Kafka Version: confluent-5.0.0 Spark Streaming Kafka Version: spark-streaming-kafka-0-8_2.11:2.3.0
– Raghav salotra
Nov 15 '18 at 6:58
So, Kafka 2.0... But you're using the wrong library for structured streaming, and make sure that the last 3 numbers matches the Spark version spark.apache.org/docs/2.3.1/…
– cricket_007
Nov 15 '18 at 7:06
So, Kafka 2.0... But you're using the wrong library for structured streaming, and make sure that the last 3 numbers matches the Spark version spark.apache.org/docs/2.3.1/…
– cricket_007
Nov 15 '18 at 7:06
@cricket_007 Thanks. fixed that issue.
– Raghav salotra
Nov 22 '18 at 12:19
@cricket_007 Thanks. fixed that issue.
– Raghav salotra
Nov 22 '18 at 12:19
Feel free to answer the question below with your solution in case it helps others
– cricket_007
Nov 22 '18 at 17:02
Feel free to answer the question below with your solution in case it helps others
– cricket_007
Nov 22 '18 at 17:02
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53302003%2fgetting-data-frame-form-kafka-using-spark-streaming%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53302003%2fgetting-data-frame-form-kafka-using-spark-streaming%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Which spark version? Which Kafka version? Which Spark Streaming Kafka version?
– cricket_007
Nov 14 '18 at 16:16
Spark version: 2.3.1 Kafka Version: confluent-5.0.0 Spark Streaming Kafka Version: spark-streaming-kafka-0-8_2.11:2.3.0
– Raghav salotra
Nov 15 '18 at 6:58
So, Kafka 2.0... But you're using the wrong library for structured streaming, and make sure that the last 3 numbers matches the Spark version spark.apache.org/docs/2.3.1/…
– cricket_007
Nov 15 '18 at 7:06
@cricket_007 Thanks. fixed that issue.
– Raghav salotra
Nov 22 '18 at 12:19
Feel free to answer the question below with your solution in case it helps others
– cricket_007
Nov 22 '18 at 17:02