Avro union - type information in resulting json
I have a part of a java application that writes an AVRO encoded object into a Kafka topic. I use the org.apache.kafka.clients.producer.Producer to send messages into the topic. The Producer is configured to use the io.confluent.kafka.serializers.KafkaAvroSerializer.
The AVRO schema contains the following field among others:
"name": "field1",
"type": [
"null",
"string"
],
"default": null
Which basically means that it's a string field which can be null.
After sending an object to the topic, I inspect its content with the following terminal command:
./kafka-avro-console-consumer --bootstrap-server broker-host:9091 --topic test-topic-1
--property schema.registry.url=http://schema-host:8081 --from-beginning
As I result I would expect to see a JSON that has the following attribute:
"field1": "something"
However, in reality I see this:
"field1": "string": "something"
Along with that, if I simply use a toString() method on the object extending the SpecificRecordBase before actually sending it to the topic, I see a JSON formatted as I expect.
Is there a way to actually get the first option in the topic, without this extra type information being a part of the JSON?
java apache-kafka avro confluent confluent-schema-registry
asked Nov 15 '18 at 17:20
Green FiremanGreen Fireman
319521
add a comment |
I have a part of a java application that writes an AVRO encoded object into a Kafka topic. I use the org.apache.kafka.clients.producer.Producer to send messages into the topic. The Producer is configured to use the io.confluent.kafka.serializers.KafkaAvroSerializer.
The AVRO schema contains the following field among others:
"name": "field1",
"type": [
"null",
"string"
],
"default": null
Which basically means that it's a string field which can be null.
After sending an object to the topic, I inspect its content with the following terminal command:
./kafka-avro-console-consumer --bootstrap-server broker-host:9091 --topic test-topic-1
--property schema.registry.url=http://schema-host:8081 --from-beginning
As I result I would expect to see a JSON that has the following attribute:
"field1": "something"
However, in reality I see this:
"field1": "string": "something"
Along with that, if I simply use a toString() method on the object extending the SpecificRecordBase before actually sending it to the topic, I see a JSON formatted as I expect.
Is there a way to actually get the first option in the topic, without this extra type information being a part of the JSON?
java apache-kafka avro confluent confluent-schema-registry
asked Nov 15 '18 at 17:20
Green FiremanGreen Fireman
319521
Are you using the case class generated by avro to produce your message to kafka? case class SomeSchema(var field1: Option[String]) extends org.apache.avro.specific.SpecificRecordBase ?
– Achilleus
Nov 16 '18 at 7:42
@AkhilanandBenkalVenkanna yes, correct
– Green Fireman
Nov 16 '18 at 7:53
add a comment |
I have a part of a java application that writes an AVRO encoded object into a Kafka topic. I use the org.apache.kafka.clients.producer.Producer to send messages into the topic. The Producer is configured to use the io.confluent.kafka.serializers.KafkaAvroSerializer.
The AVRO schema contains the following field among others:
"name": "field1",
"type": [
"null",
"string"
],
"default": null
Which basically means that it's a string field which can be null.
After sending an object to the topic, I inspect its content with the following terminal command:
./kafka-avro-console-consumer --bootstrap-server broker-host:9091 --topic test-topic-1
--property schema.registry.url=http://schema-host:8081 --from-beginning
As I result I would expect to see a JSON that has the following attribute:
"field1": "something"
However, in reality I see this:
"field1": "string": "something"
Along with that, if I simply use a toString() method on the object extending the SpecificRecordBase before actually sending it to the topic, I see a JSON formatted as I expect.
Is there a way to actually get the first option in the topic, without this extra type information being a part of the JSON?
java apache-kafka avro confluent confluent-schema-registry
asked Nov 15 '18 at 17:20
Green FiremanGreen Fireman
319521
I have a part of a java application that writes an AVRO encoded object into a Kafka topic. I use the org.apache.kafka.clients.producer.Producer to send messages into the topic. The Producer is configured to use the io.confluent.kafka.serializers.KafkaAvroSerializer.
The AVRO schema contains the following field among others:
"name": "field1",
"type": [
"null",
"string"
],
"default": null
Which basically means that it's a string field which can be null.
After sending an object to the topic, I inspect its content with the following terminal command:
./kafka-avro-console-consumer --bootstrap-server broker-host:9091 --topic test-topic-1
--property schema.registry.url=http://schema-host:8081 --from-beginning
As I result I would expect to see a JSON that has the following attribute:
"field1": "something"
However, in reality I see this:
"field1": "string": "something"
Along with that, if I simply use a toString() method on the object extending the SpecificRecordBase before actually sending it to the topic, I see a JSON formatted as I expect.
Is there a way to actually get the first option in the topic, without this extra type information being a part of the JSON?
java apache-kafka avro confluent confluent-schema-registry
java apache-kafka avro confluent confluent-schema-registry
asked Nov 15 '18 at 17:20
Green FiremanGreen Fireman
319521
asked Nov 15 '18 at 17:20
Green FiremanGreen Fireman
319521
edited Nov 15 '18 at 23:39
Green Fireman
asked Nov 15 '18 at 17:20
Green FiremanGreen Fireman
319521
asked Nov 15 '18 at 17:20
Green FiremanGreen Fireman
319521
asked Nov 15 '18 at 17:20
Green FiremanGreen Fireman
319521
319521
Are you using the case class generated by avro to produce your message to kafka? case class SomeSchema(var field1: Option[String]) extends org.apache.avro.specific.SpecificRecordBase ?
– Achilleus
Nov 16 '18 at 7:42
@AkhilanandBenkalVenkanna yes, correct
– Green Fireman
Nov 16 '18 at 7:53
add a comment |
Are you using the case class generated by avro to produce your message to kafka? case class SomeSchema(var field1: Option[String]) extends org.apache.avro.specific.SpecificRecordBase ?
– Achilleus
Nov 16 '18 at 7:42
@AkhilanandBenkalVenkanna yes, correct
– Green Fireman
Nov 16 '18 at 7:53
Are you using the case class generated by avro to produce your message to kafka? case class SomeSchema(var field1: Option[String]) extends org.apache.avro.specific.SpecificRecordBase ?
– Achilleus
Nov 16 '18 at 7:42
Are you using the case class generated by avro to produce your message to kafka? case class SomeSchema(var field1: Option[String]) extends org.apache.avro.specific.SpecificRecordBase ?
– Achilleus
Nov 16 '18 at 7:42
@AkhilanandBenkalVenkanna yes, correct
– Green Fireman
Nov 16 '18 at 7:53
@AkhilanandBenkalVenkanna yes, correct
– Green Fireman
Nov 16 '18 at 7:53
add a comment |
1 Answer
1
active
oldest
votes
Is there a way to actually get the first option in the topic, without this extra type information being a part of the JSON?
There is not for avro-console-consumer. It is using GenericRecord, not SpecificRecord
Since the field is a union-type the Avro Record JSON-encoded output must include the type of the field for that value.
answered Nov 15 '18 at 22:08
cricket_007cricket_007
83.5k1145115
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53324806%2favro-union-type-information-in-resulting-json%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Is there a way to actually get the first option in the topic, without this extra type information being a part of the JSON?
There is not for avro-console-consumer. It is using GenericRecord, not SpecificRecord
Since the field is a union-type the Avro Record JSON-encoded output must include the type of the field for that value.
answered Nov 15 '18 at 22:08
cricket_007cricket_007
83.5k1145115
add a comment |
Is there a way to actually get the first option in the topic, without this extra type information being a part of the JSON?
There is not for avro-console-consumer. It is using GenericRecord, not SpecificRecord
Since the field is a union-type the Avro Record JSON-encoded output must include the type of the field for that value.
answered Nov 15 '18 at 22:08
cricket_007cricket_007
83.5k1145115
add a comment |
Is there a way to actually get the first option in the topic, without this extra type information being a part of the JSON?
There is not for avro-console-consumer. It is using GenericRecord, not SpecificRecord
Since the field is a union-type the Avro Record JSON-encoded output must include the type of the field for that value.
answered Nov 15 '18 at 22:08
cricket_007cricket_007
83.5k1145115
Is there a way to actually get the first option in the topic, without this extra type information being a part of the JSON?
There is not for avro-console-consumer. It is using GenericRecord, not SpecificRecord
Since the field is a union-type the Avro Record JSON-encoded output must include the type of the field for that value.
answered Nov 15 '18 at 22:08
cricket_007cricket_007
83.5k1145115
answered Nov 15 '18 at 22:08
cricket_007cricket_007
83.5k1145115
answered Nov 15 '18 at 22:08
cricket_007cricket_007
83.5k1145115
answered Nov 15 '18 at 22:08
cricket_007cricket_007
83.5k1145115
83.5k1145115
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53324806%2favro-union-type-information-in-resulting-json%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Are you using the case class generated by avro to produce your message to kafka? case class SomeSchema(var field1: Option[String]) extends org.apache.avro.specific.SpecificRecordBase ?
– Achilleus
Nov 16 '18 at 7:42
@AkhilanandBenkalVenkanna yes, correct
– Green Fireman
Nov 16 '18 at 7:53