How to store the payload at any point to use it later

I've deployed spring-cloud-dataflow with Docker. These are the used containers:

springcloud/spring-cloud-dataflow-server-local:1.7.0.RELEASE

wurstmeister/zookeeper

wurstmeister/kafka:1.1.0

[other containers not involved in the issue ...]

Documentation consulted:

https://docs.spring.io/spring-cloud-dataflow/docs/1.7.0.RELEASE/reference/htmlsingle/

I am trying to create a stream in which i need to store the payload to use it later.

enter image description here

http | groovy-transform | httpclient_get: httpclient | httpclient_post: httpclient | log

There is some way to solve this need?

asked Nov 13 '18 at 12:16

tioperez

436

add a comment |

I've deployed spring-cloud-dataflow with Docker. These are the used containers:

springcloud/spring-cloud-dataflow-server-local:1.7.0.RELEASE

wurstmeister/zookeeper

wurstmeister/kafka:1.1.0

[other containers not involved in the issue ...]

Documentation consulted:

https://docs.spring.io/spring-cloud-dataflow/docs/1.7.0.RELEASE/reference/htmlsingle/

I am trying to create a stream in which i need to store the payload to use it later.

enter image description here

http | groovy-transform | httpclient_get: httpclient | httpclient_post: httpclient | log

There is some way to solve this need?

asked Nov 13 '18 at 12:16

tioperez

436

add a comment |

I've deployed spring-cloud-dataflow with Docker. These are the used containers:

springcloud/spring-cloud-dataflow-server-local:1.7.0.RELEASE

wurstmeister/zookeeper

wurstmeister/kafka:1.1.0

[other containers not involved in the issue ...]

Documentation consulted:

https://docs.spring.io/spring-cloud-dataflow/docs/1.7.0.RELEASE/reference/htmlsingle/

I am trying to create a stream in which i need to store the payload to use it later.

enter image description here

http | groovy-transform | httpclient_get: httpclient | httpclient_post: httpclient | log

There is some way to solve this need?

asked Nov 13 '18 at 12:16

tioperez

436

I've deployed spring-cloud-dataflow with Docker. These are the used containers:

springcloud/spring-cloud-dataflow-server-local:1.7.0.RELEASE

wurstmeister/zookeeper

wurstmeister/kafka:1.1.0

[other containers not involved in the issue ...]

Documentation consulted:

https://docs.spring.io/spring-cloud-dataflow/docs/1.7.0.RELEASE/reference/htmlsingle/

I am trying to create a stream in which i need to store the payload to use it later.

enter image description here

http | groovy-transform | httpclient_get: httpclient | httpclient_post: httpclient | log

There is some way to solve this need?

spring-cloud-dataflow

asked Nov 13 '18 at 12:16

tioperez

436

asked Nov 13 '18 at 12:16

tioperez

436

asked Nov 13 '18 at 12:16

tioperez

436

asked Nov 13 '18 at 12:16

tioperez

436

asked Nov 13 '18 at 12:16

tioperez

436

add a comment |

1 Answer
1

active

oldest

votes

It's not clear from your generic description what the best approach would be. A couple of options:

Fan out to the get and post

http | groovy-transform > :request
:request > httpclient-get |log
:request > httpclient-post|log

The streams can be joined:

 http | groovy-transform > :request 
 :request > httpclient-get > :response 
 :request > httpclient-post > :response
 :response > log

Use a Header enricher processor to add the original payload as a header along with some code to extract the header in httpclient_post.

Note if you are using the kafka binder with kafka < 0.11 client, you need to explicitly enable custom headers: For example:

stream create hetest --definition "time | header-enricher --headers=foo='bar' | log --expression=headers" 
stream deploy hetest --properties "app.*.spring.cloud.stream.kafka.binder.headers=foo"

edited Nov 16 '18 at 15:29

answered Nov 13 '18 at 19:26

dturanski

1,124176

Hello @dturanski, Thanks for your time. I tried option one above to ask my question, but that option does not solve my problem because the flow will be separated into two other flows and there is no mechanism to join them again after they perform their work and the result is a single message.

– tioperez
Nov 15 '18 at 7:59

Hello @dturanski, Also try option 2, but the Header-Enricher processor does not work as I thought and the documentation is not very good, and we do not have any examples of its use. When I used it I thought that the variables' foo = bar 'or foo =' bar 'or' foo = bar 'or' foo = 'bar' "or 'foo = " bar "' that were declared in the processor were for persist in the flow until the end or, at least, until the next step, but when using a record later with #root or #root.header, my variable did not exist anywhere.

– tioperez
Nov 15 '18 at 8:00

1

For option 1, you can join the two streams. I have edited the response to illustrate.

– dturanski
Nov 16 '18 at 13:42

Hello @dturanski option 2 works fine and solves my problem, thank you very much.

– tioperez
Nov 19 '18 at 8:37

add a comment |

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53280824%2fhow-to-store-the-payload-at-any-point-to-use-it-later%23new-answer', 'question_page');

);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

It's not clear from your generic description what the best approach would be. A couple of options:

Fan out to the get and post

http | groovy-transform > :request
:request > httpclient-get |log
:request > httpclient-post|log

The streams can be joined:

 http | groovy-transform > :request 
 :request > httpclient-get > :response 
 :request > httpclient-post > :response
 :response > log

Use a Header enricher processor to add the original payload as a header along with some code to extract the header in httpclient_post.

Note if you are using the kafka binder with kafka < 0.11 client, you need to explicitly enable custom headers: For example:

stream create hetest --definition "time | header-enricher --headers=foo='bar' | log --expression=headers" 
stream deploy hetest --properties "app.*.spring.cloud.stream.kafka.binder.headers=foo"

edited Nov 16 '18 at 15:29

answered Nov 13 '18 at 19:26

dturanski

1,124176

Hello @dturanski, Thanks for your time. I tried option one above to ask my question, but that option does not solve my problem because the flow will be separated into two other flows and there is no mechanism to join them again after they perform their work and the result is a single message.

– tioperez
Nov 15 '18 at 7:59

Hello @dturanski, Also try option 2, but the Header-Enricher processor does not work as I thought and the documentation is not very good, and we do not have any examples of its use. When I used it I thought that the variables' foo = bar 'or foo =' bar 'or' foo = bar 'or' foo = 'bar' "or 'foo = " bar "' that were declared in the processor were for persist in the flow until the end or, at least, until the next step, but when using a record later with #root or #root.header, my variable did not exist anywhere.

– tioperez
Nov 15 '18 at 8:00

1

For option 1, you can join the two streams. I have edited the response to illustrate.

– dturanski
Nov 16 '18 at 13:42

Hello @dturanski option 2 works fine and solves my problem, thank you very much.

– tioperez
Nov 19 '18 at 8:37

add a comment |

It's not clear from your generic description what the best approach would be. A couple of options:

Fan out to the get and post

http | groovy-transform > :request
:request > httpclient-get |log
:request > httpclient-post|log

The streams can be joined:

 http | groovy-transform > :request 
 :request > httpclient-get > :response 
 :request > httpclient-post > :response
 :response > log

Use a Header enricher processor to add the original payload as a header along with some code to extract the header in httpclient_post.

Note if you are using the kafka binder with kafka < 0.11 client, you need to explicitly enable custom headers: For example:

stream create hetest --definition "time | header-enricher --headers=foo='bar' | log --expression=headers" 
stream deploy hetest --properties "app.*.spring.cloud.stream.kafka.binder.headers=foo"

edited Nov 16 '18 at 15:29

answered Nov 13 '18 at 19:26

dturanski

1,124176

Hello @dturanski, Thanks for your time. I tried option one above to ask my question, but that option does not solve my problem because the flow will be separated into two other flows and there is no mechanism to join them again after they perform their work and the result is a single message.

– tioperez
Nov 15 '18 at 7:59

Hello @dturanski, Also try option 2, but the Header-Enricher processor does not work as I thought and the documentation is not very good, and we do not have any examples of its use. When I used it I thought that the variables' foo = bar 'or foo =' bar 'or' foo = bar 'or' foo = 'bar' "or 'foo = " bar "' that were declared in the processor were for persist in the flow until the end or, at least, until the next step, but when using a record later with #root or #root.header, my variable did not exist anywhere.

– tioperez
Nov 15 '18 at 8:00

1

For option 1, you can join the two streams. I have edited the response to illustrate.

– dturanski
Nov 16 '18 at 13:42

Hello @dturanski option 2 works fine and solves my problem, thank you very much.

– tioperez
Nov 19 '18 at 8:37

add a comment |

It's not clear from your generic description what the best approach would be. A couple of options:

Fan out to the get and post

http | groovy-transform > :request
:request > httpclient-get |log
:request > httpclient-post|log

The streams can be joined:

 http | groovy-transform > :request 
 :request > httpclient-get > :response 
 :request > httpclient-post > :response
 :response > log

Use a Header enricher processor to add the original payload as a header along with some code to extract the header in httpclient_post.

Note if you are using the kafka binder with kafka < 0.11 client, you need to explicitly enable custom headers: For example:

stream create hetest --definition "time | header-enricher --headers=foo='bar' | log --expression=headers" 
stream deploy hetest --properties "app.*.spring.cloud.stream.kafka.binder.headers=foo"

edited Nov 16 '18 at 15:29

answered Nov 13 '18 at 19:26

dturanski

1,124176

It's not clear from your generic description what the best approach would be. A couple of options:

Fan out to the get and post

http | groovy-transform > :request
:request > httpclient-get |log
:request > httpclient-post|log

The streams can be joined:

 http | groovy-transform > :request 
 :request > httpclient-get > :response 
 :request > httpclient-post > :response
 :response > log

Use a Header enricher processor to add the original payload as a header along with some code to extract the header in httpclient_post.

Note if you are using the kafka binder with kafka < 0.11 client, you need to explicitly enable custom headers: For example:

stream create hetest --definition "time | header-enricher --headers=foo='bar' | log --expression=headers" 
stream deploy hetest --properties "app.*.spring.cloud.stream.kafka.binder.headers=foo"

edited Nov 16 '18 at 15:29

answered Nov 13 '18 at 19:26

dturanski

1,124176

edited Nov 16 '18 at 15:29

answered Nov 13 '18 at 19:26

dturanski

1,124176

answered Nov 13 '18 at 19:26

dturanski

1,124176

answered Nov 13 '18 at 19:26

dturanski

1,124176

Hello @dturanski, Thanks for your time. I tried option one above to ask my question, but that option does not solve my problem because the flow will be separated into two other flows and there is no mechanism to join them again after they perform their work and the result is a single message.

– tioperez
Nov 15 '18 at 7:59

Hello @dturanski, Also try option 2, but the Header-Enricher processor does not work as I thought and the documentation is not very good, and we do not have any examples of its use. When I used it I thought that the variables' foo = bar 'or foo =' bar 'or' foo = bar 'or' foo = 'bar' "or 'foo = " bar "' that were declared in the processor were for persist in the flow until the end or, at least, until the next step, but when using a record later with #root or #root.header, my variable did not exist anywhere.

– tioperez
Nov 15 '18 at 8:00

1

For option 1, you can join the two streams. I have edited the response to illustrate.

– dturanski
Nov 16 '18 at 13:42

Hello @dturanski option 2 works fine and solves my problem, thank you very much.

– tioperez
Nov 19 '18 at 8:37

add a comment |

Hello @dturanski, Thanks for your time. I tried option one above to ask my question, but that option does not solve my problem because the flow will be separated into two other flows and there is no mechanism to join them again after they perform their work and the result is a single message.

– tioperez
Nov 15 '18 at 7:59

Hello @dturanski, Also try option 2, but the Header-Enricher processor does not work as I thought and the documentation is not very good, and we do not have any examples of its use. When I used it I thought that the variables' foo = bar 'or foo =' bar 'or' foo = bar 'or' foo = 'bar' "or 'foo = " bar "' that were declared in the processor were for persist in the flow until the end or, at least, until the next step, but when using a record later with #root or #root.header, my variable did not exist anywhere.

– tioperez
Nov 15 '18 at 8:00

1

For option 1, you can join the two streams. I have edited the response to illustrate.

– dturanski
Nov 16 '18 at 13:42

Hello @dturanski option 2 works fine and solves my problem, thank you very much.

– tioperez
Nov 19 '18 at 8:37

Hello @dturanski, Thanks for your time. I tried option one above to ask my question, but that option does not solve my problem because the flow will be separated into two other flows and there is no mechanism to join them again after they perform their work and the result is a single message.

– tioperez
Nov 15 '18 at 7:59

Hello @dturanski, Also try option 2, but the Header-Enricher processor does not work as I thought and the documentation is not very good, and we do not have any examples of its use. When I used it I thought that the variables' foo = bar 'or foo =' bar 'or' foo = bar 'or' foo = 'bar' "or 'foo = " bar "' that were declared in the processor were for persist in the flow until the end or, at least, until the next step, but when using a record later with #root or #root.header, my variable did not exist anywhere.

– tioperez
Nov 15 '18 at 8:00

For option 1, you can join the two streams. I have edited the response to illustrate.

– dturanski
Nov 16 '18 at 13:42

Hello @dturanski option 2 works fine and solves my problem, thank you very much.

– tioperez
Nov 19 '18 at 8:37

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Odtnhj