Consequences of using use StepExecutionContext/JobExecutionContext to share Hashmap with large values
up vote
1
down vote
favorite
I have a requirement in which I am retrieving values in one Reader of the Step using SQL statements and doing the same request in next reader.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step.
For this I have gone through the following link on SO :
How can we share data between the different steps of a Job in Spring Batch?
In many of the comments it is mentioned that 'data must be short'.
Also it is mentioned in one response that: these contexts are good to share strings or simple values, but not for sharing collections or huge amounts of data.
By passing that HashMap, I believe it automatically infers that the reference of the HashMap will be passed.
It would be good to know the possible consequences of passing it before hand and any better alternative approach.
java spring spring-batch batch-processing
asked Nov 11 at 17:53
Ramandeep Singh
4829
add a comment |
up vote
1
down vote
favorite
I have a requirement in which I am retrieving values in one Reader of the Step using SQL statements and doing the same request in next reader.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step.
For this I have gone through the following link on SO :
How can we share data between the different steps of a Job in Spring Batch?
In many of the comments it is mentioned that 'data must be short'.
Also it is mentioned in one response that: these contexts are good to share strings or simple values, but not for sharing collections or huge amounts of data.
By passing that HashMap, I believe it automatically infers that the reference of the HashMap will be passed.
It would be good to know the possible consequences of passing it before hand and any better alternative approach.
java spring spring-batch batch-processing
asked Nov 11 at 17:53
Ramandeep Singh
4829
1
That thread is outmoded. Which version are you currently using?
– Andrew Tobilko
Nov 11 at 18:07
@AndrewTobilko : it is 3.0.7
– Ramandeep Singh
Nov 11 at 18:11
I am guessng the problem is that Spring Batch saves its state in a table during execution - and this Map of yours would be saved as part of that context. If the data in the map is large enough, it could be more than the column in the table can hold (by default, anyway). IIRC, in one of our applications we just resized that column larger than it is by default...
– moilejter
Nov 11 at 19:12
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I have a requirement in which I am retrieving values in one Reader of the Step using SQL statements and doing the same request in next reader.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step.
For this I have gone through the following link on SO :
How can we share data between the different steps of a Job in Spring Batch?
In many of the comments it is mentioned that 'data must be short'.
Also it is mentioned in one response that: these contexts are good to share strings or simple values, but not for sharing collections or huge amounts of data.
By passing that HashMap, I believe it automatically infers that the reference of the HashMap will be passed.
It would be good to know the possible consequences of passing it before hand and any better alternative approach.
java spring spring-batch batch-processing
asked Nov 11 at 17:53
Ramandeep Singh
4829
I have a requirement in which I am retrieving values in one Reader of the Step using SQL statements and doing the same request in next reader.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step.
For this I have gone through the following link on SO :
How can we share data between the different steps of a Job in Spring Batch?
In many of the comments it is mentioned that 'data must be short'.
Also it is mentioned in one response that: these contexts are good to share strings or simple values, but not for sharing collections or huge amounts of data.
By passing that HashMap, I believe it automatically infers that the reference of the HashMap will be passed.
It would be good to know the possible consequences of passing it before hand and any better alternative approach.
java spring spring-batch batch-processing
java spring spring-batch batch-processing
asked Nov 11 at 17:53
Ramandeep Singh
4829
asked Nov 11 at 17:53
Ramandeep Singh
4829
asked Nov 11 at 17:53
Ramandeep Singh
4829
asked Nov 11 at 17:53
Ramandeep Singh
4829
asked Nov 11 at 17:53
Ramandeep Singh
4829
4829
1
That thread is outmoded. Which version are you currently using?
– Andrew Tobilko
Nov 11 at 18:07
@AndrewTobilko : it is 3.0.7
– Ramandeep Singh
Nov 11 at 18:11
I am guessng the problem is that Spring Batch saves its state in a table during execution - and this Map of yours would be saved as part of that context. If the data in the map is large enough, it could be more than the column in the table can hold (by default, anyway). IIRC, in one of our applications we just resized that column larger than it is by default...
– moilejter
Nov 11 at 19:12
add a comment |
1
That thread is outmoded. Which version are you currently using?
– Andrew Tobilko
Nov 11 at 18:07
@AndrewTobilko : it is 3.0.7
– Ramandeep Singh
Nov 11 at 18:11
I am guessng the problem is that Spring Batch saves its state in a table during execution - and this Map of yours would be saved as part of that context. If the data in the map is large enough, it could be more than the column in the table can hold (by default, anyway). IIRC, in one of our applications we just resized that column larger than it is by default...
– moilejter
Nov 11 at 19:12
1
1
That thread is outmoded. Which version are you currently using?
– Andrew Tobilko
Nov 11 at 18:07
That thread is outmoded. Which version are you currently using?
– Andrew Tobilko
Nov 11 at 18:07
@AndrewTobilko : it is 3.0.7
– Ramandeep Singh
Nov 11 at 18:11
@AndrewTobilko : it is 3.0.7
– Ramandeep Singh
Nov 11 at 18:11
I am guessng the problem is that Spring Batch saves its state in a table during execution - and this Map of yours would be saved as part of that context. If the data in the map is large enough, it could be more than the column in the table can hold (by default, anyway). IIRC, in one of our applications we just resized that column larger than it is by default...
– moilejter
Nov 11 at 19:12
I am guessng the problem is that Spring Batch saves its state in a table during execution - and this Map of yours would be saved as part of that context. If the data in the map is large enough, it could be more than the column in the table can hold (by default, anyway). IIRC, in one of our applications we just resized that column larger than it is by default...
– moilejter
Nov 11 at 19:12
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
Passing data between step is indeed done via the execution context. However, you should be careful about the size of data you put in the execution context as it is persisted between steps.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step
You can read the data from the database only once and put it in a cache. The second reader can then get the data from the cache. This would be faster that reading the data from the database a second time.
Hope this helps.
answered Nov 11 at 19:30
Mahmoud Ben Hassine
3,5301714
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
Passing data between step is indeed done via the execution context. However, you should be careful about the size of data you put in the execution context as it is persisted between steps.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step
You can read the data from the database only once and put it in a cache. The second reader can then get the data from the cache. This would be faster that reading the data from the database a second time.
Hope this helps.
answered Nov 11 at 19:30
Mahmoud Ben Hassine
3,5301714
add a comment |
up vote
0
down vote
Passing data between step is indeed done via the execution context. However, you should be careful about the size of data you put in the execution context as it is persisted between steps.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step
You can read the data from the database only once and put it in a cache. The second reader can then get the data from the cache. This would be faster that reading the data from the database a second time.
Hope this helps.
answered Nov 11 at 19:30
Mahmoud Ben Hassine
3,5301714
add a comment |
up vote
0
down vote
up vote
0
down vote
Passing data between step is indeed done via the execution context. However, you should be careful about the size of data you put in the execution context as it is persisted between steps.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step
You can read the data from the database only once and put it in a cache. The second reader can then get the data from the cache. This would be faster that reading the data from the database a second time.
Hope this helps.
answered Nov 11 at 19:30
Mahmoud Ben Hassine
3,5301714
Passing data between step is indeed done via the execution context. However, you should be careful about the size of data you put in the execution context as it is persisted between steps.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step
You can read the data from the database only once and put it in a cache. The second reader can then get the data from the cache. This would be faster that reading the data from the database a second time.
Hope this helps.
answered Nov 11 at 19:30
Mahmoud Ben Hassine
3,5301714
answered Nov 11 at 19:30
Mahmoud Ben Hassine
3,5301714
answered Nov 11 at 19:30
Mahmoud Ben Hassine
3,5301714
answered Nov 11 at 19:30
Mahmoud Ben Hassine
3,5301714
3,5301714
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53251535%2fconsequences-of-using-use-stepexecutioncontext-jobexecutioncontext-to-share-hash%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
That thread is outmoded. Which version are you currently using?
– Andrew Tobilko
Nov 11 at 18:07
@AndrewTobilko : it is 3.0.7
– Ramandeep Singh
Nov 11 at 18:11
I am guessng the problem is that Spring Batch saves its state in a table during execution - and this Map of yours would be saved as part of that context. If the data in the map is large enough, it could be more than the column in the table can hold (by default, anyway). IIRC, in one of our applications we just resized that column larger than it is by default...
– moilejter
Nov 11 at 19:12