Uploading a very large JSON file via REST using Java
I have a very large json file (about 500MB) that I am trying to upload using REST and java. It works using curl like this -
curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations
However, I get "Software caused connection abort: socket write error" when I do it using REST like this-
String filePath = "C:\2018-02-28.json";
String filename = "2018-02-28.json";
File uploadedFile = new File("C:\2018-02-28.json");
try
// HttpClient httpclient = HttpClientBuilder.create().build();
String authHeader = authToken();
HttpEntity entity = MultipartEntityBuilder
.create()
.addTextBody("name", "fileDate")
.addTextBody("fileName", filename)
.addTextBody("Content-Type", "application/json")
.addBinaryBody("fileData", new File(filePath), ContentType.create("application/json"), filename)
.build();
HttpClient httpClient = HttpClients.custom()
.setConnectionTimeToLive(2700, TimeUnit.SECONDS)
.setMaxConnTotal(400).setMaxConnPerRoute(400)
.setDefaultRequestConfig(RequestConfig.custom()
.setSocketTimeout(30000).setConnectTimeout(5000).build())
.setRetryHandler(new DefaultHttpRequestRetryHandler(5, true))
.build();
HttpPost request = new HttpPost("http://md:8086/InputGateway/rest/input-gateway-service/ABC/invocations");
request.setEntity(entity);
request.setHeader(HttpHeaders.AUTHORIZATION, authHeader);
HttpResponse resp = httpClient.execute(request);
HttpEntity entity1 = resp.getEntity();
System.out.println(EntityUtils.toString(entity1, "utf-8"));
System.out.println("File has been Uploaded successfully: " + uploadedFile);
catch (Exception ex)
throw new Exception( ex.toString());
}
what am I doing wrong here
java rest
add a comment |
I have a very large json file (about 500MB) that I am trying to upload using REST and java. It works using curl like this -
curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations
However, I get "Software caused connection abort: socket write error" when I do it using REST like this-
String filePath = "C:\2018-02-28.json";
String filename = "2018-02-28.json";
File uploadedFile = new File("C:\2018-02-28.json");
try
// HttpClient httpclient = HttpClientBuilder.create().build();
String authHeader = authToken();
HttpEntity entity = MultipartEntityBuilder
.create()
.addTextBody("name", "fileDate")
.addTextBody("fileName", filename)
.addTextBody("Content-Type", "application/json")
.addBinaryBody("fileData", new File(filePath), ContentType.create("application/json"), filename)
.build();
HttpClient httpClient = HttpClients.custom()
.setConnectionTimeToLive(2700, TimeUnit.SECONDS)
.setMaxConnTotal(400).setMaxConnPerRoute(400)
.setDefaultRequestConfig(RequestConfig.custom()
.setSocketTimeout(30000).setConnectTimeout(5000).build())
.setRetryHandler(new DefaultHttpRequestRetryHandler(5, true))
.build();
HttpPost request = new HttpPost("http://md:8086/InputGateway/rest/input-gateway-service/ABC/invocations");
request.setEntity(entity);
request.setHeader(HttpHeaders.AUTHORIZATION, authHeader);
HttpResponse resp = httpClient.execute(request);
HttpEntity entity1 = resp.getEntity();
System.out.println(EntityUtils.toString(entity1, "utf-8"));
System.out.println("File has been Uploaded successfully: " + uploadedFile);
catch (Exception ex)
throw new Exception( ex.toString());
}
what am I doing wrong here
java rest
What library are you using? Have you tried Java's built-inHttpClient
?
– Jacob G.
Nov 14 '18 at 20:30
Some servers may implement traffic/size limits to prevent DOS issues. Contact the server owner for infos.
– Konrad
Nov 14 '18 at 20:59
@Konrad if it works with curl it's not a server side issue
– mavriksc
Nov 14 '18 at 21:50
add a comment |
I have a very large json file (about 500MB) that I am trying to upload using REST and java. It works using curl like this -
curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations
However, I get "Software caused connection abort: socket write error" when I do it using REST like this-
String filePath = "C:\2018-02-28.json";
String filename = "2018-02-28.json";
File uploadedFile = new File("C:\2018-02-28.json");
try
// HttpClient httpclient = HttpClientBuilder.create().build();
String authHeader = authToken();
HttpEntity entity = MultipartEntityBuilder
.create()
.addTextBody("name", "fileDate")
.addTextBody("fileName", filename)
.addTextBody("Content-Type", "application/json")
.addBinaryBody("fileData", new File(filePath), ContentType.create("application/json"), filename)
.build();
HttpClient httpClient = HttpClients.custom()
.setConnectionTimeToLive(2700, TimeUnit.SECONDS)
.setMaxConnTotal(400).setMaxConnPerRoute(400)
.setDefaultRequestConfig(RequestConfig.custom()
.setSocketTimeout(30000).setConnectTimeout(5000).build())
.setRetryHandler(new DefaultHttpRequestRetryHandler(5, true))
.build();
HttpPost request = new HttpPost("http://md:8086/InputGateway/rest/input-gateway-service/ABC/invocations");
request.setEntity(entity);
request.setHeader(HttpHeaders.AUTHORIZATION, authHeader);
HttpResponse resp = httpClient.execute(request);
HttpEntity entity1 = resp.getEntity();
System.out.println(EntityUtils.toString(entity1, "utf-8"));
System.out.println("File has been Uploaded successfully: " + uploadedFile);
catch (Exception ex)
throw new Exception( ex.toString());
}
what am I doing wrong here
java rest
I have a very large json file (about 500MB) that I am trying to upload using REST and java. It works using curl like this -
curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations
However, I get "Software caused connection abort: socket write error" when I do it using REST like this-
String filePath = "C:\2018-02-28.json";
String filename = "2018-02-28.json";
File uploadedFile = new File("C:\2018-02-28.json");
try
// HttpClient httpclient = HttpClientBuilder.create().build();
String authHeader = authToken();
HttpEntity entity = MultipartEntityBuilder
.create()
.addTextBody("name", "fileDate")
.addTextBody("fileName", filename)
.addTextBody("Content-Type", "application/json")
.addBinaryBody("fileData", new File(filePath), ContentType.create("application/json"), filename)
.build();
HttpClient httpClient = HttpClients.custom()
.setConnectionTimeToLive(2700, TimeUnit.SECONDS)
.setMaxConnTotal(400).setMaxConnPerRoute(400)
.setDefaultRequestConfig(RequestConfig.custom()
.setSocketTimeout(30000).setConnectTimeout(5000).build())
.setRetryHandler(new DefaultHttpRequestRetryHandler(5, true))
.build();
HttpPost request = new HttpPost("http://md:8086/InputGateway/rest/input-gateway-service/ABC/invocations");
request.setEntity(entity);
request.setHeader(HttpHeaders.AUTHORIZATION, authHeader);
HttpResponse resp = httpClient.execute(request);
HttpEntity entity1 = resp.getEntity();
System.out.println(EntityUtils.toString(entity1, "utf-8"));
System.out.println("File has been Uploaded successfully: " + uploadedFile);
catch (Exception ex)
throw new Exception( ex.toString());
}
what am I doing wrong here
java rest
java rest
edited Nov 14 '18 at 20:27
Justin Pearce
3,89021932
3,89021932
asked Nov 14 '18 at 20:26
anumehaanumeha
157
157
What library are you using? Have you tried Java's built-inHttpClient
?
– Jacob G.
Nov 14 '18 at 20:30
Some servers may implement traffic/size limits to prevent DOS issues. Contact the server owner for infos.
– Konrad
Nov 14 '18 at 20:59
@Konrad if it works with curl it's not a server side issue
– mavriksc
Nov 14 '18 at 21:50
add a comment |
What library are you using? Have you tried Java's built-inHttpClient
?
– Jacob G.
Nov 14 '18 at 20:30
Some servers may implement traffic/size limits to prevent DOS issues. Contact the server owner for infos.
– Konrad
Nov 14 '18 at 20:59
@Konrad if it works with curl it's not a server side issue
– mavriksc
Nov 14 '18 at 21:50
What library are you using? Have you tried Java's built-in
HttpClient
?– Jacob G.
Nov 14 '18 at 20:30
What library are you using? Have you tried Java's built-in
HttpClient
?– Jacob G.
Nov 14 '18 at 20:30
Some servers may implement traffic/size limits to prevent DOS issues. Contact the server owner for infos.
– Konrad
Nov 14 '18 at 20:59
Some servers may implement traffic/size limits to prevent DOS issues. Contact the server owner for infos.
– Konrad
Nov 14 '18 at 20:59
@Konrad if it works with curl it's not a server side issue
– mavriksc
Nov 14 '18 at 21:50
@Konrad if it works with curl it's not a server side issue
– mavriksc
Nov 14 '18 at 21:50
add a comment |
1 Answer
1
active
oldest
votes
curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations
One thing you may want to do is run this command in verbose mode, to see what is really happening. It is possible that the example that works because curl is using an Expect header that allows the server to prepare for the data dump.
A packet analyzer, to see if the remote server is actually sending you a RST, or if something else is going on in the network stack.
(Obvious, but just in case: try using the same code to send a smaller file. Make sure that the size it the limiting factor. Doing a binary search to find out how big a file is acceptable might provide an extra clue.)
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53308238%2fuploading-a-very-large-json-file-via-rest-using-java%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations
One thing you may want to do is run this command in verbose mode, to see what is really happening. It is possible that the example that works because curl is using an Expect header that allows the server to prepare for the data dump.
A packet analyzer, to see if the remote server is actually sending you a RST, or if something else is going on in the network stack.
(Obvious, but just in case: try using the same code to send a smaller file. Make sure that the size it the limiting factor. Doing a binary search to find out how big a file is acceptable might provide an extra clue.)
add a comment |
curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations
One thing you may want to do is run this command in verbose mode, to see what is really happening. It is possible that the example that works because curl is using an Expect header that allows the server to prepare for the data dump.
A packet analyzer, to see if the remote server is actually sending you a RST, or if something else is going on in the network stack.
(Obvious, but just in case: try using the same code to send a smaller file. Make sure that the size it the limiting factor. Doing a binary search to find out how big a file is acceptable might provide an extra clue.)
add a comment |
curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations
One thing you may want to do is run this command in verbose mode, to see what is really happening. It is possible that the example that works because curl is using an Expect header that allows the server to prepare for the data dump.
A packet analyzer, to see if the remote server is actually sending you a RST, or if something else is going on in the network stack.
(Obvious, but just in case: try using the same code to send a smaller file. Make sure that the size it the limiting factor. Doing a binary search to find out how big a file is acceptable might provide an extra clue.)
curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations
One thing you may want to do is run this command in verbose mode, to see what is really happening. It is possible that the example that works because curl is using an Expect header that allows the server to prepare for the data dump.
A packet analyzer, to see if the remote server is actually sending you a RST, or if something else is going on in the network stack.
(Obvious, but just in case: try using the same code to send a smaller file. Make sure that the size it the limiting factor. Doing a binary search to find out how big a file is acceptable might provide an extra clue.)
answered Nov 15 '18 at 3:21
VoiceOfUnreasonVoiceOfUnreason
20.8k22047
20.8k22047
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53308238%2fuploading-a-very-large-json-file-via-rest-using-java%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What library are you using? Have you tried Java's built-in
HttpClient
?– Jacob G.
Nov 14 '18 at 20:30
Some servers may implement traffic/size limits to prevent DOS issues. Contact the server owner for infos.
– Konrad
Nov 14 '18 at 20:59
@Konrad if it works with curl it's not a server side issue
– mavriksc
Nov 14 '18 at 21:50