Uploading a very large JSON file via REST using Java










0















I have a very large json file (about 500MB) that I am trying to upload using REST and java. It works using curl like this -



curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations


However, I get "Software caused connection abort: socket write error" when I do it using REST like this-



String filePath = "C:\2018-02-28.json";
String filename = "2018-02-28.json";
File uploadedFile = new File("C:\2018-02-28.json");
try
// HttpClient httpclient = HttpClientBuilder.create().build();
String authHeader = authToken();
HttpEntity entity = MultipartEntityBuilder
.create()
.addTextBody("name", "fileDate")
.addTextBody("fileName", filename)
.addTextBody("Content-Type", "application/json")
.addBinaryBody("fileData", new File(filePath), ContentType.create("application/json"), filename)
.build();

HttpClient httpClient = HttpClients.custom()
.setConnectionTimeToLive(2700, TimeUnit.SECONDS)
.setMaxConnTotal(400).setMaxConnPerRoute(400)
.setDefaultRequestConfig(RequestConfig.custom()
.setSocketTimeout(30000).setConnectTimeout(5000).build())
.setRetryHandler(new DefaultHttpRequestRetryHandler(5, true))
.build();

HttpPost request = new HttpPost("http://md:8086/InputGateway/rest/input-gateway-service/ABC/invocations");
request.setEntity(entity);
request.setHeader(HttpHeaders.AUTHORIZATION, authHeader);
HttpResponse resp = httpClient.execute(request);
HttpEntity entity1 = resp.getEntity();
System.out.println(EntityUtils.toString(entity1, "utf-8"));
System.out.println("File has been Uploaded successfully: " + uploadedFile);
catch (Exception ex)
throw new Exception( ex.toString());

}


what am I doing wrong here










share|improve this question
























  • What library are you using? Have you tried Java's built-in HttpClient?

    – Jacob G.
    Nov 14 '18 at 20:30











  • Some servers may implement traffic/size limits to prevent DOS issues. Contact the server owner for infos.

    – Konrad
    Nov 14 '18 at 20:59











  • @Konrad if it works with curl it's not a server side issue

    – mavriksc
    Nov 14 '18 at 21:50















0















I have a very large json file (about 500MB) that I am trying to upload using REST and java. It works using curl like this -



curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations


However, I get "Software caused connection abort: socket write error" when I do it using REST like this-



String filePath = "C:\2018-02-28.json";
String filename = "2018-02-28.json";
File uploadedFile = new File("C:\2018-02-28.json");
try
// HttpClient httpclient = HttpClientBuilder.create().build();
String authHeader = authToken();
HttpEntity entity = MultipartEntityBuilder
.create()
.addTextBody("name", "fileDate")
.addTextBody("fileName", filename)
.addTextBody("Content-Type", "application/json")
.addBinaryBody("fileData", new File(filePath), ContentType.create("application/json"), filename)
.build();

HttpClient httpClient = HttpClients.custom()
.setConnectionTimeToLive(2700, TimeUnit.SECONDS)
.setMaxConnTotal(400).setMaxConnPerRoute(400)
.setDefaultRequestConfig(RequestConfig.custom()
.setSocketTimeout(30000).setConnectTimeout(5000).build())
.setRetryHandler(new DefaultHttpRequestRetryHandler(5, true))
.build();

HttpPost request = new HttpPost("http://md:8086/InputGateway/rest/input-gateway-service/ABC/invocations");
request.setEntity(entity);
request.setHeader(HttpHeaders.AUTHORIZATION, authHeader);
HttpResponse resp = httpClient.execute(request);
HttpEntity entity1 = resp.getEntity();
System.out.println(EntityUtils.toString(entity1, "utf-8"));
System.out.println("File has been Uploaded successfully: " + uploadedFile);
catch (Exception ex)
throw new Exception( ex.toString());

}


what am I doing wrong here










share|improve this question
























  • What library are you using? Have you tried Java's built-in HttpClient?

    – Jacob G.
    Nov 14 '18 at 20:30











  • Some servers may implement traffic/size limits to prevent DOS issues. Contact the server owner for infos.

    – Konrad
    Nov 14 '18 at 20:59











  • @Konrad if it works with curl it's not a server side issue

    – mavriksc
    Nov 14 '18 at 21:50













0












0








0








I have a very large json file (about 500MB) that I am trying to upload using REST and java. It works using curl like this -



curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations


However, I get "Software caused connection abort: socket write error" when I do it using REST like this-



String filePath = "C:\2018-02-28.json";
String filename = "2018-02-28.json";
File uploadedFile = new File("C:\2018-02-28.json");
try
// HttpClient httpclient = HttpClientBuilder.create().build();
String authHeader = authToken();
HttpEntity entity = MultipartEntityBuilder
.create()
.addTextBody("name", "fileDate")
.addTextBody("fileName", filename)
.addTextBody("Content-Type", "application/json")
.addBinaryBody("fileData", new File(filePath), ContentType.create("application/json"), filename)
.build();

HttpClient httpClient = HttpClients.custom()
.setConnectionTimeToLive(2700, TimeUnit.SECONDS)
.setMaxConnTotal(400).setMaxConnPerRoute(400)
.setDefaultRequestConfig(RequestConfig.custom()
.setSocketTimeout(30000).setConnectTimeout(5000).build())
.setRetryHandler(new DefaultHttpRequestRetryHandler(5, true))
.build();

HttpPost request = new HttpPost("http://md:8086/InputGateway/rest/input-gateway-service/ABC/invocations");
request.setEntity(entity);
request.setHeader(HttpHeaders.AUTHORIZATION, authHeader);
HttpResponse resp = httpClient.execute(request);
HttpEntity entity1 = resp.getEntity();
System.out.println(EntityUtils.toString(entity1, "utf-8"));
System.out.println("File has been Uploaded successfully: " + uploadedFile);
catch (Exception ex)
throw new Exception( ex.toString());

}


what am I doing wrong here










share|improve this question
















I have a very large json file (about 500MB) that I am trying to upload using REST and java. It works using curl like this -



curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations


However, I get "Software caused connection abort: socket write error" when I do it using REST like this-



String filePath = "C:\2018-02-28.json";
String filename = "2018-02-28.json";
File uploadedFile = new File("C:\2018-02-28.json");
try
// HttpClient httpclient = HttpClientBuilder.create().build();
String authHeader = authToken();
HttpEntity entity = MultipartEntityBuilder
.create()
.addTextBody("name", "fileDate")
.addTextBody("fileName", filename)
.addTextBody("Content-Type", "application/json")
.addBinaryBody("fileData", new File(filePath), ContentType.create("application/json"), filename)
.build();

HttpClient httpClient = HttpClients.custom()
.setConnectionTimeToLive(2700, TimeUnit.SECONDS)
.setMaxConnTotal(400).setMaxConnPerRoute(400)
.setDefaultRequestConfig(RequestConfig.custom()
.setSocketTimeout(30000).setConnectTimeout(5000).build())
.setRetryHandler(new DefaultHttpRequestRetryHandler(5, true))
.build();

HttpPost request = new HttpPost("http://md:8086/InputGateway/rest/input-gateway-service/ABC/invocations");
request.setEntity(entity);
request.setHeader(HttpHeaders.AUTHORIZATION, authHeader);
HttpResponse resp = httpClient.execute(request);
HttpEntity entity1 = resp.getEntity();
System.out.println(EntityUtils.toString(entity1, "utf-8"));
System.out.println("File has been Uploaded successfully: " + uploadedFile);
catch (Exception ex)
throw new Exception( ex.toString());

}


what am I doing wrong here







java rest






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 14 '18 at 20:27









Justin Pearce

3,89021932




3,89021932










asked Nov 14 '18 at 20:26









anumehaanumeha

157




157












  • What library are you using? Have you tried Java's built-in HttpClient?

    – Jacob G.
    Nov 14 '18 at 20:30











  • Some servers may implement traffic/size limits to prevent DOS issues. Contact the server owner for infos.

    – Konrad
    Nov 14 '18 at 20:59











  • @Konrad if it works with curl it's not a server side issue

    – mavriksc
    Nov 14 '18 at 21:50

















  • What library are you using? Have you tried Java's built-in HttpClient?

    – Jacob G.
    Nov 14 '18 at 20:30











  • Some servers may implement traffic/size limits to prevent DOS issues. Contact the server owner for infos.

    – Konrad
    Nov 14 '18 at 20:59











  • @Konrad if it works with curl it's not a server side issue

    – mavriksc
    Nov 14 '18 at 21:50
















What library are you using? Have you tried Java's built-in HttpClient?

– Jacob G.
Nov 14 '18 at 20:30





What library are you using? Have you tried Java's built-in HttpClient?

– Jacob G.
Nov 14 '18 at 20:30













Some servers may implement traffic/size limits to prevent DOS issues. Contact the server owner for infos.

– Konrad
Nov 14 '18 at 20:59





Some servers may implement traffic/size limits to prevent DOS issues. Contact the server owner for infos.

– Konrad
Nov 14 '18 at 20:59













@Konrad if it works with curl it's not a server side issue

– mavriksc
Nov 14 '18 at 21:50





@Konrad if it works with curl it's not a server side issue

– mavriksc
Nov 14 '18 at 21:50












1 Answer
1






active

oldest

votes


















0














curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations


One thing you may want to do is run this command in verbose mode, to see what is really happening. It is possible that the example that works because curl is using an Expect header that allows the server to prepare for the data dump.



A packet analyzer, to see if the remote server is actually sending you a RST, or if something else is going on in the network stack.



(Obvious, but just in case: try using the same code to send a smaller file. Make sure that the size it the limiting factor. Doing a binary search to find out how big a file is acceptable might provide an extra clue.)






share|improve this answer






















    Your Answer






    StackExchange.ifUsing("editor", function ()
    StackExchange.using("externalEditor", function ()
    StackExchange.using("snippets", function ()
    StackExchange.snippets.init();
    );
    );
    , "code-snippets");

    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "1"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader:
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    ,
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );













    draft saved

    draft discarded


















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53308238%2fuploading-a-very-large-json-file-via-rest-using-java%23new-answer', 'question_page');

    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations


    One thing you may want to do is run this command in verbose mode, to see what is really happening. It is possible that the example that works because curl is using an Expect header that allows the server to prepare for the data dump.



    A packet analyzer, to see if the remote server is actually sending you a RST, or if something else is going on in the network stack.



    (Obvious, but just in case: try using the same code to send a smaller file. Make sure that the size it the limiting factor. Doing a binary search to find out how big a file is acceptable might provide an extra clue.)






    share|improve this answer



























      0














      curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations


      One thing you may want to do is run this command in verbose mode, to see what is really happening. It is possible that the example that works because curl is using an Expect header that allows the server to prepare for the data dump.



      A packet analyzer, to see if the remote server is actually sending you a RST, or if something else is going on in the network stack.



      (Obvious, but just in case: try using the same code to send a smaller file. Make sure that the size it the limiting factor. Doing a binary search to find out how big a file is acceptable might provide an extra clue.)






      share|improve this answer

























        0












        0








        0







        curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations


        One thing you may want to do is run this command in verbose mode, to see what is really happening. It is possible that the example that works because curl is using an Expect header that allows the server to prepare for the data dump.



        A packet analyzer, to see if the remote server is actually sending you a RST, or if something else is going on in the network stack.



        (Obvious, but just in case: try using the same code to send a smaller file. Make sure that the size it the limiting factor. Doing a binary search to find out how big a file is acceptable might provide an extra clue.)






        share|improve this answer













        curl -H "content-type: application/json" --data-binary @2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations


        One thing you may want to do is run this command in verbose mode, to see what is really happening. It is possible that the example that works because curl is using an Expect header that allows the server to prepare for the data dump.



        A packet analyzer, to see if the remote server is actually sending you a RST, or if something else is going on in the network stack.



        (Obvious, but just in case: try using the same code to send a smaller file. Make sure that the size it the limiting factor. Doing a binary search to find out how big a file is acceptable might provide an extra clue.)







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Nov 15 '18 at 3:21









        VoiceOfUnreasonVoiceOfUnreason

        20.8k22047




        20.8k22047





























            draft saved

            draft discarded
















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid


            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.

            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53308238%2fuploading-a-very-large-json-file-via-rest-using-java%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            這個網誌中的熱門文章

            How to read a connectionString WITH PROVIDER in .NET Core?

            In R, how to develop a multiplot heatmap.2 figure showing key labels successfully

            Museum of Modern and Contemporary Art of Trento and Rovereto