how can i read file fastest way in c++ ? to send data to 'Vertex Buffer Object









up vote
0
down vote

favorite












i have 4-5 gigabyte ".obj" files like:



Sword_1 folder has 2000 obj files , Sword_2 folder has 2000, Dagger_1 folder has 2000 ... etc totaly i have 4-5 gigabyte ".obj" files.



files contents these :




v 16.418303 40.112064 0.078153



vt 0.445198 0.462720



vn 0.264392 0.654428 0.708394



f 27/72/32 38/73/43 23/74/13




they are classic texts



and i am reading each them like this:



char text[10000][60];
ifstream in(filename);
if(!in.is_open())

return false;

while(!in.eof())

in.getline(text[i],60);
i++;



after read one folder i am sending datas to 'VBO'.



all files take 10 minutes to complete.
when i make multithreading with '3 thread' (my cpu 4core 4 thread) , then same process take 3-4 minutes to complete.
but still too long.



how can i make it faster ?



if i need to use binary, how can i read binary with c++ ? can you make example for me :) because i don't know anything about binaries. and if i made binary i needed to convert 'char' format ? and if i need to convert, convert process will take same minutes to complete ? sorry for bad english.










share|improve this question























  • So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.
    – S.M.
    Nov 12 at 3:31











  • Have you looked into using a memory mapped file?
    – user4942583
    Nov 12 at 3:46











  • If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.
    – o11c
    Nov 12 at 4:10










  • thanks for answers guys i think i need to do another method like said wagner
    – Emre Kaya
    Nov 12 at 4:19














up vote
0
down vote

favorite












i have 4-5 gigabyte ".obj" files like:



Sword_1 folder has 2000 obj files , Sword_2 folder has 2000, Dagger_1 folder has 2000 ... etc totaly i have 4-5 gigabyte ".obj" files.



files contents these :




v 16.418303 40.112064 0.078153



vt 0.445198 0.462720



vn 0.264392 0.654428 0.708394



f 27/72/32 38/73/43 23/74/13




they are classic texts



and i am reading each them like this:



char text[10000][60];
ifstream in(filename);
if(!in.is_open())

return false;

while(!in.eof())

in.getline(text[i],60);
i++;



after read one folder i am sending datas to 'VBO'.



all files take 10 minutes to complete.
when i make multithreading with '3 thread' (my cpu 4core 4 thread) , then same process take 3-4 minutes to complete.
but still too long.



how can i make it faster ?



if i need to use binary, how can i read binary with c++ ? can you make example for me :) because i don't know anything about binaries. and if i made binary i needed to convert 'char' format ? and if i need to convert, convert process will take same minutes to complete ? sorry for bad english.










share|improve this question























  • So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.
    – S.M.
    Nov 12 at 3:31











  • Have you looked into using a memory mapped file?
    – user4942583
    Nov 12 at 3:46











  • If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.
    – o11c
    Nov 12 at 4:10










  • thanks for answers guys i think i need to do another method like said wagner
    – Emre Kaya
    Nov 12 at 4:19












up vote
0
down vote

favorite









up vote
0
down vote

favorite











i have 4-5 gigabyte ".obj" files like:



Sword_1 folder has 2000 obj files , Sword_2 folder has 2000, Dagger_1 folder has 2000 ... etc totaly i have 4-5 gigabyte ".obj" files.



files contents these :




v 16.418303 40.112064 0.078153



vt 0.445198 0.462720



vn 0.264392 0.654428 0.708394



f 27/72/32 38/73/43 23/74/13




they are classic texts



and i am reading each them like this:



char text[10000][60];
ifstream in(filename);
if(!in.is_open())

return false;

while(!in.eof())

in.getline(text[i],60);
i++;



after read one folder i am sending datas to 'VBO'.



all files take 10 minutes to complete.
when i make multithreading with '3 thread' (my cpu 4core 4 thread) , then same process take 3-4 minutes to complete.
but still too long.



how can i make it faster ?



if i need to use binary, how can i read binary with c++ ? can you make example for me :) because i don't know anything about binaries. and if i made binary i needed to convert 'char' format ? and if i need to convert, convert process will take same minutes to complete ? sorry for bad english.










share|improve this question















i have 4-5 gigabyte ".obj" files like:



Sword_1 folder has 2000 obj files , Sword_2 folder has 2000, Dagger_1 folder has 2000 ... etc totaly i have 4-5 gigabyte ".obj" files.



files contents these :




v 16.418303 40.112064 0.078153



vt 0.445198 0.462720



vn 0.264392 0.654428 0.708394



f 27/72/32 38/73/43 23/74/13




they are classic texts



and i am reading each them like this:



char text[10000][60];
ifstream in(filename);
if(!in.is_open())

return false;

while(!in.eof())

in.getline(text[i],60);
i++;



after read one folder i am sending datas to 'VBO'.



all files take 10 minutes to complete.
when i make multithreading with '3 thread' (my cpu 4core 4 thread) , then same process take 3-4 minutes to complete.
but still too long.



how can i make it faster ?



if i need to use binary, how can i read binary with c++ ? can you make example for me :) because i don't know anything about binaries. and if i made binary i needed to convert 'char' format ? and if i need to convert, convert process will take same minutes to complete ? sorry for bad english.







c++






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 12 at 4:02









eyllanesc

72.4k93054




72.4k93054










asked Nov 12 at 3:05









Emre Kaya

103




103











  • So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.
    – S.M.
    Nov 12 at 3:31











  • Have you looked into using a memory mapped file?
    – user4942583
    Nov 12 at 3:46











  • If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.
    – o11c
    Nov 12 at 4:10










  • thanks for answers guys i think i need to do another method like said wagner
    – Emre Kaya
    Nov 12 at 4:19
















  • So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.
    – S.M.
    Nov 12 at 3:31











  • Have you looked into using a memory mapped file?
    – user4942583
    Nov 12 at 3:46











  • If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.
    – o11c
    Nov 12 at 4:10










  • thanks for answers guys i think i need to do another method like said wagner
    – Emre Kaya
    Nov 12 at 4:19















So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.
– S.M.
Nov 12 at 3:31





So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.
– S.M.
Nov 12 at 3:31













Have you looked into using a memory mapped file?
– user4942583
Nov 12 at 3:46





Have you looked into using a memory mapped file?
– user4942583
Nov 12 at 3:46













If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.
– o11c
Nov 12 at 4:10




If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.
– o11c
Nov 12 at 4:10












thanks for answers guys i think i need to do another method like said wagner
– Emre Kaya
Nov 12 at 4:19




thanks for answers guys i think i need to do another method like said wagner
– Emre Kaya
Nov 12 at 4:19












1 Answer
1






active

oldest

votes

















up vote
0
down vote



accepted










There is no "precise final answer to your question", once it depends on a couple variables here.



C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).



Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.



Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)



Google for it and learn a little more.



About reading binary data in C++ is easy:



std::ifstream file("somedata.obj", ios::binary);
if (file)

char buffer[1024];
file.read(buffer, sizeof(buffer));
// buffer has you binary data... you can interpret it!



But again, this is not enough. You need to wisely interpret the data you just read.






share|improve this answer




















    Your Answer






    StackExchange.ifUsing("editor", function ()
    StackExchange.using("externalEditor", function ()
    StackExchange.using("snippets", function ()
    StackExchange.snippets.init();
    );
    );
    , "code-snippets");

    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "1"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader:
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    ,
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );













    draft saved

    draft discarded


















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53255477%2fhow-can-i-read-file-fastest-way-in-c-to-send-data-to-vertex-buffer-object%23new-answer', 'question_page');

    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes








    up vote
    0
    down vote



    accepted










    There is no "precise final answer to your question", once it depends on a couple variables here.



    C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).



    Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.



    Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)



    Google for it and learn a little more.



    About reading binary data in C++ is easy:



    std::ifstream file("somedata.obj", ios::binary);
    if (file)

    char buffer[1024];
    file.read(buffer, sizeof(buffer));
    // buffer has you binary data... you can interpret it!



    But again, this is not enough. You need to wisely interpret the data you just read.






    share|improve this answer
























      up vote
      0
      down vote



      accepted










      There is no "precise final answer to your question", once it depends on a couple variables here.



      C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).



      Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.



      Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)



      Google for it and learn a little more.



      About reading binary data in C++ is easy:



      std::ifstream file("somedata.obj", ios::binary);
      if (file)

      char buffer[1024];
      file.read(buffer, sizeof(buffer));
      // buffer has you binary data... you can interpret it!



      But again, this is not enough. You need to wisely interpret the data you just read.






      share|improve this answer






















        up vote
        0
        down vote



        accepted







        up vote
        0
        down vote



        accepted






        There is no "precise final answer to your question", once it depends on a couple variables here.



        C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).



        Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.



        Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)



        Google for it and learn a little more.



        About reading binary data in C++ is easy:



        std::ifstream file("somedata.obj", ios::binary);
        if (file)

        char buffer[1024];
        file.read(buffer, sizeof(buffer));
        // buffer has you binary data... you can interpret it!



        But again, this is not enough. You need to wisely interpret the data you just read.






        share|improve this answer












        There is no "precise final answer to your question", once it depends on a couple variables here.



        C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).



        Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.



        Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)



        Google for it and learn a little more.



        About reading binary data in C++ is easy:



        std::ifstream file("somedata.obj", ios::binary);
        if (file)

        char buffer[1024];
        file.read(buffer, sizeof(buffer));
        // buffer has you binary data... you can interpret it!



        But again, this is not enough. You need to wisely interpret the data you just read.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Nov 12 at 3:50









        Wagner Patriota

        3,9681840




        3,9681840



























            draft saved

            draft discarded
















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid


            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.

            To learn more, see our tips on writing great answers.





            Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


            Please pay close attention to the following guidance:


            • Please be sure to answer the question. Provide details and share your research!

            But avoid


            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.

            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53255477%2fhow-can-i-read-file-fastest-way-in-c-to-send-data-to-vertex-buffer-object%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            這個網誌中的熱門文章

            How to read a connectionString WITH PROVIDER in .NET Core?

            Node.js Script on GitHub Pages or Amazon S3

            Museum of Modern and Contemporary Art of Trento and Rovereto