How can i listen in real-time for changes on my MongoDB?










0















I have a DB where i am sending some data. At the same time i'm running a Python script and i would like this script to send on my console the last entry to the MongoDB database as soon as it's added.



I've been looking for a solution to this for days without finding anything.



I made some research and found about:
a) tailable cursor, but the only problem is that my database is not capped, and since i will be putting data every 5 seconds i'm afraid that a capped database would not be enough for my needs since when the max size it's reached it will start overwriting the oldest records b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS.



Any advice?



This is what i got so far:



from pymongo import MongoClient
import pymongo
import time
import random
from pprint import pprint

#Step 1: Connect to MongoDB - Note: Change connection string as needed
client = MongoClient(port=27017)


db = client.one

mycol = client["coll"]


highest_previous_primary_key = 1

while True:
cursor = db.mycol.find()
for document in cursor:

# get the current primary key, and if it's greater than the previous one
# we print the results and increment the variable to that value
current_primary_key = document['num']
if current_primary_key > highest_previous_primary_key:
print(document['num'])
highest_previous_primary_key = current_primary_key


But the problem with this it's that it will stop printing after the 4th record, plus i don't know if it's the best solution for when my db will have a lot of data.



Any advice?










share|improve this question


























    0















    I have a DB where i am sending some data. At the same time i'm running a Python script and i would like this script to send on my console the last entry to the MongoDB database as soon as it's added.



    I've been looking for a solution to this for days without finding anything.



    I made some research and found about:
    a) tailable cursor, but the only problem is that my database is not capped, and since i will be putting data every 5 seconds i'm afraid that a capped database would not be enough for my needs since when the max size it's reached it will start overwriting the oldest records b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS.



    Any advice?



    This is what i got so far:



    from pymongo import MongoClient
    import pymongo
    import time
    import random
    from pprint import pprint

    #Step 1: Connect to MongoDB - Note: Change connection string as needed
    client = MongoClient(port=27017)


    db = client.one

    mycol = client["coll"]


    highest_previous_primary_key = 1

    while True:
    cursor = db.mycol.find()
    for document in cursor:

    # get the current primary key, and if it's greater than the previous one
    # we print the results and increment the variable to that value
    current_primary_key = document['num']
    if current_primary_key > highest_previous_primary_key:
    print(document['num'])
    highest_previous_primary_key = current_primary_key


    But the problem with this it's that it will stop printing after the 4th record, plus i don't know if it's the best solution for when my db will have a lot of data.



    Any advice?










    share|improve this question
























      0












      0








      0








      I have a DB where i am sending some data. At the same time i'm running a Python script and i would like this script to send on my console the last entry to the MongoDB database as soon as it's added.



      I've been looking for a solution to this for days without finding anything.



      I made some research and found about:
      a) tailable cursor, but the only problem is that my database is not capped, and since i will be putting data every 5 seconds i'm afraid that a capped database would not be enough for my needs since when the max size it's reached it will start overwriting the oldest records b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS.



      Any advice?



      This is what i got so far:



      from pymongo import MongoClient
      import pymongo
      import time
      import random
      from pprint import pprint

      #Step 1: Connect to MongoDB - Note: Change connection string as needed
      client = MongoClient(port=27017)


      db = client.one

      mycol = client["coll"]


      highest_previous_primary_key = 1

      while True:
      cursor = db.mycol.find()
      for document in cursor:

      # get the current primary key, and if it's greater than the previous one
      # we print the results and increment the variable to that value
      current_primary_key = document['num']
      if current_primary_key > highest_previous_primary_key:
      print(document['num'])
      highest_previous_primary_key = current_primary_key


      But the problem with this it's that it will stop printing after the 4th record, plus i don't know if it's the best solution for when my db will have a lot of data.



      Any advice?










      share|improve this question














      I have a DB where i am sending some data. At the same time i'm running a Python script and i would like this script to send on my console the last entry to the MongoDB database as soon as it's added.



      I've been looking for a solution to this for days without finding anything.



      I made some research and found about:
      a) tailable cursor, but the only problem is that my database is not capped, and since i will be putting data every 5 seconds i'm afraid that a capped database would not be enough for my needs since when the max size it's reached it will start overwriting the oldest records b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS.



      Any advice?



      This is what i got so far:



      from pymongo import MongoClient
      import pymongo
      import time
      import random
      from pprint import pprint

      #Step 1: Connect to MongoDB - Note: Change connection string as needed
      client = MongoClient(port=27017)


      db = client.one

      mycol = client["coll"]


      highest_previous_primary_key = 1

      while True:
      cursor = db.mycol.find()
      for document in cursor:

      # get the current primary key, and if it's greater than the previous one
      # we print the results and increment the variable to that value
      current_primary_key = document['num']
      if current_primary_key > highest_previous_primary_key:
      print(document['num'])
      highest_previous_primary_key = current_primary_key


      But the problem with this it's that it will stop printing after the 4th record, plus i don't know if it's the best solution for when my db will have a lot of data.



      Any advice?







      python mongodb






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Nov 15 '18 at 19:13









      Jack022Jack022

      12811




      12811






















          1 Answer
          1






          active

          oldest

          votes


















          1















          b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS




          Replica sets provide redundancy and high availability, and are the basis for all production MongoDB deployments. Having said that, for testing and/or deployment you can deploy a replica set with only a single member. For local testing example:



          mongod --dbpath /path/data/test --replSet test


          Once the local test server started, connect with mongo shell to perform rs.initiate():



          mongo
          > rs.initiate()


          See related Deploy a replica set for testing and deployment



          try:
          # Only catch insert operations
          with client.watch(['$match': 'operationType': 'insert']) as stream:
          for insert_change in stream:
          print(insert_change)
          except pymongo.errors.PyMongoError:
          # The ChangeStream encountered an unrecoverable error or the
          # resume attempt failed to recreate the cursor.
          logging.error('...')


          See also pymongo.mongo_client.MongoClient.watch()






          share|improve this answer























          • Thank you! I'm looking more into replica sets.

            – Jack022
            Dec 3 '18 at 22:46











          Your Answer






          StackExchange.ifUsing("editor", function ()
          StackExchange.using("externalEditor", function ()
          StackExchange.using("snippets", function ()
          StackExchange.snippets.init();
          );
          );
          , "code-snippets");

          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "1"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader:
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          ,
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );













          draft saved

          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53326452%2fhow-can-i-listen-in-real-time-for-changes-on-my-mongodb%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          1















          b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS




          Replica sets provide redundancy and high availability, and are the basis for all production MongoDB deployments. Having said that, for testing and/or deployment you can deploy a replica set with only a single member. For local testing example:



          mongod --dbpath /path/data/test --replSet test


          Once the local test server started, connect with mongo shell to perform rs.initiate():



          mongo
          > rs.initiate()


          See related Deploy a replica set for testing and deployment



          try:
          # Only catch insert operations
          with client.watch(['$match': 'operationType': 'insert']) as stream:
          for insert_change in stream:
          print(insert_change)
          except pymongo.errors.PyMongoError:
          # The ChangeStream encountered an unrecoverable error or the
          # resume attempt failed to recreate the cursor.
          logging.error('...')


          See also pymongo.mongo_client.MongoClient.watch()






          share|improve this answer























          • Thank you! I'm looking more into replica sets.

            – Jack022
            Dec 3 '18 at 22:46















          1















          b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS




          Replica sets provide redundancy and high availability, and are the basis for all production MongoDB deployments. Having said that, for testing and/or deployment you can deploy a replica set with only a single member. For local testing example:



          mongod --dbpath /path/data/test --replSet test


          Once the local test server started, connect with mongo shell to perform rs.initiate():



          mongo
          > rs.initiate()


          See related Deploy a replica set for testing and deployment



          try:
          # Only catch insert operations
          with client.watch(['$match': 'operationType': 'insert']) as stream:
          for insert_change in stream:
          print(insert_change)
          except pymongo.errors.PyMongoError:
          # The ChangeStream encountered an unrecoverable error or the
          # resume attempt failed to recreate the cursor.
          logging.error('...')


          See also pymongo.mongo_client.MongoClient.watch()






          share|improve this answer























          • Thank you! I'm looking more into replica sets.

            – Jack022
            Dec 3 '18 at 22:46













          1












          1








          1








          b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS




          Replica sets provide redundancy and high availability, and are the basis for all production MongoDB deployments. Having said that, for testing and/or deployment you can deploy a replica set with only a single member. For local testing example:



          mongod --dbpath /path/data/test --replSet test


          Once the local test server started, connect with mongo shell to perform rs.initiate():



          mongo
          > rs.initiate()


          See related Deploy a replica set for testing and deployment



          try:
          # Only catch insert operations
          with client.watch(['$match': 'operationType': 'insert']) as stream:
          for insert_change in stream:
          print(insert_change)
          except pymongo.errors.PyMongoError:
          # The ChangeStream encountered an unrecoverable error or the
          # resume attempt failed to recreate the cursor.
          logging.error('...')


          See also pymongo.mongo_client.MongoClient.watch()






          share|improve this answer














          b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS




          Replica sets provide redundancy and high availability, and are the basis for all production MongoDB deployments. Having said that, for testing and/or deployment you can deploy a replica set with only a single member. For local testing example:



          mongod --dbpath /path/data/test --replSet test


          Once the local test server started, connect with mongo shell to perform rs.initiate():



          mongo
          > rs.initiate()


          See related Deploy a replica set for testing and deployment



          try:
          # Only catch insert operations
          with client.watch(['$match': 'operationType': 'insert']) as stream:
          for insert_change in stream:
          print(insert_change)
          except pymongo.errors.PyMongoError:
          # The ChangeStream encountered an unrecoverable error or the
          # resume attempt failed to recreate the cursor.
          logging.error('...')


          See also pymongo.mongo_client.MongoClient.watch()







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Dec 3 '18 at 6:23









          Wan BachtiarWan Bachtiar

          9,46732038




          9,46732038












          • Thank you! I'm looking more into replica sets.

            – Jack022
            Dec 3 '18 at 22:46

















          • Thank you! I'm looking more into replica sets.

            – Jack022
            Dec 3 '18 at 22:46
















          Thank you! I'm looking more into replica sets.

          – Jack022
          Dec 3 '18 at 22:46





          Thank you! I'm looking more into replica sets.

          – Jack022
          Dec 3 '18 at 22:46



















          draft saved

          draft discarded
















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid


          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.

          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53326452%2fhow-can-i-listen-in-real-time-for-changes-on-my-mongodb%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          這個網誌中的熱門文章

          How to read a connectionString WITH PROVIDER in .NET Core?

          Node.js Script on GitHub Pages or Amazon S3

          Museum of Modern and Contemporary Art of Trento and Rovereto