How can i listen in real-time for changes on my MongoDB?
I have a DB where i am sending some data. At the same time i'm running a Python script and i would like this script to send on my console the last entry to the MongoDB database as soon as it's added.
I've been looking for a solution to this for days without finding anything.
I made some research and found about:
a) tailable cursor, but the only problem is that my database is not capped, and since i will be putting data every 5 seconds i'm afraid that a capped database would not be enough for my needs since when the max size it's reached it will start overwriting the oldest records b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS.
Any advice?
This is what i got so far:
from pymongo import MongoClient
import pymongo
import time
import random
from pprint import pprint
#Step 1: Connect to MongoDB - Note: Change connection string as needed
client = MongoClient(port=27017)
db = client.one
mycol = client["coll"]
highest_previous_primary_key = 1
while True:
cursor = db.mycol.find()
for document in cursor:
# get the current primary key, and if it's greater than the previous one
# we print the results and increment the variable to that value
current_primary_key = document['num']
if current_primary_key > highest_previous_primary_key:
print(document['num'])
highest_previous_primary_key = current_primary_key
But the problem with this it's that it will stop printing after the 4th record, plus i don't know if it's the best solution for when my db will have a lot of data.
Any advice?
python mongodb
add a comment |
I have a DB where i am sending some data. At the same time i'm running a Python script and i would like this script to send on my console the last entry to the MongoDB database as soon as it's added.
I've been looking for a solution to this for days without finding anything.
I made some research and found about:
a) tailable cursor, but the only problem is that my database is not capped, and since i will be putting data every 5 seconds i'm afraid that a capped database would not be enough for my needs since when the max size it's reached it will start overwriting the oldest records b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS.
Any advice?
This is what i got so far:
from pymongo import MongoClient
import pymongo
import time
import random
from pprint import pprint
#Step 1: Connect to MongoDB - Note: Change connection string as needed
client = MongoClient(port=27017)
db = client.one
mycol = client["coll"]
highest_previous_primary_key = 1
while True:
cursor = db.mycol.find()
for document in cursor:
# get the current primary key, and if it's greater than the previous one
# we print the results and increment the variable to that value
current_primary_key = document['num']
if current_primary_key > highest_previous_primary_key:
print(document['num'])
highest_previous_primary_key = current_primary_key
But the problem with this it's that it will stop printing after the 4th record, plus i don't know if it's the best solution for when my db will have a lot of data.
Any advice?
python mongodb
add a comment |
I have a DB where i am sending some data. At the same time i'm running a Python script and i would like this script to send on my console the last entry to the MongoDB database as soon as it's added.
I've been looking for a solution to this for days without finding anything.
I made some research and found about:
a) tailable cursor, but the only problem is that my database is not capped, and since i will be putting data every 5 seconds i'm afraid that a capped database would not be enough for my needs since when the max size it's reached it will start overwriting the oldest records b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS.
Any advice?
This is what i got so far:
from pymongo import MongoClient
import pymongo
import time
import random
from pprint import pprint
#Step 1: Connect to MongoDB - Note: Change connection string as needed
client = MongoClient(port=27017)
db = client.one
mycol = client["coll"]
highest_previous_primary_key = 1
while True:
cursor = db.mycol.find()
for document in cursor:
# get the current primary key, and if it's greater than the previous one
# we print the results and increment the variable to that value
current_primary_key = document['num']
if current_primary_key > highest_previous_primary_key:
print(document['num'])
highest_previous_primary_key = current_primary_key
But the problem with this it's that it will stop printing after the 4th record, plus i don't know if it's the best solution for when my db will have a lot of data.
Any advice?
python mongodb
I have a DB where i am sending some data. At the same time i'm running a Python script and i would like this script to send on my console the last entry to the MongoDB database as soon as it's added.
I've been looking for a solution to this for days without finding anything.
I made some research and found about:
a) tailable cursor, but the only problem is that my database is not capped, and since i will be putting data every 5 seconds i'm afraid that a capped database would not be enough for my needs since when the max size it's reached it will start overwriting the oldest records b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS.
Any advice?
This is what i got so far:
from pymongo import MongoClient
import pymongo
import time
import random
from pprint import pprint
#Step 1: Connect to MongoDB - Note: Change connection string as needed
client = MongoClient(port=27017)
db = client.one
mycol = client["coll"]
highest_previous_primary_key = 1
while True:
cursor = db.mycol.find()
for document in cursor:
# get the current primary key, and if it's greater than the previous one
# we print the results and increment the variable to that value
current_primary_key = document['num']
if current_primary_key > highest_previous_primary_key:
print(document['num'])
highest_previous_primary_key = current_primary_key
But the problem with this it's that it will stop printing after the 4th record, plus i don't know if it's the best solution for when my db will have a lot of data.
Any advice?
python mongodb
python mongodb
asked Nov 15 '18 at 19:13
Jack022Jack022
12811
12811
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS
Replica sets provide redundancy and high availability, and are the basis for all production MongoDB deployments. Having said that, for testing and/or deployment you can deploy a replica set with only a single member. For local testing example:
mongod --dbpath /path/data/test --replSet test
Once the local test server started, connect with mongo shell to perform rs.initiate():
mongo
> rs.initiate()
See related Deploy a replica set for testing and deployment
try:
# Only catch insert operations
with client.watch(['$match': 'operationType': 'insert']) as stream:
for insert_change in stream:
print(insert_change)
except pymongo.errors.PyMongoError:
# The ChangeStream encountered an unrecoverable error or the
# resume attempt failed to recreate the cursor.
logging.error('...')
See also pymongo.mongo_client.MongoClient.watch()
Thank you! I'm looking more into replica sets.
– Jack022
Dec 3 '18 at 22:46
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53326452%2fhow-can-i-listen-in-real-time-for-changes-on-my-mongodb%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS
Replica sets provide redundancy and high availability, and are the basis for all production MongoDB deployments. Having said that, for testing and/or deployment you can deploy a replica set with only a single member. For local testing example:
mongod --dbpath /path/data/test --replSet test
Once the local test server started, connect with mongo shell to perform rs.initiate():
mongo
> rs.initiate()
See related Deploy a replica set for testing and deployment
try:
# Only catch insert operations
with client.watch(['$match': 'operationType': 'insert']) as stream:
for insert_change in stream:
print(insert_change)
except pymongo.errors.PyMongoError:
# The ChangeStream encountered an unrecoverable error or the
# resume attempt failed to recreate the cursor.
logging.error('...')
See also pymongo.mongo_client.MongoClient.watch()
Thank you! I'm looking more into replica sets.
– Jack022
Dec 3 '18 at 22:46
add a comment |
b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS
Replica sets provide redundancy and high availability, and are the basis for all production MongoDB deployments. Having said that, for testing and/or deployment you can deploy a replica set with only a single member. For local testing example:
mongod --dbpath /path/data/test --replSet test
Once the local test server started, connect with mongo shell to perform rs.initiate():
mongo
> rs.initiate()
See related Deploy a replica set for testing and deployment
try:
# Only catch insert operations
with client.watch(['$match': 'operationType': 'insert']) as stream:
for insert_change in stream:
print(insert_change)
except pymongo.errors.PyMongoError:
# The ChangeStream encountered an unrecoverable error or the
# resume attempt failed to recreate the cursor.
logging.error('...')
See also pymongo.mongo_client.MongoClient.watch()
Thank you! I'm looking more into replica sets.
– Jack022
Dec 3 '18 at 22:46
add a comment |
b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS
Replica sets provide redundancy and high availability, and are the basis for all production MongoDB deployments. Having said that, for testing and/or deployment you can deploy a replica set with only a single member. For local testing example:
mongod --dbpath /path/data/test --replSet test
Once the local test server started, connect with mongo shell to perform rs.initiate():
mongo
> rs.initiate()
See related Deploy a replica set for testing and deployment
try:
# Only catch insert operations
with client.watch(['$match': 'operationType': 'insert']) as stream:
for insert_change in stream:
print(insert_change)
except pymongo.errors.PyMongoError:
# The ChangeStream encountered an unrecoverable error or the
# resume attempt failed to recreate the cursor.
logging.error('...')
See also pymongo.mongo_client.MongoClient.watch()
b) change_streams, but my db is not a replica set, i'm fairly new to this so i still have to learn about more advanced topics like RS
Replica sets provide redundancy and high availability, and are the basis for all production MongoDB deployments. Having said that, for testing and/or deployment you can deploy a replica set with only a single member. For local testing example:
mongod --dbpath /path/data/test --replSet test
Once the local test server started, connect with mongo shell to perform rs.initiate():
mongo
> rs.initiate()
See related Deploy a replica set for testing and deployment
try:
# Only catch insert operations
with client.watch(['$match': 'operationType': 'insert']) as stream:
for insert_change in stream:
print(insert_change)
except pymongo.errors.PyMongoError:
# The ChangeStream encountered an unrecoverable error or the
# resume attempt failed to recreate the cursor.
logging.error('...')
See also pymongo.mongo_client.MongoClient.watch()
answered Dec 3 '18 at 6:23
Wan BachtiarWan Bachtiar
9,46732038
9,46732038
Thank you! I'm looking more into replica sets.
– Jack022
Dec 3 '18 at 22:46
add a comment |
Thank you! I'm looking more into replica sets.
– Jack022
Dec 3 '18 at 22:46
Thank you! I'm looking more into replica sets.
– Jack022
Dec 3 '18 at 22:46
Thank you! I'm looking more into replica sets.
– Jack022
Dec 3 '18 at 22:46
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53326452%2fhow-can-i-listen-in-real-time-for-changes-on-my-mongodb%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown