Cassandra - Heavy reads and moderate updates with large data in one column - Performance issue
up vote
2
down vote
favorite
I have initial data which when loaded in Cassandra, the total size shown is 3.5GB
After that I do heavy read like 10k reads per second and moderate update like 1k updates per second but in update one of the column size is updated to larger size such that the total size of the row increases from 1k to almost 5k
The storage thing is that size of the Node keeps increasing non stop to even going up to 300GB and only heading up
Compaction is running, read performances keeps on degrading nonstop. Is Cassandra having trouble with updates of row to 5k from 2k ?
This is a AWS 30GB RAM machine with 16 processors and 1TB SSD with 32k IOPS activated. Compaction throughput set at 256 and compactors at 8, still the deviation of reads continue and size of SS Tables increasing
Reads per second degrades from 22k per second to as bad as 5k per second over a period of 1 day
Please let me know what is wrong in the configuration
cassandra datastax
asked Nov 11 at 12:08
Tahseen
315220
|
show 3 more comments
up vote
2
down vote
favorite
I have initial data which when loaded in Cassandra, the total size shown is 3.5GB
After that I do heavy read like 10k reads per second and moderate update like 1k updates per second but in update one of the column size is updated to larger size such that the total size of the row increases from 1k to almost 5k
The storage thing is that size of the Node keeps increasing non stop to even going up to 300GB and only heading up
Compaction is running, read performances keeps on degrading nonstop. Is Cassandra having trouble with updates of row to 5k from 2k ?
This is a AWS 30GB RAM machine with 16 processors and 1TB SSD with 32k IOPS activated. Compaction throughput set at 256 and compactors at 8, still the deviation of reads continue and size of SS Tables increasing
Reads per second degrades from 22k per second to as bad as 5k per second over a period of 1 day
Please let me know what is wrong in the configuration
cassandra datastax
asked Nov 11 at 12:08
Tahseen
315220
On the update are you updating the same row repeatedly? Are you updating the entire row or just a column? Can you provide your schema and queries here?
– bechbd
Nov 11 at 13:57
There are 30 million rows first pre populated and then sequentially updating them one by one from initial size of 2k to 5k. All values randomised and especially that one column which is most sized compared to all columns in the row
– Tahseen
Nov 11 at 14:01
can you include your existing schema (including compaction settings) and tablestats output?
– Chris Lohfink
Nov 11 at 21:18
Which compaction strategy are you using? Can you post the output of "nodetool compactionstats" and also ssh to the node and check how many SSTable files are present in the data directory? Also, which version of Cassandra are you using?
– Justin Cameron
Nov 12 at 0:31
Levelled Compaction pending tasks: 582 - t.cc: 299 - t.ss: 283
– Tahseen
Nov 12 at 12:13
|
show 3 more comments
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I have initial data which when loaded in Cassandra, the total size shown is 3.5GB
After that I do heavy read like 10k reads per second and moderate update like 1k updates per second but in update one of the column size is updated to larger size such that the total size of the row increases from 1k to almost 5k
The storage thing is that size of the Node keeps increasing non stop to even going up to 300GB and only heading up
Compaction is running, read performances keeps on degrading nonstop. Is Cassandra having trouble with updates of row to 5k from 2k ?
This is a AWS 30GB RAM machine with 16 processors and 1TB SSD with 32k IOPS activated. Compaction throughput set at 256 and compactors at 8, still the deviation of reads continue and size of SS Tables increasing
Reads per second degrades from 22k per second to as bad as 5k per second over a period of 1 day
Please let me know what is wrong in the configuration
cassandra datastax
asked Nov 11 at 12:08
Tahseen
315220
I have initial data which when loaded in Cassandra, the total size shown is 3.5GB
After that I do heavy read like 10k reads per second and moderate update like 1k updates per second but in update one of the column size is updated to larger size such that the total size of the row increases from 1k to almost 5k
The storage thing is that size of the Node keeps increasing non stop to even going up to 300GB and only heading up
Compaction is running, read performances keeps on degrading nonstop. Is Cassandra having trouble with updates of row to 5k from 2k ?
This is a AWS 30GB RAM machine with 16 processors and 1TB SSD with 32k IOPS activated. Compaction throughput set at 256 and compactors at 8, still the deviation of reads continue and size of SS Tables increasing
Reads per second degrades from 22k per second to as bad as 5k per second over a period of 1 day
Please let me know what is wrong in the configuration
cassandra datastax
cassandra datastax
asked Nov 11 at 12:08
Tahseen
315220
asked Nov 11 at 12:08
Tahseen
315220
asked Nov 11 at 12:08
Tahseen
315220
asked Nov 11 at 12:08
Tahseen
315220
asked Nov 11 at 12:08
Tahseen
315220
315220
On the update are you updating the same row repeatedly? Are you updating the entire row or just a column? Can you provide your schema and queries here?
– bechbd
Nov 11 at 13:57
There are 30 million rows first pre populated and then sequentially updating them one by one from initial size of 2k to 5k. All values randomised and especially that one column which is most sized compared to all columns in the row
– Tahseen
Nov 11 at 14:01
can you include your existing schema (including compaction settings) and tablestats output?
– Chris Lohfink
Nov 11 at 21:18
Which compaction strategy are you using? Can you post the output of "nodetool compactionstats" and also ssh to the node and check how many SSTable files are present in the data directory? Also, which version of Cassandra are you using?
– Justin Cameron
Nov 12 at 0:31
Levelled Compaction pending tasks: 582 - t.cc: 299 - t.ss: 283
– Tahseen
Nov 12 at 12:13
|
show 3 more comments
On the update are you updating the same row repeatedly? Are you updating the entire row or just a column? Can you provide your schema and queries here?
– bechbd
Nov 11 at 13:57
There are 30 million rows first pre populated and then sequentially updating them one by one from initial size of 2k to 5k. All values randomised and especially that one column which is most sized compared to all columns in the row
– Tahseen
Nov 11 at 14:01
can you include your existing schema (including compaction settings) and tablestats output?
– Chris Lohfink
Nov 11 at 21:18
Which compaction strategy are you using? Can you post the output of "nodetool compactionstats" and also ssh to the node and check how many SSTable files are present in the data directory? Also, which version of Cassandra are you using?
– Justin Cameron
Nov 12 at 0:31
Levelled Compaction pending tasks: 582 - t.cc: 299 - t.ss: 283
– Tahseen
Nov 12 at 12:13
On the update are you updating the same row repeatedly? Are you updating the entire row or just a column? Can you provide your schema and queries here?
– bechbd
Nov 11 at 13:57
On the update are you updating the same row repeatedly? Are you updating the entire row or just a column? Can you provide your schema and queries here?
– bechbd
Nov 11 at 13:57
There are 30 million rows first pre populated and then sequentially updating them one by one from initial size of 2k to 5k. All values randomised and especially that one column which is most sized compared to all columns in the row
– Tahseen
Nov 11 at 14:01
There are 30 million rows first pre populated and then sequentially updating them one by one from initial size of 2k to 5k. All values randomised and especially that one column which is most sized compared to all columns in the row
– Tahseen
Nov 11 at 14:01
can you include your existing schema (including compaction settings) and tablestats output?
– Chris Lohfink
Nov 11 at 21:18
can you include your existing schema (including compaction settings) and tablestats output?
– Chris Lohfink
Nov 11 at 21:18
Which compaction strategy are you using? Can you post the output of "nodetool compactionstats" and also ssh to the node and check how many SSTable files are present in the data directory? Also, which version of Cassandra are you using?
– Justin Cameron
Nov 12 at 0:31
Which compaction strategy are you using? Can you post the output of "nodetool compactionstats" and also ssh to the node and check how many SSTable files are present in the data directory? Also, which version of Cassandra are you using?
– Justin Cameron
Nov 12 at 0:31
Levelled Compaction pending tasks: 582 - t.cc: 299 - t.ss: 283
– Tahseen
Nov 12 at 12:13
Levelled Compaction pending tasks: 582 - t.cc: 299 - t.ss: 283
– Tahseen
Nov 12 at 12:13
|
show 3 more comments
1 Answer
1
active
oldest
votes
up vote
0
down vote
Basically observe pending compactions and if they increase first start increasing the compaction throughput using nodetool.
If you see that the pending compactions has started reducing then increase it a bit more from that configuration as a buffer.
Also you can increase the concurrent compactors also by small amount if you usee CPU is not overloaded during operations + compaction
answered Nov 18 at 19:11
Tahseen
315220
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
Basically observe pending compactions and if they increase first start increasing the compaction throughput using nodetool.
If you see that the pending compactions has started reducing then increase it a bit more from that configuration as a buffer.
Also you can increase the concurrent compactors also by small amount if you usee CPU is not overloaded during operations + compaction
answered Nov 18 at 19:11
Tahseen
315220
add a comment |
up vote
0
down vote
Basically observe pending compactions and if they increase first start increasing the compaction throughput using nodetool.
If you see that the pending compactions has started reducing then increase it a bit more from that configuration as a buffer.
Also you can increase the concurrent compactors also by small amount if you usee CPU is not overloaded during operations + compaction
answered Nov 18 at 19:11
Tahseen
315220
add a comment |
up vote
0
down vote
up vote
0
down vote
Basically observe pending compactions and if they increase first start increasing the compaction throughput using nodetool.
If you see that the pending compactions has started reducing then increase it a bit more from that configuration as a buffer.
Also you can increase the concurrent compactors also by small amount if you usee CPU is not overloaded during operations + compaction
answered Nov 18 at 19:11
Tahseen
315220
Basically observe pending compactions and if they increase first start increasing the compaction throughput using nodetool.
If you see that the pending compactions has started reducing then increase it a bit more from that configuration as a buffer.
Also you can increase the concurrent compactors also by small amount if you usee CPU is not overloaded during operations + compaction
answered Nov 18 at 19:11
Tahseen
315220
answered Nov 18 at 19:11
Tahseen
315220
answered Nov 18 at 19:11
Tahseen
315220
answered Nov 18 at 19:11
Tahseen
315220
315220
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53248581%2fcassandra-heavy-reads-and-moderate-updates-with-large-data-in-one-column-per%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
On the update are you updating the same row repeatedly? Are you updating the entire row or just a column? Can you provide your schema and queries here?
– bechbd
Nov 11 at 13:57
There are 30 million rows first pre populated and then sequentially updating them one by one from initial size of 2k to 5k. All values randomised and especially that one column which is most sized compared to all columns in the row
– Tahseen
Nov 11 at 14:01
can you include your existing schema (including compaction settings) and tablestats output?
– Chris Lohfink
Nov 11 at 21:18
Which compaction strategy are you using? Can you post the output of "nodetool compactionstats" and also ssh to the node and check how many SSTable files are present in the data directory? Also, which version of Cassandra are you using?
– Justin Cameron
Nov 12 at 0:31
Levelled Compaction pending tasks: 582 - t.cc: 299 - t.ss: 283
– Tahseen
Nov 12 at 12:13