Displaying #cassandra-dev/2017-02-27.log:

Mon Feb 27 00:27:22 2017  Vijay:Joined the channel
Mon Feb 27 00:31:58 2017  tolbertam:Joined the channel
Mon Feb 27 00:36:18 2017  EdwardCapriolo:Joined the channel
Mon Feb 27 00:36:27 2017  adamholmberg:Joined the channel
Mon Feb 27 00:53:23 2017  mstepura:Joined the channel
Mon Feb 27 00:56:10 2017  alourie:Joined the channel
Mon Feb 27 01:02:36 2017  tolbertam:Joined the channel
Mon Feb 27 01:25:56 2017  adamholmberg:Joined the channel
Mon Feb 27 01:38:33 2017  Vijay:Joined the channel
Mon Feb 27 01:54:37 2017  tolbertam:Joined the channel
Mon Feb 27 02:15:51 2017  Vijay:Joined the channel
Mon Feb 27 02:16:25 2017  adamholmberg:Joined the channel
Mon Feb 27 02:19:12 2017  tolbertam:Joined the channel
Mon Feb 27 02:34:04 2017  adamholmberg:Joined the channel
Mon Feb 27 02:44:48 2017  tolbertam:Joined the channel
Mon Feb 27 02:45:23 2017  mstepura:Joined the channel
Mon Feb 27 02:47:03 2017  mstepura:Joined the channel
Mon Feb 27 02:48:48 2017  mstepura:Joined the channel
Mon Feb 27 02:48:48 2017  kohlisankalp:Joined the channel
Mon Feb 27 02:49:35 2017  mstepura:Joined the channel
Mon Feb 27 02:57:16 2017  tolbertam:Joined the channel
Mon Feb 27 03:12:56 2017  mstepura:Joined the channel
Mon Feb 27 03:22:01 2017  tolbertam:Joined the channel
Mon Feb 27 03:43:31 2017  dikang:Joined the channel
Mon Feb 27 03:45:06 2017  kohlisankalp:Joined the channel
Mon Feb 27 03:45:54 2017  kohlisankalp:Joined the channel
Mon Feb 27 04:00:30 2017  mstepura:Joined the channel
Mon Feb 27 04:00:48 2017  Vijay:Joined the channel
Mon Feb 27 04:08:01 2017  alourie:Joined the channel
Mon Feb 27 04:32:48 2017  adamholmberg:Joined the channel
Mon Feb 27 05:01:29 2017  mstepura:Joined the channel
Mon Feb 27 05:05:07 2017  adamholmberg:Joined the channel
Mon Feb 27 05:17:47 2017  mstepura:Joined the channel
Mon Feb 27 05:32:53 2017  mstepura:Joined the channel
Mon Feb 27 05:36:16 2017  dikang:Joined the channel
Mon Feb 27 05:48:29 2017  r005t3r:Joined the channel
Mon Feb 27 05:57:04 2017  tolbertam:Joined the channel
Mon Feb 27 06:20:44 2017  Vijay:Joined the channel
Mon Feb 27 06:32:59 2017  dikang:Joined the channel
Mon Feb 27 06:35:42 2017  adamholmberg:Joined the channel
Mon Feb 27 07:05:47 2017  adamholmberg:Joined the channel
Mon Feb 27 07:06:54 2017  clohfink:Joined the channel
Mon Feb 27 07:24:41 2017  mstepura:Joined the channel
Mon Feb 27 07:32:28 2017  makrusak:Joined the channel
Mon Feb 27 07:56:10 2017  kvaster:Joined the channel
Mon Feb 27 07:58:20 2017  mstepura:Joined the channel
Mon Feb 27 07:59:20 2017  mstepura:Joined the channel
Mon Feb 27 08:02:31 2017  mstepura:Joined the channel
Mon Feb 27 08:03:21 2017  mstepura:Joined the channel
Mon Feb 27 08:04:03 2017  mstepura:Joined the channel
Mon Feb 27 08:05:41 2017  mstepura:Joined the channel
Mon Feb 27 08:06:31 2017  mstepura:Joined the channel
Mon Feb 27 08:07:17 2017  mstepura:Joined the channel
Mon Feb 27 08:08:07 2017  mstepura:Joined the channel
Mon Feb 27 08:08:57 2017  mstepura:Joined the channel
Mon Feb 27 08:09:42 2017  mstepura:Joined the channel
Mon Feb 27 08:11:23 2017  mstepura:Joined the channel
Mon Feb 27 08:12:08 2017  mstepura:Joined the channel
Mon Feb 27 08:12:58 2017  mstepura:Joined the channel
Mon Feb 27 08:13:43 2017  mstepura:Joined the channel
Mon Feb 27 08:14:28 2017  mstepura:Joined the channel
Mon Feb 27 08:15:18 2017  mstepura:Joined the channel
Mon Feb 27 08:16:09 2017  mstepura:Joined the channel
Mon Feb 27 08:22:07 2017  mstepura:Joined the channel
Mon Feb 27 08:23:17 2017  mstepura:Joined the channel
Mon Feb 27 08:24:42 2017  mstepura:Joined the channel
Mon Feb 27 08:24:53 2017  r005t3r:Joined the channel
Mon Feb 27 09:03:54 2017  NoobNode:Joined the channel
Mon Feb 27 09:11:06 2017  stephenc:Joined the channel
Mon Feb 27 10:32:18 2017  clohfink:Joined the channel
Mon Feb 27 11:06:15 2017  r005t3r:Joined the channel
Mon Feb 27 11:19:22 2017  NoobNode:Left the channel
Mon Feb 27 11:49:41 2017  r005t3r:Joined the channel
Mon Feb 27 12:33:31 2017  clohfink:Joined the channel
Mon Feb 27 13:05:45 2017  adamholmberg:Joined the channel
Mon Feb 27 13:09:16 2017  alourie:Joined the channel
Mon Feb 27 13:55:55 2017  EdwardCapriolo:Joined the channel
Mon Feb 27 14:05:47 2017  adamholmberg:Joined the channel
Mon Feb 27 14:34:08 2017  clohfink:Joined the channel
Mon Feb 27 14:45:42 2017  clohfink:Joined the channel
Mon Feb 27 14:49:56 2017  adamholmberg:Joined the channel
Mon Feb 27 14:56:12 2017  tolbertam:Joined the channel
Mon Feb 27 15:00:19 2017  EdwardCapriolo:Joined the channel
Mon Feb 27 15:30:35 2017  ostefano:Upgraded to 3.0.11 and I have the feeling (i.e., graphite charts over the last 2 days) that the cluster might be creating more hints than usual (usually 0 but when repairing)
Mon Feb 27 15:30:52 2017  jmckenzie:Joined the channel
Mon Feb 27 15:31:13 2017  ostefano:Any known performance regression which might have been introduced lately?
Mon Feb 27 15:31:28 2017  ostefano:must say these stats are really non-scientific, as in "I just have 2 days of data"
Mon Feb 27 15:32:02 2017  ostefano:the application load seems to be same but I'll have the final numbers in a few days
Mon Feb 27 15:33:32 2017  snazy:Joined the channel
Mon Feb 27 15:48:58 2017  tolbertam:Joined the channel
Mon Feb 27 15:57:18 2017  jmckenzie:Joined the channel
Mon Feb 27 15:57:55 2017  nickmbailey:Joined the channel
Mon Feb 27 15:58:46 2017  jeffj:ostefano: upgraded from what to 3.0.11?
Mon Feb 27 15:59:11 2017  jeffj:ostefano: the otc coalescing change in 2.2+ was likely a regression for most people, but without knowing version, hard to say
Mon Feb 27 15:59:12 2017  ostefano:jeffj 3.0.10
Mon Feb 27 15:59:22 2017  ostefano:should be a really small diff, so that's why I am blaming my data
Mon Feb 27 15:59:30 2017  ostefano:yes, backported that setting a couple of upgrades ago
Mon Feb 27 15:59:44 2017  ostefano:so, no change on that side
Mon Feb 27 16:01:13 2017  ostefano:(otc_coalescing_strategy: DISABLED)
Mon Feb 27 16:33:05 2017  r005t3r:Joined the channel
Mon Feb 27 16:43:45 2017  clohfink:Joined the channel
Mon Feb 27 16:45:10 2017  r005t3r:Joined the channel
Mon Feb 27 17:04:45 2017  kohlisankalp:Joined the channel
Mon Feb 27 17:07:33 2017  Vijay:Joined the channel
Mon Feb 27 17:16:18 2017  r005t3r:Joined the channel
Mon Feb 27 17:18:13 2017  JayZhuang:Joined the channel
Mon Feb 27 17:31:23 2017  mstepura:Joined the channel
Mon Feb 27 17:40:00 2017  kohlisankalp:Joined the channel
Mon Feb 27 17:49:07 2017  kvaster:Joined the channel
Mon Feb 27 17:49:12 2017  tolbertam:Joined the channel
Mon Feb 27 17:50:32 2017  kohlisankalp:Joined the channel
Mon Feb 27 17:51:38 2017  dikang:Joined the channel
Mon Feb 27 18:03:21 2017  JayZhuang:Joined the channel
Mon Feb 27 18:19:29 2017  jmckenzie:Joined the channel
Mon Feb 27 18:28:30 2017  cassci:Joined the channel
Mon Feb 27 18:34:35 2017  mikeromeo:Joined the channel
Mon Feb 27 18:36:19 2017  kvaster:Joined the channel
Mon Feb 27 18:36:36 2017  dikang:Joined the channel
Mon Feb 27 18:40:28 2017  JayZhuang:Joined the channel
Mon Feb 27 18:55:40 2017  jmckenzie:Joined the channel
Mon Feb 27 18:58:10 2017  jmckenzie:Joined the channel
Mon Feb 27 19:02:40 2017  kohlisankalp:Joined the channel
Mon Feb 27 19:02:40 2017  JayZhuang:Joined the channel
Mon Feb 27 19:04:52 2017  dikang:Joined the channel
Mon Feb 27 19:07:57 2017  jmckenzie:Joined the channel
Mon Feb 27 19:20:07 2017  kvaster:Joined the channel
Mon Feb 27 19:29:29 2017  Vijay:Joined the channel
Mon Feb 27 19:49:22 2017  tolbertam:Joined the channel
Mon Feb 27 19:56:33 2017  JayZhuang:Joined the channel
Mon Feb 27 20:06:48 2017  mstepura:Joined the channel
Mon Feb 27 20:14:44 2017  dikang:Joined the channel
Mon Feb 27 20:36:06 2017  Vijay:Joined the channel
Mon Feb 27 20:41:53 2017  bbromhead:Joined the channel
Mon Feb 27 20:44:10 2017  bbromhead:Patch for https://issues.apache.org/jira/browse/CASSANDRA-13241 is available
Mon Feb 27 20:44:50 2017  bbromhead:Broader question, any reason why a default chunk_length_in_kb of ~64kb was chosen?
Mon Feb 27 20:47:22 2017  jmckenzie:Joined the channel
Mon Feb 27 20:52:51 2017  jmckenzie:Joined the channel
Mon Feb 27 20:54:33 2017  kohlisankalp:Joined the channel
Mon Feb 27 20:57:04 2017  jmckenzie_:Joined the channel
Mon Feb 27 20:57:31 2017  jeffj:probably for compression ratio efficiency
Mon Feb 27 20:57:42 2017  jeffj:though tbh 4k chunks suck in prod, too, at least with dense nodes.
Mon Feb 27 20:58:02 2017  aweisberg:Did anyone ever test the spectrum to see what the tradeoff actually was or did it get picked out of a hat?
Mon Feb 27 20:58:55 2017  kohlisankalp:Joined the channel
Mon Feb 27 21:04:59 2017  jeffj:i think 64k was picked out of a hat
Mon Feb 27 21:05:05 2017  jeffj:we tested a bunch at $last_employer, settled on 16k
Mon Feb 27 21:05:24 2017  jeffj:for our data set, 16k gave us approximately the same compression ratio as 64k
Mon Feb 27 21:05:46 2017  driftx:64k is magic.
Mon Feb 27 21:05:47 2017  driftx:2fdc5ef5 (Brandon Williams 2011-02-01 16:24:56 +0000 100) private static final int LARGE_NUMBER = 65536; // 64k nodes ought to be enough for anybody.
Mon Feb 27 21:06:34 2017  jeffj:so three things to trade - compression ratio, disk IO over-read, and offheap compressioninfo metadata use.
Mon Feb 27 21:12:44 2017  EdwardCapriolo:When testing with old versions: I found the magic was 8KB. (too large lots of memory pressure)
Mon Feb 27 21:18:09 2017  jeffj:EdwardCapriolo: was that pre- #10680 (2.1.10'ish) ? that was the pain we found with large dense nodes and small chunk size (~4T of data and ~16k chunks would just destroy an 8gb heap)
Mon Feb 27 21:18:10 2017  CassBotJr:https://issues.apache.org/jira/browse/CASSANDRA-10680 (Resolved; Fixed; 2.1.12, 2.2.4, 3.0.1, 3.1): "Deal with small compression chunk size better during streaming plan setup"
Mon Feb 27 21:19:41 2017  EdwardCapriolo:I do not remember the version. It was a long time ago. However for my data i found the same compression ration for 4K vs 64K so the large blocks did not buy me anything.
Mon Feb 27 21:22:12 2017  EdwardCapriolo:http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/cassandra_compression_is_like_getting
Mon Feb 27 21:22:24 2017  EdwardCapriolo:It broke the internet back then: http://highscalability.com/blog/2012/5/30/strategy-get-servers-for-free-and-make-users-happy-by-turnin.html
Mon Feb 27 21:22:32 2017  aweisberg:I don't have the structure of compression metadata on the top of my head, but wouldn't that be trivial to put in off heap memory? Are the records variable length?
Mon Feb 27 21:23:58 2017  EdwardCapriolo:In practice this is super easy to test. update metadata and nodetool cleanup or something should force the issue
Mon Feb 27 21:27:39 2017  EdwardCapriolo:Back in the day it was a different world because it was not a default option. I am correct in saying it is now on by default right?
Mon Feb 27 21:34:33 2017  jeffj:aweisberg: it is offheap.
Mon Feb 27 21:34:55 2017  jeffj:aweisberg: can still be 8-10gb for very dense nodes
Mon Feb 27 21:35:19 2017  jeffj:(with 16k blocks, 4k blocks would be much higher, of course)
Mon Feb 27 21:37:56 2017  driftx:never forget, #10680
Mon Feb 27 21:37:57 2017  CassBotJr:https://issues.apache.org/jira/browse/CASSANDRA-10680 (Resolved; Fixed; 2.1.12, 2.2.4, 3.0.1, 3.1): "Deal with small compression chunk size better during streaming plan setup"
Mon Feb 27 21:46:32 2017  bbromhead:our experience has generally shown that prioritising reducing disk IO (number of reads, size of reads) generally gives the best bang for buck result in end performance (op/s, latency)
Mon Feb 27 21:46:55 2017  bbromhead:we don't run nodes over 2TB at all, so we've never hit the heap pressure issues mentioned above
Mon Feb 27 21:48:17 2017  aweisberg:jeffj: Huh... so this just says we need to store it in-memory more efficiently :-) That should be trivial since values or are going to share most high order bits.
Mon Feb 27 21:48:28 2017  bbromhead:on AWS environments with EBS, making chunk_size as small as possible without being smaller than the read was often the best way to drive max op/s and get the best out of our EBS IOPs
Mon Feb 27 21:49:01 2017  bbromhead:but yeah 8-10GB of compression metadata does not sound awesome
Mon Feb 27 21:49:35 2017  tolbertam:Joined the channel
Mon Feb 27 21:50:20 2017  jeffj:bbromhead: back when i was running clusters in aws+ebs, we used to think 4t was the largest volume we were comfortable with based on the time-to-stream on bootstrap/replace
Mon Feb 27 21:50:48 2017  jeffj:bbromhead: but with 4t volumes, you end up running out of instance options if you use 4k chunks
Mon Feb 27 21:50:59 2017  jeffj:you wont fit compression metadata + bloom filter both in an m4.2xl
Mon Feb 27 21:51:27 2017  jeffj:(at least in my experience, which is anecdotal / not scientific)
Mon Feb 27 21:52:20 2017  aweisberg:jeffj: Huh. It's tempting to just fix the compression offset space issue. It's not persisted so it's super LHF to do a brain dead encoding scheme. Like store a base offset every N entries and then use a smaller relative offset for the subsequent N values. Or a proper integer encoding. Maybe align it on a cache line.
Mon Feb 27 21:52:43 2017  jeffj:jira it up, i'm sure someone will love you for it. :)
Mon Feb 27 21:55:20 2017  dikang:Joined the channel
Mon Feb 27 21:56:18 2017  Vijay:Joined the channel
Mon Feb 27 21:56:38 2017  zznate:(R4.4xl ftw!)
Mon Feb 27 21:58:15 2017  jeffj:didnt exist when i needed it friend.
Mon Feb 27 21:59:25 2017  driftx:heh
Mon Feb 27 22:02:25 2017  bbromhead:dem i3s tho
Mon Feb 27 22:02:43 2017  bbromhead:non ebs...but still drool
Mon Feb 27 22:03:13 2017  kohlisankalp:Joined the channel
Mon Feb 27 22:43:50 2017  alourie:Joined the channel
Mon Feb 27 22:53:19 2017  bbromhead:Joined the channel
Mon Feb 27 22:57:42 2017  kohlisan_:Joined the channel
Mon Feb 27 23:08:09 2017  jmckenzie:Joined the channel
Mon Feb 27 23:10:42 2017  clohfink:Joined the channel
Mon Feb 27 23:12:47 2017  dikang:Joined the channel
Mon Feb 27 23:24:04 2017  cassci:Joined the channel
Mon Feb 27 23:24:08 2017  Vijay:Joined the channel
Mon Feb 27 23:28:08 2017  jshook:Joined the channel
Mon Feb 27 23:28:14 2017  bbromhead:Joined the channel
Mon Feb 27 23:35:14 2017  JayZhuang:Joined the channel
Mon Feb 27 23:44:11 2017  kohlisankalp:Joined the channel
Mon Feb 27 23:48:45 2017  tolbertam:Joined the channel

Comments