Displaying #cassandra-dev/2016-12-05.log:

Mon Dec 5 00:10:17 2016  clohfink:Joined the channel
Mon Dec 5 00:22:04 2016  clohfink:Joined the channel
Mon Dec 5 01:05:51 2016  clohfink:Joined the channel
Mon Dec 5 01:10:21 2016  dikang:Joined the channel
Mon Dec 5 02:17:59 2016  clohfink:Joined the channel
Mon Dec 5 02:18:38 2016  clohfink_:Joined the channel
Mon Dec 5 03:06:03 2016  kvaster_:Joined the channel
Mon Dec 5 03:17:31 2016  dikang:Joined the channel
Mon Dec 5 07:34:22 2016  kohlisankalp:Joined the channel
Mon Dec 5 07:44:41 2016  clohfink:Joined the channel
Mon Dec 5 08:11:43 2016  spodkowinski:Joined the channel
Mon Dec 5 08:36:42 2016  JoeyI_:Joined the channel
Mon Dec 5 08:41:52 2016  zhaiyuyong:Joined the channel
Mon Dec 5 08:42:11 2016  zhaiyuyong:https://issues.apache.org/jira/browse/CASSANDRA-12992
Mon Dec 5 08:42:22 2016  zhaiyuyong:when mapreduce create sstables and load to cassandra cluster,then drop the table there are much data file not moved to snapshot
Mon Dec 5 08:42:54 2016  zhaiyuyong:i must manual delete the files
Mon Dec 5 08:43:04 2016  zhaiyuyong:it is a bug?
Mon Dec 5 09:03:41 2016  ASFBot:Joined the channel
Mon Dec 5 09:28:41 2016  minimarcel:Joined the channel
Mon Dec 5 09:33:15 2016  clohfink:Joined the channel
Mon Dec 5 10:18:25 2016  spodkowinski:could someone update the 2.x+3.0 EOL dates on the download page please?
Mon Dec 5 11:21:23 2016  clohfink:Joined the channel
Mon Dec 5 11:24:20 2016  ostefano:Joined the channel
Mon Dec 5 11:36:26 2016  beobal:could somebody please add jira user iksaif to the contributors group?
Mon Dec 5 11:38:04 2016  marcuse:beobal: done
Mon Dec 5 11:38:46 2016  beobal:marcuse: thx
Mon Dec 5 11:53:42 2016  pcmanus:ptnapoleon: for info, I'm struggling a bit with this custom dtest branch: http://cassci.datastax.com/job/pcmanus-11115-dtest/6/console.
Mon Dec 5 11:54:19 2016  pcmanus:It seems to be complaining about a --with-flaky option not existing, but I'm not sure what I did wrong to get that.
Mon Dec 5 12:10:14 2016  mikewallace:Joined the channel
Mon Dec 5 12:41:53 2016  mikewallace:Joined the channel
Mon Dec 5 13:09:31 2016  clohfink:Joined the channel
Mon Dec 5 13:19:05 2016  chbatey:Joined the channel
Mon Dec 5 13:27:24 2016  mikewallace:Joined the channel
Mon Dec 5 13:59:52 2016  jfarrell:Joined the channel
Mon Dec 5 14:27:58 2016  adamholmberg:Joined the channel
Mon Dec 5 14:36:39 2016  exlt:spodkowinski: https://lists.apache.org/thread.html/fd9f9d4ab1fc7f498c2d57b8a8d77dc5f85e7e39881554960eccbfa8@%3Ccommits.cassandra.apache.org%3E
Mon Dec 5 14:37:07 2016  exlt:s/November 2016/4.0 release (date TBD)/ in summary
Mon Dec 5 14:48:06 2016  spodkowinski:exlt: perfect, thanks alot! :)
Mon Dec 5 14:51:27 2016  ptnapoleon:pcmanus: looking now
Mon Dec 5 14:55:14 2016  ptnapoleon:pcmanus: It was my fault. I told you the requirements.txt line needed to look like
Mon Dec 5 14:55:14 2016  ptnapoleon:-e git+https://github.com/pcmanus/ccm.git@thrift_removal
Mon Dec 5 14:55:16 2016  ptnapoleon:which you used
Mon Dec 5 14:55:22 2016  ptnapoleon:but apparently the syntax the pip version on cassci uses wants
Mon Dec 5 14:55:30 2016  ptnapoleon:-e git+https://github.com/pcmanus/ccm.git@thrift_removal#egg=ccm
Mon Dec 5 14:57:04 2016  vjarnot:Joined the channel
Mon Dec 5 14:57:23 2016  ptnapoleon:Note, the only difference is the #egg=ccm at the end, there
Mon Dec 5 14:57:39 2016  clohfink:Joined the channel
Mon Dec 5 15:49:13 2016  exlt:I have not seen so much 419 spam in years, since adding myself as a dev@ moderator - there must be absolutely zero spam filtering to the list
Mon Dec 5 15:50:53 2016  driftx:yeah, pretty much, I gave up on moderation on client-dev for that reason... plus nobody posts there, heh
Mon Dec 5 15:50:59 2016  jasonstack:Joined the channel
Mon Dec 5 15:59:20 2016  kohlisankalp:Joined the channel
Mon Dec 5 16:08:34 2016  pcmanus:ptnapoleon: thanks, restarted. We'll see how it goes.
Mon Dec 5 16:09:25 2016  pcmanus:exlt: no excuse, that think must be moderated promptly not matter what
Mon Dec 5 16:10:04 2016  exlt::)
Mon Dec 5 16:20:11 2016  kohlisankalp:Joined the channel
Mon Dec 5 16:35:44 2016  mstepura:Joined the channel
Mon Dec 5 17:18:36 2016  mstepura:Joined the channel
Mon Dec 5 17:50:19 2016  minimarcel:Joined the channel
Mon Dec 5 17:51:36 2016  zaller:Joined the channel
Mon Dec 5 17:51:57 2016  jasobrown:exlt: yeah, *lots* of spam in dev@
Mon Dec 5 17:52:08 2016  jasobrown:glad to know there's another moderator :D
Mon Dec 5 17:52:25 2016  iamaleksey:make sure to stick to the SLA
Mon Dec 5 17:52:27 2016  iamaleksey:or else
Mon Dec 5 17:53:56 2016  jasobrown:iamaleksey: yeah, clearly the gods become enraged when the SLA is broken
Mon Dec 5 17:54:11 2016  iamaleksey:waving their hats angrily
Mon Dec 5 17:55:43 2016  driftx:just the feather in the cap, not the cap itself.
Mon Dec 5 18:01:07 2016  urandom:exlt: in the past, i've considered putting a procmail rule to work in rubber-stamping everything
Mon Dec 5 18:01:37 2016  urandom:and when the inevitable wtfs start rolling in, be all like:
Mon Dec 5 18:01:39 2016  urandom:¯\_(ツ)_/¯
Mon Dec 5 18:03:41 2016  urandom:it's well understood that you should greenlight anything from a human, so "moderation" here is a fancy way of saying that you are a human spam filter
Mon Dec 5 18:04:09 2016  driftx:that's so true
Mon Dec 5 18:04:34 2016  exlt:isn't King Faruque a human, too? ;)
Mon Dec 5 18:04:45 2016  urandom:nope
Mon Dec 5 18:05:02 2016  kvaster_:Joined the channel
Mon Dec 5 18:06:17 2016  dikang:Joined the channel
Mon Dec 5 18:13:41 2016  mebigfatguy:Joined the channel
Mon Dec 5 18:36:19 2016  kohlisankalp:Joined the channel
Mon Dec 5 18:42:59 2016  kohlisankalp:Joined the channel
Mon Dec 5 18:50:47 2016  kohlisankalp:Joined the channel
Mon Dec 5 19:20:29 2016  wei_:Joined the channel
Mon Dec 5 19:21:00 2016  chbatey_:Joined the channel
Mon Dec 5 19:23:26 2016  lqid_:Joined the channel
Mon Dec 5 19:26:23 2016  mambocab_:Joined the channel
Mon Dec 5 19:26:33 2016  tjake_:Joined the channel
Mon Dec 5 19:26:49 2016  wadey_:Joined the channel
Mon Dec 5 19:27:34 2016  carlyeks-:Joined the channel
Mon Dec 5 19:34:02 2016  ben__:Joined the channel
Mon Dec 5 19:35:53 2016  mambocab:Left the channel
Mon Dec 5 19:39:54 2016  paulomotta:should we tag #12905 as 3.10? It may prevent bootstrap of nodes with MVs. I was planning to review it this week.
Mon Dec 5 19:39:54 2016  CassBotJr:https://issues.apache.org/jira/browse/CASSANDRA-12905 (Patch Available; Unresolved; 3.0.x, 3.x): "Retry acquire MV lock on failure instead of throwing WTE on streaming"
Mon Dec 5 19:42:40 2016  exlt:paulomotta: yeah, that would be great to keep it on my release radar, thanks
Mon Dec 5 19:44:51 2016  exlt:I know setting fixver at commit time is the norm, but when someone brings up JIRAs as release critical, I don't know a better way to keep track of them than hard-setting fixver pre-commit - "this must be included in version X.Y and is not complete"
Mon Dec 5 19:49:04 2016  jeffj:paulomotta: think you could glance at #12888 at the same time? same author, similar part of the code, looks like you've already glanced at it?
Mon Dec 5 19:49:04 2016  CassBotJr:https://issues.apache.org/jira/browse/CASSANDRA-12888 (Patch Available; Unresolved; 3.0.x, 3.x): "Incremental repairs broken for MVs and CDC"
Mon Dec 5 19:54:17 2016  paulomotta:jeffj: sure, but I don't think we should block release on it since it's less critical and there's a workaround and I suspect the proposed solution is not viable because it changes MV design a bit and may affect correctness
Mon Dec 5 19:54:25 2016  jeffj:dont disagree with that
Mon Dec 5 20:46:12 2016  urandom:i'm (still) wondering what #9754 and #11206 mean for wide partitions, maybe a better way of approaching it is, what still suffers as a result of having wide partitions?
Mon Dec 5 20:46:13 2016  CassBotJr:https://issues.apache.org/jira/browse/CASSANDRA-11206 (Resolved; Fixed; 3.6): "Support large partitions on the 3.0 sstable format"
Mon Dec 5 20:46:23 2016  CassBotJr:https://issues.apache.org/jira/browse/CASSANDRA-9754 (Awaiting Feedback; Unresolved; 4.x): "Make index info heap friendly for large CQL partitions"
Mon Dec 5 20:46:27 2016  urandom:streaming operations? repair?
Mon Dec 5 20:47:53 2016  urandom:assuming heap/gc overhead was reasonable, what becomes the next big suck?
Mon Dec 5 20:49:14 2016  driftx:repair, but only because if partially damaged, we have to send the whole partition
Mon Dec 5 20:49:32 2016  urandom:even w/ incremental repair?
Mon Dec 5 20:50:20 2016  iamaleksey:the whole partition from the unrepaired set
Mon Dec 5 20:50:56 2016  urandom:which is something less than the whole thing, probably, if repairs are run frequently
Mon Dec 5 20:51:01 2016  urandom:?
Mon Dec 5 20:55:13 2016  urandom:i have a use-case with some wide partitions and all of the fun that goes with them (like OOMS), and i'm trying to convince others that we should remodel storage, and while i think that's inevitable, i do want to do due diligence here
Mon Dec 5 20:57:52 2016  urandom:what happens in LCS if the width of a partition exceeds the SSTable size? is the file made bigger, or does the partition span files?
Mon Dec 5 21:03:15 2016  driftx:spans
Mon Dec 5 21:03:31 2016  urandom:oh it does?
Mon Dec 5 21:03:38 2016  urandom:hrmm
Mon Dec 5 21:03:55 2016  driftx:when being written during compaction, yeah. Of course it can do whatever in L0
Mon Dec 5 21:04:04 2016  kohlisankalp:Joined the channel
Mon Dec 5 21:05:02 2016  zznate:big partitions and LCS (past sstable_size_in_mb) kinda blows. We see performance issues there were people have a few 'outlier' partitions causing fragmentation
Mon Dec 5 21:05:42 2016  urandom:zznate: just because it increases the sstables/read?
Mon Dec 5 21:05:50 2016  zznate:yep.
Mon Dec 5 21:06:53 2016  driftx:aren't you mostly timeseries under DTCS though? If you align PK with window in TWCS, it doesn't matter much how big they are, single sstable per read
Mon Dec 5 21:07:22 2016  zznate:^+1
Mon Dec 5 21:07:32 2016  urandom:driftx: repairs don't seem tractable for us, too much data
Mon Dec 5 21:07:38 2016  urandom:so we have read-repair enabled
Mon Dec 5 21:07:48 2016  driftx:with dtcs? heh
Mon Dec 5 21:07:51 2016  urandom:twcs
Mon Dec 5 21:08:04 2016  driftx:still bad for twcs, just not a rat's nest of tiers
Mon Dec 5 21:08:12 2016  urandom:right
Mon Dec 5 21:08:27 2016  urandom:but as a result, there is pressure to try LCS again
Mon Dec 5 21:08:45 2016  urandom:since it (current discussion nothwithstanding) would better bound sstables/read
Mon Dec 5 21:08:53 2016  driftx:you ttl too, don't you?
Mon Dec 5 21:09:00 2016  urandom:yes. :(
Mon Dec 5 21:09:08 2016  driftx:then lcs is going to suck at eviction
Mon Dec 5 21:09:32 2016  urandom:yes, i think so too, but i'm arguing w/ someone who thinks the opposite is true
Mon Dec 5 21:09:35 2016  driftx:as is basically any other strategy besides twcs
Mon Dec 5 21:10:26 2016  urandom:that because no overlap can exist within a level, that eventually data will work its way into the higher levels and overlaps will resolve
Mon Dec 5 21:10:50 2016  urandom:but that seems to miss the out-of-order writes entering the lower levels
Mon Dec 5 21:19:00 2016  mstepura:Joined the channel
Mon Dec 5 21:21:28 2016  zznate:urandom - i just gave it a quick glance, but you could try hacking MaxSSTableSizeWriter to skip LCS spanning behavior.
Mon Dec 5 21:24:50 2016  urandom:that would be an interesting experiment, but it seems like it might create as many problems as it solved
Mon Dec 5 21:25:44 2016  urandom:i mean it would pretty much shatter current assumptions about the amount of data per level
Mon Dec 5 21:26:04 2016  urandom:of course, spanning changes some assumptions too
Mon Dec 5 21:26:31 2016  urandom:like having partitions that consumed an entire level
Mon Dec 5 21:26:40 2016  urandom:that seems entirely possible
Mon Dec 5 21:29:44 2016  zznate:good point - probably do that hand in hand with turning up max_sstable_size_in_mb. either way, agree would be interesting
Mon Dec 5 21:30:23 2016  driftx:pretty you want to stick with twcs here, heh
Mon Dec 5 21:31:25 2016  jeffj:pretty sure your use case is weird, evertyhing is awful, and you're choosing between least awful options. which may mean writing your own compaction strategy if the tradeoffs are insufficient
Mon Dec 5 21:31:36 2016  jeffj:(which isnt to say your use case is wrong, just unusual)
Mon Dec 5 21:32:03 2016  urandom:jeffj: so diplomatic!
Mon Dec 5 21:32:12 2016  urandom::)
Mon Dec 5 21:32:32 2016  urandom:jeffj: much of it feels pretty wrong
Mon Dec 5 21:32:40 2016  jeffj:tone is hard, but that wasnt meant to be negative :) just that you're walking weird edge cases, and it's all going to suck.
Mon Dec 5 21:33:13 2016  urandom:jeffj: i know :)
Mon Dec 5 21:34:22 2016  urandom:jeffj: it's a shit sandwich, and i'm just trying to find an angle to take a bite that is predominately bread
Mon Dec 5 21:40:34 2016  bbromhead:Joined the channel
Mon Dec 5 21:41:15 2016  kohlisankalp:Joined the channel
Mon Dec 5 21:48:08 2016  urandom:driftx, zznate: the way the docs read, partitions do not span sstables: http://cassandra.apache.org/doc/latest/operating/compaction.html?highlight=sstable_size_in_mb
Mon Dec 5 21:48:21 2016  urandom:"sstables can end up being larger if there are very large partitions on the node."
Mon Dec 5 21:48:51 2016  driftx:not true ime, maybe it's changed recently
Mon Dec 5 21:49:02 2016  kohlisankalp:Joined the channel
Mon Dec 5 21:57:52 2016  zznate:MaxSSTableSizeWriter calls sstableWriter.switchTable when the masSStableSize boundary is breeched.
Mon Dec 5 21:59:45 2016  zznate:just realized I had 2.2-latest open in my IDE
Mon Dec 5 22:00:41 2016  zznate:still doing a switch in trunk though
Mon Dec 5 22:03:35 2016  urandom:huh, so the docs are wrong...
Mon Dec 5 22:04:04 2016  driftx:surprise.
Mon Dec 5 22:12:43 2016  zznate:actually, i could have read this wrong - MaxSSTableWriter#realAppend does the size check+swicthWriter *after* it appends the whole UnfilteredRowIterator (3.x, trunk) so perhaps docs are right
Mon Dec 5 22:25:26 2016  urandom:zznate: i guess that wouldn't have an sstables/read impact then, what do you figure caused the performance issues you've seen?
Mon Dec 5 22:26:18 2016  urandom:i really don't know which would be worse here; either way it seems to underscore wide partitions being an issue w/ LCS
Mon Dec 5 22:26:32 2016  urandom:whether they span tables or not
Mon Dec 5 22:27:11 2016  urandom:increasing sstables_size_in_mb only seems like a good idea if partitions are generally wide
Mon Dec 5 22:27:32 2016  urandom:we have some that are quite wide, but our 98p size is quite reasonable
Mon Dec 5 22:28:29 2016  urandom:i suspect that's true for a lot of people with wide partitions
Mon Dec 5 22:33:15 2016  jeffj:wei was mentioning he did a ton of research into the right defaults for that
Mon Dec 5 22:35:48 2016  jeffj:#12591 for more notes on sstable_size_in_mb (may or may not be directly related to your question)
Mon Dec 5 22:35:50 2016  CassBotJr:https://issues.apache.org/jira/browse/CASSANDRA-12591 (Open; Unresolved; Unscheduled): "Re-evaluate the default 160MB sstable_size_in_mb choice in LCS"
Mon Dec 5 22:36:30 2016  bbromhead:î
Mon Dec 5 22:36:31 2016  zznate:urandom biggest issues we've seen have been L1 thrashing for high-traffic tables (that recent thread on dev@)
Mon Dec 5 22:37:03 2016  zznate:continual re-writting of L1 after L0 dumps some huge STCS'ed tables in
Mon Dec 5 22:37:26 2016  urandom:zznate: specific to wide partitions?
Mon Dec 5 22:37:43 2016  urandom:i could see this though
Mon Dec 5 22:38:02 2016  zznate:most recently that was some of it - wide in this case was outliers ~ 150ish mb
Mon Dec 5 22:38:36 2016  urandom:jeffj: reading...
Mon Dec 5 22:39:09 2016  zznate:if seen it with not as wide partitions though. result of big batch jobs that run over several minutes dumping in data for multiple partitions across that time
Mon Dec 5 22:40:21 2016  zznate:setting sstable_size_in_mb 'fairly high' tends to help a lot
Mon Dec 5 23:11:48 2016  zanson:zznate driftx urandom pretty sure you get a giant sstable with the giant partition as the last thing in it, we do not split partitions across sstables in a given level, ever
Mon Dec 5 23:12:08 2016  zanson:"1 partition is in 1 sstable" is a big thing for LCS assumptions
Mon Dec 5 23:12:20 2016  zanson:(in a given level that is not L0)
Mon Dec 5 23:12:56 2016  zanson:so even if you have max size of 160, if you have a 1 GB partition you will have a 1 GB sstable
Mon Dec 5 23:16:23 2016  iamaleksey:that
Mon Dec 5 23:20:56 2016  zaller:Joined the channel
Mon Dec 5 23:59:19 2016  kohlisankalp:Joined the channel

Comments