Displaying #cassandra-dev/2017-08-10.log:

Thu Aug 10 00:49:30 2017  kohlisankalp:Joined the channel
Thu Aug 10 01:05:22 2017  kohlisankalp:Joined the channel
Thu Aug 10 01:09:41 2017  robusto:Joined the channel
Thu Aug 10 01:45:26 2017  robusto:Joined the channel
Thu Aug 10 01:57:52 2017  robusto:Joined the channel
Thu Aug 10 03:34:36 2017  kohlisankalp:Joined the channel
Thu Aug 10 03:43:15 2017  kohlisankalp:Joined the channel
Thu Aug 10 03:49:31 2017  kohlisankalp:Joined the channel
Thu Aug 10 04:03:27 2017  kohlisankalp:Joined the channel
Thu Aug 10 04:05:15 2017  kohlisankalp:Joined the channel
Thu Aug 10 04:05:51 2017  kohlisankalp:Joined the channel
Thu Aug 10 04:23:52 2017  jeffj:Joined the channel
Thu Aug 10 04:37:08 2017  jayzhuang:Joined the channel
Thu Aug 10 05:33:13 2017  robusto:Joined the channel
Thu Aug 10 05:58:08 2017  spodkowinski:Joined the channel
Thu Aug 10 06:37:35 2017  kvaster:Joined the channel
Thu Aug 10 07:35:34 2017  cosql:Joined the channel
Thu Aug 10 09:04:26 2017  kvaster:Joined the channel
Thu Aug 10 09:33:34 2017  robusto:Joined the channel
Thu Aug 10 09:41:27 2017  kvaster:Joined the channel
Thu Aug 10 10:06:31 2017  hkroger:Possible data corruption bug in 3.11: https://issues.apache.org/jira/browse/CASSANDRA-13752
Thu Aug 10 10:06:34 2017  hkroger:In case you are interested
Thu Aug 10 11:36:04 2017  cosql:Joined the channel
Thu Aug 10 11:37:14 2017  cosql:Joined the channel
Thu Aug 10 11:38:22 2017  cosql:Joined the channel
Thu Aug 10 11:39:23 2017  cosql:Joined the channel
Thu Aug 10 11:40:51 2017  cosql:Joined the channel
Thu Aug 10 11:42:15 2017  cosql:Joined the channel
Thu Aug 10 11:43:25 2017  cosql:Joined the channel
Thu Aug 10 11:44:30 2017  cosql:Joined the channel
Thu Aug 10 11:45:54 2017  cosql:Joined the channel
Thu Aug 10 11:58:04 2017  hkroger:Linking also 3.11 upgradesstables bug ticket, I created 2 weeks ago or so: https://issues.apache.org/jira/browse/CASSANDRA-13718
Thu Aug 10 12:24:27 2017  adamholmberg:Joined the channel
Thu Aug 10 12:33:22 2017  robusto:Joined the channel
Thu Aug 10 13:15:54 2017  exlt:hkroger: a way to reproduce would be most excellent, as well as possibly some more hw, fs, env details - we have crc32 checksums, to they match? - offline non-destructive fsck? - this sort of frequency in your cluster reminds me of some disk controller issues years ago with spent months with Dell to find..
Thu Aug 10 13:17:29 2017  exlt:several misspellings there.. moar coffee :)
Thu Aug 10 13:21:19 2017  hkroger:exlt: I will check the checksum
Thu Aug 10 13:21:29 2017  hkroger:However the problem seems to be in stats file not the main data file
Thu Aug 10 13:22:38 2017  marcuse:yup, the stats file does not have any checksum yet (#13321)
Thu Aug 10 13:22:38 2017  CassBotJr:https://issues.apache.org/jira/browse/CASSANDRA-13321 (Open; Unresolved; 4.x): "Add a checksum component for the sstable metadata (-Statistics.db) file"
Thu Aug 10 13:24:11 2017  exlt:ah, I see what you mean
Thu Aug 10 13:25:23 2017  hkroger:honestly, I don't know if the possible bug is in writer or reader. might be that sstables are fine but the reader has some issues.
Thu Aug 10 13:26:02 2017  marcuse:hkroger: would probably help if you could attach the sstable to the ticket
Thu Aug 10 13:27:22 2017  hkroger:exlt: digest matches
Thu Aug 10 13:28:01 2017  hkroger:marcuse: I think it is too big to be attached
Thu Aug 10 13:28:05 2017  hkroger:gigs
Thu Aug 10 13:28:39 2017  marcuse:k, all 3 of them?
Thu Aug 10 13:29:38 2017  hkroger:oh, that contains also some sensitive data
Thu Aug 10 14:06:53 2017  clohfink:Joined the channel
Thu Aug 10 14:49:06 2017  RussSpitzer_:Joined the channel
Thu Aug 10 15:57:58 2017  kohlisankalp:Joined the channel
Thu Aug 10 16:13:38 2017  kohlisankalp:Joined the channel
Thu Aug 10 16:24:01 2017  kohlisankalp:Joined the channel
Thu Aug 10 16:33:03 2017  cosql:Joined the channel
Thu Aug 10 16:40:08 2017  kohlisan_:Joined the channel
Thu Aug 10 16:42:02 2017  jayzhuang:Joined the channel
Thu Aug 10 16:49:36 2017  jeffj:urandom: what version of netty for your threadlocal issue?
Thu Aug 10 17:33:11 2017  jayzhuang:Joined the channel
Thu Aug 10 18:13:57 2017  jeffj:hkroger: are you doing anything a typical? any other errors in the logs before those hit? have you run incremental repair on those tables recently?
Thu Aug 10 18:41:44 2017  kohlisankalp:Joined the channel
Thu Aug 10 18:49:33 2017  jayzhuang:Joined the channel
Thu Aug 10 19:19:43 2017  jayzhuang1:Joined the channel
Thu Aug 10 20:00:31 2017  urandom:jeffj: mmm, whatever is embedded in 3.11.0... let me look
Thu Aug 10 20:01:52 2017  urandom:jeffj: 4.0.44?
Thu Aug 10 20:03:48 2017  kohlisankalp:Joined the channel
Thu Aug 10 20:10:31 2017  jayzhuang:Joined the channel
Thu Aug 10 20:33:17 2017  jayzhuang1:Joined the channel
Thu Aug 10 20:38:49 2017  jayzhuang:Joined the channel
Thu Aug 10 20:50:52 2017  damons:Joined the channel
Thu Aug 10 20:52:28 2017  jayzhuang:Joined the channel
Thu Aug 10 21:04:33 2017  jayzhuang1:Joined the channel
Thu Aug 10 21:54:21 2017  RussSpitzer:Joined the channel
Thu Aug 10 21:56:17 2017  jayzhuang:Joined the channel
Thu Aug 10 22:01:19 2017  hkroger:jeffj: I commented on the ticket as well
Thu Aug 10 22:01:38 2017  hkroger:incremental repair is being run (also full because of some ugprade difficulties)
Thu Aug 10 22:03:01 2017  kohlisan_:Joined the channel
Thu Aug 10 22:03:03 2017  adamholmberg:Joined the channel
Thu Aug 10 22:03:55 2017  jeffj:hkroger: appreciated. it'll be hard to really understand unless we can get some hints to reproduce.
Thu Aug 10 22:33:24 2017  jayzhuang:Joined the channel
Thu Aug 10 22:50:13 2017  jayzhuang:Joined the channel
Thu Aug 10 23:02:52 2017  jayzhuang1:Joined the channel
Thu Aug 10 23:33:21 2017  kohlisankalp:Joined the channel
Thu Aug 10 23:33:55 2017  kohlisan_:Joined the channel
Thu Aug 10 23:39:49 2017  kohlisankalp:Joined the channel

Comments