Displaying #cassandra-dev/2016-11-15.log:

Tue Nov 15 00:02:34 2016  aaron_morton:Joined the channel
Tue Nov 15 00:28:34 2016  aaron_morton:Joined the channel
Tue Nov 15 00:32:31 2016  mstepura:Joined the channel
Tue Nov 15 00:33:36 2016  lqid:Joined the channel
Tue Nov 15 00:37:50 2016  dikang_:Joined the channel
Tue Nov 15 01:30:35 2016  aaron_morton:Joined the channel
Tue Nov 15 01:42:28 2016  clohfink:Joined the channel
Tue Nov 15 02:05:57 2016  aaron_morton:Joined the channel
Tue Nov 15 02:30:46 2016  mstepura:Joined the channel
Tue Nov 15 02:54:57 2016  mstepura:Joined the channel
Tue Nov 15 03:01:39 2016  mstepura:Joined the channel
Tue Nov 15 03:02:42 2016  mstepura:Joined the channel
Tue Nov 15 03:04:18 2016  mstepura:Joined the channel
Tue Nov 15 03:05:04 2016  mstepura:Joined the channel
Tue Nov 15 03:05:50 2016  mstepura:Joined the channel
Tue Nov 15 03:06:41 2016  mstepura:Joined the channel
Tue Nov 15 03:07:29 2016  mstepura:Joined the channel
Tue Nov 15 03:15:39 2016  mstepura:Joined the channel
Tue Nov 15 03:39:10 2016  mstepura:Joined the channel
Tue Nov 15 03:45:09 2016  mstepura:Joined the channel
Tue Nov 15 04:22:51 2016  dikang:Joined the channel
Tue Nov 15 04:24:56 2016  vjarnot:Joined the channel
Tue Nov 15 04:28:22 2016  vince:Joined the channel
Tue Nov 15 04:34:29 2016  dikang:Joined the channel
Tue Nov 15 04:39:41 2016  mstepura:Joined the channel
Tue Nov 15 05:36:16 2016  mstepura:Joined the channel
Tue Nov 15 05:56:18 2016  jshook_:Joined the channel
Tue Nov 15 06:07:58 2016  mstepura:Joined the channel
Tue Nov 15 06:18:13 2016  vince:Joined the channel
Tue Nov 15 06:18:50 2016  ostefano:Joined the channel
Tue Nov 15 06:21:03 2016  dikang:Joined the channel
Tue Nov 15 06:22:50 2016  jshook:Joined the channel
Tue Nov 15 06:31:14 2016  nickmbailey:Joined the channel
Tue Nov 15 06:33:27 2016  jshook:Joined the channel
Tue Nov 15 06:44:03 2016  clohfink:Joined the channel
Tue Nov 15 07:02:57 2016  nickmbailey:Joined the channel
Tue Nov 15 07:12:24 2016  vince:Joined the channel
Tue Nov 15 07:21:26 2016  mstepura:Joined the channel
Tue Nov 15 07:31:18 2016  exlt:jeffj: the links you posted may, indeed, be some of the ones I was thinking about - I just read your eamil after a movie and beers and had to get that out :)
Tue Nov 15 07:31:54 2016  exlt:I'll try to read through in the morning and see if there were others
Tue Nov 15 07:42:13 2016  exlt:zznate: are you and friends OK? just hadn't heard your proximity.. 7.8 is huge..
Tue Nov 15 07:57:10 2016  nickmbailey:Joined the channel
Tue Nov 15 08:17:37 2016  spodkowinski:Joined the channel
Tue Nov 15 09:20:37 2016  minimarcel_:Joined the channel
Tue Nov 15 09:40:25 2016  vince:Joined the channel
Tue Nov 15 10:38:54 2016  nickmbailey:Joined the channel
Tue Nov 15 11:30:23 2016  vince:Joined the channel
Tue Nov 15 11:33:41 2016  nickmbailey:Joined the channel
Tue Nov 15 12:09:34 2016  kvaster:Joined the channel
Tue Nov 15 12:27:36 2016  nickmbailey:Joined the channel
Tue Nov 15 13:03:29 2016  minimarcel_:Joined the channel
Tue Nov 15 13:40:17 2016  minimarcel_:Joined the channel
Tue Nov 15 14:10:32 2016  kvaster:Joined the channel
Tue Nov 15 14:11:06 2016  jfarrell:Joined the channel
Tue Nov 15 14:37:05 2016  kvaster:Joined the channel
Tue Nov 15 14:38:15 2016  kvaster:Joined the channel
Tue Nov 15 14:56:38 2016  EdwardCapriolo:Joined the channel
Tue Nov 15 15:11:16 2016  vjarnot:Joined the channel
Tue Nov 15 15:20:21 2016  jfarrell_:Joined the channel
Tue Nov 15 15:34:21 2016  nickmbailey:Joined the channel
Tue Nov 15 15:44:41 2016  clohfink:Joined the channel
Tue Nov 15 15:47:31 2016  thobbs:Joined the channel
Tue Nov 15 16:10:39 2016  minimarcel_:Joined the channel
Tue Nov 15 16:25:33 2016  minimarcel_:Joined the channel
Tue Nov 15 17:04:28 2016  EdwardCapriolo:Joined the channel
Tue Nov 15 17:15:57 2016  mstepura:Joined the channel
Tue Nov 15 17:21:51 2016  JayZhuang:Joined the channel
Tue Nov 15 17:30:26 2016  JayZhuang:Joined the channel
Tue Nov 15 17:33:26 2016  aweisberg:Has anyone ever done any major surgery on the system tables? Stuff like changing the primary key and reloading the data? I am thinking for 4.0 I will need to use LegacySchemaMigrator to migrate from 3.0 to 4.0 to address issues like the system tables referring to nodes ambiguously via InetAddress instead of InetAddress + ports.
Tue Nov 15 17:40:43 2016  iamaleksey:I did
Tue Nov 15 17:41:03 2016  iamaleksey:also Branimir when rewriting batchlog
Tue Nov 15 17:41:14 2016  iamaleksey:and a little with Auth
Tue Nov 15 17:41:36 2016  iamaleksey:LSM is not for that though, and it'll be ditched in 4.0 as no longer necessary
Tue Nov 15 17:42:11 2016  iamaleksey:and I'm pretty sure you'll have to come up with new tables to live side by side with older ones for a while
Tue Nov 15 17:42:18 2016  iamaleksey:can't just modify them in place like that
Tue Nov 15 17:52:55 2016  dikang:Joined the channel
Tue Nov 15 18:07:20 2016  minimarcel_:Joined the channel
Tue Nov 15 18:12:58 2016  aweisberg:iamaleksey: Are the system tables local or distributed? I thought they were private local and other nodes didn't read from them?
Tue Nov 15 18:13:24 2016  iamaleksey:some are some aren't
Tue Nov 15 18:13:31 2016  iamaleksey:the ones in system.* are all local
Tue Nov 15 18:13:41 2016  iamaleksey:the ones in system_schema.* (3.0) are in between
Tue Nov 15 18:13:47 2016  iamaleksey:all others are global
Tue Nov 15 18:13:53 2016  aweisberg:system.peers is 100% broken for running multiple nodes with different ports. The key is ambiguous and can't be fixed without at least changing the type to blob so I can add the missing ports.
Tue Nov 15 18:13:56 2016  iamaleksey:(system_auth.* system_distributed.*)
Tue Nov 15 18:14:09 2016  iamaleksey:but there is external tooling reading from them
Tue Nov 15 18:14:16 2016  iamaleksey:drivers and such.
Tue Nov 15 18:14:42 2016  iamaleksey:for compatibility, may or may not to need keep the old table in place alongside the new ones
Tue Nov 15 18:15:52 2016  aweisberg:I think it would be compatible with existing tools even with the new columns and additional clustering key columns. Existing queries would work.
Tue Nov 15 18:16:32 2016  aweisberg:I need to look at system_schema system_auth and system_distributed to see how often they refer to nodes by IP.
Tue Nov 15 18:17:04 2016  iamaleksey:the first two don't at all
Tue Nov 15 18:17:12 2016  iamaleksey:system_distributed might.
Tue Nov 15 18:17:39 2016  iamaleksey:it's also currently bugged, the table storing views has gc gs of 0
Tue Nov 15 18:17:42 2016  iamaleksey:and does deletes
Tue Nov 15 18:24:28 2016  aweisberg:iamaleksey: Are all modifications to existing tables off the table? You can't add columns with default values like you would in the SQL world (I know C* doesn't support default values), but I was thinking old code would just insert null for those columns.
Tue Nov 15 18:24:43 2016  aweisberg:Or will it error saying you didn't supply enough columns.
Tue Nov 15 18:25:48 2016  iamaleksey:aweisberg: not all; can modify params. can probably add regular and static columns relatively safely
Tue Nov 15 18:26:24 2016  iamaleksey:can modify primary key columns. can probably rename them, but then backward compat is potentially an issue
Tue Nov 15 18:26:42 2016  iamaleksey:the cleanest way is to add a new table and deprecate the old one
Tue Nov 15 18:26:49 2016  iamaleksey:that's how we prefer to deal with it
Tue Nov 15 18:30:20 2016  aweisberg:Makes sense. So just dupe the data in both tables?
Tue Nov 15 18:53:15 2016  iamaleksey:in your case, yes, if we decide that keeping compat is important or possible
Tue Nov 15 18:53:49 2016  iamaleksey:if not, you just migrate data on startup then purge the old table and leave it empty;
Tue Nov 15 18:53:58 2016  iamaleksey:but the new table should have a different name
Tue Nov 15 18:59:21 2016  Vijay:Joined the channel
Tue Nov 15 19:24:43 2016  kvaster:Joined the channel
Tue Nov 15 19:33:40 2016  JayZhuang:Joined the channel
Tue Nov 15 19:39:57 2016  aweisberg:We refer to nodes by inet in so much schema. The straw... it's so short.
Tue Nov 15 19:40:35 2016  iamaleksey:well, in schema we don't at all
Tue Nov 15 19:40:40 2016  iamaleksey:in some system tables we do
Tue Nov 15 19:42:58 2016  mstepura:Joined the channel
Tue Nov 15 19:46:56 2016  Vijay:Joined the channel
Tue Nov 15 19:58:58 2016  vince:Joined the channel
Tue Nov 15 20:02:18 2016  dikang:Joined the channel
Tue Nov 15 20:04:23 2016  Vijay_:Joined the channel
Tue Nov 15 20:24:36 2016  aweisberg:There are 7 system tables that use inet addresses to refer to nodes, but only system uses them as a key. So I could add columns to make the rest work and system is local so I could just migrate system.peer and system.peer_events to have the necessary key and add columns to the rest. I really feel like trying to hand write migration code is going to be more
Tue Nov 15 20:24:36 2016  aweisberg:error prone than letting the DB and query language do what they are built to do. Dealing with an additional column is pretty straightforward, old nodes ignore it, and new nodes use the value if it's not null otherwise they use the default from cassandra.yaml. When nodes have their ports changed it can always remove the old value in the tables that have null
Tue Nov 15 20:24:36 2016  aweisberg:for the new columns. https://www.irccloud.com/pastebin/dnkGF9qw/
Tue Nov 15 20:38:40 2016  nickmbailey:Joined the channel
Tue Nov 15 20:45:01 2016  Vijay:Joined the channel
Tue Nov 15 20:55:31 2016  dikang:Joined the channel
Tue Nov 15 21:00:29 2016  Vijay:Joined the channel
Tue Nov 15 21:02:16 2016  EdwardCapriolo:Joined the channel
Tue Nov 15 21:22:34 2016  aaron_morton:Joined the channel
Tue Nov 15 21:28:41 2016  JayZhuang:Joined the channel
Tue Nov 15 21:32:01 2016  aweisberg:Do I even need to migrate the contents of system.peers and system.peer_events, it looks like I could drop and recreate the tables and system.peer would get populated by gossip.
Tue Nov 15 21:35:10 2016  iamaleksey:you do
Tue Nov 15 21:35:26 2016  iamaleksey:it's an old ticket, driftx might remember the #
Tue Nov 15 21:35:59 2016  driftx:hmm, nope.
Tue Nov 15 21:40:48 2016  dikang:Joined the channel
Tue Nov 15 21:44:55 2016  Vijay:Joined the channel
Tue Nov 15 21:49:55 2016  aweisberg:Just looking at the way hardcoded schema is handled I don't think I can really change the key without further machinations because at startup it won't know whether to use the old or the new schema. Immutable schema FTW I guess.
Tue Nov 15 22:01:13 2016  EdwardCapriolo:Joined the channel
Tue Nov 15 22:03:50 2016  aweisberg:Fun fact. If you physically mail an individual CLA to Apache it will get returned because the mailbox they have is apparently closed.
Tue Nov 15 22:05:18 2016  JayZhuang:Joined the channel
Tue Nov 15 22:19:23 2016  Vijay:Joined the channel
Tue Nov 15 22:37:42 2016  iamaleksey:I told you not to change the key
Tue Nov 15 22:42:26 2016  zanson:aweisberg if you don't migrate them then you can have issues at startup with GPFS or any other gossip based snitch
Tue Nov 15 22:43:14 2016  zanson:for peers anyway, as we use peers to store DC/rack info for use when we don't have the data in gossip yet
Tue Nov 15 22:44:52 2016  dikang:Joined the channel
Tue Nov 15 22:47:40 2016  Vijay:Joined the channel
Tue Nov 15 22:55:07 2016  EdwardCapriolo:Joined the channel
Tue Nov 15 22:58:55 2016  JayZhuang:Joined the channel
Tue Nov 15 23:14:45 2016  adamholmberg:Joined the channel
Tue Nov 15 23:44:28 2016  EdwardCapriolo:Joined the channel
Tue Nov 15 23:52:44 2016  EdwardCapriolo:Joined the channel

Comments