Displaying #cassandra-dev/2017-05-17.log:

Wed May 17 00:02:12 2017  kohlisankalp:Joined the channel
Wed May 17 00:07:49 2017  kohlisankalp:Joined the channel
Wed May 17 00:14:44 2017  kohlisankalp:Joined the channel
Wed May 17 00:17:57 2017  kohlisankalp:Joined the channel
Wed May 17 00:31:52 2017  clohfink:Joined the channel
Wed May 17 01:04:53 2017  mebigfatguy:Joined the channel
Wed May 17 01:23:01 2017  clohfink:Joined the channel
Wed May 17 03:15:48 2017  dikang:Joined the channel
Wed May 17 04:02:32 2017  makrusak:Joined the channel
Wed May 17 04:39:33 2017  dikang:Joined the channel
Wed May 17 04:40:13 2017  nickmbailey:Joined the channel
Wed May 17 04:43:25 2017  nickmbailey:Joined the channel
Wed May 17 04:55:16 2017  nickmbailey:Joined the channel
Wed May 17 04:55:34 2017  bbromhead:Joined the channel
Wed May 17 05:12:01 2017  nickmbailey:Joined the channel
Wed May 17 05:13:51 2017  nickmbailey:Joined the channel
Wed May 17 06:01:25 2017  kohlisankalp:Joined the channel
Wed May 17 06:04:27 2017  bbromhead:Joined the channel
Wed May 17 06:05:48 2017  bbromhea_:Joined the channel
Wed May 17 06:07:31 2017  nickmbailey:Joined the channel
Wed May 17 06:18:11 2017  nickmbailey:Joined the channel
Wed May 17 06:19:06 2017  nickmbailey:Joined the channel
Wed May 17 06:38:05 2017  bbromhead:Joined the channel
Wed May 17 06:44:35 2017  spodkowinski:Joined the channel
Wed May 17 06:50:17 2017  kvaster:Joined the channel
Wed May 17 07:01:55 2017  nickmbailey:Joined the channel
Wed May 17 07:19:07 2017  clohfink:Joined the channel
Wed May 17 08:14:02 2017  zaller:Joined the channel
Wed May 17 08:35:57 2017  ostefano:Joined the channel
Wed May 17 08:44:37 2017  ostefano:Joined the channel
Wed May 17 08:49:54 2017  nickmbailey:Joined the channel
Wed May 17 08:50:20 2017  ostefano:Joined the channel
Wed May 17 08:52:47 2017  ostefano:Joined the channel
Wed May 17 08:54:10 2017  ostefano:Joined the channel
Wed May 17 08:59:24 2017  ostefano:Joined the channel
Wed May 17 09:00:43 2017  ostefano:Joined the channel
Wed May 17 09:03:46 2017  ostefano:Joined the channel
Wed May 17 09:40:18 2017  bbromhead:Joined the channel
Wed May 17 10:01:22 2017  clohfink:Joined the channel
Wed May 17 10:37:58 2017  nickmbailey:Joined the channel
Wed May 17 11:01:21 2017  Vijay_:Joined the channel
Wed May 17 11:31:01 2017  spodkowinski:exlt: can we increase the number of global concurrent builds per job?
Wed May 17 11:31:38 2017  spodkowinski:I like to run 3 devbranch jobs and two of them end up queued, while there are still idle slaves
Wed May 17 11:32:50 2017  nickmbailey:Joined the channel
Wed May 17 11:35:33 2017  spodkowinski:how about using "throttleConcurrentBuilds { maxPerNode(1) maxTotal(5) }" for the devbranch job dsl ?
Wed May 17 11:41:35 2017  clohfink:Joined the channel
Wed May 17 11:43:33 2017  bbromhead:Joined the channel
Wed May 17 11:50:01 2017  exlt:they are already running with 1 build per node, 0 (unlimited) total - that's what the "Cassandra" throttle category has configured
Wed May 17 11:50:04 2017  exlt:hmm..
Wed May 17 11:50:27 2017  spodkowinski:https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/
Wed May 17 11:50:34 2017  exlt:the queued jobs are blocking on the same job.. so..
Wed May 17 11:50:39 2017  exlt:oh yeah, I see that..
Wed May 17 11:51:59 2017  exlt:http://12.am/tmp/throttle_configs.png
Wed May 17 11:52:17 2017  exlt:I'm thinking that parameter concurrency one might be the one we want?
Wed May 17 11:52:30 2017  exlt:allow jobs with different params to run concurrently
Wed May 17 11:54:29 2017  exlt:seeing if I can find that option in the DSL API doc
Wed May 17 11:55:28 2017  spodkowinski:the throttle plugin description also mentions a "Execute concurrent builds if necessary" option for that https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin
Wed May 17 11:58:31 2017  exlt::( I think all the latest security foo with jenkins is preventing the api from being fully populated - looks completely disabled on the asf jenkins, now..
Wed May 17 12:00:38 2017  exlt:https://jenkinsci.github.io/job-dsl-plugin/#path/job-throttleConcurrentBuilds
Wed May 17 12:01:17 2017  exlt:I don't see which option that might be
Wed May 17 12:02:03 2017  exlt:it's not a matrix job.. and "disabled" doesn't seem to fit
Wed May 17 12:02:51 2017  exlt:just wants to check a box..
Wed May 17 12:03:43 2017  exlt:that box can probably be ticked - there are some DSL xml hacking tips around, but I'm not very savvy with that
Wed May 17 12:05:41 2017  exlt:maybe there's another way - there's a generic "Execute concurrent builds if necessary" - let's see where that is in the DSL
Wed May 17 12:06:12 2017  exlt:concurrentBuild(boolean allowConcurrentBuild = true)
Wed May 17 12:08:08 2017  exlt:let me check another place - I know I'm using a similar setup somewhere
Wed May 17 12:13:25 2017  exlt:yep. `concurrentBuild()` along with throttleConcurrentBuilds { categories(['Cassandra']) }` should do the trick, spodkowinski - committed. building new configs. then we should see those queued jobs start building
Wed May 17 12:15:33 2017  exlt:man, infra's own jobs are hosed - 10 days waiting..
Wed May 17 12:15:49 2017  exlt:nukes infra's jdk test job..
Wed May 17 12:16:20 2017  exlt:there, now the build queue looks a little better :)
Wed May 17 12:16:57 2017  exlt:success - your jobs are building concurrently, spodkowinski :-D
Wed May 17 12:21:40 2017  spodkowinski:great, let's keep those nodes busy :) thanks for your help
Wed May 17 12:26:30 2017  nickmbailey:Joined the channel
Wed May 17 13:39:11 2017  clohfink:Joined the channel
Wed May 17 13:45:19 2017  bbromhead:Joined the channel
Wed May 17 13:56:29 2017  clohfink:Joined the channel
Wed May 17 14:14:30 2017  nickmbailey:Joined the channel
Wed May 17 14:49:35 2017  thobbs:Joined the channel
Wed May 17 16:00:11 2017  nickmbailey:Joined the channel
Wed May 17 16:10:40 2017  jfarrell:Joined the channel
Wed May 17 16:25:02 2017  nickmbailey:Joined the channel
Wed May 17 16:31:13 2017  kohlisankalp:Joined the channel
Wed May 17 16:48:57 2017  dikang:Joined the channel
Wed May 17 16:49:11 2017  spodkowinski:Joined the channel
Wed May 17 17:01:17 2017  nickmbailey:Joined the channel
Wed May 17 17:04:30 2017  bbromhead:Joined the channel
Wed May 17 17:18:56 2017  bbromhead:Joined the channel
Wed May 17 17:27:20 2017  nickmbailey:Joined the channel
Wed May 17 17:32:17 2017  kohlisankalp:Joined the channel
Wed May 17 17:33:00 2017  aweisberg:Was there a bad news ccm or dtest change recently? https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-dtest/142/#showFailuresLink
Wed May 17 17:34:48 2017  dikang_:Joined the channel
Wed May 17 17:34:57 2017  kohlisankalp:Joined the channel
Wed May 17 18:01:45 2017  kohlisankalp:Joined the channel
Wed May 17 18:02:00 2017  Vijay:Joined the channel
Wed May 17 18:10:59 2017  nickmbailey:Joined the channel
Wed May 17 18:16:53 2017  ptnapoleon:aweisberg: we had an issue with cloning from the asf git
Wed May 17 18:17:07 2017  ptnapoleon:if you retry now, it should work, but let me know
Wed May 17 18:17:22 2017  aweisberg:ptnapoleon: Thanks
Wed May 17 18:17:38 2017  tjake:aweisberg: hows the dtests on circleci going?
Wed May 17 18:18:06 2017  aweisberg:tjake: I have a circle.yml that does it you just can't do it on free circleci. Not enough RAM.
Wed May 17 18:18:31 2017  tjake:yeah I had that with regular unit tests :/
Wed May 17 18:28:04 2017  nickmbailey:Joined the channel
Wed May 17 18:31:06 2017  nickmbailey:Joined the channel
Wed May 17 18:36:25 2017  exlt:aweisberg: those errors look like an orphaned ccm cluster from a dtest, since it start right off not being able to bind to ports..
Wed May 17 18:36:35 2017  exlt:I'll poke the slaves with a stick
Wed May 17 18:36:46 2017  exlt:and this is exactly why slaves suck
Wed May 17 18:39:36 2017  ptnapoleon:Oh wow I didnt even read where your link was from, my bad ariel
Wed May 17 18:40:31 2017  exlt:all the jobs try to clean up after themselves, but if someone gets abort happy, they won't clean up
Wed May 17 18:40:52 2017  dikang:Joined the channel
Wed May 17 18:41:10 2017  exlt:yeah, there are ccm nodes running on some, even though there are no jobs
Wed May 17 18:42:59 2017  exlt:and for some reason, some of the slaves are running like a dozen jenkins slave.jar processes sucking up ram
Wed May 17 18:44:40 2017  exlt:lol
Wed May 17 18:44:41 2017  exlt:java -Xmx4g -Xms512m -XX:MaxPermSize=256m -XX:-UseConcMarkSweepGC -jar slave.jar
Wed May 17 18:47:30 2017  exlt:ccm nodes should all be gone, but I'll see if I can get the slave.jar processes cleaned up, too
Wed May 17 18:47:43 2017  aweisberg:I would have figured we would have each job start by nuking all the other cassandra processes as well as their files in /tmp or where ever. The slave jars thing is a trippy.
Wed May 17 18:48:30 2017  exlt:well, you can't exactly do a `killall java` on jenkins machines
Wed May 17 18:49:29 2017  exlt:I started looking at test jobs in containers as a better option, but got pulled to other things
Wed May 17 18:59:53 2017  nickmbailey:Joined the channel
Wed May 17 19:05:34 2017  bbromhead:Joined the channel
Wed May 17 19:06:42 2017  bbromhea_:Joined the channel
Wed May 17 19:06:47 2017  exlt:INFRA-14153 isn't exactly done, the new slaves won't connect, now, after killing the numerous slave.jar processes :P
Wed May 17 19:07:24 2017  kohlisankalp:Joined the channel
Wed May 17 19:07:46 2017  exlt:I'm guessing that's why there were so many running
Wed May 17 19:10:18 2017  kohlisankalp:Joined the channel
Wed May 17 19:13:39 2017  Vijay:Joined the channel
Wed May 17 19:16:03 2017  kohlisankalp:Joined the channel
Wed May 17 19:17:50 2017  exlt:I'm looking at the one slave I have in the lab, and it has no heap settings passed to the slave.jar, so infra has explicitly set that huge heap for some reason - I'm willing to bet we ran out of memory somewhere and left some hung dtest clusters - ariel's test ran on cassandra14 which is one of the new slaves that are mid-setup
Wed May 17 19:18:06 2017  zaller:Joined the channel
Wed May 17 19:18:48 2017  exlt:what should really happen is that infra gets the ssh keys set up correctly in jenkins, then krut should reboot all those servers to make sure they come up clean and actually connect properly on boot :)
Wed May 17 19:28:16 2017  Vijay:Joined the channel
Wed May 17 19:41:04 2017  dikang:Joined the channel
Wed May 17 19:42:12 2017  bbromhead:Joined the channel
Wed May 17 19:45:36 2017  bbromhea_:Joined the channel
Wed May 17 19:50:10 2017  bbromhead:Joined the channel
Wed May 17 19:51:59 2017  nickmbailey:Joined the channel
Wed May 17 20:01:58 2017  sPollk:Joined the channel
Wed May 17 20:04:09 2017  sPollk:anybody build a rpm version of cassandra 3.10?
Wed May 17 20:05:02 2017  driftx:you'll have to build it yourself, but the spec file is there
Wed May 17 20:07:03 2017  sPollk:ok thks...but I avoid build process because I have not a build resource on my host
Wed May 17 20:07:07 2017  sPollk:...
Wed May 17 20:15:41 2017  nickmbailey:Joined the channel
Wed May 17 20:21:59 2017  kohlisankalp:Joined the channel
Wed May 17 20:38:54 2017  exlt:sPollk: all you need is docker - https://github.com/apache/cassandra-builds
Wed May 17 20:42:08 2017  sPollk:thks @exlt...I'm cloning project.
Wed May 17 21:01:11 2017  dnick:Joined the channel
Wed May 17 21:02:05 2017  nickmbailey:Joined the channel
Wed May 17 21:10:49 2017  kohlisankalp:Joined the channel
Wed May 17 21:40:07 2017  clohfink:Joined the channel
Wed May 17 22:04:56 2017  Vijay:Joined the channel
Wed May 17 22:14:53 2017  zaller:Joined the channel
Wed May 17 22:27:40 2017  kohlisan_:Joined the channel
Wed May 17 22:29:20 2017  bbromhead:Joined the channel
Wed May 17 22:34:16 2017  clohfink:Joined the channel
Wed May 17 22:50:04 2017  nickmbailey:Joined the channel
Wed May 17 23:01:46 2017  kohlisankalp:Joined the channel
Wed May 17 23:30:31 2017  mebigfatguy:Joined the channel
Wed May 17 23:36:37 2017  kohlisankalp:Joined the channel
Wed May 17 23:44:52 2017  nickmbailey:Joined the channel

Comments