Displaying #traffic-server/2015-11-04.log:

Wed Nov 4 00:00:14 2015  jpeach:zwoop: you can set up travis-ci to do that, but you would need asfinfra to do it
Wed Nov 4 00:00:28 2015  zwoop:jpeach make it so #1!
Wed Nov 4 00:00:35 2015  jpeach:Humbedooh: ^^^
Wed Nov 4 00:10:23 2015  amc:dxu - are you still there
Wed Nov 4 00:10:24 2015  amc:>
Wed Nov 4 00:10:25 2015  amc:?
Wed Nov 4 00:11:06 2015  dxu:amc: Yes. I can look at it later tonight.
Wed Nov 4 00:11:59 2015  amc:Allright. If you can't get to it soon I need to work on it. zwoop is correct that it needs to get fixed as soon as possible.
Wed Nov 4 00:12:36 2015  amc:You may as well just do whatever zwoop tells you, because if you don't he'll just tell me to tell you :-)
Wed Nov 4 00:13:02 2015  dxu:Haha ok
Wed Nov 4 00:13:22 2015  zwoop:lol
Wed Nov 4 00:13:24 2015  dxu:I can begin on it at 10pm CT. I've got to go to this thing for class in a little
Wed Nov 4 00:13:47 2015  amc:That's fine. I'll try to take a bit of a look too.
Wed Nov 4 00:14:27 2015  zwoop:that's good. Let me know if you need any help seeing the issues (I sent the coverity list of errors to amc, but I can send it to dxu too if you give me the email address.). That info is on the issues@ mailing list as well I think.
Wed Nov 4 00:14:40 2015  dxu:It seems to me atm that it's just the unsued variable breaking the rest of the builds. The coverity things (at the least the ones I looked at) are just code missing NULL checks
Wed Nov 4 00:15:13 2015  zwoop:the coverity checks are a few differnet things, mostly minor, but should be fixed (such as checking return codes from system calls)
Wed Nov 4 00:15:26 2015  zwoop:the clang analyzer were missing NULL checks I think
Wed Nov 4 00:16:48 2015  dxu:zwoop: If you could send the coverity list to dlxu2@yahoo.com that would be great
Wed Nov 4 00:18:21 2015  jpeach:zwoop: ci is hosed?
Wed Nov 4 00:18:32 2015  zwoop:jpeach it is ?
Wed Nov 4 00:18:46 2015  jpeach:it doesn't load
Wed Nov 4 00:18:52 2015  zwoop:ok, I'll take a look
Wed Nov 4 00:19:33 2015  jpeach:no it is my safari that is hosed :(
Wed Nov 4 00:19:38 2015  jpeach:works with curl and chrome
Wed Nov 4 00:19:49 2015  zwoop:might be ATS :)
Wed Nov 4 00:20:05 2015  zwoop:I needed to reboot that box anyways, it did not get rebooted when Rackspace rebooted everything 5 days ago
Wed Nov 4 00:44:11 2015  zwoop:I think the CI should be up again, try it out
Wed Nov 4 00:44:18 2015  zwoop:it seems really slow though, not sure why
Wed Nov 4 00:44:40 2015  zwoop:restarting Jenkins again
Wed Nov 4 00:47:35 2015  zwoop:I'm setting up new builds for Ubuntu 15.10 btw
Wed Nov 4 02:24:41 2015  _klk_:Joined the channel
Wed Nov 4 02:29:00 2015  amc:Thanks bcall - I'm waiting for the current build to finish
Wed Nov 4 02:42:58 2015  msekimura:Joined the channel
Wed Nov 4 04:14:18 2015  leprechau:Joined the channel
Wed Nov 4 04:16:23 2015  danielxu:Joined the channel
Wed Nov 4 04:29:17 2015  psp:Joined the channel
Wed Nov 4 04:33:27 2015  psp1:Joined the channel
Wed Nov 4 04:44:12 2015  psp:Joined the channel
Wed Nov 4 04:48:49 2015  dxu:For coverity issue, CID 1338075, it's giving a FORWARD_NULL error saying that my NULL check implies that the thing I'm checking might be NULL.
Wed Nov 4 04:49:19 2015  dxu:I'm not sure why that's flagged (or why that might be a bad thing...)
Wed Nov 4 04:49:30 2015  dxu:Anyone knows what the issue means?
Wed Nov 4 04:52:20 2015  jpeach:dxu: do you have coverity access?
Wed Nov 4 04:53:42 2015  jpeach:the problem is that when you do "if (stderr_log && stderr_log->is_init())", this implies that stderr_log might me null
Wed Nov 4 04:54:02 2015  jpeach:then a few lines later stderr_log is dereferenced withou a null check
Wed Nov 4 04:55:26 2015  dxu:jpeach: I requested access ~20m ago, so not yet
Wed Nov 4 04:55:53 2015  jpeach:dxu: what email address did you use?
Wed Nov 4 04:56:29 2015  dxu:jpeach: I see. I checked the program flow, and it shouldn't be possible for stderr_log to null at that point in code. Do you think it would be better to add more null checks or remove all of them?
Wed Nov 4 04:56:36 2015  dxu:jpeach: I signed in with my github account
Wed Nov 4 04:56:56 2015  dxu:jpeach: I think my github is linked to danobi0803@gmail.com
Wed Nov 4 04:57:28 2015  jpeach:ack
Wed Nov 4 04:57:34 2015  jpeach:you need to be consistent
Wed Nov 4 04:57:55 2015  jpeach:if it can't be NULL, then add a ink_assert() and remove the additional checks
Wed Nov 4 04:58:32 2015  dxu:I've never considered that before, but that makes a lot of sense. I'll do that
Wed Nov 4 04:59:26 2015  jpeach:I think you should do the same for stdout_log
Wed Nov 4 05:14:15 2015  jpeach:dxu: still there?
Wed Nov 4 05:15:31 2015  dxu:jpeach: Yes
Wed Nov 4 05:16:07 2015  jpeach:look at Diags::print_va() ... there's an unlock() in an #if !freebsd
Wed Nov 4 05:16:21 2015  jpeach:does that seems pretty WTF to you too?
Wed Nov 4 05:17:29 2015  jpeach:oh I see ... freebsd holds the lock until after the syslog :-/
Wed Nov 4 05:17:38 2015  dxu:I think that's because of syslog stuff
Wed Nov 4 05:17:53 2015  dxu:Yeah
Wed Nov 4 05:19:43 2015  jpeach:does this look good to you http://fpaste.org/286712/14466143/ ?
Wed Nov 4 05:25:24 2015  dxu:It seems reasonable, except I think you have a typo on line 23 with the '=='
Wed Nov 4 05:25:41 2015  dxu:I'm a little short on time atm so I can't test it out
Wed Nov 4 05:25:52 2015  jpeach:yup thanks
Wed Nov 4 05:47:55 2015  psp1:Joined the channel
Wed Nov 4 06:05:23 2015  felicity:Joined the channel
Wed Nov 4 06:25:04 2015  dxu:Has there been any changes to clang-format recently? I just ran it and it (badly) formatted a bunch of files
Wed Nov 4 11:59:19 2015  ASFBot:Joined the channel
Wed Nov 4 12:00:49 2015  ASFBot:Joined the channel
Wed Nov 4 12:03:09 2015  ASFBot:Joined the channel
Wed Nov 4 12:16:06 2015  JSeymour:Joined the channel
Wed Nov 4 12:54:22 2015  malfaic:Joined the channel
Wed Nov 4 13:43:12 2015  felicity:related question: is there a way to mark a cache entry as being stale?
Wed Nov 4 13:52:52 2015  bahumbug:Joined the channel
Wed Nov 4 14:02:07 2015  shinrich1:Joined the channel
Wed Nov 4 14:12:35 2015  gancho:Joined the channel
Wed Nov 4 14:28:52 2015  davet_:Joined the channel
Wed Nov 4 14:34:06 2015  esproul:Joined the channel
Wed Nov 4 15:07:55 2015  dcarlin:felicity: maybe this plugin will help with stale while error: https://github.com/apache/trafficserver/tree/master/plugins/experimental/stale_while_revalidate
Wed Nov 4 15:08:44 2015  felicity:dcarlin: yes, i found that a little bit after i asked :)
Wed Nov 4 15:08:59 2015  dcarlin:I don't know about marching cache entry as stale
Wed Nov 4 15:09:02 2015  felicity:TS-3993
Wed Nov 4 15:09:02 2015  felicity:then filed that ^ for the other one
Wed Nov 4 15:09:02 2015  felicity:nudges ASFBot 
Wed Nov 4 15:09:54 2015  dcarlin:https://github.com/apache/trafficserver/tree/master/plugins/experimental/regex_revalidate
Wed Nov 4 15:10:17 2015  dcarlin:that sounds sorta what you want to do in ts-3993 - but not the serve while error part
Wed Nov 4 15:10:43 2015  dcarlin:but we use it to revalidate objects before their cache expiration time
Wed Nov 4 15:11:01 2015  dcarlin:not sure if there is some way to use both plugins together
Wed Nov 4 15:11:12 2015  dcarlin:regex_revalidate needs docs
Wed Nov 4 15:11:52 2015  felicity:i'm trying to work out what it does, but the source code is not very enlightening :)
Wed Nov 4 15:14:42 2015  dcarlin:so someone at yahoo was reading source to develop docs
Wed Nov 4 15:14:55 2015  dcarlin:these are notes they have so far: https://paste.apache.org/24Ia
Wed Nov 4 15:17:12 2015  sudheerv:felicity: hoody: open_write_fail_action doesn't quite replace the SWR (at least, as required by the RFC) yet
Wed Nov 4 15:17:42 2015  sudheerv:the notable things missing are: there's still one request that revalidates the cache
Wed Nov 4 15:17:54 2015  sudheerv:and the SWR headers are not supported yet
Wed Nov 4 15:18:27 2015  sudheerv:the good news is that those are easily solvable..just need to find some time to work on it
Wed Nov 4 15:19:21 2015  felicity:dcarlin: hmm, updating a file and reloading ATS every time we want to purge a page is going to be a hassle :(
Wed Nov 4 15:20:30 2015  felicity:(we probably purge pages several times per minute)
Wed Nov 4 15:20:43 2015  dcarlin:I don't think you have to reload ats
Wed Nov 4 15:20:47 2015  dcarlin:it checks file periodically
Wed Nov 4 15:20:49 2015  dcarlin:we purge a lot too
Wed Nov 4 15:20:59 2015  sudheerv:yeah, the plugin loads the config dynamically
Wed Nov 4 15:21:11 2015  sudheerv:no need to reload traffic_server
Wed Nov 4 15:21:47 2015  felicity:hmm, maybe i'll change it a bit to use a database like cache_key_genid does
Wed Nov 4 15:22:15 2015  felicity:dcarlin: how many entries do you have in the config? matching 500k+ regexps on every request sounds like it'll be slow :)
Wed Nov 4 15:24:12 2015  dcarlin:it was my understanding (man the docs suck) that once a match occurs, its removed from config file - since its been revalidated. this keeps # of regexes low
Wed Nov 4 15:24:31 2015  dcarlin:but yes, we don't have that number of entries
Wed Nov 4 15:24:32 2015  felicity:hmm, so its like a queue
Wed Nov 4 15:25:02 2015  sudheerv:dcarlin: umm..is that true? i dont think the entry is physically removed
Wed Nov 4 15:25:41 2015  sudheerv:ah, actually you are right - it doesn't remove from the config file, but, it does remove from the context
Wed Nov 4 15:27:04 2015  dcarlin:hmm
Wed Nov 4 15:27:13 2015  dcarlin:but is it removed after its hit the first time? or removed when it expires
Wed Nov 4 15:27:19 2015  sudheerv:after it expires
Wed Nov 4 15:27:49 2015  sudheerv:at least, afair - been a while since I looked at it though and I may not be accurate
Wed Nov 4 15:28:54 2015  dcarlin:Rule deletion - while reloading rules file, plugin keeps already loaded rules in memory even if they are gone from the rules file.
Wed Nov 4 15:28:54 2015  dcarlin:To handle rule deletion, the same rule is added with few minutes expiry. Plugin updates the expiry change in memory and removes from matching list when it expires.
Wed Nov 4 15:29:00 2015  dcarlin:lame
Wed Nov 4 15:29:27 2015  sudheerv:yeah, that makes sense, no?
Wed Nov 4 15:29:38 2015  sudheerv:coz, you want to keep the rule until it actually expires
Wed Nov 4 15:30:07 2015  dcarlin:if the rule was hit tho why not remove it from memory
Wed Nov 4 15:30:17 2015  dcarlin:i thought it did that
Wed Nov 4 15:30:24 2015  dcarlin:PSUdaemon would know best :)
Wed Nov 4 15:30:36 2015  sudheerv:yeah, he knows best :)
Wed Nov 4 15:31:18 2015  sudheerv:but, afaict, the reason you want to keep it until it expires is that, you want to make sure the revalidation was successful?
Wed Nov 4 15:32:02 2015  sudheerv:(but, there may already be some code to actually remove it after revalidation is successful)
Wed Nov 4 15:32:06 2015  sudheerv:docs may not be accurate
Wed Nov 4 15:33:42 2015  dcarlin:yeah since they were reverse engineered
Wed Nov 4 15:54:55 2015  jpeach:zwoop: I don't see any build emails?
Wed Nov 4 16:00:05 2015  jpeach:dcarlin: the file is relative to the config dir, not necessarily /usr/local/...
Wed Nov 4 16:06:06 2015  dcarlin:jpeach: sure, just copy/pasting doc :)
Wed Nov 4 16:06:38 2015  jpeach:could you paste it into a .rst file for the docs? ;)
Wed Nov 4 16:07:03 2015  blattj:Joined the channel
Wed Nov 4 16:07:28 2015  dcarlin:considering it
Wed Nov 4 16:07:37 2015  dcarlin:there are two docs I should merge together
Wed Nov 4 16:07:47 2015  dcarlin:not clear how accurate they are, but better than bothing
Wed Nov 4 16:07:49 2015  dcarlin:nothing
Wed Nov 4 16:08:18 2015  jpeach:well as long as they are not misleading
Wed Nov 4 16:22:36 2015  dcarlin:I'll make a PR for docs and ask PSUdaemon to review for merge
Wed Nov 4 16:33:48 2015  shinrich2:Joined the channel
Wed Nov 4 16:40:54 2015  bahumbug:Joined the channel
Wed Nov 4 16:41:05 2015  jpeach:dcarlin: thanks!
Wed Nov 4 16:42:31 2015  Becoming:Joined the channel
Wed Nov 4 16:56:29 2015  psp1:Joined the channel
Wed Nov 4 17:04:22 2015  jrushford:Joined the channel
Wed Nov 4 17:07:51 2015  amc:dxu - ping
Wed Nov 4 17:15:22 2015  es:Joined the channel
Wed Nov 4 17:17:10 2015  es1:Joined the channel
Wed Nov 4 17:21:24 2015  _klk_:Joined the channel
Wed Nov 4 17:30:37 2015  gancho:Joined the channel
Wed Nov 4 17:32:18 2015  jrushford:jpeach - I saw your comment on pull request 321 and I understand.
Wed Nov 4 17:32:49 2015  jpeach:thanks jrushford; I'm happy to help break it down if you like
Wed Nov 4 17:33:53 2015  jrushford:I could sure use the help. I've already spent alot of time. I have a 5.3.x version we're currently using and this request was for the 6.0 master branch
Wed Nov 4 17:34:03 2015  gancho_:Joined the channel
Wed Nov 4 17:34:37 2015  jrushford:over the next few days, i can chat with you on this. thanks for the help offer.
Wed Nov 4 17:35:32 2015  jpeach:hopefully on the weekend I can look at the patch and suggest some ways to carve it up
Wed Nov 4 17:35:49 2015  jrushford:okay, i'd appreciate that very much.
Wed Nov 4 17:36:09 2015  jpeach:the best case for review+commit is a series of small patches that are all "obviously correct" :)
Wed Nov 4 17:36:19 2015  jrushford:gotcha
Wed Nov 4 17:36:23 2015  jpeach:that can be challenging to construct however :)
Wed Nov 4 17:37:58 2015  jrushford:have a look and then we can get together over email or irc. I'll be at the ATS summit too and if you have time, we can go over it and discuss
Wed Nov 4 17:38:27 2015  jpeach:maybe this in an unrelated question, but could you use a series of traffic control deployments as layers in the cache heierarchy?
Wed Nov 4 17:38:44 2015  jpeach:I'll be at the summit and also at the sunday hackathon
Wed Nov 4 17:39:17 2015  jrushford:I'll be at the hackathon too.
Wed Nov 4 17:41:49 2015  jrushford:looking forward to talking with you there.
Wed Nov 4 17:44:33 2015  jpeach:so the reason I ask about traffic control is that I'm wondering whether that gives you the parent features, so I'm curious why you chose to implement them in ATS?
Wed Nov 4 17:46:01 2015  jrushford:yes, our current version of traffic control lets you configure the multi-site origin servers. It doesn't let you configure the secondary hash list yet.
Wed Nov 4 17:47:59 2015  jrushford:we had requirements from our video group to provide round robin or consistent hash selection over multiple origin end points for vod and linear video. thats why we implemented this in ats
Wed Nov 4 17:48:29 2015  jrushford:the simple retry and dead server retry were part of the requirements.
Wed Nov 4 17:49:19 2015  jrushford:i'm working on some slides as I was going to present this at ATS, I'll email you a copy of what I have so far.
Wed Nov 4 17:49:57 2015  jpeach:yeh I was thinking that if the traffic control router was the origin, then it could do the hashing, etc ... ATS would still need to to retries and redirect following
Wed Nov 4 17:54:58 2015  jrushford:jpeach - i emailed you my preliminary slide deck. Maybe its helpful but we can chat about it as you have time.
Wed Nov 4 17:55:14 2015  jpeach:oh great, thanks!
Wed Nov 4 18:00:37 2015  _klk_:Joined the channel
Wed Nov 4 18:36:16 2015  es:Joined the channel
Wed Nov 4 18:50:36 2015  _klk_:Joined the channel
Wed Nov 4 18:58:43 2015  davet:Joined the channel
Wed Nov 4 19:13:06 2015  jrushford:Joined the channel
Wed Nov 4 20:27:35 2015  _klk_:Joined the channel
Wed Nov 4 20:49:08 2015  bahumbug:Joined the channel
Wed Nov 4 20:49:08 2015  bahumbug:Joined the channel
Wed Nov 4 21:03:48 2015  _klk_:Joined the channel
Wed Nov 4 21:51:55 2015  es1:Joined the channel
Wed Nov 4 21:55:12 2015  shinrich1:Joined the channel
Wed Nov 4 22:02:07 2015  shinrich2:Joined the channel
Wed Nov 4 22:02:34 2015  _klk_:Joined the channel
Wed Nov 4 22:03:50 2015  shinrich1:Joined the channel
Wed Nov 4 22:12:14 2015  amc:dxu - ping
Wed Nov 4 22:46:03 2015  dxu:amc: I'm back
Wed Nov 4 22:50:57 2015  _klk_:Joined the channel
Wed Nov 4 23:00:36 2015  amc:Still having problems with the logging. Working on a patch.
Wed Nov 4 23:01:05 2015  amc:There also seems to be a buffering problem where the output to traffic.out is buffered. I'm going to look at that next but if you want to give me a hint thatwould be cool.
Wed Nov 4 23:01:28 2015  amc:Looks like for the elevate I will need to make it work both priviledged and unprivileded because it gets called both ways.
Wed Nov 4 23:05:03 2015  dxu:I suspect that it's a missing setvbuf() call somewhere
Wed Nov 4 23:06:16 2015  dxu:I thought I had made sure to port those calls over for the patch
Wed Nov 4 23:12:08 2015  dxu:IIRC if you open() a file, the file is atomically line buffered??
Wed Nov 4 23:12:09 2015  dxu:https://github.com/apache/trafficserver/compare/master...danobi:log-rotation-squashed#diff-772cdf9b526f76c710242c7c05047503L796
Wed Nov 4 23:12:43 2015  dxu:That line gets replaced with an fopen() call here:
Wed Nov 4 23:12:44 2015  dxu:https://github.com/apache/trafficserver/compare/master...danobi:log-rotation-squashed#diff-97e43af051300a575ae62f64032f9012R326
Wed Nov 4 23:12:52 2015  amc:I think it would be, but I'm going to set O_SYNC on it anyway.
Wed Nov 4 23:13:16 2015  amc:Tracking down the fopen() vs. setvbuf() now.
Wed Nov 4 23:13:57 2015  dxu:I don't think setvbuf() was call at all originally for traffic.out. It was only open()'d in TC
Wed Nov 4 23:14:07 2015  dxu:So it must be a difference between fopen() and open()
Wed Nov 4 23:15:03 2015  amc:Quite possibly.
Wed Nov 4 23:15:17 2015  amc:Presumably setvbuf() is effectively called on stderr/stdout by glibc.
Wed Nov 4 23:16:05 2015  amc:If you want to try it, enable debug and set the tags to "http" and run some transactions through. You'll get a lot of output.
Wed Nov 4 23:16:25 2015  amc:jrushford was the first to report this but I can see it now too.
Wed Nov 4 23:16:52 2015  amc:So, once you've observed it, you can play with setlinebuf() in BaseLogFile file opening.
Wed Nov 4 23:19:17 2015  dxu:Ok. I'll take a look at that tonight
Wed Nov 4 23:21:00 2015  amc:Just for you own edification, I ahve to get it fixed ASAP.
Wed Nov 4 23:21:24 2015  amc:except my vm host melted down, trying to get it back up. Sigh. I blame zwoop.
Wed Nov 4 23:32:35 2015  amc:The setlinebuf() seems to make it much better, without the O_SYNC on the open() call.
Wed Nov 4 23:44:49 2015  niq:Joined the channel
Wed Nov 4 23:58:29 2015  zwoop:jpeach do we have double targets for this man page?
Wed Nov 4 23:58:32 2015  zwoop: /usr/bin/install: will not overwrite just-created ‘/var/jenkins/workspace/debian_jessie-master/compiler/gcc/label/debian_jessie/type/release/1040/install/man/man8/traffic_cop.8’ with ‘docbuild/man/traffic_cop.8’
Wed Nov 4 23:58:32 2015  zwoop:Makefile:720: recipe for target 'install-man8' failed
Wed Nov 4 23:58:41 2015  zwoop:(it fails on some modern platforms)

Comments