Displaying #traffic-server/2015-11-30.log:

Mon Nov 30 00:49:21 2015  biilmann:Joined the channel
Mon Nov 30 00:51:30 2015  leprechau:Joined the channel
Mon Nov 30 01:10:17 2015  felicity:Joined the channel
Mon Nov 30 01:29:50 2015  skaji:Joined the channel
Mon Nov 30 01:32:37 2015  leprechau:Joined the channel
Mon Nov 30 02:09:14 2015  Top_Cat:Joined the channel
Mon Nov 30 03:11:00 2015  msekimura:Joined the channel
Mon Nov 30 03:38:53 2015  psp:Joined the channel
Mon Nov 30 03:43:00 2015  BlackCobra:Joined the channel
Mon Nov 30 03:52:02 2015  leprechau:Joined the channel
Mon Nov 30 03:57:21 2015  ibrezac:Joined the channel
Mon Nov 30 04:15:13 2015  felicity:Joined the channel
Mon Nov 30 04:19:06 2015  sGoico:Joined the channel
Mon Nov 30 04:22:18 2015  leprechau:Joined the channel
Mon Nov 30 04:24:01 2015  rhand:Joined the channel
Mon Nov 30 04:29:16 2015  leprechau:Joined the channel
Mon Nov 30 05:04:04 2015  sGoico:Joined the channel
Mon Nov 30 05:17:48 2015  leprechau:Joined the channel
Mon Nov 30 05:42:57 2015  leprechau:Joined the channel
Mon Nov 30 06:05:30 2015  psp:Joined the channel
Mon Nov 30 06:32:25 2015  felicity:Joined the channel
Mon Nov 30 08:11:36 2015  leprechau:Joined the channel
Mon Nov 30 08:18:04 2015  sGoico:Joined the channel
Mon Nov 30 08:21:57 2015  perry:Joined the channel
Mon Nov 30 08:21:57 2015  perry:Joined the channel
Mon Nov 30 08:27:37 2015  Humbedooh:Joined the channel
Mon Nov 30 08:28:11 2015  Humbedooh:Joined the channel
Mon Nov 30 08:54:59 2015  felicity:Joined the channel
Mon Nov 30 09:19:22 2015  felicity:Joined the channel
Mon Nov 30 09:38:04 2015  leprechau:Joined the channel
Mon Nov 30 09:47:34 2015  mturk:Joined the channel
Mon Nov 30 09:47:34 2015  mturk:Joined the channel
Mon Nov 30 09:50:04 2015  BlackCobra:Joined the channel
Mon Nov 30 10:12:00 2015  skaji:Joined the channel
Mon Nov 30 10:27:48 2015  skaji:Hi, I am writing a plugin that treats headers. If I use TSMimeHdrFieldNext() and while loop, headers are sorted by their name. Is it possible to get headers in the same order as http request?
Mon Nov 30 11:38:04 2015  JSeymour:Joined the channel
Mon Nov 30 12:55:14 2015  BlackCobra:Joined the channel
Mon Nov 30 13:47:56 2015  mturk:Joined the channel
Mon Nov 30 13:47:56 2015  mturk:Joined the channel
Mon Nov 30 13:55:11 2015  jrickman:Joined the channel
Mon Nov 30 13:56:11 2015  mturk:Joined the channel
Mon Nov 30 13:56:11 2015  mturk:Joined the channel
Mon Nov 30 14:00:06 2015  Shireikan:Joined the channel
Mon Nov 30 14:10:54 2015  shinrich1:Joined the channel
Mon Nov 30 14:11:34 2015  dcarlin:Left the channel
Mon Nov 30 14:46:36 2015  davet:Joined the channel
Mon Nov 30 15:34:35 2015  skaji:Joined the channel
Mon Nov 30 15:35:18 2015  jrickman:Joined the channel
Mon Nov 30 15:35:37 2015  jrushford:Joined the channel
Mon Nov 30 15:37:00 2015  perry:Joined the channel
Mon Nov 30 15:37:00 2015  perry:Joined the channel
Mon Nov 30 15:40:13 2015  JSeymour:Joined the channel
Mon Nov 30 15:40:40 2015  jrushford:Joined the channel
Mon Nov 30 15:45:41 2015  jrushfor1:Joined the channel
Mon Nov 30 15:46:40 2015  reveller1:Joined the channel
Mon Nov 30 15:51:46 2015  ibrezac:Joined the channel
Mon Nov 30 15:53:50 2015  esproul:Joined the channel
Mon Nov 30 15:56:34 2015  danielxu:Joined the channel
Mon Nov 30 16:04:34 2015  zwoop:Joined the channel
Mon Nov 30 16:05:04 2015  zwoop_:Joined the channel
Mon Nov 30 16:05:26 2015  zwoop:Joined the channel
Mon Nov 30 16:06:29 2015  zwoop:Did FreeNode do something strange? My session through the ZNC server didn't succeed until I modified my ZNC settings to s/irc.freenode.org/chat.freenode.org/
Mon Nov 30 16:08:09 2015  shinrich1:dcarlin was having issues.
Mon Nov 30 16:08:38 2015  reveller:Last week, I was finally able to deploy trafficserver-6.0.0 to our production environment. During testing, I noticed that I could reduce my remap.config down to a single line - a simple regex_map that just directs all https traffic to an http backend. About an hour after the 6.0.0 deployment and remap.config change to a single line, I got reports of service impacting slowness and had to revert the entire deployment. All traffic through https was
Mon Nov 30 16:09:16 2015  reveller:Could it have been this simpler remap config, or something I have mis-configured in 6.0.0? Any thoughts?
Mon Nov 30 16:10:44 2015  amc:Joined the channel
Mon Nov 30 16:10:55 2015  bcall:Joined the channel
Mon Nov 30 16:11:26 2015  zwoop:I doubt it's the simpler remap, if it's only one regex that shouldn't be bad
Mon Nov 30 16:12:57 2015  sudheerv:Joined the channel
Mon Nov 30 16:14:34 2015  reveller:zwoop: that was my thought - going from a remap config file with 26k lines of config to a single line shouldn't have been a problem. Granted, each lookup would have to go through a regex, but that compared to a lookup across 26k entires, I should come out ahead.
Mon Nov 30 16:15:28 2015  jpeach:Joined the channel
Mon Nov 30 16:15:49 2015  zwoop:if those 26K lines are all unique host names (like, poppaspopsicles.com), the number of host lines in the file doesn't really matter (other than maintenance). It's a hash-map so O(1) on the lookup. But yes, a single regex should be just fine.
Mon Nov 30 16:16:28 2015  zwoop:the way "map" works is that it hashes the hostname, and then uses a trie if there are more than one map rule for a host (i.e. multiple prefixes)
Mon Nov 30 16:17:17 2015  dcarlin1:Joined the channel
Mon Nov 30 16:17:23 2015  dcarlin1:yay
Mon Nov 30 16:17:25 2015  amc:Noooo! He found us!
Mon Nov 30 16:17:59 2015  dcarlin1:adams.freenode.net works, asimov.freenode.net does not
Mon Nov 30 16:18:35 2015  zwoop:dcarlin1 I changed to chat.freenode.org and then it worked
Mon Nov 30 16:18:44 2015  zwoop:meh
Mon Nov 30 16:18:49 2015  zwoop:chat.freenode.net
Mon Nov 30 16:19:00 2015  dcarlin:znc isn't working
Mon Nov 30 16:19:18 2015  dcarlin:on asimov I saw only yahoo japan people :) seems like a netsplit
Mon Nov 30 16:19:46 2015  zwoop:dcarlin it worked for me when I changed from irc.freenode to chat.freenode in the settings on the ZNC server
Mon Nov 30 16:19:55 2015  dcarlin:ah, neat
Mon Nov 30 16:20:00 2015  briang:Joined the channel
Mon Nov 30 16:20:00 2015  briang:Joined the channel
Mon Nov 30 16:20:11 2015  dcarlin:briang! :)
Mon Nov 30 16:20:22 2015  zwoop:I even rebooted the ZNC server thinking something was broken there, but it looks like something is b0rken on Freenode ?
Mon Nov 30 16:21:14 2015  dcarlin:their twitter account mentions ddos on friday, maybe something going on today too
Mon Nov 30 16:21:43 2015  zwoop:dcarlin ah
Mon Nov 30 16:21:56 2015  zwoop:well, the official docs page says to use chat.freenode.net
Mon Nov 30 16:22:02 2015  dcarlin:why we can't have nice things
Mon Nov 30 16:22:05 2015  zwoop:https://freenode.net/irc_servers.shtml
Mon Nov 30 16:22:50 2015  zwoop:reveller so, we should figure out why your site collapsed with 6.0.0, that's not good :/
Mon Nov 30 16:23:34 2015  zwoop:amc I'm starting to think we'll have to wait with 6.1.0 until Feb / March, until we can get some stability here... it seems 6.0.0 isn't even usable ?
Mon Nov 30 16:23:49 2015  amc:I thought some people were running on it.
Mon Nov 30 16:24:01 2015  zwoop:who ?
Mon Nov 30 16:24:29 2015  amc:Aren't you guys?
Mon Nov 30 16:24:44 2015  dcarlin-:Joined the channel
Mon Nov 30 16:26:38 2015  zwoop:nope
Mon Nov 30 16:27:22 2015  reveller:I thought "all the cool kids" were running 6.0.x and I was the one lagging.
Mon Nov 30 16:27:36 2015  zwoop:You're the coolest kid on the block
Mon Nov 30 16:28:22 2015  amc:Allright, time for my morning of meetings.
Mon Nov 30 16:28:33 2015  reveller:Only on my little island ;)
Mon Nov 30 16:28:53 2015  bcall:I have one server running master :)
Mon Nov 30 16:29:36 2015  zwoop:bcall well, at this point, I'm not sure it makes sense to do 6.1.0 in Mid January. Wdyt ?
Mon Nov 30 16:33:06 2015  felicity:we're running on 6.0... aside from stale-while-revalidate being broken and a nasty memory leak it seems fine. (i suspect the memory leak may be due to our local patches as i haven't seen anyone else report it...)
Mon Nov 30 16:33:22 2015  Humbedooh:uh oh
Mon Nov 30 16:33:27 2015  Humbedooh:zwoop is alive and awake
Mon Nov 30 16:34:09 2015  zwoop:felicity well, that's good (curious aboutt he leak though, if you can verify if that's upstream or not, it'd be great)
Mon Nov 30 16:36:32 2015  amc:So, having spent some time working on the internal RPC logic, I have to say it's ugly even for TS.
Mon Nov 30 16:36:33 2015  bcall:zwoop: I think the adoption rate would be better on a 6.1.0 release - I think people are afraid of the 6.0.0
Mon Nov 30 16:36:53 2015  amc:I've been looking at alternatives, mainly Thrift / ProtocolBuffers / Avro
Mon Nov 30 16:36:54 2015  jpeach:amc: ?
Mon Nov 30 16:37:11 2015  bcall:zwoop: I would create the branch in mid December and do a release in January as planned
Mon Nov 30 16:37:28 2015  amc:Well, for one you have to touch so many files to make it work.
Mon Nov 30 16:37:47 2015  amc:Second there's really no RPC from TM to TS - I had to hand roll that to make it work for something other than a simple string.
Mon Nov 30 16:37:47 2015  jpeach:amc: yeh, protobuf would work fine ... if you remember I floated that about a year ago
Mon Nov 30 16:37:49 2015  reveller:I will be doing some more testing in our load test environment. If anyone has any thoughts on items to test, I would be very interested in any input.
Mon Nov 30 16:38:16 2015  jpeach:protobuf doesn't give you RPC, you still need to build that yourself
Mon Nov 30 16:38:20 2015  zwoop:bcall ok
Mon Nov 30 16:38:44 2015  amc:Yes, but I presume having the pipes / named sockets is enough. All we really need is serialization support.
Mon Nov 30 16:39:08 2015  jpeach:the message marshalling stuff is fine for structs, but it hurts that it doesn't do lists and nested messages
Mon Nov 30 16:39:13 2015  amc:I liked Avro because it was more flexible/dynamic and Apache, but it requires Boost.
Mon Nov 30 16:39:26 2015  jpeach:we do have serialization support; I added it
Mon Nov 30 16:39:41 2015  amc:But it doesn't seem available in traffic_server.
Mon Nov 30 16:39:50 2015  jpeach:admittedly it is quite primitive
Mon Nov 30 16:39:52 2015  zwoop:Avro seems *way* overkill
Mon Nov 30 16:40:00 2015  zwoop:Avro solves a lot of problems that we do not have
Mon Nov 30 16:40:06 2015  jpeach:no, it is available in traffic_server; it was written to be separable
Mon Nov 30 16:40:06 2015  zwoop:CBOR
Mon Nov 30 16:40:20 2015  zwoop:https://tools.ietf.org/html/rfc7049
Mon Nov 30 16:40:21 2015  amc:Depends on how much work it is to add. But I'm now in a meeting.
Mon Nov 30 16:40:22 2015  zwoop:that
Mon Nov 30 16:40:36 2015  amc:jpeach - I'll look at that, but it didn't compile for me.
Mon Nov 30 16:41:07 2015  zwoop:or protobuffs
Mon Nov 30 16:41:14 2015  felicity:actually, i just disabled stale-while-revalidate and it looks like the memory leak has gone away
Mon Nov 30 16:41:22 2015  jpeach:if you need to marshall something, use mgmt/utils/MgmtMarshall.h
Mon Nov 30 16:41:38 2015  zwoop:Avro relies on "schemas", I don't see why that has value to us
Mon Nov 30 16:42:15 2015  jpeach:seriously, I added marshalling that takes care of the simple cases
Mon Nov 30 16:42:26 2015  zwoop:nod
Mon Nov 30 16:43:49 2015  felicity:https://people.torchbox.com/~felicity/mem.png < you can probably spot where i disabled stale-while-revalidate this morning
Mon Nov 30 16:44:25 2015  zwoop:felicity so the leak is related to stale-while-revalidate ?
Mon Nov 30 16:44:32 2015  felicity:zwoop: yes
Mon Nov 30 16:44:48 2015  felicity:zwoop: it's the 6.0 version with a custom patch for TS-4009
Mon Nov 30 16:44:50 2015  felicity:TS-4009
Mon Nov 30 16:44:55 2015  zwoop:gotcha
Mon Nov 30 16:45:01 2015  zwoop:that sucks
Mon Nov 30 16:45:13 2015  zwoop:felicity and that leak was not there in 5.3.x ?
Mon Nov 30 16:45:26 2015  felicity:(our patch: https://github.com/torchbox/trafficserver/commit/bab20102d7d95f6a36c4c0b1824ad9ad3afab793)
Mon Nov 30 16:45:36 2015  felicity:zwoop: we didn't use it in 5.3, so i'm not sure
Mon Nov 30 16:45:43 2015  zwoop:gotcha
Mon Nov 30 16:46:38 2015  felicity:do i need to free the mutex when the continuation is finished? if it's that simple, i can test it again easily enough
Mon Nov 30 16:46:43 2015  zwoop:felicity just a quick glance, I see a lot of mutex = TSMutexCreate(); but no Destroy ?
Mon Nov 30 16:46:47 2015  zwoop:yes
Mon Nov 30 16:46:49 2015  felicity:(i based it on the patch from that bug which doesn't free it)
Mon Nov 30 16:46:52 2015  felicity:aha
Mon Nov 30 16:46:54 2015  zwoop:pretty sure at least
Mon Nov 30 16:47:05 2015  zwoop:jpeach she'd have to TSMutexDestroy() those, no ?
Mon Nov 30 16:47:48 2015  zwoop:hmmm
Mon Nov 30 16:47:50 2015  zwoop:maybe not
Mon Nov 30 16:47:56 2015  jpeach:the continuation takes ownership of the mutex (ref count)
Mon Nov 30 16:48:09 2015  zwoop:yeah
Mon Nov 30 16:48:09 2015  jpeach:which is why there was never a destroy API initiallt
Mon Nov 30 16:48:23 2015  jpeach:TSContCreate(consume_resource, TSMutexCreate())
Mon Nov 30 16:48:40 2015  zwoop:yah
Mon Nov 30 16:49:02 2015  zwoop:felicity so, that should be ok
Mon Nov 30 16:49:11 2015  felicity:since stale-while-revalidate was presumably never tested in 6.0 (given it crashes out of the box) i imagine it's possible some other API change caused the leak?
Mon Nov 30 16:49:13 2015  felicity:(i can't imagine this leak was in 5.3 as it makes it pretty unusable)
Mon Nov 30 16:50:28 2015  zwoop:it's strange
Mon Nov 30 16:50:30 2015  felicity:i see at least one other possible leak in a quick glance: fetch_resource() allocates a mutex and never does anything with it (including freeing it)
Mon Nov 30 16:50:45 2015  felicity:oh. no, never mind
Mon Nov 30 16:51:58 2015  zwoop:felicity yeah, all those TSMutexCreate() in the patch are transferred to the continuation it seems.
Mon Nov 30 16:52:24 2015  felicity:yeah
Mon Nov 30 16:52:25 2015  felicity:is there a way to get some debug output with where allocations were made?
Mon Nov 30 16:54:53 2015  amc:felicity - I'd ask about ASAN. I know we found another mutex leak using that.
Mon Nov 30 16:55:10 2015  amc:We're doing a lot of work internally at Y! to get that field deployable.
Mon Nov 30 16:56:40 2015  felicity:it may be something to do with our usage pattern, we do ESI, so once an ESI resource becomes stale, it instantly gets about 40 req/s trying to fetch the same resource. maybe that's a less well tested code path in stale-while-revalidate
Mon Nov 30 16:56:47 2015  amc:jpeach - There's also something screwy about having to a define a function that's never called.
Mon Nov 30 16:57:13 2015  jpeach:linker dependencies need to be broken
Mon Nov 30 16:57:25 2015  jpeach:in some cases you can get away without doing that
Mon Nov 30 16:57:40 2015  jpeach:or figure out how to break the dependencies :)
Mon Nov 30 16:57:44 2015  amc:felicity - see if you can ping shinrich1 - I think she just worked through an issue with the SWR plugin and thread scheduling stuff.
Mon Nov 30 16:58:18 2015  amc:jpeach - Jason is working feverishly on that (which is what he talked about at the summit).
Mon Nov 30 16:58:35 2015  felicity:shinrich1: ping :-)
Mon Nov 30 16:58:45 2015  jpeach:yeh I strongly tried to tell him that a big bang approach is not desirable
Mon Nov 30 16:59:17 2015  amc:Yes, we'll be having him make some smaller scale changes to test the waters.
Mon Nov 30 17:00:54 2015  blattj:Joined the channel
Mon Nov 30 17:01:06 2015  shinrich2:Joined the channel
Mon Nov 30 17:05:21 2015  amc:shinrich2 is in a meeting with me now. She'll be distracted until 3pm CST or so.
Mon Nov 30 17:05:46 2015  amc:IIRC it was due to SWR scheduling on ET_TASK instead of ET_NET causing a crash.
Mon Nov 30 17:06:19 2015  es:Joined the channel
Mon Nov 30 17:07:02 2015  felicity:ah, i don't think we've seen that one
Mon Nov 30 17:07:16 2015  felicity:i may hold off putting it back into production though :)
Mon Nov 30 17:08:25 2015  biilmann:Joined the channel
Mon Nov 30 17:09:05 2015  es1:Joined the channel
Mon Nov 30 17:22:12 2015  amc:jpeach - any reason to keep that marshalling logic in mgmt and not move it to lib/rpc ?
Mon Nov 30 17:23:02 2015  jpeach:no, that could be done
Mon Nov 30 17:27:30 2015  amc:I think that would help in breaking some of the linker cycles. If it's really independent then it should be split out of having to link any mgmt code.
Mon Nov 30 17:31:11 2015  biilmann:Joined the channel
Mon Nov 30 17:34:58 2015  jpeach:since it is alrady independent, it doesn't really break the linker cycles
Mon Nov 30 17:36:04 2015  jpeach:the problem is caused by CoreAPI stuff
Mon Nov 30 17:39:00 2015  zwoop:felicity you can get a dump of the freelist memory usage, see if your memory is in there.
Mon Nov 30 17:39:20 2015  zwoop:felicity you also should look into the work sudheerv has put into this, which might make your use of the plugin obsolete
Mon Nov 30 17:41:35 2015  psp:Joined the channel
Mon Nov 30 17:42:05 2015  psp1:Joined the channel
Mon Nov 30 17:43:50 2015  psp2:Joined the channel
Mon Nov 30 18:13:04 2015  blattj:Joined the channel
Mon Nov 30 18:17:07 2015  Shireikan:Joined the channel
Mon Nov 30 18:22:55 2015  es:Joined the channel
Mon Nov 30 18:37:12 2015  kichan:Joined the channel
Mon Nov 30 18:42:01 2015  perry:Joined the channel
Mon Nov 30 18:42:01 2015  perry:Joined the channel
Mon Nov 30 18:45:35 2015  oag:Joined the channel
Mon Nov 30 18:58:45 2015  phibs:Joined the channel
Mon Nov 30 19:07:39 2015  oag:Joined the channel
Mon Nov 30 19:17:10 2015  biilmann:Joined the channel
Mon Nov 30 19:22:25 2015  danielxu:TS-306 can probably be closed now
Mon Nov 30 19:33:14 2015  amc:danielxu - done.
Mon Nov 30 19:33:38 2015  zwoop:amc see my comment on the actual Jira :)
Mon Nov 30 19:33:47 2015  zwoop:TS-4043
Mon Nov 30 19:33:50 2015  zwoop:^
Mon Nov 30 19:34:31 2015  amc:I was looking at TS-306. Hold on.
Mon Nov 30 19:34:39 2015  zwoop:I also emailed dev@, we have to decide where discussions should be tracked, to avoid split discussions (since the sync from github is uni-directional to Jira). Maybe the answer is to comment only on Github, and rely the Github -> Jira sync?
Mon Nov 30 19:45:24 2015  _klk_:Joined the channel
Mon Nov 30 19:50:47 2015  amc:Hmmm. I can see preferring using JIRA, the one point of github being better is commenting on specific pieces of code.
Mon Nov 30 19:54:19 2015  zwoop:agreed
Mon Nov 30 19:54:31 2015  zwoop:I'm ok either way honestly, we should just make a decision
Mon Nov 30 20:02:14 2015  _klk_:Joined the channel
Mon Nov 30 20:03:33 2015  dcarlin:will traffic_line -x pickup new plugins in plugins.config ?
Mon Nov 30 20:16:44 2015  danielxu:zwoop: It looks like I read the outdated RFC for TS-4043, I'll update it soon
Mon Nov 30 20:20:57 2015  amc:dcarlin - I think that's unlikely.
Mon Nov 30 20:26:20 2015  amc:zwoop, bcall - does H2 pay attention to IP allow and its methods?
Mon Nov 30 20:27:39 2015  dcarlin:amc: OK, that was one of the reasons I prefer the per-ip debugging in core
Mon Nov 30 20:50:20 2015  bcall:amc: no
Mon Nov 30 20:50:34 2015  bcall:there is a bug open for it
Mon Nov 30 20:52:21 2015  amc:I'm trying to convince Susan to fix it in her 3612 work. Can you link that other bug to 3612?
Mon Nov 30 20:52:41 2015  amc:I presume this means it should be done.
Mon Nov 30 21:09:45 2015  jpeach:amc: https://github.com/apache/trafficserver/blob/master/proxy/http2/Http2SessionAccept.cc#L41
Mon Nov 30 21:15:21 2015  swoc:Joined the channel
Mon Nov 30 21:16:52 2015  _klk_:Joined the channel
Mon Nov 30 21:17:14 2015  yamc:Joined the channel
Mon Nov 30 21:20:42 2015  Shireikan:Joined the channel
Mon Nov 30 21:40:05 2015  jacksontj:Joined the channel
Mon Nov 30 22:02:34 2015  _klk_:Joined the channel
Mon Nov 30 22:47:05 2015  jrickman:Joined the channel
Mon Nov 30 22:48:19 2015  _klk_:Joined the channel
Mon Nov 30 22:48:34 2015  biilmann:Joined the channel
Mon Nov 30 23:13:45 2015  shinrich1:Joined the channel
Mon Nov 30 23:15:31 2015  shinrich2:Joined the channel
Mon Nov 30 23:30:46 2015  masaori:Joined the channel
Mon Nov 30 23:50:56 2015  masaori_:Joined the channel

Comments