Displaying #traffic-server/2015-10-26.log:

Mon Oct 26 00:20:44 2015  jpeach:you would end up with 2 cache entries however
Mon Oct 26 00:24:04 2015  oag:Ah, so both sides are cached?
Mon Oct 26 00:24:14 2015  oag:I checked and it does work.
Mon Oct 26 00:24:19 2015  oag:Adding a query.
Mon Oct 26 00:25:38 2015  oag:That won't work for us in general since we need to pass requests through unharmed to a bunch of backends we don't know much about.
Mon Oct 26 00:25:51 2015  oag:But it feels like progress.
Mon Oct 26 00:26:28 2015  jpeach:I guess you are intercepting because you need to transform the body in some way?
Mon Oct 26 00:26:34 2015  oag:That's right.
Mon Oct 26 00:26:45 2015  jpeach:maybe you can write a transform plugin?
Mon Oct 26 00:27:08 2015  oag:What I'm actually trying to write is a plugin which runs in Javascript using spidermonkey.
Mon Oct 26 00:27:31 2015  oag:It intercepts the incoming request and allows you , in js to make as many outgoing requests as you need to build the eventual content.
Mon Oct 26 00:27:55 2015  oag:So it isn't always strictly a transform plugin.
Mon Oct 26 00:28:04 2015  jpeach:so it is a bit like ESI but using javascript?
Mon Oct 26 00:28:09 2015  oag:Yep.
Mon Oct 26 00:28:27 2015  oag:3rd parties will write the js code.
Mon Oct 26 00:28:45 2015  oag:We should look at the lua plugin but I think JS would be more saleable.
Mon Oct 26 00:29:38 2015  jpeach:does it really look at the request? or the response body?
Mon Oct 26 00:29:50 2015  oag:It has access to both .. hang on ..
Mon Oct 26 00:30:26 2015  jpeach:just thinking that for a ESI-style feature you would embed JS in the page you are serving
Mon Oct 26 00:32:18 2015  jpeach:there is per-txn config that you can use to turn the caching off ... not sure whether is would help this specific case
Mon Oct 26 00:32:42 2015  oag:http://pastebin.com/Lb3K8iYN
Mon Oct 26 00:32:44 2015  jpeach:proxy.config.http.cache.http
Mon Oct 26 00:33:09 2015  oag:Ah I will have a look
Mon Oct 26 00:33:46 2015  jpeach:oh so the plugin runs the JS code provided by the operator?
Mon Oct 26 00:33:59 2015  oag:Yes
Mon Oct 26 00:34:16 2015  oag:There would be a layer where it chooses based on the vhost which js handler to run.
Mon Oct 26 00:35:02 2015  jpeach:ok, remap.config could do that
Mon Oct 26 00:40:00 2015  jpeach:in this example, a better approach (though more complex) might be to do a transform
Mon Oct 26 00:40:36 2015  jpeach:you would transform on the page that is being fetched, but you could still kick off sub-requests to pull in extra content
Mon Oct 26 00:42:35 2015  oag:Ah, a transform can still make http requests?
Mon Oct 26 00:43:14 2015  oag:I guess I was really feeling my way with this stuff at first. (Well I still am, but I am getting the idea that I can put the pieces together in more than one way to get this job done.)
Mon Oct 26 00:44:27 2015  jpeach:yeh a transform is basically spliced into the IO buffer chain (a bit like a libevent bufferevent object)
Mon Oct 26 00:44:52 2015  jpeach:so you can alwys read some data, parse it, do some other work and keep going
Mon Oct 26 00:46:59 2015  oag:I wonder if that relates to another slightly odd thing I noticed, which is that if I gave a made-up host on the client request, TS attempted a dns lookup and complained about it, even though from my point of view it wasn't clear that it would ever have to try to connect to the host.
Mon Oct 26 00:47:23 2015  oag:It's like TS wants to behave like a proxy even when I intercept the entire request.
Mon Oct 26 00:47:52 2015  jpeach:there's a few examples in the source tree of transforms though I've never written one myself
Mon Oct 26 00:48:20 2015  oag:I tried one myself at an early stage, but for some reason I concluded then that the intercept plugin was more flexible.
Mon Oct 26 00:48:59 2015  oag:Do you know where in the source this cache locking might be happening?
Mon Oct 26 00:49:33 2015  jpeach:wait, I think I got lost ... are calling TSHttpTxnIntercept?
Mon Oct 26 00:49:58 2015  oag:I'm using the C++ api, so for me it's adding an InterceptPlugin via a GlobalPlugin.
Mon Oct 26 00:50:20 2015  jpeach:ah, yeh that makes your plugin appear to the core as though it is an origin server
Mon Oct 26 00:50:31 2015  oag:And there is a flag, so it's a 'server' intercept or something.. like it applies after cache lookup.
Mon Oct 26 00:50:46 2015  oag:Which is what we want.
Mon Oct 26 00:50:51 2015  jpeach:yeh that's how I've done it too
Mon Oct 26 00:52:20 2015  jpeach:DNS happens after cache lookup, so I'm a little surprised you see the DNS request
Mon Oct 26 00:52:44 2015  oag:Yes, it would be good to track that down too. Let me just check that again.
Mon Oct 26 00:53:32 2015  jpeach:AFAICT the subrequests that you send should be visible to your plugin (which should avoid intercepting them since they are internal)
Mon Oct 26 00:54:46 2015  oag:Yes the DNS thing is really odd. TS replies ahead of my plugin saying the server does not have a DNS entry, but it actually does install the plugin!
Mon Oct 26 00:55:52 2015  jpeach:you see that in logs?
Mon Oct 26 00:56:02 2015  oag:TS sends a response to the client..
Mon Oct 26 00:56:13 2015  oag:curl -i -H host:oliverdman.com http://mva-dev:8080/index.html
Mon Oct 26 00:56:34 2015  oag:That host name is wrong (i could have made it more obvious!)
Mon Oct 26 00:57:09 2015  jpeach:are you installing the intercept at HOOK_READ_REQUEST_HEADERS_PRE_REMAP ?
Mon Oct 26 00:57:28 2015  oag:And I get back HTTP/1.1 502 Cannot find server. (followed by a full explanation).
Mon Oct 26 00:58:06 2015  oag:Yes, exactly.
Mon Oct 26 00:58:22 2015  jpeach:did you see https://docs.trafficserver.apache.org/en/latest/sdk/http-hooks-and-transactions.en.html#http-txn-state-diagram
Mon Oct 26 00:59:46 2015  oag:Aha! Well I guess that might explain a few things..
Mon Oct 26 01:00:55 2015  jpeach:I wonder why DNS state happens before the cache lookup hook there ... maybe zwoop knows
Mon Oct 26 01:01:16 2015  oag:I see, right, exactly, DNS happens very early (from my POV)
Mon Oct 26 01:03:18 2015  oag:Oh actually now I don't understand.
Mon Oct 26 01:03:51 2015  oag:HOOK_READ_REQUEST_HEADERS_PRE_REMAP looks like it ought to be up the top somewhere, although it doesn't actually appear in the picture.
Mon Oct 26 01:04:14 2015  oag:But since it's supposed to be post cache lookup.. I'm not sure where it comes.
Mon Oct 26 01:04:34 2015  jpeach:pre-remap happens before cache lookup
Mon Oct 26 01:05:05 2015  jpeach:it has to, because if the URL is remapped then you might end up using a different cache key
Mon Oct 26 01:08:58 2015  jpeach:fwiw you can fetch pages manually using TSNetConnect, though I don't know that I'd recommend that as a first try :)
Mon Oct 26 01:10:01 2015  jpeach:more hacky, but probably ok for a proof of concept
Mon Oct 26 01:10:25 2015  jpeach:define a remap rule trampoline that is only accessible to internal requests
Mon Oct 26 01:11:07 2015  jpeach:then when you want to send a request, use the hostname in the trampoline rule
Mon Oct 26 01:11:39 2015  jpeach:you can turn off caching in the trampoline, and also rewrite the URL to the real outbound one
Mon Oct 26 01:12:08 2015  jpeach:it is a bit convoluted but it will work and get you unblocked
Mon Oct 26 01:12:53 2015  oag:Ah, yes I see what you're saying.
Mon Oct 26 01:23:17 2015  jpeach:I gotta get dinner, let us know how you get on :)
Mon Oct 26 01:23:41 2015  oag:Cool, thanks so much for all the time and suggestions!
Mon Oct 26 01:40:36 2015  steveq:Joined the channel
Mon Oct 26 03:00:10 2015  Becoming_:Joined the channel
Mon Oct 26 03:38:03 2015  hello1:Joined the channel
Mon Oct 26 03:39:35 2015  hello1:anyone awake?
Mon Oct 26 03:47:53 2015  psp:Joined the channel
Mon Oct 26 04:52:45 2015  psp:Joined the channel
Mon Oct 26 05:21:43 2015  psp:Joined the channel
Mon Oct 26 05:59:49 2015  Becoming___:Joined the channel
Mon Oct 26 06:23:44 2015  psp:Joined the channel
Mon Oct 26 08:35:34 2015  \ask:Joined the channel
Mon Oct 26 08:40:13 2015  [Jok]:Joined the channel
Mon Oct 26 09:12:29 2015  Lethalman:Joined the channel
Mon Oct 26 10:53:32 2015  dR0M3st3R:Joined the channel
Mon Oct 26 11:16:33 2015  JSeymour:Joined the channel
Mon Oct 26 12:19:30 2015  bahumbug:Joined the channel
Mon Oct 26 12:50:45 2015  es:Joined the channel
Mon Oct 26 13:09:33 2015  shinrich1:Joined the channel
Mon Oct 26 13:11:24 2015  es:Joined the channel
Mon Oct 26 13:23:02 2015  esproul:Joined the channel
Mon Oct 26 14:38:47 2015  davet_:Joined the channel
Mon Oct 26 15:49:19 2015  bahumbug:Joined the channel
Mon Oct 26 15:55:00 2015  es:Joined the channel
Mon Oct 26 16:02:16 2015  blattj:Joined the channel
Mon Oct 26 16:05:52 2015  reveller:Joined the channel
Mon Oct 26 16:13:35 2015  msekimura:Joined the channel
Mon Oct 26 16:36:06 2015  esproul:Joined the channel
Mon Oct 26 17:11:36 2015  gancho:Joined the channel
Mon Oct 26 17:18:53 2015  shinrich1:Joined the channel
Mon Oct 26 17:23:33 2015  es:Joined the channel
Mon Oct 26 17:28:16 2015  psp1:Joined the channel
Mon Oct 26 17:53:05 2015  es:Joined the channel
Mon Oct 26 18:01:46 2015  jpeach:bcall: did you see my comments about read-only config?
Mon Oct 26 18:01:53 2015  bcall:yes
Mon Oct 26 18:02:43 2015  bcall:if you mean in the ticket
Mon Oct 26 18:03:01 2015  jpeach:I mean email to dev@ :)
Mon Oct 26 18:03:23 2015  bcall:no, I will look
Mon Oct 26 18:03:50 2015  jpeach:basically, I think that you should be able to change the setting on disk then reload
Mon Oct 26 18:04:31 2015  bcall:k
Mon Oct 26 18:04:41 2015  bcall:that makes sense
Mon Oct 26 18:06:27 2015  bcall:I updated the ticket
Mon Oct 26 18:12:51 2015  jpeach:thanks bcall
Mon Oct 26 18:15:06 2015  [Jok]:Joined the channel
Mon Oct 26 18:18:40 2015  _klk_:Joined the channel
Mon Oct 26 18:23:14 2015  danielxu:Joined the channel
Mon Oct 26 18:45:39 2015  blattj:Joined the channel
Mon Oct 26 18:57:00 2015  gasolwu:Joined the channel
Mon Oct 26 19:04:18 2015  danielxu:Joined the channel
Mon Oct 26 19:12:09 2015  _klk_:Joined the channel
Mon Oct 26 19:43:03 2015  _klk_:Joined the channel
Mon Oct 26 19:45:36 2015  es:Joined the channel
Mon Oct 26 20:02:01 2015  _klk_:Joined the channel
Mon Oct 26 20:02:48 2015  blattj:Joined the channel
Mon Oct 26 20:16:26 2015  gancho:Joined the channel
Mon Oct 26 20:30:38 2015  zwoop:bcall the multiplexer build fails
Mon Oct 26 20:30:41 2015  zwoop:In file included from ../../../../plugins/experimental/multiplexer/dispatch.cc:26:
Mon Oct 26 20:30:41 2015  zwoop:../../../../plugins/experimental/multiplexer/fetcher.h:275:22: error: implicit instantiation of undefined member 'sockaddr_in'
Mon Oct 26 20:30:41 2015  zwoop: struct sockaddr_in socket;
Mon Oct 26 20:30:41 2015  zwoop: ^
Mon Oct 26 20:30:46 2015  zwoop:on e.g. FreeBSD
Mon Oct 26 20:33:15 2015  bcall:k, I will look at it
Mon Oct 26 20:37:00 2015  zwoop:thanks
Mon Oct 26 20:41:10 2015  blattj:Joined the channel
Mon Oct 26 20:41:39 2015  blattj:Joined the channel
Mon Oct 26 20:42:24 2015  blattj:Joined the channel
Mon Oct 26 20:43:26 2015  zwoop:http://blog.streamingmedia.com/2015/10/yahoo-keynote-streaming-show.html
Mon Oct 26 20:43:45 2015  zwoop:meh, this one http://blog.streamingmedia.com/2015/10/yahoo-nfl-stream.html
Mon Oct 26 20:44:46 2015  bcall:yeah, I had to be around during the game if there was any issues
Mon Oct 26 20:45:01 2015  zwoop:double overtime pay!
Mon Oct 26 20:45:07 2015  bcall:lol
Mon Oct 26 20:45:11 2015  phibs:Joined the channel
Mon Oct 26 20:45:13 2015  bcall:it was actually a good game
Mon Oct 26 20:45:17 2015  zwoop:haha
Mon Oct 26 20:45:34 2015  zwoop:tough job, but someone had to do it ?
Mon Oct 26 20:49:25 2015  bcall:I was expecting to fight fires, but there was nothing to do beside monitor the IRC channels and watch the game
Mon Oct 26 20:50:27 2015  zwoop:cool
Mon Oct 26 20:59:40 2015  blattj:Joined the channel
Mon Oct 26 21:04:03 2015  _klk_:Joined the channel
Mon Oct 26 21:35:29 2015  jpeach:bcall: the multiplexer plugin also assumes clock_gettime is available but it is not there on OS X
Mon Oct 26 21:36:13 2015  PSUdaemon:srsly OSX, WTF?!
Mon Oct 26 21:37:00 2015  bcall:jpeach: k
Mon Oct 26 21:37:27 2015  jpeach:PSUdaemon: it's probably an optional bit of posix
Mon Oct 26 21:39:23 2015  PSUdaemon:isn't "optional standard" an oxymoron?
Mon Oct 26 21:52:31 2015  bcall:jpeach: switching it to use getimteofday - it didn't use the nsec resolution anywyas
Mon Oct 26 21:57:15 2015  _klk_:Joined the channel
Mon Oct 26 22:23:18 2015  zwoop:TS-3980
Mon Oct 26 22:23:26 2015  zwoop:sudheerv is that the bug for removing the config option ?
Mon Oct 26 22:48:00 2015  sudheerv:TS-3981
Mon Oct 26 22:48:08 2015  sudheerv:TS-3979
Mon Oct 26 22:48:15 2015  sudheerv:zwoop: I think it's TS-3979
Mon Oct 26 22:48:16 2015  sudheerv:TS-3979
Mon Oct 26 22:48:32 2015  zwoop:then what is TS-3981 vs TS-3978
Mon Oct 26 22:48:48 2015  sudheerv:TS-3981 - don't know, unrelated :)
Mon Oct 26 22:48:51 2015  sudheerv:TS-3979
Mon Oct 26 22:48:53 2015  sudheerv:TS-3980
Mon Oct 26 22:48:58 2015  sudheerv:those are the two I opened
Mon Oct 26 22:49:02 2015  sudheerv:TS-3978
Mon Oct 26 22:49:17 2015  sudheerv:I think felicity opened TS-3978
Mon Oct 26 22:49:30 2015  zwoop:the description on TS-3980 is weird
Mon Oct 26 22:49:32 2015  zwoop:"Instead of using special config settings (e.g allow_empty_doc) to control caching of empty docs, the consensus is to always cache any doc that is cacheable and safe to cache (meaning, unbroken response, including Content-Length(0), Chunked Encoding (0), SPDY/H2(0) etc)."
Mon Oct 26 22:49:41 2015  zwoop:that sounds like you want to remove the config
Mon Oct 26 22:54:13 2015  sudheerv:zwoop: yeah..so, in a way yes
Mon Oct 26 22:54:32 2015  sudheerv:but, I wanted to not say, we can not *simply* remove the config without any other changes
Mon Oct 26 22:54:36 2015  zwoop:ok
Mon Oct 26 22:54:43 2015  sudheerv:seems like I didn't capture the intent well
Mon Oct 26 22:54:53 2015  zwoop:can you edit that, and also add appropriate links to dependencies ?
Mon Oct 26 22:54:58 2015  sudheerv:what I wanted to say is, *investigate* how to remove the config *safely*
Mon Oct 26 22:55:02 2015  sudheerv:yeah, will do
Mon Oct 26 23:04:58 2015  steveq:Joined the channel
Mon Oct 26 23:19:20 2015  zwoop:thanks
Mon Oct 26 23:22:29 2015  sudheerv:TS-3979
Mon Oct 26 23:22:29 2015  sudheerv:TS-3980
Mon Oct 26 23:22:29 2015  sudheerv:zwoop: can you check now pls
Mon Oct 26 23:22:29 2015  sudheerv:TS-3979
Mon Oct 26 23:22:29 2015  sudheerv:ASFBot: what happened :)
Mon Oct 26 23:22:29 2015  sudheerv:not able to describe jiras?
Mon Oct 26 23:22:29 2015  Amaryllis:RIP ASFBot
Mon Oct 26 23:22:29 2015  akotani-:Joined the channel
Mon Oct 26 23:22:29 2015  zwoop:lol
Mon Oct 26 23:25:29 2015  sudheerv:Amaryllis: ASFBot was jus asleep apparently :)
Mon Oct 26 23:30:50 2015  _klk_:Joined the channel
Mon Oct 26 23:51:54 2015  Amaryllis:1445903488.753 0 10.190.254.90 TCP_MISS/200 257 GET http:///__ats_status_check - DIRECT/- text/html
Mon Oct 26 23:52:07 2015  Amaryllis:i mean, what do we need Host: headers for? it's not like HTTP/1.1 will ever catch on

Comments