Displaying #maven-dev/2017-02-19.log:

Sun Feb 19 08:26:25 2017  Michael-O:Joined the channel
Sun Feb 19 10:30:25 2017  tibor_:Joined the channel
Sun Feb 19 10:34:16 2017  tibor_:I still do not see branch 2.19.2-experimental in GitHub.
Sun Feb 19 10:38:03 2017  Michael-O:GitHub is async I think
Sun Feb 19 10:38:42 2017  tibor_:Michael-O: I tried to clone the branch successfully. Sometimes a push to a new branch does not appear in remote server.
Sun Feb 19 10:39:04 2017  Michael-O:wip-us-git should show up immediately
Sun Feb 19 10:39:26 2017  tibor_:yes i used our asf git repo
Sun Feb 19 10:40:00 2017  tibor_:I hard coded ForkModeIT - faster - 80 seconds tests
Sun Feb 19 10:40:44 2017  tibor_:now there is 1 sec delay between sending Z event and exit(0)
Sun Feb 19 10:41:01 2017  rfscholte:Joined the channel
Sun Feb 19 10:41:15 2017  tibor_:if the test will pass I have to additionally read the pending data
Sun Feb 19 10:41:32 2017  Michael-O:check my latest findings regarding exit and flush
Sun Feb 19 10:41:39 2017  tibor_:i will
Sun Feb 19 10:43:29 2017  Michael-O:basically, we need an unbuffered version of stdio I think
Sun Feb 19 10:44:43 2017  tibor_:I and users wanted to use separate file i/o for intra process communication. Christian Rosenwold was skeptical, that it is more risky than std/out, and realistically JUnit3 throws SecurotyException if using file system. His advice to open file stream before junit starts, possible.
Sun Feb 19 10:45:09 2017  tibor_:unbuffered version?
Sun Feb 19 10:45:20 2017  tibor_:but the stream is embedded injre
Sun Feb 19 10:45:54 2017  tibor_:why to unbuffer stdin in Utils?
Sun Feb 19 10:46:26 2017  tibor_:one Thread is reader and second closes the stream
Sun Feb 19 10:46:31 2017  Michael-O:http://stackoverflow.com/q/11138355/696632
Sun Feb 19 10:46:49 2017  Michael-O:we work with pipes and child processes, it is likely subject to this
Sun Feb 19 10:47:01 2017  Michael-O:from without Java we cannot set setbuf on stdio
Sun Feb 19 10:48:21 2017  Michael-O:of course an additing \n after Z might help also
Sun Feb 19 10:51:28 2017  tibor_:we add \n
Sun Feb 19 10:51:56 2017  tibor_:if we flush the stream after Z it would not propagate the data to the pipe?
Sun Feb 19 10:52:19 2017  tibor_:Did you start the build?
Sun Feb 19 10:53:15 2017  tibor_:Is there this guarantee of happens-before-after relationship?
Sun Feb 19 10:54:15 2017  tibor_:Child: write Z, write flush, exit. Parent process: reads, waitForProcessExit, read from pipe unread data
Sun Feb 19 10:55:55 2017  tibor_:If we do not have the guarantee to read unread data after exit, then we cannot support freebsd and we should rework surefire to use other mechanism
Sun Feb 19 10:55:57 2017  Michael-O:According to my research, if stdout is a pipe (which is here) and you do write Z which is in the buffer and immediate terminate the child process you are likely to lose the buffered content
Sun Feb 19 10:56:53 2017  Michael-O:I highly doubt that you can read data from a process via pipe after the process is gone, you would have broken pipe issue.
Sun Feb 19 10:57:20 2017  Michael-O:This issue is not limited to FreeBSD, Christian noted on the list that sporadic failures also happen on Jenkins
Sun Feb 19 10:57:24 2017  tibor_:we can add a mechanism to confirm events with short timeout
Sun Feb 19 10:57:40 2017  Michael-O:Some NodeJS guy noted that you have to drain the pipe before exiting the process
Sun Feb 19 10:58:27 2017  Michael-O:http://unix.stackexchange.com/q/145526/40618
Sun Feb 19 10:58:28 2017  tibor_:this means to read out all data on the side of parent, right?
Sun Feb 19 10:58:39 2017  Michael-O:Yes, I guess so.
Sun Feb 19 10:59:00 2017  Michael-O:Currently, have no guaratnee that all data arrives like with TCP
Sun Feb 19 10:59:09 2017  tibor_:yes
Sun Feb 19 10:59:38 2017  tibor_:pls try to run the build. 1 sec before exit is long period to procee the buffer by master
Sun Feb 19 11:00:08 2017  tibor_:I think confirmation of sending Z can make it
Sun Feb 19 11:00:33 2017  tibor_:of course we send Z if exit = 0, so 1 does not
Sun Feb 19 11:00:43 2017  tibor_:we should add extra event
Sun Feb 19 11:01:12 2017  tibor_:because ForkStarter is sensitive to Z if exit 0
Sun Feb 19 11:01:54 2017  Michael-O:Can we unconditionally request a confirmation on both sides for Z before exiting?
Sun Feb 19 11:03:10 2017  Michael-O:I have another build running current, will run your new branch as soon as possible
Sun Feb 19 11:03:33 2017  tibor_:we do not have such code to call it yet. We have to write it and some waiting for the event to arrive.
Sun Feb 19 11:04:42 2017  tibor_:From the link you sent me, socket is bad choice as well.
Sun Feb 19 11:06:42 2017  Michael-O:so looks stdout to me
Sun Feb 19 11:06:53 2017  tibor_:No sorry socket is waiting to drain the buffer, but port can be used by other proces which is another issue.
Sun Feb 19 11:06:55 2017  Michael-O:maybe a unix domain socket is the best way to solve this
Sun Feb 19 11:11:09 2017  tibor_:This is a nightmare for me, but in surefire it is not the first time for me. The worst is that i am a blocker for Maven.
Sun Feb 19 11:13:24 2017  Michael-O:Joined the channel
Sun Feb 19 11:13:31 2017  Michael-O:http://carminedimascio.com/2014/01/named-pipes-with-java/
Sun Feb 19 11:23:09 2017  Michael-O:tibor_: tests are running now
Sun Feb 19 11:23:22 2017  tibor_:thank you
Sun Feb 19 11:24:16 2017  tibor_:I am just reading the lates article.
Sun Feb 19 11:25:31 2017  tibor_:we should use isolated bidirectional pipes. The threads should consistently write lines, one after another.
Sun Feb 19 11:32:17 2017  Michael-O:build done, see new tarball
Sun Feb 19 11:40:53 2017  Michael-O_:Joined the channel
Sun Feb 19 11:40:57 2017  tibor_:Tests run: 12, Failures: 0, Errors: 3, Skipped: 0
Sun Feb 19 11:41:06 2017  tibor_:maybe 1 sec is short
Sun Feb 19 11:42:52 2017  Michael-O:add 3
Sun Feb 19 11:43:37 2017  tibor_:non
Sun Feb 19 11:43:40 2017  tibor_:good news
Sun Feb 19 11:43:41 2017  tibor_:ThreadedStreamConsumer#23504796 run() :: items :: Z,0,BYE!
Sun Feb 19 11:43:50 2017  tibor_:we received Z
Sun Feb 19 11:44:37 2017  tibor_:now I have to find why ForkStarter is faster to check the Z event
Sun Feb 19 11:45:29 2017  tibor_:because ForkStarter does not see this event for some reason, it is stored in volatile variable, so maybe timing issue
Sun Feb 19 11:49:29 2017  Michael-O:Joined the channel
Sun Feb 19 11:50:54 2017  Michael-O:Joined the channel
Sun Feb 19 11:51:15 2017  Michael-O:tibor_: I have currently problems connecting to the internet
Sun Feb 19 11:51:30 2017  Michael-O:"we received Z" this was the last message I received from you
Sun Feb 19 11:52:05 2017  tibor_:I go for lunch, enjoy yours.
Sun Feb 19 11:56:13 2017  tibor_:now I have to find why ForkStarter is faster to check the Z event
Sun Feb 19 11:56:20 2017  tibor_:because ForkStarter does not see this event for some reason, it is stored in volatile variable, so maybe timing issue
Sun Feb 19 11:57:10 2017  tibor_:there are two async threads, and one of them throws exception if the first did not set volatile variable to true that Z appears
Sun Feb 19 11:57:22 2017  tibor_:so we should synchronize these
Sun Feb 19 11:57:43 2017  tibor_:ForkClient thread is processing these events
Sun Feb 19 11:58:05 2017  tibor_:and ForkStarter additionally reads this variable
Sun Feb 19 11:58:52 2017  tibor_:the code is built on the assumtion that finished procees finished processing the events, which is not guaranteed
Sun Feb 19 11:59:05 2017  Michael-O:Joined the channel
Sun Feb 19 14:56:31 2017  tibor_:Joined the channel
Sun Feb 19 15:10:12 2017  Michael-O:Joined the channel
Sun Feb 19 15:14:39 2017  Michael-O_:Joined the channel
Sun Feb 19 15:41:56 2017  Michael-O_:Joined the channel
Sun Feb 19 16:17:02 2017  Michael-O:tibor_: I am back
Sun Feb 19 16:17:05 2017  Michael-O:see updated tarball
Sun Feb 19 16:17:13 2017  Michael-O:failures are really less
Sun Feb 19 16:17:18 2017  Michael-O:so the delay does work
Sun Feb 19 16:17:34 2017  Michael-O:should we increase the delay?
Sun Feb 19 16:17:36 2017  tibor_:I place the delay after close()
Sun Feb 19 16:17:42 2017  tibor_:It should be before :)
Sun Feb 19 16:18:05 2017  tibor_:How long time you spent with testing?
Sun Feb 19 16:18:08 2017  tibor_:now
Sun Feb 19 16:18:34 2017  tibor_:Is it a problem to repeat?
Sun Feb 19 16:18:59 2017  Michael-O:ForkModeIT needs 7 min to run
Sun Feb 19 16:19:13 2017  Michael-O:shall I increase wait time?
Sun Feb 19 16:20:29 2017  tibor_:no
Sun Feb 19 16:20:34 2017  tibor_:1 sec is enough
Sun Feb 19 16:20:46 2017  tibor_:pls git pull
Sun Feb 19 16:21:00 2017  Michael-O:btw, I have another machien with FreeBSD 11-STABLE, very old. It passed ForkModeIT
Sun Feb 19 16:21:18 2017  tibor_:ForkModeIT is enough for this test
Sun Feb 19 16:21:32 2017  tibor_:I think we have it
Sun Feb 19 16:21:55 2017  Michael-O:rerunning tests now on two distinct machines
Sun Feb 19 16:22:39 2017  tibor_:ok, so I will meanwhile make nice code with semaphore and without these delays
Sun Feb 19 16:23:34 2017  tibor_:when I am writing multithreading code it takes hours, unlike normal.
Sun Feb 19 16:23:51 2017  Michael-O:multithreading code is really challenging
Sun Feb 19 16:24:04 2017  Michael-O:we will have a result in 10 min
Sun Feb 19 16:24:40 2017  tibor_:you know, even the call sequences matter, what happens before, reordering, guarantees, memory model
Sun Feb 19 16:31:05 2017  Michael-O:Joined the channel
Sun Feb 19 16:40:12 2017  Michael-O_:Joined the channel
Sun Feb 19 16:46:01 2017  Michael-O:tibor_: rerunning all IT again with your branch
Sun Feb 19 16:46:10 2017  Michael-O:it should be much less failures/errors
Sun Feb 19 16:46:15 2017  tibor_:thx
Sun Feb 19 16:47:08 2017  Michael-O:if you think you will have something testable today, let me know!
Sun Feb 19 16:53:01 2017  Michael-O:Joined the channel
Sun Feb 19 16:56:10 2017  Michael-O_:Joined the channel
Sun Feb 19 17:06:46 2017  Michael-O:Joined the channel
Sun Feb 19 18:04:26 2017  Michael-O:tibor_: I have updated the tarball, all ITs have run. We now have even more test failures
Sun Feb 19 18:04:32 2017  Michael-O:Please have look
Sun Feb 19 18:45:55 2017  Michael-O:Joined the channel
Sun Feb 19 19:27:40 2017  rfscholte:Joined the channel
Sun Feb 19 20:11:21 2017  tibor_:Michael-O: Theread surefire-forkedjvm-command-thread in class CommandReader either does not run or does not receive any data. That's why tests with reuse "ForkModeIT_**One|TwoReuse" dont'receve what to run and : [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
Sun Feb 19 20:12:20 2017  Michael-O:Is this thread never spawned?
Sun Feb 19 20:12:29 2017  Michael-O:Can we check that somehow?
Sun Feb 19 20:17:10 2017  Michael-O:tibor_: I have an idea.
Sun Feb 19 20:17:18 2017  tibor_:you can check it by jstack
Sun Feb 19 20:17:25 2017  tibor_:or i can print it
Sun Feb 19 20:17:31 2017  Michael-O:I just checked the code of java.lang.System
Sun Feb 19 20:17:35 2017  Michael-O:print is easier
Sun Feb 19 20:17:55 2017  Michael-O:out is initalized with a FileDescriptor and wrapped with a buffered stream
Sun Feb 19 20:18:15 2017  tibor_:println flushes stream or?
Sun Feb 19 20:18:23 2017  Michael-O:what if we repace that with a new stream to FileDescriptor#out and us an unbuffered OutputStream
Sun Feb 19 20:18:48 2017  Michael-O:look here: http://www.grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/java/lang/System.java#1185
Sun Feb 19 20:18:57 2017  tibor_:it's native stream
Sun Feb 19 20:19:26 2017  tibor_:we can but it will be a warning and meybe exception, see ForkCliant switch-case and catch afterwards
Sun Feb 19 20:19:57 2017  tibor_:strange is that flush goes only sometimes
Sun Feb 19 20:20:07 2017  Michael-O:No, it is a wrapper to FileDescriptor#out
Sun Feb 19 20:20:08 2017  tibor_:OutputStreamFlushReceiver flush OutputStream
Sun Feb 19 20:20:20 2017  Michael-O:the printwriter does some buffering too which we can remove for us
Sun Feb 19 20:21:32 2017  tibor_:essentially you say that FileDescriptor#out would bypass Java bufferring
Sun Feb 19 20:22:09 2017  tibor_:is Utils with flush still there?
Sun Feb 19 20:22:29 2017  Michael-O:Yes, because the buffering is happening in the PrintSteam attached to out
Sun Feb 19 20:22:43 2017  Michael-O:Yes, I used utls 0.9.1-SNAPSHOT
Sun Feb 19 20:24:30 2017  tibor_: synchronized (this) { ensureOpen(); textOut.newLine(); textOut.flushBuffer(); charOut.flushBuffer(); if (autoFlush) out.flush(); }
Sun Feb 19 20:25:02 2017  Michael-O:where is this piece from?
Sun Feb 19 20:25:33 2017  tibor_:Oracle names flushing when the contnt of buffer is entirelly written, but they do not flush stream unless autoFlush=true
Sun Feb 19 20:25:39 2017  tibor_:private void newLine() {
Sun Feb 19 20:26:07 2017  tibor_:public void println(String x) { calls print(x); newLine();
Sun Feb 19 20:26:20 2017  tibor_:so yes, we have to use descriptor
Sun Feb 19 20:27:16 2017  Michael-O:When using the descriptor directly and not applying buffering to the PrintStream would be a breach of contract, but I do not care in this case.
Sun Feb 19 20:27:30 2017  Michael-O:STDOUT hasn't been simply designed for our usecase.
Sun Feb 19 20:27:47 2017  Michael-O:For Surefir 4.0 we should consider named pipes or sockets.
Sun Feb 19 20:29:43 2017  tibor_:How is java.langProcess implemented with getOutputStream ?
Sun Feb 19 20:29:49 2017  tibor_:is there buffereing to?
Sun Feb 19 20:30:15 2017  Michael-O:output steam is never buffered
Sun Feb 19 20:30:46 2017  Michael-O:see: http://stackoverflow.com/q/12945736/696632
Sun Feb 19 20:32:59 2017  tibor_:Having a look in our Jenkins. Old builds with Ubuntu and Windows passed. I have no idea why new ones with Jenkins file fail.
Sun Feb 19 20:33:50 2017  tibor_:We should not break build and make release and rework the physical layer of sending data.
Sun Feb 19 20:34:21 2017  tibor_:https://builds.apache.org/job/maven-surefire/
Sun Feb 19 20:34:31 2017  tibor_:https://builds.apache.org/job/maven-surefire-windows/
Sun Feb 19 20:34:38 2017  tibor_:both pass - old build
Sun Feb 19 20:35:47 2017  Michael-O:What about this: https://mail-archives.apache.org/mod_mbox/maven-dev/201702.mbox/%3Ca124d948-39f7-2dab-e4ff-d60a8a85a735%40schulte.it%3E?
Sun Feb 19 20:36:26 2017  Michael-O:Just because it passes on Ubuntu/Windows it does not mean that the code is correct
Sun Feb 19 20:39:33 2017  tibor_:but we are working one 2.19.2 one year
Sun Feb 19 20:39:46 2017  tibor_:and users need to have their fixes
Sun Feb 19 20:39:59 2017  tibor_:it is not a problem with 2.19.1, or?
Sun Feb 19 20:41:02 2017  tibor_:did you try with 2.19.1, maybe individual test
Sun Feb 19 20:41:12 2017  Michael-O:No, I did not
Sun Feb 19 20:41:25 2017  Michael-O:shall I? Christian always asked for master
Sun Feb 19 20:41:47 2017  Michael-O:I will run ForkModeIT on 2.19.1
Sun Feb 19 20:42:00 2017  tibor_:if you simply checkout release tag from git
Sun Feb 19 20:42:51 2017  tibor_:if we distinguish between platform and s/w we can say it is not regression and it is platform
Sun Feb 19 20:43:54 2017  tibor_:maybe rename <surefire.version>${project.version}</surefire.version> to 2.19.1
Sun Feb 19 20:44:00 2017  tibor_:in surefire-i-t POM
Sun Feb 19 20:46:24 2017  Michael-O:Joined the channel
Sun Feb 19 20:58:39 2017  Michael-O:tibor_: running ForkModeIT on 2.19.1 tag
Sun Feb 19 20:58:54 2017  tibor_:ok
Sun Feb 19 20:59:29 2017  tibor_:Are you tired from this?
Sun Feb 19 20:59:56 2017  tibor_:We are working on it one week I guess.
Sun Feb 19 21:05:16 2017  Michael-O:tibor_: not really, I see it as a challenge
Sun Feb 19 21:05:33 2017  Michael-O:portability is extremely important for stuff like Surefire
Sun Feb 19 21:05:40 2017  Michael-O:test has completed
Sun Feb 19 21:05:46 2017  Michael-O:Less failures
Sun Feb 19 21:05:53 2017  Michael-O:I will prepare a tarbal
Sun Feb 19 21:05:56 2017  Michael-O:tarball*
Sun Feb 19 21:07:56 2017  tibor_:The Jenkins file based build has only 3 error, something like a text not found in log, etc. But still it is the same Ubuntu like in the old Jenkins build. So if I fix those issues thenwe can release 2.19.2 or 2.20.0 and continue on this.
Sun Feb 19 21:10:35 2017  Michael-O:Uploaded a new tarbal for 2.19.1. Have a look.
Sun Feb 19 21:13:08 2017  Michael-O:tibor_: you don't want to know how may weeks I have wasted for bugs in JAXB RI and JAX-WS RI
Sun Feb 19 21:14:07 2017  tibor_:Are you in Java Community JSR?
Sun Feb 19 21:17:46 2017  tibor_:Only 2 tests failed with The forked VM terminated without properly saying goodbye.
Sun Feb 19 21:18:55 2017  tibor_:I guess the improvement of the most recent class ThreadedStreamConsumer in master increased the number of issues
Sun Feb 19 21:21:56 2017  tibor_:Michael-O: These are symptoms that event are not coming from plugin to fork
Sun Feb 19 21:21:58 2017  tibor_:Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
Sun Feb 19 21:22:19 2017  tibor_:[INFO] Total time: 22.328 s
Sun Feb 19 21:22:45 2017  tibor_:Howeve 2 issues but it's the same problem I would say.
Sun Feb 19 21:25:19 2017  tibor_:I can use FileDescriptot. I will check the communication diagram, because if the forked jvm first sends *next*test* and then plugin sends the class to test, then FileDescriptor makes pretty sense.
Sun Feb 19 21:29:03 2017  Michael-O:No, I am not ont he JSRs. The FD idea sounds good. You can safely flush after every new line.
Sun Feb 19 21:29:45 2017  Michael-O:If you going the FD route, check the default impl for STDIN, you likely have to replace STDIN with the FD too in the fork.
Sun Feb 19 21:31:07 2017  tibor_:>>You can safely flush after every new line.: Exactly.
Sun Feb 19 21:31:46 2017  tibor_:>>you likely have to replace STDIN with the FD too in the fork.: yes, i will check it.
Sun Feb 19 21:32:04 2017  Michael-O:for consistency, you know...
Sun Feb 19 21:46:28 2017  Michael-O:tibor_: I am going offline now. If you have something, just mail me tomorrow. Good night.
Sun Feb 19 21:48:02 2017  tibor_:Good Night.
Sun Feb 19 22:27:26 2017  Michael-O:Joined the channel

Comments