Openjdk-infrastructure: Configure auto logon for macos to support non-headless tests

[sxa]

Extracted from https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/184
@smlambert I'm splitting this out because the solution is different

[smlambert]

Thanks @sxa555, appreciated!

[jdekonin]

Try XQuartz
https://www.xquartz.org/

[sxa]

@gdams and I initially tried using xquartz but didn't manage to succeed within the jenkins environment for the JCK but we haven't got back to working it further again - if you have knowledge/experience of it @jdekonin that would be great :-)

[jdekonin]

@sxa555 I don't have experience running via jenkins on osx, so I can't help there. But that comment leads me to believe that if you ran manually and not through a jenkins agent/environment that it was working??

[sxa]

I think so yeah but it was a while ago :-) The xquartz server is probably still on the box I was using for testing

[geraintwjones]

I'm not convinced it helps, but I found this conversation in another community that talks about this issue. I'll keep digging.

[geraintwjones]

So I just typed which Xvfb on my macbook. It returned /opt/X11/bin/Xvfb. On most Unix systems it's in /usr/bin/Xvfb. I'm probably being naïve here, but is it not as simple as doing something with the path, or setting up a symbolic link?

[sxa]

@geraintwjones No idea -you tell us - does it work? ;-) I don't mind whether it's called Xvfb or xvfb

[sxa]

@smlambert A thought - do we require remote access to your mac GUI display to be able to run GUI tests? (Ref: https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/267 which I've raised for JCK purposes)

[sxa]

@Haroon-Khel We never managed to get this to a solution - since you looked at the Xvfb stuff on other platforms and are familiar with running the tests can you see if you can make an equivalent work on macos?

[Haroon-Khel]

I've tried to use the Xvfb jenkins plugin, like we do with Linux, but I couldn't quite get it to work. I tested this in this grinder on test-macincloud-macos1014-x64-1, which does have Xvfb/Xquartz installed, using the jdk test awt/BasicStroke/DashOffset.java which fails if Xvfb isn't correctly setup.

According to https://stackoverflow.com/questions/31857203/jenkins-cannot-run-program-xvfb-java-io-ioexception-no-such-file-or-director, the error in that grinder suggests that Xvfb isn't added to the PATH variable, but it certainly is on the machine; running which Xvfb returns /opt/X11/bin/Xvfb, so I'm not sure what is causing the error

[Haroon-Khel]

My plan now is to use the test scripts to use bash commands to setup an Xvfb server and export DISPLAY accordingly, during a test run, and then shut down the server after the run sort of how we do with aix. However to implement this in the test scripts, I first need to achieve this locally on the machine, using test-macincloud-macos1014-x64-1 again.

I'm using Xvfb :0 -ac -screen 0 1024x768x8 to run an Xvfb process, which seems to run fine, and exporting DISPLAY=:0. However when I run the test awt/BasicStroke/DashOffset.java locally I get the following error:

java.awt.AWTError: WindowServer is not available
        at java.desktop/sun.lwawt.macosx.LWCToolkit.<clinit>(LWCToolkit.java:166)
        at java.base/java.lang.Class.forNameImpl(Native Method)
        at java.base/java.lang.Class.forName(Class.java:342)

which sort of suggests that more than just an Xvfb process needs to be running. Has anyone here managed to get an X11 based test to run locally on a mac?

[gdams]

the error in that grinder suggests that Xvfb isn't added to the PATH variable, but it certainly is on the machine; running which Xvfb returns /opt/X11/bin/Xvfb

@Haroon-Khel looking at that grinder run it appears that /opt/X11/bin isn't in the PATH:

17:58:13  ITERATIONS=1
17:58:13  PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin/
17:58:13  MAIL=/var/mail/jenkins

I've kicked off this grinder with the PATH updated - let's see how it goes:

18:56:49  ITERATIONS=1
18:56:49  PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin/:/opt/X11/bin
18:56:49  MAIL=/var/mail/jenkins

[smlambert]

From grinder launched above:
Caused: java.io.IOException: Cannot run program "Xvfb": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at hudson.Proc$LocalProc.(Proc.java:252)
at hudson.Proc$LocalProc.(Proc.java:221)

[smlambert]

Is xquartz installed on the mac machine that Grinder is being run on, I am presuming not as I do not think that is in any ansible playbook or regular config setup at present. As I mentioned in #1450 (closed as dup of this issue), there were some attempts internally to enable, and eventual success on a machine (referenced in internal backlog/280 issue).

[gdams]

It's there:

test-macincloud-macos1014-x64-1:~ admin$ which xvfb
/opt/X11/bin/xvfb

[Haroon-Khel]

I created a link between /opt/X11/bin/Xvfb and /usr/bin/Xvfb. In this grinder, it seems jenkins is now using the plugin, but is now getting the same error that I would get by running the test locally (which i've posted in one of my comments above):

14:18:21  java.awt.AWTError: WindowServer is not available
14:18:21    at java.desktop/sun.lwawt.macosx.LWCToolkit.<clinit>(LWCToolkit.java:166)
14:18:21    at java.base/java.lang.Class.forNameImpl(Native Method)
14:18:21    at java.base/java.lang.Class.forName(Class.java:342)
14:18:21    at java.desktop/java.awt.Toolkit$2.run(Toolkit.java:588)
14:18:21    at java.desktop/java.awt.Toolkit$2.run(Toolkit.java:583)

This suggests that more than simply an Xvfb session needs to be running for these tests to work

[gdams]

just tried running this on the mac in question as the Jenkins user:

bash-3.2$ Xvfb
_XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed
_XSERVTransMakeAllCOTSServerListeners: server already running
(EE) 
Fatal server error:
(EE) Cannot establish any listening sockets - Make sure an X server isn't already running(EE) 

this could well be the issue, just investigating

[sxa]

Suggest not using the first display i.e. try Xvfb :1 or a higher number in case it's defaulting to zero which will be the physical display on the machine (Not sure if I have access to the machine to look myself)

[smlambert]

How is this one looking? Please let me know if you want us to test or try stuff out.

[Haroon-Khel]

@gdams and I were troubleshooting this a while back.
The display number doesnt seem to be the issue. The process continues to complain about a WindowServer not being available, despite

 _windowserver     145   0.3  0.2  4764296  14052   ??  Ss   13Aug20 168:39.78 /System/Library/PrivateFrameworks/SkyLight.framework/Resources/WindowServer -daemon

running in the background, a supposed WindowServer program

[sxa]

Could that process be a windowserver executing on the physical display of the machine instead of the virtual one?

[Mesbah-Alam]

@Haroon-Khel @sxa - Any update on this ?

[psoujany]

We still see display issue on MAC osxrt4 machine. Is there a way I can run JCK display tests on MAC machines? Could anyone please help me on this issue. Thank you.

[smlambert]

Noting that this issue blocks the enablement of jdk_awt and jdk_swing tests (from the openjdk group) at AdoptOpenJDK project (and also affect automation of some tests from the jck group).

jdk_awt is disabled for os.osx
jdk_swing is disabled for os.osx

[sxa]

I was able to start up an Xvfb on one of the machines and successfully execute "xterm" pointing at the DISPLAY, however this does not appear to work adequately for the purposes of the automated tests running through our test harnesses via jenkins. Needs further debugging. I'd be tempted to use strace/truss/dtrace to try and track down whether any particular system call is causing a problem here.

[aahlenst]

I have serious doubts that any implementation of X is needed on macOS, at all. macOS does not use X to display things, it uses Quartz. So it would be weird for the tests of the native macOS back-end of Java to require X.

Shelley ran a Grinder yesterday with jdk_awt and it's full of errors like:

18:07:19  java.lang.RuntimeException: Error: unable to create robot
18:07:19    at test.java.awt.regtesthelpers.Util.createRobot(Util.java:91)
18:07:19    at WindowJumpingTest.main(WindowJumpingTest.java:38)
18:07:19    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
18:07:19    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
18:07:19    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
18:07:19    at java.lang.reflect.Method.invoke(Method.java:498)
18:07:19    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
18:07:19    at java.lang.Thread.run(Thread.java:748)
18:07:19  Caused by: java.awt.AWTException: headless environment
18:07:19    at java.awt.Robot.<init>(Robot.java:91)
18:07:19    at test.java.awt.regtesthelpers.Util.createRobot(Util.java:87)
18:07:19    ... 7 more

This exception is thrown by Java if AWT code is touched that is not compatible with headless environments while AWT is instructed to run headless (java.awt.headless=true). https://www.oracle.com/technical-resources/articles/javase/headless.html provides all the details. So what this means: We're trying to run tests that draw windows while putting AWT in its windowless mode. If we wanted to run headless AWT tests, we would have to filter them out (some support for that case seems to exist according to https://bugs.openjdk.java.net/browse/JDK-8198612). But that's not what we want anyway.

I can easily reproduce the exception above by running the tests locally on my Mac (note -vmoptions:"-Djava.awt.headless=true"):

$ /usr/local/jtreg/bin/jtreg -jdk:build/macosx-x86_64-normal-server-release/images/j2sdk-bundle/jdk1.8.0.jdk/Contents/Home/ -vmoptions:"-Djava.awt.headless=true" -a -v:fail,error,time,nopass jdk/test/java/awt/Window/

--------------------------------------------------
TEST: java/awt/Window/8027025/Test8027025.java
TEST JDK: /Users/andreas/Projects/OpenJDK/openjdk-jdk8u/build/macosx-x86_64-normal-server-release/images/j2sdk-bundle/jdk1.8.0.jdk/Contents/Home

ACTION: build -- Passed. All files up to date
REASON: Named class compiled on demand
TIME:   0.0 seconds
messages:
command: build Test8027025
reason: Named class compiled on demand
elapsed time (seconds): 0.0

ACTION: main -- Failed. Execution failed: `main' threw exception: java.lang.reflect.InvocationTargetException
REASON: User specified action: run main Test8027025 
TIME:   0.526 seconds
messages:
command: main Test8027025
reason: User specified action: run main Test8027025 
Mode: othervm
elapsed time (seconds): 0.526
configuration:
STDOUT:
STDERR:
java.lang.reflect.InvocationTargetException
    at java.awt.EventQueue.invokeAndWait(EventQueue.java:1349)
    at java.awt.EventQueue.invokeAndWait(EventQueue.java:1324)
    at javax.swing.SwingUtilities.invokeAndWait(SwingUtilities.java:1353)
    at Test8027025.main(Test8027025.java:42)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.awt.HeadlessException
    at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:204)
    at java.awt.Window.<init>(Window.java:536)
    at java.awt.Frame.<init>(Frame.java:420)
    at Test8027025.lambda$main$0(Test8027025.java:43)
    at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:301)
    at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
    at java.awt.EventQueue.access$500(EventQueue.java:97)
    at java.awt.EventQueue$3.run(EventQueue.java:709)
    at java.awt.EventQueue$3.run(EventQueue.java:703)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

JavaTest Message: Test threw exception: java.lang.reflect.InvocationTargetException
JavaTest Message: shutting down test

STATUS:Failed.`main' threw exception: java.lang.reflect.InvocationTargetException
rerun:
cd /Users/andreas/Projects/OpenJDK/openjdk-jdk8u/JTwork/scratch && \
HOME=/Users/andreas \
LANG=de_CH.UTF-8 \
LC_CTYPE=de_CH.UTF-8 \
PATH=/bin:/usr/bin:/usr/sbin \
CLASSPATH=/Users/andreas/Projects/OpenJDK/openjdk-jdk8u/JTwork/classes/java/awt/Window/8027025:/Users/andreas/Projects/OpenJDK/openjdk-jdk8u/jdk/test/java/awt/Window/8027025:/Users/andreas/Projects/OpenJDK/openjdk-jdk8u/build/macosx-x86_64-normal-server-release/images/j2sdk-bundle/jdk1.8.0.jdk/Contents/Home/lib/tools.jar:/usr/local/jtreg/lib/javatest.jar:/usr/local/jtreg/lib/jtreg.jar \
    /Users/andreas/Projects/OpenJDK/openjdk-jdk8u/build/macosx-x86_64-normal-server-release/images/j2sdk-bundle/jdk1.8.0.jdk/Contents/Home/bin/java \
        -Dtest.vm.opts=-Djava.awt.headless=true \
        -Dtest.tool.vm.opts=-J-Djava.awt.headless=true \
        -Dtest.compiler.opts= \
        -Dtest.java.opts= \
        -Dtest.jdk=/Users/andreas/Projects/OpenJDK/openjdk-jdk8u/build/macosx-x86_64-normal-server-release/images/j2sdk-bundle/jdk1.8.0.jdk/Contents/Home \
        -Dcompile.jdk=/Users/andreas/Projects/OpenJDK/openjdk-jdk8u/build/macosx-x86_64-normal-server-release/images/j2sdk-bundle/jdk1.8.0.jdk/Contents/Home \
        -Dtest.timeout.factor=1.0 \
        -Dtest.root=/Users/andreas/Projects/OpenJDK/openjdk-jdk8u/jdk/test \
        -Dtest.name=java/awt/Window/8027025/Test8027025.java \
        -Dtest.file=/Users/andreas/Projects/OpenJDK/openjdk-jdk8u/jdk/test/java/awt/Window/8027025/Test8027025.java \
        -Dtest.src=/Users/andreas/Projects/OpenJDK/openjdk-jdk8u/jdk/test/java/awt/Window/8027025 \
        -Dtest.src.path=/Users/andreas/Projects/OpenJDK/openjdk-jdk8u/jdk/test/java/awt/Window/8027025 \
        -Dtest.classes=/Users/andreas/Projects/OpenJDK/openjdk-jdk8u/JTwork/classes/java/awt/Window/8027025 \
        -Dtest.class.path=/Users/andreas/Projects/OpenJDK/openjdk-jdk8u/JTwork/classes/java/awt/Window/8027025 \
        -Djava.awt.headless=true \
        com.sun.javatest.regtest.agent.MainWrapper /Users/andreas/Projects/OpenJDK/openjdk-jdk8u/JTwork/java/awt/Window/8027025/Test8027025.d/main.0.jta

TEST RESULT: Failed. Execution failed: `main' threw exception: java.lang.reflect.InvocationTargetException
--------------------------------------------------

I didn't see any similar errors when running with java.awt.headless=false. And I don't have X on my Mac.

I tried to run a Grinder with java.awt.headless=false. It stopped providing output and hung until I aborted it. This looks more like a Jenkins than a test problem.

[aahlenst]

There are tests that require manual interaction. https://openjdk.java.net/jtreg/runtests.html explains how to skip those:

Some tests, typically GUI tests, may require manual interaction in order to determine whether they pass or fail. To run just those tests, use the -m or -manual option. To avoid running such tests, use the -a or -automatic option.

[aahlenst]

When SSHing into our machines, I can observe that the tests hang, too.

Further information about some behaviour like tests being automatically forced into headless mode on macOS:

http://mail.openjdk.java.net/pipermail/macosx-port-dev/2011-December/001883.html
http://mail.openjdk.java.net/pipermail/macosx-port-dev/2013-May/005751.html

[smlambert]

re: https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/211#issuecomment-775853319, I am confirming that we run in -automatic mode (see https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/openjdk/openjdk.mk#L68 where we set that option for all openjdk tests run under automation)

[aahlenst]

Tests run fine locally when UI session is present (I opened the shell via SSH):

build-macstadium-macos1014-x64-2:openjdk-jdk8u jenkins$ ../jtreg/bin/jtreg -jdk:../jdk8u282-b08/Contents/Home -vmoptions:"-Djava.awt.headless=false" -a -verbose jdk/test/java/awt/window
Directory "JTwork" not found: creating
Directory "JTreport" not found: creating
runner starting test: java/awt/Window/8027025/Test8027025.java
runner finished test: java/awt/Window/8027025/Test8027025.java
Passed. Execution successful
runner starting test: java/awt/Window/AlwaysOnTop/AlwaysOnTopEvenOfWindow.java
runner finished test: java/awt/Window/AlwaysOnTop/AlwaysOnTopEvenOfWindow.java
Passed. Execution successful
runner starting test: java/awt/Window/AlwaysOnTop/AlwaysOnTopFieldTest.java
runner finished test: java/awt/Window/AlwaysOnTop/AlwaysOnTopFieldTest.java
Passed. Execution successful
runner starting test: java/awt/Window/AlwaysOnTop/AutoTestOnTop.java

[aahlenst]

To be able to run UI tests on macOS, a UI session needs to be started as the user that runs the tests. In our case, this means we have to enable auto-login on each macOS for the user jenkins (= after a reboot, macOS has to automatically log in and open the desktop for Jenkins). Also, we have to allow sshd-keygen-wrapper to control the screen.

I altered the configuration on build-macstadium-macos1014-x64-2 and a grinder completed successfully.

[aahlenst]

@sxa How should we proceed here? You voiced (valid) concerns. Still we need to run those tests because upstream does not and a UI session is needed for TCK, too. So either we roll this change out to more machines or I remove the change from the one machine I reconfigured for verifying the config change. If you prefer that we do not open the existing machines, how would a setup look like that you'd approve?

[sxa]

Sorry, overwhelmed with other stuff this week and I didn't manage to progress it - would like to get input from others so was going to open a discussion in the private infra slack channel. Will do so on Monday (if I don't forget - if I do, feel free to prod me) :-)

[aahlenst]

Summarizing the discussion with @karianna and @sxa that happened elsewhere as good as I can:

We roll out the changes as outlined in https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/211#issuecomment-776673592 across all our testing machines:

https://ci.adoptopenjdk.net/label/ci.role.test&&mac/

These commands (followed by a reboot to check whether everything is working as supposed) should be sufficient:

sudo defaults -currentHost write com.apple.screensaver idleTime -0
sysadminctl -screenLock off -password <password here>
sudo defaults write /Library/Preferences/com.apple.loginwindow autoLoginUser jenkins

In addition, you manually have to allow sshd-keygen-wrapper to control the screen.

We briefly discussed whether to use Parallels Desktop so that we don't have to unlock any user. Parallels Desktop is able to start a VM when the computer starts and can open a window that qualifies as an UI session without having to unlock the desktop of the VM host. That would generally be a nice thing to have and by using the Jenkins Parallels Desktop Cloud plug-in we could even get on-demand agents (at least in theory). But so far, the appetite to test and implement the required changes is low.

I don't have the necessary credentials, so somebody else has to implement the changes.

[smlambert]

I believe that the jdk_imageio test target is also failing due to this config issue.

How can this issue be addressed/progressed? Who has the credentials to be able to do the clearly identified steps above?
Related: https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/2014

[aahlenst]

@smlambert AFAIK, only George has them. Stewart didn't the last time I asked him (some weeks ago). You need passwords for both the admin user and jenkins user.

[smlambert]

@gdams - can you please help here?

Best if you can get the credentials shared to a couple more people, worst case please address this particular issue, without it, 3 large jtreg targets (equivalent of a few thousand test cases) and 2 large jck targets are excluded (and would have to be run manually rather than in automation).

[gdams]

@aahlenst the passwords are available in the secrets repo. What particular password do you need?

[aahlenst]

@gdams Great. Administrator and Jenkins should suffice.

[sxa]

@aahlenst the passwords are available in the secrets repo. What particular password do you need?

@gdams There has never been more than the two 1010 systems at each provider in there AFAIK unless you've put the others in another location

[gdams]

The jenkins password has always been there. The rest are there too now

[sxa]

Thank you ... But please use PRs when pushing to the repositories.

[gdams]

But please use PRs when pushing to the repositories.

We've discussed this before, there is no point in creating a PR for a GPG encrypted message? To the human eye it's complete nonsense...

[sxa]

That doesn't stop the reviewer from decrypting it locally and making sure it hasn't corrupted the content, which has happened in the past