Jetty.project: Add systemd service file

Created on 19 Apr 2017  ·  56Comments  ·  Source: eclipse/jetty.project

jetty.sh looks broken if used under systemd.

While starting jetty seems to work via sudo service jetty start, stopping it does not seem to work, via sudo service jetty stop.

Also, systemd may need additional metadata to make jetty "known" to systemd so that systemd tools work with jetty too.

Most helpful comment

I'm currently investigating:
https://stackoverflow.com/questions/44661232/starting-jetty-9-4-6-as-a-service-fails-under-debian-jessie

Note that I had to copy the init script from /etc/init.d/service into /etc/systemd/system/jetty.service.

All 56 comments

I would have expected the commands when running under systemd to have been ...

$ sudo systemctl start jetty
$ sudo systemctl stop jetty

The service command is a leftover from System V init scripts days.
The systemctl command is what's used under systemd for starting / stopping systemd units.

Do we document (and/or ship) a systemd unit that references the jetty.sh?
Do we continue to support System V init script behavior?
Do we deprecate System V init script behavior from jetty.sh?

I'm currently investigating:
https://stackoverflow.com/questions/44661232/starting-jetty-9-4-6-as-a-service-fails-under-debian-jessie

Note that I had to copy the init script from /etc/init.d/service into /etc/systemd/system/jetty.service.

This is my first try:

[Unit]
Description=Jetty Web Application Server
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/var/run/jetty.pid
ExecStart=/etc/init.d/jetty start
ExecStop=/etc/init.d/jetty stop
User=jetty

[Install]
WantedBy=multi-user.target

I have to spend much more time in supporting other features. It would be nice to be able to reload Jetty too. Note that it calls your script jetty-distribution-9.4.6.v20170531/bin/jetty.sh.

Nice.

This should go into jetty-home-<ver>/bin/jetty.service (which jetty-distribution builds from anyway)

Isn't service reload just stop+start ?

Ok. Then, I'll add:
ExecReload=/etc/init.d/jetty restart

Can you confirm that it makes sense? restart != reload?
https://www.freedesktop.org/software/systemd/man/systemd.service.html#ExecReload=

I'm not sure that "Environment=" is required.

Correct - there is no general way to reload configuration other than a restart (or stop+start).

root@A20-OLinuXino:/etc/systemd/system# systemctl status jetty.service
● jetty.service - Jetty Web Application Server
   Loaded: loaded (/etc/systemd/system/jetty.service; disabled)
   Active: inactive (dead)
root@A20-OLinuXino:/etc/systemd/system# systemctl enable jetty.service
Synchronizing state for jetty.service with sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d jetty defaults
Executing /usr/sbin/update-rc.d jetty enable
root@A20-OLinuXino:/etc/systemd/system# systemctl status jetty.service
● jetty.service - Jetty Web Application Server
   Loaded: loaded (/etc/systemd/system/jetty.service; enabled)
   Active: inactive (dead)

After a reboot:

root@A20-OLinuXino:~# systemctl status jetty.service
● jetty.service - Jetty Web Application Server
   Loaded: loaded (/etc/systemd/system/jetty.service; enabled)
   Active: failed (Result: exit-code) since Wed 2017-06-21 19:26:18 CEST; 5min ago
  Process: 279 ExecStart=/etc/init.d/jetty start (code=exited, status=1/FAILURE)

Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1515)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1439)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: ... 9 more
Jun 21 19:26:18 A20-OLinuXino jetty[279]: Usage: java -jar $JETTY_HOME/start.jar [options] [properties] [configs]
Jun 21 19:26:18 A20-OLinuXino jetty[279]: java -jar $JETTY_HOME/start.jar --help  # for more information
Jun 21 19:26:18 A20-OLinuXino jetty[279]: FAILED Wed Jun 21 19:26:18 CEST 2017
Jun 21 19:26:18 A20-OLinuXino systemd[1]: jetty.service: control process exited, code=exited status=1
Jun 21 19:26:18 A20-OLinuXino systemd[1]: Failed to start Jetty Web Application Server.
Jun 21 19:26:18 A20-OLinuXino systemd[1]: Unit jetty.service entered failed state.
Jun 21 19:26:16 A20-OLinuXino jetty[279]: . 2017-06-21 19:26:16.356:INFO:oejs.Server:main: jetty-9.4.6.v20170531
Jun 21 19:26:16 A20-OLinuXino jetty[279]: 2017-06-21 19:26:16.638:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///opt/web/mybase/webapps/] at interval 1
Jun 21 19:26:17 A20-OLinuXino jetty[279]: 2017-06-21 19:26:17.567:INFO:oejsh.ContextHandler:main: Started o.e.j.s.h.ContextHandler@3f4d3d{/,null,AVAILABLE,www.cooperativeintegrale.com}
Jun 21 19:26:17 A20-OLinuXino jetty[279]: 2017-06-21 19:26:17.829:INFO:oejsh.ContextHandler:main: Started o.e.j.s.h.ContextHandler@bf9759{/,null,AVAILABLE,www.sel-de-paname.org}
Jun 21 19:26:18 A20-OLinuXino jetty[279]: 2017-06-21 19:26:18.057:WARN:oejx.XmlConfiguration:main:
Jun 21 19:26:18 A20-OLinuXino jetty[279]: java.security.PrivilegedActionException: java.net.SocketException: Permission denied
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at java.security.AccessController.doPrivileged(Native Method)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1438)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at java.lang.reflect.Method.invoke(Method.java:498)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.start.Main.invokeMain(Main.java:221)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.start.Main.start(Main.java:506)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.start.Main.main(Main.java:78)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: Caused by:
Jun 21 19:26:18 A20-OLinuXino jetty[279]: java.net.SocketException: Permission denied
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at sun.nio.ch.Net.bind0(Native Method)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at sun.nio.ch.Net.bind(Net.java:433)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at sun.nio.ch.Net.bind(Net.java:425)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:298)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.server.Server.doStart(Server.java:431)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1515)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1439)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at java.security.AccessController.doPrivileged(Native Method)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1438)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at java.lang.reflect.Method.invoke(Method.java:498)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.start.Main.invokeMain(Main.java:221)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.start.Main.start(Main.java:506)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.start.Main.main(Main.java:78)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: java.lang.reflect.InvocationTargetException
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at java.lang.reflect.Method.invoke(Method.java:498)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.start.Main.invokeMain(Main.java:221)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.start.Main.start(Main.java:506)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.start.Main.main(Main.java:78)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: Caused by: java.security.PrivilegedActionException: java.net.SocketException: Permission denied
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at java.security.AccessController.doPrivileged(Native Method)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1438)
Jun 21 19:26:18 A20-OLinuXino jetty[279]: ... 7 more

There is a problem of permission.

You are either attempting to bind to a privileged port (1-1024) with the wrong userid. (in which case you'll probably need to setup setuid)

Or your system has some kind of security enabled to prevent random apps from binding to ports?

Does systemd need to declare that the service will bind to a port?

The ports in the range 1-1023 are privileged. Only root is allowed to bind to them. You're right. The user "jetty" isn't allowed to use those ports. I'm trying to make it work with root first. Then, I'll solve this minor problem with the user "jetty".

Jan 01 01:00:18 A20-OLinuXino jetty[269]: Starting Jetty: FAILED Fri Jan 1 01:00:18 CET 2010
Jan 01 01:00:18 A20-OLinuXino systemd[1]: jetty.service: control process exited, code=exited status=1
Jan 01 01:00:18 A20-OLinuXino systemd[1]: Failed to start Jetty Web Application Server.
Jan 01 01:00:18 A20-OLinuXino systemd[1]: Unit jetty.service entered failed state.

I have to use the port 8080 for HTTP, otherwise it doesn't work even outside the init script. Now, I get this:

root@A20-OLinuXino:~# systemctl status jetty.service
● jetty.service - Jetty Web Application Server
   Loaded: loaded (/etc/systemd/system/jetty.service; enabled)
   Active: failed (Result: timeout) since Wed 2017-06-21 23:50:19 CEST; 1min 51s ago
  Process: 272 ExecStart=/etc/init.d/jetty start (code=exited, status=0/SUCCESS)

Jun 21 23:48:25 A20-OLinuXino jetty[272]: Starting Jetty: OK Wed Jun 21 23:48:25 CEST 2017
Jun 21 23:48:25 A20-OLinuXino systemd[1]: PID file /var/run/jetty.pid not readable (yet?) after start.
Jun 21 23:50:19 A20-OLinuXino systemd[1]: jetty.service start operation timed out. Terminating.
Jun 21 23:50:19 A20-OLinuXino systemd[1]: Failed to start Jetty Web Application Server.
Jun 21 23:50:19 A20-OLinuXino systemd[1]: Unit jetty.service entered failed state.

It works:

root@A20-OLinuXino:~# systemctl status jetty.service
● jetty.service - Jetty Web Application Server
   Loaded: loaded (/etc/systemd/system/jetty.service; enabled)
   Active: active (running) since Thu 2017-06-22 00:22:07 CEST; 2min 2s ago
  Process: 276 ExecStart=/etc/init.d/jetty start (code=exited, status=0/SUCCESS)
 Main PID: 318 (java)
   CGroup: /system.slice/jetty.service
           └─318 /usr/bin/java -Djetty.home=/opt/jetty/jetty-distribution-9.4.6.v20170531 -Djetty.base=/opt/web/mybase -Djava.io.tmpdir=/opt/jetty/temp -jar /opt/jetty/jetty-distr...

I had to add the following line into /etc/default/jetty:
JETTY_PID=/opt/web/mybase/jetty/jetty.pid

I had to modify jetty.service:
PIDFile=/opt/web/mybase/jetty/jetty.pid

Traditionally we made the pid show up in the jetty.base directory.

I wonder if we should reevaluate that decision and just use the /var/run/<instanceid>.pid default location instead.

Using /var/run/<instanceid>.pid would require to change the owner of this file or (even worse) to change the owner of /var/run if we follow the documentation wisely advising to create a separate user "jetty" who only owns /opt/jetty and /opt/web.

At first, I'll try to set PIDFile=/opt/web/mybase/jetty.pid in /etc/systemd/system/jetty.service and JETTY_PID=$JETTY_BASE/jetty.pid in /etc/default/jetty. Maybe the latter isn't necessary.

Then, I'll retry to use the user "jetty" instead of the superuser.

Finally, I'll try your suggestion concerning the setuid (with the privileged HTTP port 80 and the privileged HTTPS port 443). I know that the documentation can't contain everything but note that it doesn't mention the need of setting the setuid bit on /etc/init.d/jetty.sh or I've missed something.

With regards to listening on port 80 and 443, there are two basic
approaches:

1) Start jetty as root, bind to the port and then setuid to the jetty user.
This works fine, but does rely on a little bit of native code. There
can also be some ownership issues if any files are created before the
setuid is done.

2) Run jetty as user jetty and just bind to ports like 8080 and 8443. Use
a mechanism like ipchains or iptables to map ports 80 to 8080 and 443 to

  1. This means that the server never runs as root and there is less
    risk of security and/or ownership problems. This is my preferred approach.

For further information see
http://www.eclipse.org/jetty/documentation/9.4.x/setting-port80-access.html

regards

On 22 June 2017 at 10:14, Julien Gouesse notifications@github.com wrote:

Using /var/run/.pid would require to change the owner of this file or
(even worse) to change the owner of /var/run if we follow the documentation
wisely advising to create a separate user "jetty" who only owns /opt/jetty
and /opt/web.

At first, I'll try to set PIDFile=/opt/web/mybase/jetty.pid in
/etc/systemd/system/jetty.service and JETTY_PID=$JETTY_BASE/jetty.pid in
/etc/default/jetty. Maybe the latter isn't necessary.

Then, I'll retry to use the user "jetty" instead of the superuser.

Finally, I'll try your suggestion concerning the setuid. I known that the
documentation can't contain everything but note that it doesn't mention the
need of setting the setuid bit on /etc/init.d/jetty.sh or I've missed
something.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/eclipse/jetty.project/issues/1485#issuecomment-310309363,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEUrVrTFxf6ubOpB8PMxD_vgTLkBqtSks5sGiKCgaJpZM4NCR2N
.

--
Greg Wilkins gregw@webtide.com CTO http://webtide.com

Thank you. I currently use the second approach. I set a NAT/PAT rule in my residential gateway which probably uses iptables under the hood. The NAT/PAT rule would be necessary even if I used the port 80. Sorry, I should have searched a bit further concerning setuid. Maybe a link to this article should be put into the instructions to create a Unix service for Jetty.

I had to call that to fix the time in the logs:
timedatectl set-ntp true

My file /etc/default/jetty contains:

JETTY_HOME=/opt/jetty/jetty-distribution-9.4.6.v20170531
JETTY_BASE=/opt/web/mybase
JETTY_PID=/opt/web/mybase/jetty.pid
TMPDIR=/opt/jetty/temp

The temporary file start_1276664672082575285.properties contains:

#start.jar properties
#Thu Jun 22 21:58:18 CEST 2017
java.version.platform=8
jetty.deploy.scanInterval=1
java.version=1.8.0_111
java.version.micro=0
jetty.home=/opt/jetty/jetty-distribution-9.4.6.v20170531
jetty.http.compliance=RFC7230
jetty.server.stopAtShutdown=true
java.version.minor=8
jetty.home.uri=file\:///opt/jetty/jetty-distribution-9.4.6.v20170531
jetty.http.port=8080
java.version.update=111
jetty.base=/opt/web/mybase
java.version.major=1
jetty.base.uri=file\:///opt/web/mybase
jetty.deploy.monitoredDir=webapps
jetty.state=/opt/web/mybase/jetty.state

The file jetty.service contains:

[Unit]
Description=Jetty Web Application Server
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/opt/web/mybase/jetty.pid
ExecStart=/etc/init.d/jetty start
ExecStop=/etc/init.d/jetty stop
ExecReload=/etc/init.d/jetty restart
User=jetty

[Install]
WantedBy=multi-user.target

Why aren't there any logs in $JETTY_BASE/logs/?

jetty-logging.xml isn't in /opt/jetty/jetty-distribution-9.4.6.v20170531/etc/, is there anything missing in the instructions?
http://www.eclipse.org/jetty/documentation/current/startup-unix-service.html#_practical_setup_of_a_jetty_service

@gouessej jetty-logging.xml was changed to console-capture.xml in Jetty 9.4 but it looks like the documentation was not updated for this section. I'll work on making sure it is up to date.

Thank you so much. I'll add it into jetty.conf.

@WalkerWatch Thank you for the update in the documentation:
https://github.com/eclipse/jetty.project/commit/0bfc41e0680cd309617889a61f53460315141a7c

By the way, /opt/jetty/temp is created twice, you can find mkdir -p /opt/jetty/temp and mkdir /opt/jetty/temp, it's a bit confusing. chown should be called with jetty:jetty as we use the user "jetty" in the group "jetty".

Adding JETTY_PID=/opt/web/mybase/jetty.pid into /etc/default/jetty is necessary. I get this when I remove it:

Jun 23 10:44:26 A20-OLinuXino systemd[1]: jetty.service: PID file /opt/web/mybase/jetty.pid not readable (yet?) after start: No such file or directory
Jun 23 10:44:52 A20-OLinuXino systemd[1]: jetty.service: Daemon never wrote its PID file. Failing.

I had to create $JETTY_BASE/logs, change its owner to jetty:jetty and ensure it's writable to make it work:
mkdir -p /opt/web/mybase/logs
chown --recursive jetty /opt/web/mybase/logs
chmod u+w /opt/web/mybase/logs

Hi
Now Jetty ignores all my web application configuration files when it's run as a service but it works as expected if I run it with cd /opt/web/mybase && java -jar /opt/jetty/jetty-distribution-9.4.6.v20170531/start.jar. What can cause that?

I only get this in the logs:
2017-08-05 21:41:31.019:INFO:oejs.Server:main: jetty-9.4.6.v20170531
2017-08-05 21:41:31.345:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///opt/web/mybase/webapps/] at interval 1
2017-08-05 21:41:31.409:INFO:oejs.AbstractNCSARequestLog:main: Opened /opt/web/mybase/logs/2017_08_05.request.log
2017-08-05 21:41:31.659:INFO:oejs.AbstractConnector:main: Started ServerConnector@125fce0{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2017-08-05 21:41:31.675:INFO:oejs.Server:main: Started @32804ms

Nothing in the debug logs :(

It works anew when I add jetty.dump.start=true into /opt/web/mybase/start.ini (joakim's advice), it's strange.

It fails anew, no webapp configuration is taken into account:

2017-08-06 17:25:07.918:DBUG:oejx.XmlConfiguration:main: XML call setOut
2017-08-06 17:25:07.921:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.server.Server@1082746
2017-08-06 17:25:07.934:DBUG:oejuc.ContainerLifeCycle:main: org.eclipse.jetty.server.Server@1082746 added {org.eclipse.jetty.server.handler.ErrorHandler@1774679,AUTO}
2017-08-06 17:25:07.973:INFO:oejs.Server:main: jetty-9.4.6.v20170531
2017-08-06 17:25:08.273:DBUG:oejsh.AbstractHandler:main: starting org.eclipse.jetty.server.Server@1082746
2017-08-06 17:25:08.275:DBUG:oejuc.AbstractLifeCycle:main: starting qtp31137656{STOPPED,10<=0<=200,i=0,q=0}
2017-08-06 17:25:08.297:DBUG:oejuc.AbstractLifeCycle:main: STARTED @32147ms qtp31137656{STARTED,10<=10<=200,i=10,q=0}
2017-08-06 17:25:08.303:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@a71081
2017-08-06 17:25:08.323:DBUG:oejuc.AbstractLifeCycle:main: STARTED @32174ms org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@a71081
2017-08-06 17:25:08.324:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.server.handler.HandlerCollection@5bef66[org.eclipse.jetty.server.handler.ContextHandlerCollection@88ff2c[], org.eclipse.jetty.server.handler.DefaultHa
ndler@1851384]
2017-08-06 17:25:08.326:DBUG:oejsh.AbstractHandler:main: starting org.eclipse.jetty.server.handler.HandlerCollection@5bef66[org.eclipse.jetty.server.handler.ContextHandlerCollection@88ff2c[], org.eclipse.jetty.server.handler.DefaultHand
ler@1851384]
2017-08-06 17:25:08.328:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.server.handler.ContextHandlerCollection@88ff2c[]
2017-08-06 17:25:08.330:DBUG:oejsh.AbstractHandler:main: starting org.eclipse.jetty.server.handler.ContextHandlerCollection@88ff2c[]
2017-08-06 17:25:08.331:DBUG:oejuc.AbstractLifeCycle:main: STARTED @32182ms org.eclipse.jetty.server.handler.ContextHandlerCollection@88ff2c[]
2017-08-06 17:25:08.332:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.server.handler.DefaultHandler@1851384
2017-08-06 17:25:08.333:DBUG:oejsh.AbstractHandler:main: starting org.eclipse.jetty.server.handler.DefaultHandler@1851384
2017-08-06 17:25:08.335:DBUG:oejuc.AbstractLifeCycle:main: STARTED @32185ms org.eclipse.jetty.server.handler.DefaultHandler@1851384
2017-08-06 17:25:08.336:DBUG:oejuc.AbstractLifeCycle:main: STARTED @32186ms org.eclipse.jetty.server.handler.HandlerCollection@5bef66[org.eclipse.jetty.server.handler.ContextHandlerCollection@88ff2c[], org.eclipse.jetty.server.handler.D
efaultHandler@1851384]
2017-08-06 17:25:08.338:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.deploy.DeploymentManager@6ae04d
2017-08-06 17:25:08.339:DBUG:oejd.DeploymentManager:main: DeploymentManager using standard bindings
2017-08-06 17:25:08.389:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.deploy.providers.WebAppProvider@1975490
2017-08-06 17:25:08.391:DBUG:oejdp.ScanningAppProvider:main: WebAppProvider.doStart()
2017-08-06 17:25:08.393:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///opt/web/mybase/webapps/] at interval 1
2017-08-06 17:25:08.420:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.util.Scanner@11053e1
2017-08-06 17:25:08.433:DBUG:oeju.Scanner:main: scan accepted /opt/web/mybase/webapps/**.org.war
2017-08-06 17:25:08.445:DBUG:oeju.Scanner:main: scan accepted /opt/web/mybase/webapps/beta.
***.org.war
2017-08-06 17:25:08.449:DBUG:oeju.Scanner:main: scan accepted /opt/web/mybase/webapps/
****.org.xml
2017-08-06 17:25:08.453:DBUG:oeju.Scanner:main: scan accepted /opt/web/mybase/webapps/beta.
***.org.xml
2017-08-06 17:25:08.465:DBUG:oeju.Scanner:main: scanned [/opt/web/mybase/webapps]: {/opt/web/mybase/webapps/beta.**.org.xml=ADDED, /opt/web/mybase/webapps/beta.**.org.war=ADDED, /opt/web/mybase/webapps/**.org.war=ADDED, /opt/web/mybase/webapps/**.org.xml=ADDED}
2017-08-06 17:25:08.481:DBUG:oeju.Scanner:main: scan accepted /opt/web/mybase/webapps/
**.org.war
2017-08-06 17:25:08.485:DBUG:oeju.Scanner:main: scan accepted /opt/web/mybase/webapps/beta.
***.org.war
2017-08-06 17:25:08.488:DBUG:oeju.Scanner:main: scan accepted /opt/web/mybase/webapps/***.org.xml
2017-08-06 17:25:08.491:DBUG:oeju.Scanner:main: scan accepted /opt/web/mybase/webapps/beta.***.org.xml
2017-08-06 17:25:08.494:DBUG:oeju.Scanner:main: scanned [/opt/web/mybase/webapps]: {/opt/web/mybase/webapps/beta.
*.org.xml=ADDED, /opt/web/mybase/webapps/beta.*.org.war=ADDED, /opt/web/mybase/webapps/**.org.war=ADDED, /opt/web/mybase/webapps/**.org.xml=ADDED}
2017-08-06 17:25:08.508:DBUG:oejdp.ScanningAppProvider:main: added /opt/web/mybase/webapps/beta.
**.org.xml
2017-08-06 17:25:08.510:DBUG:oejd.DeploymentManager:main: Deployable added: /opt/web/mybase/webapps/beta.
*.org.xml
2017-08-06 17:25:08.537:DBUG:oejd.DeploymentManager:main: Executing Node Node[deploying]
2017-08-06 17:25:08.540:DBUG:oejd.AppLifeCycle:main: Calling org.eclipse.jetty.deploy.bindings.StandardDeployer for App[null,/opt/web/mybase/webapps/
*.org.xml]
2017-08-06 17:25:08.565:DBUG:oejx.XmlParser:main: parse: file:/opt/web/mybase/webapps/beta.
**.org.xml
2017-08-06 17:25:08.567:DBUG:oejx.XmlParser:main: parsing: sid=file:/opt/web/mybase/webapps/beta.
**.org.xml,pid=null
2017-08-06 17:25:08.579:DBUG:oejx.XmlParser:main: resolveEntity(-//Jetty//Configure//EN, http://www.eclipse.org/jetty/configure_9_3.dtd)
2017-08-06 17:25:08.584:DBUG:oejx.XmlParser:main: Redirected entity http://www.eclipse.org/jetty/configure_9_3.dtd --> jar:file:/opt/jetty/jetty-distribution-9.4.6.v20170531/lib/jetty-xml-9.4.6.v20170531.jar!/org/eclipse/jetty/xml/configure_9_3.dtd
2017-08-06 17:25:08.879:DBUG:oeju.DecoratedObjectFactory:main: Adding Decorator: org.eclipse.jetty.util.DeprecationWarning@1ea3742
2017-08-06 17:25:09.042:DBUG:oejuc.ContainerLifeCycle:main: o.e.j.w.WebAppContext@183967b{/,null,UNAVAILABLE} added {org.eclipse.jetty.server.session.SessionHandler12743635==dftMaxIdleSec=-1,MANAGED}
2017-08-06 17:25:09.132:DBUG:oejuc.ContainerLifeCycle:main: org.eclipse.jetty.security.ConstraintSecurityHandler@136e911 added {org.eclipse.jetty.security.DefaultAuthenticatorFactory@a3426,POJO}
2017-08-06 17:25:09.149:DBUG:oejuc.ContainerLifeCycle:main: org.eclipse.jetty.server.session.SessionHandler12743635==dftMaxIdleSec=-1 added {org.eclipse.jetty.security.ConstraintSecurityHandler@136e911,MANAGED}
2017-08-06 17:25:09.199:DBUG:oejuc.ContainerLifeCycle:main: org.eclipse.jetty.security.ConstraintSecurityHandler@136e911 added {org.eclipse.jetty.servlet.ServletHandler@1df90b0,MANAGED}
2017-08-06 17:25:09.202:DBUG:oejuc.ContainerLifeCycle:main: o.e.j.w.WebAppContext@183967b{/,null,UNAVAILABLE} added {org.eclipse.jetty.servlet.ErrorPageErrorHandler@1536d79,MANAGED}
2017-08-06 17:25:09.267:DBUG:oejx.XmlConfiguration:main: XML o.e.j.w.WebAppContext@183967b{/,null,UNAVAILABLE}.setContextPath(/)
2017-08-06 17:25:09.272:DBUG:oejx.XmlConfiguration:main: XML o.e.j.w.WebAppContext@183967b{/,null,UNAVAILABLE}.setWar(/opt/web/mybase/webapps/beta.
*.org.war)
2017-08-06 17:25:09.276:DBUG:oejx.XmlConfiguration:main: XML o.e.j.w.WebAppContext@183967b{/,null,UNAVAILABLE}{/beta.*.org.war}.setExtractWAR(true)
2017-08-06 17:25:09.281:IGNORED:oejx.XmlConfiguration:main:
java.lang.NoSuchMethodException: org.eclipse.jetty.webapp.WebAppContext.setExtractWAR(java.lang.String)
at java.lang.Class.getMethod(Class.java:1786)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.set(XmlConfiguration.java:488)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:412)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:359)
at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:260)
at org.eclipse.jetty.deploy.providers.WebAppProvider.createContextHandler(WebAppProvider.java:301)
at org.eclipse.jetty.deploy.App.getContextHandler(App.java:105)
at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:36)
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:502)
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:150)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:564)
at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:239)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.server.Server.start(Server.java:452)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:419)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1515)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1439)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1438)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:221)
at org.eclipse.jetty.start.Main.start(Main.java:506)
at org.eclipse.jetty.start.Main.main(Main.java:78)

Note that this method exists:
http://download.eclipse.org/jetty/stable-9/apidocs/org/eclipse/jetty/webapp/WebAppContext.html#setExtractWAR-boolean-

I added some options into /etc/init.d/jetty. Now I get this:

2017-08-06 17:53:51.062:DBUG:oejx.XmlConfiguration:main: XML call setOut
2017-08-06 17:53:51.065:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.server.Server@1479e3f
2017-08-06 17:53:51.082:DBUG:oejuc.ContainerLifeCycle:main: org.eclipse.jetty.server.Server@1479e3f added {org.eclipse.jetty.server.handler.ErrorHandler@1832929,AUTO}
2017-08-06 17:53:51.133:INFO:oejs.Server:main: jetty-9.4.6.v20170531
2017-08-06 17:53:51.523:DBUG:oejsh.AbstractHandler:main: starting org.eclipse.jetty.server.Server@1479e3f
2017-08-06 17:53:51.524:DBUG:oejuc.AbstractLifeCycle:main: starting qtp22713019{STOPPED,10<=0<=200,i=0,q=0}
2017-08-06 17:53:51.553:DBUG:oejuc.AbstractLifeCycle:main: STARTED @35432ms qtp22713019{STARTED,10<=10<=200,i=10,q=0}
2017-08-06 17:53:51.561:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@16a9952
2017-08-06 17:53:51.586:DBUG:oejuc.AbstractLifeCycle:main: STARTED @35465ms org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@16a9952
2017-08-06 17:53:51.588:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.server.handler.HandlerCollection@33ce2c[org.eclipse.jetty.server.handler.ContextHandlerCollection@3c1e69[], org.eclipse.jetty.server.handler.DefaultHa
ndler@88ff2c]
2017-08-06 17:53:51.590:DBUG:oejsh.AbstractHandler:main: starting org.eclipse.jetty.server.handler.HandlerCollection@33ce2c[org.eclipse.jetty.server.handler.ContextHandlerCollection@3c1e69[], org.eclipse.jetty.server.handler.DefaultHand
ler@88ff2c]
2017-08-06 17:53:51.593:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.server.handler.ContextHandlerCollection@3c1e69[]
2017-08-06 17:53:51.595:DBUG:oejsh.AbstractHandler:main: starting org.eclipse.jetty.server.handler.ContextHandlerCollection@3c1e69[]
2017-08-06 17:53:51.597:DBUG:oejuc.AbstractLifeCycle:main: STARTED @35476ms org.eclipse.jetty.server.handler.ContextHandlerCollection@3c1e69[]
2017-08-06 17:53:51.599:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.server.handler.DefaultHandler@88ff2c
2017-08-06 17:53:51.600:DBUG:oejsh.AbstractHandler:main: starting org.eclipse.jetty.server.handler.DefaultHandler@88ff2c
2017-08-06 17:53:51.602:DBUG:oejuc.AbstractLifeCycle:main: STARTED @35480ms org.eclipse.jetty.server.handler.DefaultHandler@88ff2c
2017-08-06 17:53:51.603:DBUG:oejuc.AbstractLifeCycle:main: STARTED @35482ms org.eclipse.jetty.server.handler.HandlerCollection@33ce2c[org.eclipse.jetty.server.handler.ContextHandlerCollection@3c1e69[], org.eclipse.jetty.server.handler.D
efaultHandler@88ff2c]
2017-08-06 17:53:51.606:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.deploy.DeploymentManager@6a3f77
2017-08-06 17:53:51.608:DBUG:oejd.DeploymentManager:main: DeploymentManager using standard bindings
2017-08-06 17:53:51.675:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.deploy.providers.WebAppProvider@7f6473
2017-08-06 17:53:51.677:DBUG:oejdp.ScanningAppProvider:main: WebAppProvider.doStart()
2017-08-06 17:53:51.679:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///opt/web/mybase/webapps/] at interval 1
2017-08-06 17:53:51.715:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.util.Scanner@17d0008
2017-08-06 17:53:51.732:DBUG:oeju.Scanner:main: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:53:51.748:DBUG:oeju.Scanner:main: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:53:51.752:DBUG:oejuc.AbstractLifeCycle:main: STARTED @35631ms org.eclipse.jetty.util.Scanner@17d0008
2017-08-06 17:53:51.754:DBUG:oejuc.AbstractLifeCycle:main: STARTED @35633ms org.eclipse.jetty.deploy.providers.WebAppProvider@7f6473
2017-08-06 17:53:51.756:DBUG:oejuc.AbstractLifeCycle:main: STARTED @35635ms org.eclipse.jetty.deploy.DeploymentManager@6a3f77
2017-08-06 17:53:51.758:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.server.AsyncNCSARequestLog@feb3fa
2017-08-06 17:53:51.771:INFO:oejs.AbstractNCSARequestLog:main: Opened /opt/web/mybase/logs/2017_08_06.request.log
2017-08-06 17:53:51.789:DBUG:oejuc.AbstractLifeCycle:main: STARTED @35668ms org.eclipse.jetty.server.AsyncNCSARequestLog@feb3fa
2017-08-06 17:53:51.791:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.server.handler.ErrorHandler@1832929
2017-08-06 17:53:51.793:DBUG:oejsh.AbstractHandler:main: starting org.eclipse.jetty.server.handler.ErrorHandler@1832929
2017-08-06 17:53:51.794:DBUG:oejuc.AbstractLifeCycle:main: STARTED @35673ms org.eclipse.jetty.server.handler.ErrorHandler@1832929
2017-08-06 17:53:51.796:DBUG:oejuc.AbstractLifeCycle:main: starting ServerConnector@f5e5e3{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2017-08-06 17:53:51.875:DBUG:oejuc.ContainerLifeCycle:main: ServerConnector@f5e5e3{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} added {sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:8080],POJO}
2017-08-06 17:53:51.919:DBUG:oejuc.AbstractLifeCycle:main: starting HttpConnectionFactory@fbd816[HTTP/1.1]
2017-08-06 17:53:51.923:DBUG:oejuc.AbstractLifeCycle:main: STARTED @35802ms HttpConnectionFactory@fbd816[HTTP/1.1]
2017-08-06 17:53:51.927:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.server.ServerConnector$ServerConnectorManager@34c75a
2017-08-06 17:53:52.058:DBUG:oejuc.ContainerLifeCycle:main: org.eclipse.jetty.io.ManagedSelector@49fd9b id=0 keys=-1 selected=-1 added {EatWhatYouKill@adede5/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@1928643/IDLE/0/1,AUTO}
2017-08-06 17:53:52.069:DBUG:oejuc.ContainerLifeCycle:main: org.eclipse.jetty.server.ServerConnector$ServerConnectorManager@34c75a added {org.eclipse.jetty.io.ManagedSelector@49fd9b id=0 keys=-1 selected=-1,AUTO}
2017-08-06 17:53:52.077:DBUG:oejuc.AbstractLifeCycle:main: starting org.eclipse.jetty.io.ManagedSelector@49fd9b id=0 keys=-1 selected=-1
2017-08-06 17:53:52.082:DBUG:oejuc.AbstractLifeCycle:main: starting EatWhatYouKill@adede5/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@1928643/IDLE/0/1
2017-08-06 17:53:52.084:DBUG:oejuc.AbstractLifeCycle:main: STARTED @35963ms EatWhatYouKill@adede5/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@1928643/IDLE/0/1
2017-08-06 17:53:52.130:DBUG:oejut.QueuedThreadPool:main: queue org.eclipse.jetty.io.ManagedSelector$$Lambda$25/19597309@bf3bbf
2017-08-06 17:53:52.133:DBUG:oejut.QueuedThreadPool:qtp22713019-9: run org.eclipse.jetty.io.ManagedSelector$$Lambda$25/19597309@bf3bbf
2017-08-06 17:53:52.140:DBUG:oejuc.AbstractLifeCycle:main: STARTED @36019ms org.eclipse.jetty.io.ManagedSelector@49fd9b id=0 keys=0 selected=0
2017-08-06 17:53:52.146:DBUG:oejuc.AbstractLifeCycle:main: STARTED @36025ms org.eclipse.jetty.server.ServerConnector$ServerConnectorManager@34c75a
2017-08-06 17:53:52.156:DBUG:oejuts.EatWhatYouKill:qtp22713019-9: EatWhatYouKill@adede5/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@1928643/PRODUCING/0/1 execute true
2017-08-06 17:53:52.190:DBUG:oejuc.ContainerLifeCycle:main: ServerConnector@f5e5e3{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} added {acceptor-0@1c24521,POJO}
2017-08-06 17:53:52.203:DBUG:oejuts.EatWhatYouKill:qtp22713019-9: EatWhatYouKill@adede5/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@1928643/PRODUCING/0/1 produce non-blocking
2017-08-06 17:53:52.203:DBUG:oejut.QueuedThreadPool:main: queue acceptor-0@1c24521
2017-08-06 17:53:52.206:DBUG:oeji.ManagedSelector:qtp22713019-9: Selector loop waiting on select
2017-08-06 17:53:52.207:DBUG:oejut.QueuedThreadPool:qtp22713019-12: run acceptor-0@1c24521
2017-08-06 17:53:52.214:INFO:oejs.AbstractConnector:main: Started ServerConnector@f5e5e3{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2017-08-06 17:53:52.256:DBUG:oejuc.AbstractLifeCycle:main: STARTED @36135ms ServerConnector@f5e5e3{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2017-08-06 17:53:52.267:INFO:oejs.Server:main: Started @36144ms
2017-08-06 17:53:52.268:DBUG:oejuc.AbstractLifeCycle:main: STARTED @36147ms org.eclipse.jetty.server.Server@1479e3f
2017-08-06 17:53:52.768:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:53:53.776:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:53:54.787:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:53:55.798:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:53:56.808:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:53:57.818:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:53:58.828:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:53:59.838:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:00.847:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:01.857:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:02.867:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:03.878:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:04.888:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:05.898:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:06.907:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:07.917:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:08.927:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:09.937:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:10.947:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:11.957:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:12.967:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:13.977:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:14.987:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:15.998:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:17.007:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:18.017:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:19.027:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:20.037:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:21.047:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:22.057:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:23.067:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:24.077:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:25.087:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:26.097:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:27.107:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:28.117:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:29.128:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:30.139:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:31.149:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:32.160:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}
2017-08-06 17:54:33.171:DBUG:oeju.Scanner:Scanner-0: scanned [/opt/web/mybase/webapps]: {}

When I run this as root and when I modify jetty.service so that it uses "root" as a user and "root" as a group, it works:
/usr/bin/java -Djetty.home=/opt/jetty/jetty-distribution-9.4.6.v20170531 -Djetty.base=/opt/web/mybase -Djava.io.tmpdir=/opt/jetty/temp -jar /opt/jetty/jetty-distribution-9.4.6.v20170531/start.jar jetty.state=/opt/web/mybase/jetty.state jetty-started.xml console-capture.xml

I don't understand why.

I'll try to set "extractWAR" to false because I wonder whether Jetty fails to extract the WAR into /opt/jetty/temp when I use "jetty" as a user and as a group.

Also try unpacking the war your self and deploying the unpacked directory.

regards

On 7 August 2017 at 09:23, Julien Gouesse notifications@github.com wrote:

I'll try to set "extractWAR" to false because I wonder whether Jetty fails
to extract the WAR into /opt/jetty/temp when I use "jetty" as a user and as
a group.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/eclipse/jetty.project/issues/1485#issuecomment-320539556,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEUrXTKZol8tnqlSgcPeWKX8Bl-7CQKks5sVksMgaJpZM4NCR2N
.

--
Greg Wilkins gregw@webtide.com CTO http://webtide.com

Good idea. I'll replace the WAR by a directory containing the unpacked content of this WAR in the configuration file:
http://www.eclipse.org/jetty/documentation/current/configuring-specific-webapp-deployment.html#using-basic-descriptor-files

Our both tests will help to determine whether the problem comes from the extraction.

None works, Jetty scans every second but finds nothing :(

What is it?

java.lang.NullPointerException
at org.eclipse.jetty.websocket.server.WebSocketServerFactory.toString(WebSocketServerFactory.java:641)
at java.lang.String.valueOf(String.java:2994)
at org.eclipse.jetty.util.log.StdErrLog.format(StdErrLog.java:552)
at org.eclipse.jetty.util.log.StdErrLog.format(StdErrLog.java:441)
at org.eclipse.jetty.util.log.StdErrLog.debug(StdErrLog.java:405)
at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)
at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:226)
at org.eclipse.jetty.websocket.server.WebSocketServerFactory.(WebSocketServerFactory.java:149)
at org.eclipse.jetty.websocket.server.WebSocketServerFactory.(WebSocketServerFactory.java:126)
at org.eclipse.jetty.websocket.server.WebSocketServerFactory.(WebSocketServerFactory.java:105)
at org.eclipse.jetty.websocket.server.NativeWebSocketConfiguration.(NativeWebSocketConfiguration.java:51)
at org.eclipse.jetty.websocket.server.NativeWebSocketServletContainerInitializer.getDefaultFrom(NativeWebSocketServletContainerInitializer.java:36)
at org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer.configureContext(WebSocketServerContainerInitializer.java:139)
at org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer.onStartup(WebSocketServerContainerInitializer.java:206)
at org.eclipse.jetty.plus.annotation.ContainerInitializer.callStartup(ContainerInitializer.java:136)
at org.eclipse.jetty.annotations.ServletContainerInitializersStarter.doStart(ServletContainerInitializersStarter.java:63)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:329)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1501)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1463)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:502)
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:150)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:564)
at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:239)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.server.Server.start(Server.java:452)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:419)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1515)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1439)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1438)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:221)
at org.eclipse.jetty.start.Main.start(Main.java:506)
at org.eclipse.jetty.start.Main.main(Main.java:78)

java.lang.RuntimeException: java.lang.NullPointerException
at org.eclipse.jetty.annotations.ServletContainerInitializersStarter.doStart(ServletContainerInitializersStarter.java:68)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:329)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1501)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1463)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:502)
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:150)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:564)
at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:239)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.server.Server.start(Server.java:452)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:419)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1515)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1439)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1438)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:221)
at org.eclipse.jetty.start.Main.start(Main.java:506)
at org.eclipse.jetty.start.Main.main(Main.java:78)

java.lang.NullPointerException
at org.eclipse.jetty.websocket.server.WebSocketServerFactory.toString(WebSocketServerFactory.java:641)
at java.lang.String.valueOf(String.java:2994)
at org.eclipse.jetty.util.log.StdErrLog.format(StdErrLog.java:552)
at org.eclipse.jetty.util.log.StdErrLog.format(StdErrLog.java:441)
at org.eclipse.jetty.util.log.StdErrLog.debug(StdErrLog.java:405)
at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)
at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:226)
at org.eclipse.jetty.websocket.server.WebSocketServerFactory.(WebSocketServerFactory.java:149)
at org.eclipse.jetty.websocket.server.WebSocketServerFactory.(WebSocketServerFactory.java:126)
at org.eclipse.jetty.websocket.server.WebSocketServerFactory.(WebSocketServerFactory.java:105)
at org.eclipse.jetty.websocket.server.NativeWebSocketConfiguration.(NativeWebSocketConfiguration.java:51)
at org.eclipse.jetty.websocket.server.NativeWebSocketServletContainerInitializer.getDefaultFrom(NativeWebSocketServletContainerInitializer.java:36)
at org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer.configureContext(WebSocketServerContainerInitializer.java:139)
at org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer.onStartup(WebSocketServerContainerInitializer.java:206)
at org.eclipse.jetty.plus.annotation.ContainerInitializer.callStartup(ContainerInitializer.java:136)
at org.eclipse.jetty.annotations.ServletContainerInitializersStarter.doStart(ServletContainerInitializersStarter.java:63)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:329)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1501)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1463)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:502)
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:150)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:564)
at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:239)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.server.Server.start(Server.java:452)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:419)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1515)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1439)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1438)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:221)
at org.eclipse.jetty.start.Main.start(Main.java:506)
at org.eclipse.jetty.start.Main.main(Main.java:78)

I had to disable the debug logs to make Jetty work anew.

@gouessej that's fixed in issue #1603

Julien,

Can you create a small test project that demonstrates the delay you are
seeing?
Jetty can definitely deploy expanded war files, so there is something going
on in what you are doing that we cannot diagnose as we can see what you are
doing.

regards

On 8 August 2017 at 06:25, Joakim Erdfelt notifications@github.com wrote:

@gouessej https://github.com/gouessej that's fixed in issue #1603
https://github.com/eclipse/jetty.project/issues/1603


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/eclipse/jetty.project/issues/1485#issuecomment-320770884,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEUrY0LvPIP72mcFbtSgpSl_fSOv9Xxks5sV3K5gaJpZM4NCR2N
.

--
Greg Wilkins gregw@webtide.com CTO http://webtide.com

@gregw I confirm that Jetty deploys WAR files. Actually there are two problems. At first, I succeed in deploying WARs only as root, static deployment (without WebAppContext) works both as "jetty" and as root. Secondly, there is this nasty delay appearing only once at the very first access of any JSP, it varies between 3 and 12 seconds.

Yes I can create a test project but what do you expect in it? I simply follow the steps I mention in my tutorial. Then I replace the static deployment by the deployment of a WAR.

Should "root" remain the owner of JETTY_HOME?

JETTY_USER isn't mentioned in the documentation on the startup of a Unix service:
http://www.eclipse.org/jetty/documentation/current/startup-unix-service.html

Is it intentional?

Q: I'm trying to run solr jetty under systemd as a simple service by passing -f to the jetty start script. When systemd sends it a SIGTERM (the normal way for the OS to tell a process to shut down gracefully), jetty exits with status 134 or when I start it in an interactive bash shell, it exits on CTRL-C with status 130.

Anything but status 0 means there is an error. Systemd (IMO correctly) interprets services that exit with nonzero status as broken. Is there a reason for this?

@aphor Please can you be more precise? Why do you need to pass "-f"?

Unless you can't, I think you almost certainly should try to run Jetty as a simple type service in systemd.

https://www.freedesktop.org/software/systemd/man/systemd.service.html
http://patrakov.blogspot.com/2011/01/writing-systemd-service-files.html

Systemd controls stdin, stdout, and stderr for "simple" type services. If the process exits (after forking), systemd will assume the service is down. This is the preferred way to run a service on any Linux system that uses systemd.

Systemd, provides buggy whip support for services that fork a daemon process into the background. This is significantly more complicated and introduces startup script technical debt, for basic stuff systemd does for you, including but not limited to changing to a non-privileged user, managing pid files, which are unreliable duplication of state (is the service running?), and also dependency checking, which either the daemon or a wrapper script must perform. Systemd then must be configured to accomodate the sometimes idiosyncratic startup and shutdown semantics provided by the daemon or wrapper script. Things can quickly evolve to resemble a rube goldberg machine.

I'm running solr in (bundled) jetty, and I noticed that when systemd sends SIGTERM to the simple service jvm process, the process exits with a nonzero (130, or 134) exit code, which is odd, because that's supposed to mean some kind of crash. What I (and probably most of the Linux/Unix world) expects is that when the jvm SIGTERM handler runs, it does everything required for a graceful shutdown, and exits with a status 0, unless there was a problem, in which nonzero should be the exit code, and to be nice, something about the error should be logged, conventionally to STDERR. There doesn't seem to be any documentation on the exit codes, so I can't even figure out what 130 and 134 mean.

https://stackoverflow.com/questions/2975248/how-to-handle-a-sigterm
https://docs.oracle.com/javase/9/troubleshoot/handle-signals-and-exceptions.htm#JSTGD345
https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/signals006.html
https://www.freedesktop.org/software/systemd/man/systemd.kill.html#

Exit code 130 and 134 are standard for Java and signals, handled by the Java JVM itself.

It's typically described as exit_code = 128 + <signal_id>

http://journal.thobe.org/2013/02/jvms-and-kill-signals.html

Other Java projects have added SuccessExitStatus=130 143 to their systemd files because of this.

Example: MrAlias/druid@6929383a2f61ef1beae60033b55b4a9c5048ae71

@olamy can you review the state of this issue and see if it is closeable?

Wow. Someone reverse engineered (undocumented) Java 8 signal handling exit behaviors and codes in 2013, and this is sufficient to define a standard? It would be a loss if the magnanimous blog of Tobias Ivarsson were lost.

Tuesday, February 19, 2013
JVMs and kill signals
Ever wondered how a JVM reacts to various kill signals? The (intended) behaviour might be documented somewhere already, but I found having a table of > the actual behaviour available quite useful. In particular I wanted to know which kill signals trigger the JVM to run registered shutdown hooks, and > > which kill signals don't actually terminate the JVM. So I decided to compile a table of that information.

I wrote up a small Java application that just registers a shutdown hook that I can detect whether it has executed or not, and then sleeps until I get a chance to kill it:

class Death {
  public static void main(String... args) throws Exception {
    Runtime.getRuntime().addShutdownHook( new Thread(){
      @Override
      public void run()
      {
        System.out.println("Shutting down");
      }
    } );
    for (;;) Thread.sleep(100);
  }
}

Then I ran the program in one terminal window (java Death; echo $?) while iterating through all kill signals (0-31) in another:

    kill -$SIGNAL $(jps | grep Death | cut -d\  -f1)

signal | shutdown | runs hook | exit code | comment
-------|-----------|-----------|------------|--------
default (15) | yes | yes | 143 | SIGTERM | is the default unix kill signal
0 | no | - | - |
1 (SIGHUP) | yes | yes | 129 |
2 (SIGINT) | yes | yes | 130 | SIGINT is the signal sent on ^C
3 (SIGQUIT) | no | - | - | Makes the JVM dump threads / stack-traces
4 (SIGILL) | yes | no | 134 | Makes the JVM write a core dump and abort on trap 6
5 | yes | no | 133 | Makes the JVM exit with "Trace/BPT trap: 5"
6 (SIGABRT) | yes | no | 134 | Makes the JVM exit with "Abort trap: 6"
7 | yes | no | 135 | Makes the JVM exit with "EMT trap: 7"
8 (SIGFPE) | yes | no | 134 | Makes the JVM write a core dump and abort on trap 6
9 (SIGKILL) | yes | no | 137 | The JVM is forcibly killed (exits with "Killed: 9")
10 (SIGBUS) | yes | no | 134 | Emulates a "Bus Error"
11 (SIGSEGV) | yes | no | 134 | Emulates a "Segmentation fault"
12 | yes | no | 140 | Makes the JVM exit with "Bad system call: 12"
13 | no | - | - |
14 | yes | no | 142 | Makes the JVM exit with "Alarm clock: 14"
15 (SIGTERM) | yes | yes | 143 | This is the default unix kill signal
16 | no | - | - |
17 | no | - | 145 | Stops the application (sends it to the background), same as ^Z
18 | no | - | 146 | Stops the application (sends it to the background), same as ^Z
19 | no | - | - |
20 | no | - | - |
21 | no | - | 149 | Stops the application (sends it to the background), same as ^Z
22 | no | - | 150 | Stops the application (sends it to the background), same as ^Z
23 | no | - | - |
24 | yes | no | 152 | Makes the JVM exit with "Cputime limit exceeded: 24"
25 | no | - | - |
26 | yes | no | 154 | Makes the JVM exit with "Virtual timer expired: 26"
27 | yes | no | 155 | Makes the JVM exit with "Profiling timer expired: 27"
28 | no | - | - |
29 | no | - | - |
30 | yes | no | 158 | Makes the JVM exit with "User defined signal 1: 30"
31 | yes | no | 134 | Makes the JVM exit on Segmentation fault

This list was compiled using (a quite old) Oracle Hotspot Java 8 EA on Mac OS X:

java version "1.8.0-ea" Java(TM) SE Runtime Environment (build 1.8.0-ea-b65) Java HotSpot(TM) 64-Bit Server VM (build 25.0-b09, mixed mode)
Hope this is useful to more people than myself.

Posted by Tobias at 10:42
Labels: Java, JVM

The very last joakime's suggestion is ok for me, feel free to add SuccessExitStatus=130 143. My other problems aren't caused by my systemd file but by my mistakes in the setting of the permissions in some directories, it has nothing to do with Jetty.

@aphor You could run your test with a more recent version of Java 1.8, why not using AdoptOpenJDK?

@aphor Have you tested Type=simple? systemd could identify the main process without relying on PIDFile according to the documentation you quoted. Are there any drawbacks of switching from forking to simple?

I'm traveling today, but I'll test simple mode service unit file with the success on exit config this week.

@aphor Sorry to bump you again with this. Have you had a chance of trying your suggestion?

Sorry. I'm super busy with other stuff, but I can pass this off to a colleague!

@cristian-popa since you're back from vacation... before you get ramped up, would you like to see if this jetty build works with our solr deployment automation with systemd unit file startup?

Guys you may find useful the following little project of mine:
https://github.com/cosenmarco/jetty-systemd-notifier

Jetty seems to start fine and reports status correctly to SystemD.
Note for now the .service file is just a small test and it's not containing many features of the original jetty.sh but it's a starting point.

@gregw I just realise that I made a pull request on the wrong branch, the "master" branch. Should I make another pull request against the branch "jetty-9.4.x"?

You can edit the branch in a PR.

@joakime I hope that it's correct: https://github.com/eclipse/jetty.project/pull/4259

Was this page helpful?
0 / 5 - 0 ratings