Jetty.project: Upgrade to 9.4.15-1~18.04.1ubuntu1 fails due to read-only filesystem

[vmario89]

Hi,
after upgrading jetty9 to 9.4.15-1~18.04.1ubuntu1 my previously running .war archive is not working anymore. Starting up fails due to

Apr 20 01:14:17 busy-ubuntu jetty9[17707]: 20 Apr 2019 01:14:17,107 INFO com.sismics.util.filter.RequestContextFilter.init(RequestContextFilter.java:42) Using base data directory: /var/docs
Apr 20 01:14:17 busy-ubuntu jetty9[17707]: log4j:ERROR setFile(null,true) call failed.
Apr 20 01:14:17 busy-ubuntu jetty9[17707]: java.io.FileNotFoundException: /var/docs/log/docs.log (Read-only file system)

I did not change anything else in filesystem. Do you have an idea what could cause "Read-only file system"? Please note that i tried this procedure on two different systems (Ubuntu 18) with same results. If you search on the net for this error it will lead to confusing messages about erroneoues hard drives or file systems. This IS NOT the case. I already used fsck and other tools. Its definitely some ACL or general permission problem belonging to Java or Jetty

I am running openjdk 11 and i already tried to adjust default.policy of java. Made no difference

kind regards, Mario

[joakime]

The jetty distribution does not ship with log4j.
Jetty has it's own Logging called StdErrLog (that writes to STDERR / System.err).
From your errors, it seems that you have a simple log4j configuration issue to work out on your side.

[joakime]

Looking at the sismics code you are running at https://github.com/sismics/docs
It sets up log4j environment, but it's own configurations never write to a file, all of their log4j.properties (they don't use log4j.xml) only write to the console.

[vmario89]

Hi, thanks for feedback. But it might not have to do something with Sismics itself. I am running the completely same webarchive on jetty9 on two other servers without problems. The other servers run Jetty 9.2.14 and use Java 8. I already tried to use Java 8 on the faulty server but that makes no difference.

So i am sure something in latest jetty9 upgrade causes this. At the moment i am trying to revert back to older jetty9

[vmario89]

hm okay i figured out that the same problem occures with tomcat9 which i tried now. Reverting back jetty9 failed on my system (dont know why). So it might belong to some library change in jdk11 (which was openjdk-11-jre-headless 10.0.2, now openjdk-11-jre-headless 11.0.2)

[isaacrj]

I'm facing the same error and I was able to track it down to the jetty9.service file.

This file was NOT included in previous versions of the jetty9 pkg and therefore systemd was starting, stopping, etc Jetty by relying on the old initd scripts.

The new .service file includes the following lines:
ProtectSystem=strict
ReadWritePaths=/var/lib/jetty9

I'm no expert in systemd unit files, but I'm afraid that this lines mean that you can not write anything outside /var/lib/jetty9.

I resolved the problem by removing /lib/systemd/system/jetty9.service and then running systemctl daemon-reload && systemctl restart jetty9 this restores the previous behavior and makes systemd use the old init files to manage Jetty

[josemduarte]

Thanks @isaacrj for the tip. This saved my life! My web application writes some files in another file system different from where it runs in root file system. It worked fine under jetty 9.2, but when trying it in Ubuntu 18.04 with jetty 9.4 it could not write files to the different file system location anymore.

I could not figure out how on earth was jetty or java restricting access to file system server side. Setting ProtectSystem=false in /lib/systemd/system/jetty9.service and running systemctl daemon-reload did the trick. An alternative is to explicitly exclude the file system I want to write to with ReadWritePaths=/mnt/vdb1, keeping ProtectSystem=strict

[joakime]

This seems like a 3rd party (or linux distribution) packaging issue.
Not related to the work performed here at this github project.
There's nothing for Jetty to do to address whatever is causing this issue.

[josemduarte]

@joakime indeed this is an Ubuntu packaging problem.

[ghost]

@joakime You're right and I didn't use ProtectSystem=strict and ReadWritePaths=/var/lib/jetty9 in my contribution as far as I remember.