Quarkus: QuarkusTest is downloading dependencies to default localRepository (ignores MAVEN_HOME/conf/settings.xml?)

Created on 21 May 2020 · 36Comments · Source: quarkusio/quarkus

Describe the bug
While trying out 1.5.0.CR1, I saw duplicate downloads: First the "regular" ones by the main Maven process but then a second time when the first @QuarkusTest was executed.
I then renamed ~/.m2/repository and voilà: The main Maven process does not re-download anything but the test execution does.
It seems the test execution is not picking up my settings from MAVEN_HOME/conf/settings.xml.
Please note that I have multiple local repositories for isolated development and therefore I do _not_ set localRepository via ~/.m2/settings.xml.

Expected behavior
QuarkusTests must use the same Maven settings as the main Maven process.

Actual behavior

localRepository from MAVEN_HOME/conf/settings.xml is ignored
mirrors from MAVEN_HOME/conf/settings.xml are ignored (I can see that some of the dependency downloads triggered by the QuarkusTest are performed against central)

To Reproduce
Steps to reproduce the behavior:

rename ~/.m2/repository to something else
configure a localRepository in MAVEN_HOME/conf/settings.xml
build some quarkus project incl. QuarkusTest execution (you should already see duplicate downloads)
again rename ~/.m2/repository to something else
build project again, which should now only download when executing the test

Configuration
n/a

Environment (please complete the following information):

Output of uname -a or ver:
MINGW64_NT-10.0-18363 W4DEUMSY9003463 3.0.7-338.x86_64 2019-11-21 23:07 UTC x86_64 Msys
Output of java -version:
openjdk version "11.0.6" 2020-01-14 OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.6+10) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.6+10, mixed mode)
GraalVM version (if different from Java): n/a
Quarkus version or git rev: 1.4.2.Final & 1.5.0.CR1
Build tool (ie. output of mvnw --version or gradlew --version): Apache Maven 3.6.3

Additional context
This is _not_ a regression caused by 1.5.0.CR1 since it also happens with 1.4.2.Final.

kinbug

Source

famod

All 36 comments

I just tried setting localRepository in my MAVEN_HOME/conf/settings.xml and everything worked as expected.
I saw all the extra artifacts being downloaded from the tests go into the localRepository I set.

geoand on 21 May 2020

@geoand Can you please try again with https://github.com/famod/modmono-quarkus?

Where is localRepository in your MAVEN_HOME/conf/settings.xml pointing to?

famod on 21 May 2020

Just tried with your application (but with 1.5.0.CR1 as the version) and the new dependencies did get downloaded by the test into localRepository which was set via $MAVEN_HOME/conf/settings.xml to /home/gandrian/Desktop/maven-repo

geoand on 21 May 2020

So I am going to close this for now. Feel free to reopen if you have more information.

geoand on 21 May 2020

@geoand

Well, I can definitely reproduce it. On two different systems (both Windows 10) and two different projects.
For your test, did you have a localRepository configured in ~/.m2/settings.xml?

famod on 21 May 2020

I do not. Do you?

geoand on 21 May 2020

No. Well. I guess I'll have to record the workflow somehow...

famod on 21 May 2020

I don't see how, but maybe it's a Windows issue? Have you been able to reproduce on a non-Windows machine?

geoand on 21 May 2020

I'm reopening so we can examine whether this is a Windows issue

geoand on 21 May 2020

What is MAVEN_HOME on your machine?

geoand on 21 May 2020

Another thing for you to check: Have you perhaps have a maven.home system property set anywhere?

If you are up for some debugging of the Quarkus code, I would suggest setting a break point at the constructor BootstrapMavenContext and at method getEffectiveSettings.

geoand on 21 May 2020

What is MAVEN_HOME on your machine?

C:\_dev\Maven\latest which is a directory junction:

$ ls -la /c/_dev/Maven/
total 28
drwxr-xr-x 1 Falko 197121  0 Okt 18  2016 apache-maven-3.3.9/
drwxr-xr-x 1 Falko 197121  0 Okt 29  2017 apache-maven-3.5.2/
drwxr-xr-x 1 Falko 197121  0 Jun 17  2018 apache-maven-3.5.4/
drwxr-xr-x 1 Falko 197121  0 Okt  4  2019 apache-maven-3.6.0/
drwxr-xr-x 1 Falko 197121  0 Apr  4  2019 apache-maven-3.6.1/
drwxr-xr-x 1 Falko 197121  0 Aug 27  2019 apache-maven-3.6.2/
drwxr-xr-x 1 Falko 197121  0 Nov  7  2019 apache-maven-3.6.3/
lrwxrwxrwx 1 Falko 197121 32 Nov 22 01:02 latest -> /c/_dev/Maven/apache-maven-3.6.3/

I do not have a MAVEN_HOME or M2_HOME env variable set, but MVN_HOME (which is my own "invention").
On my company laptop (on which I can also reproduce the problem with another project), I do have M2_HOME set to C:\Develop\SomeProject\SomeRepo\SomeRelease\..\Maven\latest (also via directory junction).

Have you perhaps have a maven.home system property set anywhere?

No, I don't.

famod on 21 May 2020

The default repo (~/.m2/repository) is also a junction, to D:\mvnrepos\user (ran out of space at some point).
I tried without this junction but this did not change anything (as expected).

famod on 21 May 2020

Then I am confused, if you don't have MAVEN_HOME set as an environment variable, why do expect the configuration to be picked up?

geoand on 21 May 2020

Because ${maven.home}/conf/settings.xml is a standard location supported by Maven: https://maven.apache.org/settings.html#quick-overview

Furthermore, it is not required to set MAVEN_HOME:

The only place I could find where MAVEN_HOME is mentioned in the official MAVEN docs is here:
https://maven.apache.org/configure.html#mvn-extensions-xml-file
(which looks as if it should be more like ${maven.home}, see settings link above)

See also:

https://stackoverflow.com/questions/26609922/maven-home-mvn-home-or-m2-home
bin/mvn:
# ----------------------------------------------------------------------------- # Apache Maven Startup Script # # Environment Variable Prerequisites # # JAVA_HOME Must point at your Java Development Kit installation. # MAVEN_OPTS (Optional) Java runtime options used when Maven is executed. # MAVEN_SKIP_RC (Optional) Flag to disable loading of mavenrc files. # -----------------------------------------------------------------------------
bin/mvn.cmd
@REM ----------------------------------------------------------------------------- @REM Apache Maven Startup Script @REM @REM Environment Variable Prerequisites @REM @REM JAVA_HOME Must point at your Java Development Kit installation. @REM MAVEN_BATCH_ECHO (Optional) Set to 'on' to enable the echoing of the batch commands. @REM MAVEN_BATCH_PAUSE (Optional) set to 'on' to wait for a key stroke before ending. @REM MAVEN_OPTS (Optional) Java runtime options used when Maven is executed. @REM MAVEN_SKIP_RC (Optional) Flag to disable loading of mavenrc files. @REM -----------------------------------------------------------------------------

Yes, both scripts do pick up a MAVEN_HOME (if set) but this is not a _prerequisite_.

So, long story short: The problem vanishes once I set MAVEN_HOME. 👍
Since that is a fully _optional_ environment variable from the Maven perspective, the Quarkus docs should mention that requirement explicitly. Maybe here? https://quarkus.io/guides/maven-tooling
Maybe the presence of the variable could also be validated at runtime?

famod on 21 May 2020

Aha! Thanks for the details.

I think we might be able to make things more standard compliant.
In the analysis above, where does ${maven.home} come from?
IIUC, it's not set anywhere, it's just the parent of the directory where the maven "binary" is being executed, right?
If that is the case, then we can't do anything in the tests, since the Java process doesn't generally have anything to do with the maven Java process.
If on the other hand it's just something that is configured, we can additionally read that configuration.

geoand on 21 May 2020

It seems I have to correct myself for one detail:

Yes, both scripts do pick up a MAVEN_HOME (if set) but this is not a _prerequisite_.

Both scripts seem to _always_ auto-calculate MAVEN_HOME:

famod on 21 May 2020

Btw, things are not really getting any better when running dev mode in Eclipse via the "Quarkus Eclipse Feature" as this will also re-download dependencies to the wrong local repo.

Which means you have to take care that Eclipse also has MAVEN_HOME set.

famod on 21 May 2020

Which means you have to take care that Eclipse also has MAVEN_HOME set.

Oh wow, this doesn't even work!
As long as Quarkus needs MAVEN_HOME (or something similar), the Eclipse plugin would need to pass it over (if set at all for the Eclipse process).
Depending on where we end up here, I might create an issue for this over at https://github.com/jbosstools/jbosstools-quarkus

famod on 21 May 2020

👍1

Do you want to update the maven documentation with the environment variable that addresses your issue?

I don't think there is anything else that can be done on the Quarkus side of things

geoand on 21 May 2020

Back to the main issue:
FWIW, https://github.com/shrinkwrap/resolver use and document own system properties that can be used to override the settings.xml locations:
https://github.com/shrinkwrap/resolver#system-properties

The names are rather misleading, though, because they look like official Maven properties (which they are not AFAIK): org.apache.maven.[...]-settings

famod on 21 May 2020

Do you want to update the maven documentation with the environment variable that addresses your issue?

I can do that, yes.

famod on 21 May 2020

👍1

@geoand One last thing: It _could_ be argued that Quarkus is "misusing" an internal Maven variable (that was "public" in Maven 1, but isn't anymore). But at least you are not alone, e.g.: https://issues.apache.org/jira/browse/MESOS-1743
This might be something that you guys may want to discuss, once the Quarkus issue count is down to 100. 🙂

cc @aloubyansky @stuartwdouglas

famod on 21 May 2020

Indeed, but it does seem to be pretty well known.
Also, the problem is sort of an edge case that the env var easily solves, so 🤷

geoand on 21 May 2020

We check MAVEN_HOME and maven.home system property, which is set by Maven CLI. The property isn't actually propagated to the tests but it's easy to do that by adding <maven.home>${maven.home}</maven.home> to the surefire plugin config. You could give it a try @famod, if you like.

aloubyansky on 22 May 2020

Thanks, @aloubyansky. That'll surely work too.
It would be better if Quarkus users wouldn't need to care at all but I can live with this!
Either way, Quarkus users should know about this. That's why I created #9523.

famod on 22 May 2020

Given that we generate the surefire plugin config we should simply stick maven.home in there.

aloubyansky on 22 May 2020

I like this idea. Wouldn't work for Eclipse, though: #9339
IntelliJ would pick it up automatically, I guess.

famod on 22 May 2020

It would certainly help every other use-case though, including the non-IDE one. Feel free to submit a PR ;) Or open an issue, if you don't have the time.

aloubyansky on 22 May 2020

@aloubyansky @gsmet AFAIK the correct variable for Maven 2 or higher is M2_HOME and not MAVEN_HOME (which was used in Maven 1). So Quarkus should at least evaluate M2_HOME too, as it may be already set.

sebthom on 13 Jul 2020

@sebthom see https://maven.apache.org/docs/3.5.0/release-notes.html and https://issues.apache.org/jira/browse/MNG-5607

aloubyansky on 13 Jul 2020

@aloubyansky Interesting, so this means neither M2_HOME (maven 2.x-3.4.x), nor MAVEN_HOME (maven 1.x) are supported by current Maven. Why is Quarkus evaluating MAVEN_HOME then at all? I actually came here because I was looking for a way to specify a custom maven settings.xml as I am using mvn --settings path/to/my/settings.xml.

Wouldn't it be more straightforward/less confusing if Quarkus allowed specifying a direct path to a custom settings file via a Quarkus specific ENV Variable/JVM Property, e.g. quarkus.maven.settings.path or something, instead of relying on a retired and undocumented Maven 1.x environment variable?

sebthom on 13 Jul 2020

@aloubyansky Interesting, so this means neither M2_HOME (maven 2.x-3.4.x), nor MAVEN_HOME (maven 1.x) are supported by current Maven. Why is Quarkus evaluating MAVEN_HOME then at all?

At this point, it's used only to locate the conf/settings.xml. There actually is another reason for us to get hold of the Maven home dir but we are doing it based on the value of maven.home that the Maven CLI sets. So I am wondering whether we should also check MAVEN_HOME if the property wasn't propagated (e.g. in tests).
We need to be able to configure the Maven resolver and also properly interpret the Maven CLI args. If we are able to resolve Maven home dir we will use the original Maven CLI parser to do that.

I actually came here because I was looking for a way to specify a custom maven settings.xml as I am using mvn --settings path/to/my/settings.xml.

--settings/-s path does work, doesn't it?

Wouldn't it be more straightforward/less confusing if Quarkus allowed specifying a direct path to a custom settings file via a Quarkus specific ENV Variable/JVM Property, e.g. quarkus.maven.settings.path or something, instead of relying on a retired and undocumented Maven 1.x environment variable?

That'd be very easy to add. Feel free to open an issue and perhaps a PR ;)

aloubyansky on 14 Jul 2020

@aloubyansky

I actually came here because I was looking for a way to specify a custom maven settings.xml as I am using mvn --settings path/to/my/settings.xml.

--settings/-s path does work, doesn't it?

I want to run single test cases from within Eclipse via "Run As -> JUnit Test" but with a custom settings.xml containing HTTP proxy information and a different local repo path. On the command line I have an alias for mvn like this: alias mvn=mvn --settings path/to/my/settings.xml. This allows me to switch between different Maven version by just changing the PATH variable to point to another Maven distribution without having to copy around/sync the settings.xml file or otherwise modifying the extracted Maven distribution (e.g. symlinks).

Here I read, to use a custom settings.xml I have to add a custom JVM property "maven.home" to the registered JDK within Eclipse. This is not very elegant as it

somehow gives the impression that by specifying the maven.home directory also a different Maven distribution will be used under the hood by Quarkus
I need to put my maven settings file into a folder called conf having the name settings.xml and can not point to a file named differently and in another folder.
I actually have to modify the JDK configuration only for Quarkus
additionally, my environment specifying the maven.home property pointing to a folder containing my custom settings file under conf/settings.xml doesn't work (Eclipse 2020-03, Quarkus 1.6.0.Final). Quarkus does not seem to use the file and thus in my case fails to download dependencies.

I would like to be able to either:

Ideally when I execute a JUnit Test like this, I'd prefer quarkus somehow picked up the settings.xml I specified for m2eclipse
Specify the settings.xml file to be used in the application.properties or another file in my project.
Explicitly specify the location of a settings.xml as JVM property in the JVM config. E.g. -Dmaven.settings=<path>

sebthom on 14 Jul 2020

Here I read, to use a custom settings.xml I have to add a custom JVM property "maven.home" to the registered JDK within Eclipse. This is not very elegant as it

somehow gives the impression that by specifying the maven.home directory also a different Maven distribution will be used under the hood by Quarkus

That is partly true. See my previous comments.

I need to put my maven settings file into a folder called conf having the name settings.xml and can not point to a file named differently and in another folder.

True. With maven.home you are customizing the _global maven settings_, i.e. equivalent to the -gs Maven CLI argument, not -s.

I actually have to modify the JDK configuration only for Quarkus

I'm not sure what you mean here, tbh.

additionally, my environment specifying the maven.home property pointing to a folder containing my custom settings file under conf/settings.xml doesn't work (Eclipse 2020-03, Quarkus 1.6.0.Final). Quarkus does not seem to use the file and thus in my case fails to download dependencies.

There is a trap. Quarkus creates a classpath cache for the tests which is created on the first launch, so the subsequent test launches can start faster. The cache is checking the timestamps of the pom files. If a relevant pom has changed, the cache is invalidated. But if you change e.g. the value of maven.home, it won't be noticed. Try removing target/quarkus dir and run the tests again.

I would like to be able to either:

Ideally when I execute a JUnit Test like this, I'd prefer quarkus somehow picked up the settings.xml I specified for m2eclipse

Perhaps I'm missing something but I don't see how these are connected. JUnit and m2eclipse are two separate configs, aren't they?

Specify the settings.xml file to be used in the application.properties or another file in my project.

application.properties is definitely not the place, imo. One thing to keep in mind here is that this file should be used by Maven CLI as well. Which is achieved by adding -s argument that is also picked up by Quarkus tests. Having potentially another default location for the Maven settings for Quarkus tests doesn't look like a good thing.