Nixpkgs: Maven builds are not documented in nixpgs manual

[cko]

Many Java Projects are build with Maven. There are a couple of tools around that like mvn2nix and buildMaven which should be documented in the nixpkgs manual.

[Gerschtli]

Whats the state of this?
I'm asking because I can't figure out how to do it..

I executed mvn org.nixos.mvn2nix:mvn2nix-maven-plugin:mvn2nix and my default.nix looks like

with import <nixpkgs> { };
(buildMaven ./project-info.json).build

but I' m getting

[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.springframework:gs-rest-service:0.1.0: Cannot access spring-releases (https://repo.spring.io/libs-release) in offline mode and the artifact org.springframework.boot:spring-boot-starter-parent:pom:1.5.8.RELEASE has not been downloaded from it before. and 'parent.relativePath' points at wrong local POM @ line 10, column 13
 @ 
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project org.springframework:gs-rest-service:0.1.0 (/tmp/nix-build-gs-rest-service-0.1.0.jar.drv-0/spring-rest-api/pom.xml) has 1 error
[ERROR]     Non-resolvable parent POM for org.springframework:gs-rest-service:0.1.0: Cannot access spring-releases (https://repo.spring.io/libs-release) in offline mode and the artifact org.springframework.boot:spring-boot-starter-parent:pom:1.5.8.RELEASE has not been downloaded from it before. and 'parent.relativePath' points at wrong local POM @ line 10, column 13 -> [Help 2]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
builder for ‘/nix/store/d7zmpfil3hhbx868jqsz9kvbx0ln4kxg-gs-rest-service-0.1.0.jar.drv’ failed with exit code 1
error: build of ‘/nix/store/d7zmpfil3hhbx868jqsz9kvbx0ln4kxg-gs-rest-service-0.1.0.jar.drv’ failed

on nix-build.

pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.springframework</groupId>
    <artifactId>gs-rest-service</artifactId>
    <version>0.1.0</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.8.RELEASE</version>
    </parent>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.jayway.jsonpath</groupId>
            <artifactId>json-path</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.nixos.mvn2nix</groupId>
            <artifactId>mvn2nix-maven-plugin</artifactId>
            <version>1.2.0</version>
        </dependency>
    </dependencies>

    <properties>
        <java.version>1.8</java.version>
    </properties>


    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

    <repositories>
        <repository>
            <id>spring-releases</id>
            <url>https://repo.spring.io/libs-release</url>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>spring-releases</id>
            <url>https://repo.spring.io/libs-release</url>
        </pluginRepository>
    </pluginRepositories>
</project>

If my comment should go in a separate issue, I will move it ;)

[jerith666]

Thanks for the reminder -- I'd run across this same problem and fixed it locally, but forgot to ever make a PR. Done now as https://github.com/NixOS/mvn2nix-maven-plugin/pull/11.

However, that wasn't enough to get my test case working. I actually ended up abandoning mvn2nix and wrote my own shell script, which I ran on the local maven repo after building the project outside of nix:

#!/usr/bin/env bash

#run 'mvn -Dmaven.repo.local=dir package' (note: *not* dependency:go-offline, that doesn't reliably download everything)
#then 'repo-to-fetchMaven.sh dir > repo.nix'

echo "{fetchMaven} :";
echo "";
echo "let";
echo "";

deps="";

for f in $(find $1 -type f -not -name \*.sha1 -not -name _remote.repositories | sort); do
    groupId=$(echo $(dirname $(dirname $(dirname $f))) | sed "s|^$1||" | sed "s|/|.|g");
    fName=$(basename $f);
    version=$(basename $(dirname $f));
    fNameNoExt=$(basename -s .signature $(basename -s .pom $(basename -s .jar $fName)));
    if echo $fNameNoExt | grep ".-${version}-" > /dev/null; then
        classifier=$(echo $fNameNoExt | sed -E "s/^.*-${version}-//");
        cls="      suffix = \"-$classifier\";
";
    else
        cls="";
    fi;
    artifactId=$(echo $fNameNoExt | sed -E "s/-${version}(-.*)?$//");
    type=$(echo $fName | rev | cut -d . -f 1 | rev);
    sha=$(sha512sum $f | cut -d " " -f 1);
    pkg=$(echo $(dirname $f) | sed "s|^$1||" | sed "s|/|_|g" | sed "s|\.|_|g")_$type;
    cat <<EOF
    $pkg = fetchMaven {
      groupId = "$groupId";
      artifactId = "$artifactId";
$cls      version = "$version";
      type = "$type";
      sha512 = "$sha";
    };
EOF
    deps="$deps $pkg";
done;

echo "";
echo "in";
echo "  [$deps ]";
echo "";

[jerith666]

I also made this minor change to make buildMaven a bit more flexible, but haven't thought through whether it's ready for upstream or not: https://github.com/jerith666/nixpkgs/commit/3f0940295df780f457a1ae8b09aff9753918f8d8.

[Gerschtli]

@jerith666 Looks cool, but how would a default.nix look like for such a maven project?

[jerith666]

Here's what I came up with for apache commons-lang:

$ cat default.nix
with import <nixpkgs> {};

{} :

let

  repoInfo = import ./repo.nix {
    fetchMaven=javaPackages.fetchMaven;
  };

in

javaPackages.mavenbuild rec{
  name = "commons-lang";
  version = "3.5";

  src = fetchgit {
    url = https://git-wip-us.apache.org/repos/asf/commons-lang.git;
    rev = "36f98d87b24c2f542b02abbf6ec1ee742f1b158b"; #sha1 of LANG_3_5 tag;
    sha256 = "16wx2jg8pcfb5mg28pwyihv3gkhgnr3gj2146kps2lngskbpxwpk";
  };

  mavenDeps = repoInfo;

  quiet = false;
  skipTests = false;

  meta = {};

  m2Path = "/org/apache/commons/commons-lang3/${version}";
}

[Gerschtli]

What should I do, if the hashes don't match?

trying http://repo1.maven.org/maven2/com/vaadin/external/google/android-json/0.0.20131108.vaadin1/android-json-0.0.20131108.vaadin1.pom
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2786  100  2786    0     0   2786      0  0:00:01 --:--:--  0:00:01 66333
output path ‘/nix/store/ny0fq46nsg56p5519r29cyhrr8f2x321-android-json-0.0.20131108.vaadin1.pom’ has sha512 hash ‘0z2675wz1axx7vnrvkjr981jnjsk004w2yz3mcf4dl1wkjb0qlmizlmmrm1ccgjm69l1j1vk77mx3bwsmsd8c57z4cch0b4yh75w32p’ when ‘1i0cghq7b89b23a61jl1cq78ci9axvhk21c6g15mldn6kvl3vd14jkp0qd196cdvk921m8q5fpjjxmd98w7yc34d0f23k6d7gakamrb’ was expected
cannot build derivation ‘/nix/store/11hkzx63ybb1kzv69r969z57n87xwnvl-android-json-0.0.20131108.vaadin1.drv’: 1 dependencies couldn't be built

default.nix:

with import <nixpkgs> { };

let
  repoInfo = import ./repo.nix {
    inherit (javaPackages) fetchMaven;
  };
in

javaPackages.mavenbuild rec {
  name = "spring-rest-api";

  src = ./.;

  mavenDeps = repoInfo;

  quiet = false;
  skipTests = true;

  meta = { };

  m2Path = "/de/tobias-happ/spring-rest-api";
}

[jerith666]

I don't know. I get a third value for the hash:

$ curl -O http://repo1.maven.org/maven2/com/vaadin/external/google/android-json/0.0.20131108.vaadin1/android-json-0.0.20131108.vaadin1.pom

$ sha512sum android-json-0.0.20131108.vaadin1.pom 
570c5e0ef46401c818f9a730d474d57c8d5ecf993bc84093a9f231166aae95fe5829064b4e1e68238ed5f1bde00480a9a59501a52ce7ce769fde55f8bc1c233e  android-json-0.0.20131108.vaadin1.pom

[Gerschtli]

Very strange.. :/

[jerith666]

@yegortimoshenko said this over in #32695:

Reimplementing native Maven fetcher is a bad idea: your fetcher won't handle classifiers, Maven repositories other than Central, exclusions, custom destFileName or outputDirectory, stripVersion flag, and probably a whole bunch of other corner cases I couldn't come up with right away: https://maven.apache.org/plugins/maven-dependency-plugin/usage.html

It does handle classifiers.

It does not handle maven repositories other than Central. Not sure how often that comes up in practice for open source projects; for corporate closed-source projects, it's probably most frequently the case that everything flows through an internal repo manager (nexus or artifactory); that's handled easily enough IMO, by letting https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/java-modules/m2install.nix#L10 be parameterized.

I'm not sure what the other three things are. If they're options to the maven dependency plugin, I'm not sure they really matter, since the goal is to create a valid normally-formed local repo, not download maven artifacts in a one-off fashion.

[yegortimoshenko]

It does not handle maven repositories other than Central. Not sure how often that comes up in practice for open source projects

It will come up very often: http://mvnrepository.com/repos. Sonatype Releases end up in Central, but most other repositories don't. Clojure projects use Clojars primarily. Or, just from yesterday, Hadoop: #32893. Check their POM: https://github.com/apache/hadoop/blob/release-3.0.0-RC1/pom.xml#L61

I'm not sure what the other three things are. If they're options to the maven dependency plugin, I'm not sure they really matter, since the goal is to create a valid normally-formed local repo, not download maven artifacts in a one-off fashion.

The rest specify how to name jars. If POM uses any of these options, this script will fail to generate proper Nix to fetch jars in the first place.

[jerith666]

Okay, good to know that repos other than central are pretty common -- thanks for the concrete examples!

The rest specify how to name jars. If pom.xml uses these and they are not handled, it just won't build.

I did a quick google for maven destFileName, maven outputDirectory, and maven stripVersion -- in all three cases, google is pointing me at the dependency plugin. But it would seem to me that, in order to fetch artifacts and construct a local maven repo, all we care about are the maven coordinates (5 things: GAV + packaging & classifier) ... what am I missing? Do you have an example pom.xml that uses these features?

[yegortimoshenko]

Sure:

• destFileName: https://github.com/apache/hadoop/search?q=destFileName
• outputDirectory: https://github.com/apache/hadoop/search?q=outputDirectory

However, I was wrong about these options: my understanding was that such artifcats are fetched straight from Maven and mutate name of the artifact inside the local repo, while actually they are just copies.

It's down to remote repositories, then. I think there is an ephemeral _remote.repositories file that could be used to determine which repository should be used.

[jerith666]

hmm, interesting; apparently there is also a _maven.repositories file; the difference is as yet unclear, but these are good leads. ref: https://stackoverflow.com/questions/16866978/maven-cant-find-my-local-artifacts.

[jerith666]

An earlier version of your comment also mentioned systemPath e.g. https://github.com/apache/hadoop/blob/release-3.0.0-RC1/hadoop-common-project/hadoop-annotations/pom.xml#L52. This is a bit of a separate issue since these artifacts by definition are not obtained by maven and don't reside in the local repo. But still an important consideration for full maven support, so I don't want to lose track of the issue.

[yegortimoshenko]

This is a bit of a separate issue since these artifacts by definition are not obtained by maven and don't reside in the local repo.

That's why I've dropped that comment.

You've changed my mind and now I think this is not only feasible approach, but better than what I've proposed. Thank you for your patience :-)

[jerith666]

Glad to hear it! :) The manual pre-build + shell script I have here obviously leaves several things to be desired -- I think it's worth contacting the maven folks to see if (a) dependency:go-offline can be made to work more reliably or (b) they have any other ideas.

[yegortimoshenko]

I don't think there is a way to make dependency:go-offline work any more reliably: the problem is that some plugins don't declare their dependencies but instead inject them at runtime.

I've had the same problem with Leiningen (Clojure build tool based on Maven): https://github.com/technomancy/leiningen/issues/2055

[jerith666]

Noting @icetan's https://github.com/icetan/mavenix as another approach to investigate. Cursory glance looks like it's built on top of dependency:go-offline, but I don't claim to have grokked it completely.

[asymmetric]

@jerith666 I'm giving your script a try. I've seen a couple of issues that I'm currently fixing:

• It sometimes produces duplicates in the repo.nix file - not sure why they're there, so maybe the script should just check for them
• The groupId always started with a ., which produced an invalid URL - I've changed it to groupId=$(echo $(cut -d / -f 2));

I'll let you know how building K goes.

[jluttine]

I'm trying to package Traccar which uses Maven. I'm quite lost how to write the derivation, so any help is greatly appreciated: https://github.com/NixOS/nixpkgs/pull/54309

[Gerschtli]

I've successfully packaged a maven project some time ago, maybe a look at https://github.com/Gerschtli/spring-rest-api can help.

[asymmetric]

I ended up successfully using mavenix after encountering several issues with maven2nix.

[icetan]

I'm trying to package Traccar which uses Maven. I'm quite lost how to write the derivation, so any help is greatly appreciated: #54309

@jluttine I made a PR (https://github.com/jluttine/nixpkgs/pull/1) on your fork with an example of how it could be written using mavenix.

[deliciouslytyped]

Here's an example of using mavenix as part of creating a container https://github.com/NixOS/nixpkgs/issues/79285

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/download-and-wrap-a-jar-extensively-documented-example/8049/4

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/packaging-a-maven-application/8231/1

[fzakaria]

I published https://fzakaria.com/2020/07/20/packaging-a-maven-application-with-nix.html to help anyone who wants understand _one way_ with which to package a Maven application.

I also started yet another solution to create a Nix attrset with the Maven dependencies at https://github.com/fzakaria/mvn2nix
I think it differs from mvn2nix plugin because it's not a plugin but a separate binary + can be built itself with Nix.

It's also a much smaller codebase to audit & reason through.

[RohanHart]

You mean https://fzakaria.com/2020/07/20/packaging-a-maven-application-with-nix.html

[fzakaria]

@RohanHart whoops! Updated my OP.

[fzakaria]

I added documentation in a pull-request: https://github.com/NixOS/nixpkgs/pull/100660/files