Ksql: Release build does not include beta artifacts' Maven repository

[colinhicks]

To Reproduce
Check out a release tag, e.g. v0.8.1-ksqldb, then build the project.

Expected behavior
ksqlDB should build successfully at the tag.

Actual behavior
Maven can't find Confluent Platform dependencies.

[INFO] Scanning for projects...
Downloading from confluent: https://packages.confluent.io/maven/io/confluent/rest-utils-parent/6.0.0-beta200320180305/rest-utils-parent-6.0.0-beta200320180305.pom
Downloading from confluent-snapshots: https://s3-us-west-2.amazonaws.com/confluent-snapshots/io/confluent/rest-utils-parent/6.0.0-beta200320180305/rest-utils-parent-6.0.0-beta200320180305.pom
Downloading from central: https://repo.maven.apache.org/maven2/io/confluent/rest-utils-parent/6.0.0-beta200320180305/rest-utils-parent-6.0.0-beta200320180305.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for io.confluent.ksql:ksqldb-parent:0.8.1: Could not find artifact io.confluent:rest-utils-parent:pom:6.0.0-beta200320180305 in confluent (https://packages.confluent.io/maven/) and 'parent.relativePath' points at wrong local POM @ line 22, column 13

Additional context
The release build should add and commit the repository and pluginRepository nodes respective to the artifacts. Here's the missing piece for v0.8.1-ksqldb, for example:

diff --git a/pom.xml b/pom.xml
index 20da1a270..cf07e1ca3 100644
--- a/pom.xml
+++ b/pom.xml
@@ -75,6 +75,10 @@
             <id>confluent-snapshots</id>
             <url>https://s3-us-west-2.amazonaws.com/confluent-snapshots/</url>
         </repository>
+        <repository>
+          <id>packaging-beta</id>
+          <url>https://jenkins-confluent-packages-beta-maven.s3-us-west-2.amazonaws.com/6.0.0-beta200320180305/1/maven/</url>
+        </repository>
     </repositories>

     <pluginRepositories>
@@ -89,6 +93,10 @@
                 <enabled>true</enabled>
             </snapshots>
         </pluginRepository>
+        <pluginRepository>
+          <id>packaging-beta</id>
+          <url>https://jenkins-confluent-packages-beta-maven.s3-us-west-2.amazonaws.com/6.0.0-beta200320180305/1/maven/</url>
+        </pluginRepository>
     </pluginRepositories>

     <properties>

[agavra]

It looks like there's no command to modify the POM in such a way... this may require more hacky maneuvers!

[agavra]

@colinhicks - do you think it would be sufficient to publish the settings.xml alongside the build, and then instruct users to put that in their .m2 or their {maven_home}/conf? There isn't a good way to solve this problem without super hacky sed commands with dubious reliability.

EDIT: for context, the jenkins build uses https://www.jenkins.io/doc/pipeline/steps/pipeline-maven/ to do exactly what I'm suggesting automatically

[colinhicks]

Thanks for taking a look, @agavra.

If we end up needing to create a custom settings.xml, I wonder if we should instead consider updating the docs to include passing that explicitly via mvn -s ksqldb-parent-settings.xml .... Having folks update ~/.m2/settings.xml directly feels pretty odd to me.

I'm inclined to keep considering options ...

• sed doesn't feel _that_ hacky :). I don't think it would be so unreliable, and we could also have the release job consume the altered pom, so that it fails if something is awry.
• To play nicer, could we use https://code.google.com/archive/p/maven-replacer-plugin/ to alter the pom?
• It seems like Jenkins should be able to use the Groovy XmlParser stuff to directly manipulate the file. However, I have a feeling this might be sandboxed/blocked by the Jenkins security model.

[agavra]

I think I found a way using mvn versions:set-property and using a property for the url

[colinhicks]

💯

[the-cybersapien]

I raised this on slack.

After #5497 , mvn clean package doesn't work if there's no pre-existing cache in the system, or if -U flag is used.

Error Log:

mvn clean package -DskipTests
[INFO] Scanning for projects...
Downloading from confluent: https://packages.confluent.io/maven/io/confluent/common/6.0.0-SNAPSHOT/maven-metadata.xml
[WARNING] Could not transfer metadata io.confluent:common:6.0.0-SNAPSHOT/maven-metadata.xml from/to confluent-snapshots (${plugin.unreleased.repository.url}): Cannot access ${plugin.unreleased.repository.url} with type default using the available connector factories: BasicRepositoryConnectorFactory
Downloading from confluent: https://packages.confluent.io/maven/io/confluent/common/6.0.0-SNAPSHOT/common-6.0.0-SNAPSHOT.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for io.confluent.ksql:ksqldb-parent:6.0.0-SNAPSHOT: Could not transfer artifact io.confluent:common:pom:6.0.0-SNAPSHOT from/to confluent-snapshots (${plugin.unreleased.repository.url}): Cannot access ${plugin.unreleased.repository.url} with type default using the available connector factories: BasicRepositoryConnectorFactory and 'parent.relativePath' points at wrong local POM @ line 22, column 13
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project io.confluent.ksql:ksqldb-parent:6.0.0-SNAPSHOT (/srv/ksql/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM for io.confluent.ksql:ksqldb-parent:6.0.0-SNAPSHOT: Could not transfer artifact io.confluent:common:pom:6.0.0-SNAPSHOT from/to confluent-snapshots (${plugin.unreleased.repository.url}): Cannot access ${plugin.unreleased.repository.url} with type default using the available connector factories: BasicRepositoryConnectorFactory and 'parent.relativePath' points at wrong local POM @ line 22, column 13: Cannot access ${plugin.unreleased.repository.url} using the registered transporter factories: WagonTransporterFactory: Unsupported transport protocol -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
root@a4e7e82999e5:/srv/ksql# maven -v
bash: maven: command not found
root@a4e7e82999e5:/srv/ksql# mvn -version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/share/maven
Java version: 1.8.0_252, vendor: Oracle Corporation, runtime: /usr/local/openjdk-8/jre
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1062.12.1.el7.x86_64", arch: "amd64", family: "unix"

Originally, I thought this may be due to bad cache or java/mvn version mismatch when it worked on my local but failed on a freshly spun-up CentOS server. But I tested on the official docker container from maven:3.6.3-openjdk-8 as well, which gave the same results.

To reproduce:
First method:

1. Delete ~/.m2/repository to clean all cache.
2. Run mvn clean package -DskipTests on project root.

Second method:

1. Run mvn clean package -U -DskipTestson project root.

Usually it's recommended to use profiles to change maven repositories. Can we try that?
Also, would it be possible to share the build process followed by Confluent? So we can better understand why we need multiple snapshot repos :)

[colinhicks]

Thanks, @the-cybersapien. You're right that the current configuration is incomplete. We needed to make an incremental change in order to test a corresponding build server configuration. We're still actively working on this. As you suggest, using a separate profile may be a better option once we have a solid end-to-end solution.

The need for multiple snapshot repos is related to some complexities around the packaging ecosystem at Confluent. Historically, ksqlDB was released only as part of the Confluent Platform pipeline. Since November, we have also been releasing ksqlDB independently using a pipeline that leverages Confluent Platform packaging. For the time being, this requires us to depend on multiple Maven repositories, in order to release ksqlDB features a faster cadence.

In the short term, we'll have a workaround for the issue you originally raised. In the medium term, we improve the release pipeline to remove the constraints that can cause this sort of obstacle.

[the-cybersapien]

Hey @colinhicks and @agavra , we have another similar issue with KSQL 0.9.0 branch.

[ERROR] Failed to execute goal on project ksqldb-test-util: Could not resolve dependencies for project io.confluent.ksql:ksqldb-test-util:jar:6.0.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.kafka:kafka_2.13:jar:5.5.0-ccs, org.apache.kafka:kafka_2.13:jar:test:5.5.0-ccs: Could not find artifact org.apache.kafka:kafka_2.13:jar:5.5.0-ccs in confluent (https://packages.confluent.io/maven/) -> [Help 1]

Steps to reproduce:

1. Ensure a clean maven environment with jdk 1.8. I recommend using maven:3.6.3 docker image to do the job.
2. Take a fresh pull of https://github.com/confluentinc/ksql.git and checkout 0.9.0-ksqldb.
3. Run mvn clean package install -DskipTests to compile KSQL.

I am running some tests with KSQL around #5527 and want to test some additional things using 0.9.0-ksqldb as base branch, since master has seen some very big changes recently. Please advice how I can generate a fresh build for KSQLDB 0.9.0.

PS. I understand this is due to missing kafka dependencies, Should I raise a similar issue on confluentinc/kafka ?

[colinhicks]

Thanks @the-cybersapien. Sorry for the continued obstacles. I reproduced this as well.

It appears that the org.apache.kafka:kafka_2.13:jar:test:5.5.0-ccs artifact was somehow not propagated to the confluent-snapshots repository. I am working on a workaround.

[agavra]

I just caught up on this conversation, the 0.9.0-ksqldb branch is not the release branch (we'll work on clarifying that). Instead you want to be building from this tag: https://github.com/confluentinc/ksql/releases/tag/v0.9.0-ksqldb - you'll still need to add the magic to your maven-settings, POM, or pass in a settings.xml with -s though we've simplified that for future releases:

<repository>
          <id>packaging-beta</id>
          <url>https://jenkins-confluent-packages-beta-maven.s3-us-west-2.amazonaws.com/6.0.0-beta200415181825/1/maven/</url>
</repository> 

pluginRepository>
          <id>packaging-beta</id>
          <url>https://jenkins-confluent-packages-beta-maven.s3-us-west-2.amazonaws.com/6.0.0-beta200415181825/1/maven/</url>
</pluginRepository>

[apurvam]

Should this be closed or is there More work to do to get the release branches publicly build able ?

[colinhicks]

@agavra's work in https://github.com/confluentinc/ksql/pull/5607 does a reasonable job of addressing the issue for now, considering the current constraints with the release pipeline. But we should follow up with documentation on how to build release tags with the -s maven-settings.xml flag - in CONTRIBUTING.md for example.

[agavra]

What do you think about generating a README.md in the release tag code that just has the instructions to make it jump out in GitHub when looking at that branch?

[the-cybersapien]

@agavra's work in #5607 does a reasonable job of addressing the issue for now, considering the current constraints with the release pipeline. But we should follow up with documentation on how to build release tags with the -s maven-settings.xml flag - in CONTRIBUTING.md for example.

@colinhicks This sounds useful. Will help everyone trying to rebuild.