Renovate: Support Maven registries with no metadata xml

[KRonst]

Which Renovate are you using?

Renovate Open Source CLI

Which platform are you using?

Bitbucket Server

Have you checked the logs? Don't forget to include them if relevant

Yes

What would you like to do?

In my gradle project I have some plugins

plugins {
    id 'org.springframework.boot' version '2.2.2.RELEASE' apply false

    id 'org.jetbrains.kotlin.kapt' version '1.3.61' apply false
    id 'org.jetbrains.kotlin.jvm' version '1.3.61' apply false
    id 'org.jetbrains.kotlin.plugin.allopen' version '1.3.61' apply false // provides kotlin-spring
    id 'org.jetbrains.kotlin.plugin.noarg' version '1.3.61' apply false // provides kotlin-jpa

    id 'org.sonarqube' version '2.8' apply false
    id 'io.gitlab.arturbosch.detekt' version '1.2.1' apply false
    id 'net.nemerosa.versioning' version '2.6.1' apply false
    id 'com.gorylenko.gradle-git-properties' version '1.5.2' apply false
    id 'org.hidetake.swagger.generator' version '2.18.1' apply false
}

Configuration for bot

"renovate-config": {
    "base-backend-kotlin": {
      "description": "Default base configuration for bpm kotlin backend applications",
      "extends": [
        ":ignoreUnstable",
        ":prImmediately",
        ":semanticCommits",
        ":semanticCommitTypeAll(renovatebot)",
        ":updateNotScheduled",
        ":automergeDisabled",
        ":ignoreModulesAndTests"
      ],
      "baseBranches": ["test-renovate"],
      "gradle": {
        "enable": true
      },
      "ignoreDeps": [
        "io.gitlab.arturbosch.detekt:detekt-formatting"
      ]
    }
  }

After bot competes checking my repo I have some PRs for dependencies but there is no RP for any plugin. Also I see in logs this part:

DEBUG: Branch lists (repository=BPMIN/renovate-test)
       "branchList": [
         "renovate/spring-cloud",
         "renovate/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.x",
         "renovate/org.jetbrains.kotlin-kotlin-reflect-1.x",
         "renovate/io.github.microutils-kotlin-logging-1.x",
         "renovate/net.logstash.logback-logstash-logback-encoder-6.x",
         "renovate/com.google.guava-guava-29.x",
         "renovate/org.apache.commons-commons-lang3-3.x",
         "renovate/org.jetbrains.kotlin-kotlin-annotation-processing-gradle-1.x",
         "renovate/org.jetbrains.kotlin-kotlin-scripting-compiler-embeddable-1.x",
         "renovate/org.jetbrains.kotlin-kotlin-allopen-1.x",
         "renovate/org.jetbrains.kotlin-kotlin-noarg-1.x",
         "renovate/assertj",
         "renovate/mockito",
         "renovate/gradle-6.x"
       ],
       "renovateBranches": [
         "renovate/assertj",
         "renovate/com.google.guava-guava-29.x",
         "renovate/gradle-6.x",
         "renovate/io.github.microutils-kotlin-logging-1.x",
         "renovate/io.gitlab.arturbosch.detekt-detekt-formatting-1.x",
         "renovate/mockito",
         "renovate/net.logstash.logback-logstash-logback-encoder-6.x",
         "renovate/org.apache.commons-commons-lang3-3.x",
         "renovate/org.assertj-assertj-core-3.x",
         "renovate/org.mockito-mockito-core-3.x",
         "renovate/org.mockito-mockito-junit-jupiter-3.x",
         "renovate/org.springframework.cloud-spring-cloud-dependencies-.x",
         "renovate/spring-cloud",
       ]

There is branches for update plugins in branchList, but they didn't displayed in renovateBranches block and as a result there is no branches in repo for updating this plugins.
And here is another part from logs for one of the plugins:

DEBUG: getFile(build.gradle, test-renovate) (repository=BPMIN/renovate-test, branch=renovate/org.jetbrains.kotlin-kotlin-noarg-1.x)
DEBUG: gradle.updateDependency(): packageFile:build.gradle depName:org.jetbrains.kotlin:kotlin-noarg, version:1.3.61 ==> 1.3.72 (repository=BPMIN/renovate-test, branch=renovate/org.jetbrain
s.kotlin-kotlin-noarg-1.x)
DEBUG: No package files need updating (repository=BPMIN/renovate-test, branch=renovate/org.jetbrains.kotlin-kotlin-noarg-1.x)
DEBUG: No updated lock files in branch (repository=BPMIN/renovate-test, branch=renovate/org.jetbrains.kotlin-kotlin-noarg-1.x)
DEBUG: No files to commit (repository=BPMIN/renovate-test, branch=renovate/org.jetbrains.kotlin-kotlin-noarg-1.x)

[rarkins]

Can you create a public GitHub repo that reproduces this?

[KRonst]

That's interesting... In public github repo renovate has created PRs for plugins. E.g. Spring Boot plugin update

[KRonst]

In my bitbucket project I use company's registry for searching dependencies and plugins. Could this be the cause of such a problem if our registry don't have updated versions for these plugins?

[viceice]

Yes, what software do you use?

[KRonst]

Yes, what software do you use?

Could you please clarify about what software are you asking?)

[viceice]

Sorry, I mean your companys registry software. That can be the problem

[KRonst]

We use Sonatype Nexus as our main registry.

[viceice]

@KRonst Please share a more complete debug log.

[KRonst]

Full log is too big, so I pasted it on gist.

[KRonst]

Okay. I guess I've found the problem. Plugins will not update if there is no maven-metadata.xml in it's registry. Is there any way to solve this problem?

[rarkins]

How does your registry structure look like? e.g. an example file listing or screenshot?

FYI @zharinov

[KRonst]

How does your registry structure look like? e.g. an example file listing or screenshot?

[rarkins]

Moved it into the main repo as a feature request. I guess we need to fetch /repositories/gradle/m2/org/springframework/boot/org.springframework.boot.gradleplugin/ and then parse the HTML?

@KRonst can you paste in the raw HTML for that so we can see how parseable it is?

[KRonst]

Here is HTML from page /repositories/gradle/m2/org/springframework/boot/org.springframework.boot.gradle.plugin/

[rarkins]

Are you 100% sure that's the raw HTML and not transformed by the browser? I'm a little surprised that the links are absolute (including full url/domain) and not relative.

[viceice]

If they use nexus from sonatype, they can force the metadata generation, see here

[KRonst]

It was HTML from registry based on Sonatype Nexus 2.11.3-01
We also have another registry based on Sonatype Nexus OSS 3.22.1-02. And here is HTML from one of the artifact from there: Spring Boot

[rarkins]

The latest HTML is what I was expecting.

[viceice]

As read here, nexus should auto generate the maven-metadata.xml

[KRonst]

Thanks to all for help :)