Renovate: Maven datasource skips tons of libraries

[bagage]

What Renovate type are you using?
Renovate CLI

Describe the bug
Maybe related to #3262, but in my private repository only 3 PRs were created, while 28 PRs were created with previous system. It looks like the faulty line is .find { it instanceof DefaultExternalModuleDependency } in gradle-updates-report.js.

If I commented this out, 27 PRs are created (the left one was an unstable version so this is OK). What this line should be doing?

If needed I can try to provide a reproducible repo, but is it needed in the first place?

[rarkins]

Sounds like it’s not entirely due to private modules if you get so many when you uncomment the filter. @corecanarias could you comment about this filter line?

[bagage]

FYI in my case, the class seems to be class org.gradle.api.internal.artifacts.dependencies.DefaultExternalModuleDependency_Decorated and .find { it instanceof DefaultExternalModuleDependency_Decorated } does not filter them properly.

[corecanarias]

That line is supposed to filter gradle modules, for example when you depend on another module inside your project, or local libraries, if you depend on a library in your local filesystem

Can you give me more details on what type of dependencies is skipping?

[bagage]

Indeed I'm using modules, basically my project's tree is:

build.gradle
server
    ---> build.gradle
service
    ----> build.gradle

Yet only dependencies of root build.gradle are fetched, but not the ones from my modules. I'd expect renovate to fetch both, because my root gradle is rather empty and most of the libraries used are within modules?

[corecanarias]

It should fetch dependencies from any submodule. Can you run the gradle-updates-report.js script and paste the report that it generates here? It should go through every project and report the dependencies

[bagage]

OK so I've reproduced this issue on https://github.com/bagage/gradle-template. See module1 build.gradle. There is a mention of prometheus.

Here are the logs for renovate master, and renovate with filter line being commented. You can see that on first log file, there is no mention of prometheus (and it fails to find it on second version but I believe this is another issue).
logs.txt upstream, prometheus not listed
logs2.txt filter disabled, prometheus listed (but on error)

[corecanarias]

I'm doing some testing and it seems that for some reason it's filtering some DefaultExternalModuleDependency dependencies for your project... also because you don't have any repository defined for your submodules even if you disable that filter, there are not repositories defined to look up for new dependencies.

The first part, the filter, I need to find out what happen with this project,

The second part, I guess you need to define your repositories for all your submodules

[bagage]

The second part, I guess you need to define your repositories for all your submodules

Shouldn't renovate use the repositories from top project? I mean, this is how gradle works, you do not define repositories in every module but only on top module, do you?

Eg. if users have to modify their build.gradle to make renovate work feel wrong... but maybe I'm wrong! Again, not familiar with gradle in general :). Thanks for your hard work @corecanarias!

[corecanarias]

This might solve the issue, can you check your repo with this version?

https://github.com/renovatebot/renovate/pull/3270

[corecanarias]

@ChristianMurphy I already tested this change against uPortal https://github.com/corecanarias/uPortal/pulls could you test it too? It seems it's working fine

[corecanarias]

Shouldn't renovate use the repositories from top project? I mean, this is how gradle works, you do not define repositories in every module but only on top module, do you?

Actually, how renovate works at this point is ask to gradle for the repositories defined in each module, that is why we need to run gradle as part of renovate, gradle is telling renovate that those modules don't have any repository. We are trying to honor as much as possible the configuration that gradle provides.

I'll try to investigate though if there is something that I'm missing, see how gradle behaves when there are no repositories defined. If it's using the local cache as default, it might be ok for standard use but not for renovate

[corecanarias]

This is how you could define repositories for all your submodules without having to go one by one
https://github.com/corecanarias/uPortal/blob/master/build.gradle#L94

[ChristianMurphy]

It's scanning as expected in uPortal and NotificationPortlet :ballot_box_with_check:
https://github.com/Jasig/uPortal/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+chore%28deps%29
https://github.com/Jasig/NotificationPortlet/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+chore%28deps%29

[corecanarias]

btw @bagage you need to add
"gradle": { "enabled": true },

to your renovate.json

EDIT: Ignore, saw it

[corecanarias]

Now I'm seeing this:

ERROR: Repository has unknown error (repository=corecanarias/gradle-template) "err": { "message": "Cannot read property 'split' of null", "stack": "TypeError: Cannot read property 'split' of null\n at parseRange (/Users/ibethencourt/Projects/renovate/lib/versioning/maven/compare.js:243:31)\n at isVersion (/Users/ibethencourt/Projects/renovate/lib/versioning/maven/compare.js:227:39)\n at lookupUpdates (/Users/ibethencourt/Projects/renovate/lib/workers/repository/process/lookup/index.js:32:7)\n at fetchDepUpdates (/Users/ibethencourt/Projects/renovate/lib/workers/repository/process/fetch.js:42:29)\n at /Users/ibethencourt/Projects/renovate/lib/workers/repository/process/fetch.js:75:5\n at el (/Users/ibethencourt/Projects/renovate/node_modules/p-all/index.js:4:59)\n at Promise.resolve.then.el (/Users/ibethencourt/Projects/renovate/node_modules/p-map/index.js:46:16)\n at process._tickCallback (internal/process/next_tick.js:68:7)" }

Seems something related with the versioning? @rarkins

[corecanarias]

Added a small fix to that and I got this https://github.com/corecanarias/gradle-template/pulls

[bagage]

I cannot test it until next week but sa far as I can tell it's looking good!

[rarkins]

@bagage were you able to test and confirm if this can be closed?

[bagage]

@rarkins thanks to remind me. I will test it on Monday and let you know!

[bagage]

I can confirm that the problem is solved for me @rarkins!

[rarkins]

Thanks @corecanarias :)