Renovate: Minor composer package version leads to no flattened updates

[kochen]

What Renovate type are you using?

Self-hosted Renovate.
Platform: GitLab, running Docker in CI

Describe the bug

For several days we are not getting any composer related updates.

After a lot of digging I striped down most of the packages of the project to try and eliminate the cause and I managed to narrow it down to this:
When requiring a package with composer, the package version in composer.json is (sometimes) short (i.e ^1.1). This leads to no [flattened] updates found.

Did you see anything helpful in debug logs?

In my case version 1.1.0 is installed while 1.1.1 is already released:

               {
                 "depType": "require",
                 "depName": "thecodingmachine/safe",
                 "currentValue": "^1.1",
                 "datasource": "packagist",
                 "lockedVersion": "1.1",
                 "depIndex": 17,
                 "updates": [],
                 "warnings": [],
                 "sourceUrl": "https://github.com/thecodingmachine/safe"
               },

simply changing it to ^1.1.0

               {
                 "depType": "require",
                 "depName": "thecodingmachine/safe",
                 "currentValue": "^1.1.0",
                 "datasource": "packagist",
                 "lockedVersion": "1.1",
                 "depIndex": 17,
                 "updates": [
                   {
                     "fromVersion": "1.1.0",
                     "toVersion": "1.1.1",
                     "newValue": "^1.1.1",
                     "newMajor": 1,
                     "newMinor": 1,
                     "updateType": "minor",
                     "isSingleVersion": false,
                     "isRange": true,
                     "releaseTimestamp": "2020-05-04T15:25:36+00:00",
                     "isBump": true
                   }
                 ],
                 "warnings": [],
                 "sourceUrl": "https://github.com/thecodingmachine/safe"
               },

This is just a sample package. The issue occurs on at least 10 different packages all having a short minor version in composer.json

To Reproduce

Require any [composer] package with a minor version only (i.e ^1.0).

Additional context

I ran a simple composer update (with all args as renovate would run) in both the renovate/renovate & renovate/composer images and updates where properly found & updated (no platform dependencies or any blocking issues).

I could manually change all the packages in composer.json to have a full SemVer version (including patch version), but I'd expect renovate to fix that whenever a new version of a certain package is released (same way it was until recently).

[viceice]

Do you know since which version the problem occured?

[kochen]

@viceice not sure about that.
We had several updates in the last few days but those packages had indeed the full version.

UPDATE: a week ago, we got an update to webmozart/assert (1.7 ->1.8`):

"renovateVersion": "19.219.6"

[viceice]

Ok, so we can check the changes since 19.219.6 and the if the is something composer relevant

[rarkins]

See #6057

We might want to revert that change until @hussainweb can update with a fix

[viceice]

One more question, what version on of renovate are you using? Or since which version you notice the problem?

[viceice]

I think this are the relevant changes:
https://github.com/renovatebot/renovate/releases/tag/19.226.0

https://github.com/renovatebot/renovate/releases/tag/19.225.5

https://github.com/renovatebot/renovate/releases/tag/19.217.0

[kochen]

@viceice @rarkins I just tried:
docker run --rm -v "$(pwd)/config.js:/usr/src/app/config.js" renovate/renovate:19.219.6 (dry run) and the issue still occurs!

Since I couldn't try with 19.217.0:

> docker pull renovate/renovate:19.217.0
Error response from daemon: manifest for renovate/renovate:19.217.0 not found: manifest unknown: manifest unknown`

I tried one before (19.216.9) and it also couldn't find any updated....... 😕

[hussainweb]

@kochen, could you please help with another example? The examples with ^1.1 and the screenshot you posted with the MR update are not exactly the same.

Also, what's the rangeStrategy you're using? It doesn't look like you're using the default value of "pin".

About the examples:
In your first example, you're looking for an upgrade from 1.1.0 to 1.1.1, both of which are satisfied by your version constraint ^1.1. Since the version number has not changed, Renovate doesn't know it should attempt an update (as far as my understanding goes).

In the example with the screenshot, if you're using rangeStrategy set to "bump" (which is my guess here), Renovate would recognise the opportunity to bump and upgrade.

---

I see your comment now that it's not working on earlier versions either. That is my conclusion as well, that this is not related to #6057. This is something that would work when we get update-lockfile support for composer, which I believe is in another PR.

---

Thanks for tagging me @rarkins. I hope the above is useful.

[hussainweb]

FYI: The composer update-lockfile support is worked upon in #5859.

[kochen]

@hussainweb you are correct, rangeStrategy: bump

another example:
"doctrine/doctrine-bundle": "^2.0",
installed 2.0.7
available: 2.0.8

[kochen]

I do see different version formats (composer out):

doctrine/doctrine-bundle                       2.0.7              2.0.8
prooph/event-store                             v7.5.6             v7.5.7
thecodingmachine/safe                          v1.1               v1.1.1

[hussainweb]

@kochen, do you mean to say updating "doctrine/doctrine-bundle": "^2.0" worked from 2.0.7 to 2.0.8 before?

[hussainweb]

Additional information: The above behaviour for rangeStrategy set to bump is reflected in this test:

  it('bumps short caret to same', () => {
    expect(
      semver.getNewValue({
        currentValue: '^1.0',
        rangeStrategy: 'bump',
        fromVersion: '1.0.0',
        toVersion: '1.0.7',
      })
    ).toEqual('^1.0');
  });

At this moment, I think we need update-lockfile support to fix this issue (#5859).

[kochen]

@hussainweb no.
I mean, "doctrine/doctrine-bundle": "^2.0" is set in the composer.json file.
I currently have 2.0.7 installed, but 2.0.8 is available (and not offered by renovate):

               {
                 "depType": "require",
                 "depName": "doctrine/doctrine-bundle",
                 "currentValue": "^2.0",
                 "datasource": "packagist",
                 "lockedVersion": "2.0.7",
                 "depIndex": 14,
                 "updates": [],
                 "warnings": [],
                 "sourceUrl": "https://github.com/doctrine/DoctrineBundle",
                 "homepage": "http://www.doctrine-project.org"
               },

Additional information: The above behaviour for rangeStrategy set to bump is reflected in this test:

  it('bumps short caret to same', () => {
    expect(
      semver.getNewValue({
        currentValue: '^1.0',
        rangeStrategy: 'bump',
        fromVersion: '1.0.0',
        toVersion: '1.0.7',
      })
    ).toEqual('^1.0');
  });

At this moment, I think we need update-lockfile support to fix this issue (#5859).

Yes, but the test "assumes" an update was found/considered.
The problem is somewhere in the flattened, not considering those updates

[hussainweb]

Yes, but the test "assumes" an update was found/considered.

I know what you mean as I used to think so. But when I traced the code, I found that this updates array is constructed after all the versions are loaded, found, and filtered. This filtering also takes into account the rangeStrategy aspects we are discussing here. This means that even if a later version exists, it won't be present in this array.

Thanks for pointing me to the code relating to flatten. I'll take a look.

[hussainweb]

So, I'm seeing the flatten code is mainly responsible for applying packageRules in the config, right? I see it also handles (deciding to) updating the lockfile, but I presume it won't enter that section for composer at all.

Based on my early reading, I still think this problem is more to do with update-lockfile not being supported. I mean to say, what you are seeing is the behaviour for "bump". What you want can be achieved with update-lockfile if it were supported.

I'm not saying it's not connected with flatten. I just started using Renovate, so I am still trying to understand everything.

[kochen]

Thanks for trying to help!

I might just manually update composer.json to have a full version (^1.2.3) and not just ^1.2.
We'll see...

[rarkins]

I marked this as needs-reproduction as without a demo repo to test against I'm not able to fix.

[peternijssen]

I think I am dealing with the same issue here. I have a very simple composer file;

    "require": {
        "php": "~7.4",
        "symfony/symfony": "^4.4",
        "symfony/monolog-bundle": "^3.0.2",
        "sensio/framework-extra-bundle": "^5.0",
        "incenteev/composer-parameter-handler": "^2.0",
        "cravler/maxmind-geoip-bundle": "^1.3"
    },

In my composer.lock file, I can clearly see that Symfony is on 4.4.7, while 4.4.13 is out.

"name": "symfony/symfony",
 "version": "v4.4.7",

Rather then getting a MR for both 4.4.13 and 5.1.5, I only get the 5.1.5 MR.

[rarkins]

@peternijssen check the docs for rangeStrategy. You're describing the default behaviour

[peternijssen]

@rarkins Then I suggest to update this documentation: https://docs.renovatebot.com/configuration-options/#separatemajorminor

Renovate's default behaviour is to create a separate branch/PR if both minor and major version updates exist.

[rarkins]

With default rangeStrategy, no minor update exists.

[peternijssen]

Sure. So my suggestion is to maybe include that in the documentation of the SeparateMajorMinor setting, as the current documentation says that actually the default behaviour is to create separate MRs. Anyways, I'll try the rangeStrategy then. Thanks for helping.