Renovate: [Composer] Lock file maintenance bumps incompatible dependency

[ThibaultVlacich]

I have an issue with the weekly lock file maintenance for Composer. One of my dependency has "ocramius/proxy-manager": "^2.8.0" as a dependency. The weekly lockfile maintenance bumps the dependency from 2.8.0 to 2.9.0 in the lockfile.

             "name": "ocramius/proxy-manager",
-            "version": "2.8.0",
+            "version": "2.9.0",
             "source": {
                 "type": "git",
                 "url": "https://github.com/Ocramius/ProxyManager.git",
-                "reference": "ac1dd414fd114cfc0da9930e0ab46063c2f5e62a"
+                "reference": "9475248b6521f3029b973c4cd9aad31022bfd793"
             },
             "dist": {
                 "type": "zip",
-                "url": "https://api.github.com/repos/Ocramius/ProxyManager/zipball/ac1dd414fd114cfc0da9930e0ab46063c2f5e62a",
-                "reference": "ac1dd414fd114cfc0da9930e0ab46063c2f5e62a",
+                "url": "https://api.github.com/repos/Ocramius/ProxyManager/zipball/9475248b6521f3029b973c4cd9aad31022bfd793",
+                "reference": "9475248b6521f3029b973c4cd9aad31022bfd793",
                 "shasum": ""
             },
             "require": {
+                "composer-runtime-api": "^2.0.0",
                 "laminas/laminas-code": "^3.4.1",
-                "ocramius/package-versions": "^1.8.0",
                 "php": "~7.4.1",
                 "webimpress/safe-writer": "^2.0.1"
             },

Problem is, version 2.9.0 that was released doesn't work with Composer 1.0, but only with Composer 2.0 (that is still in alpha). As you can see, there is a new dependency to "composer-runtime-api": "^2.0.0" that is added. If I run a local composer update, the dependency stays on 2.8.0 (is even downgraded to 2.8.0 if I run it on the renovate/lock-file-maintenance branch).

Cf Ocramius/ProxyManager#621

[rarkins]

I checked and confirmed that the latest renovate/compose image is still 1.x:

❯ docker run --rm renovate/composer composer --version
Composer version 1.10.10 2020-08-03 11:35:19

So I'm pretty sure it's not because we're running 2.x in the app.

Note that Renovate doesn't run composer update - it deletes the composer.lock and runs composer install. If you run that locally, so you see it making the same mistake? Otherwise it's possibly because of --ignore-platform-reqs.

Cc @viceice

[ThibaultVlacich]

Note that Renovate doesn't run composer update - it deletes the composer.lock and runs composer install. If you run that locally, so you see it making the same mistake?

It does not.

Otherwise it's possibly because of --ignore-platform-reqs.

Indeed, if I add --ignore-platform-reqs to composer install, it bumps the dependency.

[rarkins]

Transferring this to the main repo, as it's something we'd need to fix or refactor here

[rarkins]

We have an option composerIgnorePlatformReqs but it's admin-only. i.e. cannot be configured in renovate.json.

First question is: does it need to be admin-only? i.e. is there any risk/problem to the bot owner/operator if users configure this? From the docs:

The main reason why we ignored these was because we usually can't meet all the requirements in the renovate/composer image.

@ThibaultVlacich do you have Docker installed to test something related?

[viceice]

@rarkins Did you test our composer image without using --ignore-platform-deps ?

[viceice]

@ThibaultVlacich Can you please create a minimal reproduction repo, so we can run local tests against.

[rarkins]

@viceice I'm assuming you'd get the correct results if running without it, I was going to get @ThibaultVlacich to verify that locally but you're right that a reproduction repo is best.

[ThibaultVlacich]

@viceice Sure, I just did : ThibaultVlacich/renovate-composer

You can see that the "pin dependency" PR did the same thing.

[rarkins]

@viceice we need to check if running without ignore platform deps causes something else to break during composer install.

[msheakoski]

How do I stop Renovate from upgrading proxy-manager 2.8.0 to 2.9.1 every time lock file maintenance runs? I added it to my ignoreDeps but it seems to have no effect.

Edit: Please see this comment: https://github.com/renovatebot/renovate/issues/6945#issuecomment-716209190

[rarkins]

@msheakoski ignoreDeps is a Renovate instruction, not a Composer one. Renovate simply deletes the lock file and lets Composer update it and Composer is deciding to update to whatever versions there are. You can find more details in this issue above.

[msheakoski]

Edit: This fix is no longer recommended. Please see this comment: https://github.com/renovatebot/renovate/issues/6945#issuecomment-716209190

For anyone that needs a quick fix while this is being resolved, add this to your composer.json:

"conflict": {
  "ocramius/proxy-manager": ">=2.9"
}

and it will use the correct composer 1.x compatible version of proxy-manager even if composer.lock is deleted and composer install --ignore-platform-reqs is run. No need to add proxy-manager as a direct dependency of your project or ignore anything in renovate.json.

[rarkins]

@msheakoski thanks for sharing!

[viceice]

ok, i've verified that composer would update to incompatible versions if we use --ignore-platform-reqs

see https://github.com/viceice-tests/renovate-composer/pulls

@rarkins Are you seeing any issues allowing composerIgnorePlatformReqs on repo config?

https://github.com/renovatebot/renovate/blob/c33da16aba2e3fff357ead02c79343ca614f827a/docs/usage/self-hosted-configuration.md#L43-L46

[msheakoski]

Keep in mind that if --ignore-platform-reqs is not used, the docker image must have a version of PHP compatible with the project and all extensions installed in order to be successful.

Another way might be to check for a conflict with composer-runtime-api after the lock file is deleted and everything is installed. The command returns a non-zero code for any failures and it is easy to scan for a line beginning with "composer-runtime-api" to extract the error message for a PR with something actionable like "add "conflict": { "ocramius/proxy-manager": ">=2.9" } to composer.json

❯ composer --version
Composer version 1.10.15 2020-10-13 15:59:09

❯ composer check-platform-reqs
composer-plugin-api   1.1.0                                                                    success  
composer-runtime-api  1.0.0     ocramius/proxy-manager requires composer-runtime-api (^2.0.0)  failed   
ext-ctype             7.4.11                                                                   success  
ext-dom               20031129                                                                 success  
...

[rarkins]

Is composer check-platform-reqs also useful for the scenario where we're ignoring platform reqs, or including them? Would the logic be like the following?

• Run install ignoring platform reqs
• If non-zero exit code. then run check-platform-reqs afterwards?

[msheakoski]

@rarkins

• Run: composer install --ignore-platform-reqs like usual
• Run: composer check-platform-reqs and check exit code
• If check-platform-reqs exits with 0:
  
  
  
  • everything is good, nothing else to do
  
  
• If check-platform-reqs exits with non-zero:
  
  
  
  • If the line beginning with composer-runtime-api 1. is present and 2. is marked as failed
  
  
  • The package listed on this line is not compatible with the current version of composer that was used for the lock file maintenance (and probably not compatible with the latest stable version of composer anyway).
  
  
  • So far, I have only seen this happen with ocramius/proxy-manager since it changed a major dependency to an unreleased version of composer in a feature update.
  
  
  • The end result is that there was an error performing lock file maintenance. The end user would need to add something to the conflict section of composer.json to fix it.
  
  
  • If any other line is failed, it probably means that the composer/php docker image does not have one of the PHP extensions installed and the errors can be ignored
  
  

[viceice]

@rarkins Are you seeing any issues allowing composerIgnorePlatformReqs on repo config instead of bot admin?

[rarkins]

I think it should be ok

[msheakoski]

Composer 2.0 was released on October 24th: https://blog.packagist.com/composer-2-0-is-now-available/

Versions of ocramius/proxy-manager >=2.9 are no longer an issue under Composer 2. The current lock file maintenance method that Renovate uses will continue to work for users of both Composer 1 and Composer 2.

If users want to keep using Composer 1, they should specify a composer-runtime-api version in composer.json:

"require": {
  "composer-runtime-api": "^1.0"
}

This will prevent packages like proxy-manger >= 2.9 which depend on newer versions of Composer from being installed.

It is also possible to add a conflict, but I think that composer-runtime-api is the better solution because it directly addresses the problem for all affected packages.

[bytestream]

We have an option [composerIgnorePlatformReq
First question is: does it need to be admin-only? i.e. is there any risk/problem to the bot owner/operator if users configure this?

No there was no reason, that's just how I did it...

[mbrodala]

I just tried and the composer-runtime-api workaround does not work. We still get a huge composer.lock update which evidently was performed with Composer 2 indicated by the plugin-api-version bump:

We also see the following in the repository logs which confirms that Composer 2 was used:

The \"helhum/dotenv-connector\" plugin was skipped because it requires a Plugin API version (\"^1.0\") that does not match your Composer installation (\"2.0.0\"). You may need to run composer update with the \"--no-plugins\" option.

[msheakoski]

@mbrodala helhum/dotenv-connector v2.3.0 and higher supports both Composer 1 and 2 if you are able to update to that version.

[mbrodala]

@msheakoski Thanks, I know. That's not the point. ;-)

[msheakoski]

@mbrodala understood, but it is probably a good idea to start using Composer 2. It is supported by most packages at this point. I did not run into any issues with existing projects.

According to https://blog.packagist.com/composer-2-0-is-now-available/

As for Composer 1.x, it is now more or less EOL. It will also receive critical fixes if anything comes up but the goal for everyone should be to migrate to 2.x as soon as possible.

and https://github.com/composer/composer/issues/9329

However, Composer 1.x will not be maintained for very long, so if you do this please remember it and aim to upgrade to Composer 2.x in a timely manner.

[mbrodala]

@msheakoski Yes, I am aware of all of this and we do not really struggle to upgrade to Composer 2.

My point still is that Renovate must respect requests to use Composer 1, otherwise it's not reliable and cannot be used to manage PHP dependencies.

[viceice]

I'm preparing a pr to support some composer constraints now like the ruby ones

[rarkins]

If we can reasonably support both then we should, even if it's effectively EOL.

[renovate-release]

:tada: This issue has been resolved in version 23.88.0 :tada:

The release is available on:

• GitHub release
• 23.88.0

Your semantic-release bot :package::rocket:

[mbrodala]

@viceice From the reading the code this means that the composer-runtime-api workaround can be dropped since you also check for the plugin-api-version (which is added to the composer.lock automatically)?

[viceice]

@mbrodala Feel free to send a pr. 🙃

I'm not an composer expert. 😄
I just wanted to catch if composer.lock doesn't exist yet. 🤷‍♂️

[mbrodala]

Yep, sounds reasonable, thanks for caring.

Now I just need to wait, a retry just now still used version 23.87.1 ...

[viceice]

Yep, sounds reasonable, thanks for caring.

Now I just need to wait, a retry just now still used version 23.87.1 ...

Yes, @rarkins will update the app regularly, so be patient 🙃 After that you would need to use the force retry checkbox, so renovate will update your existing pr.

[mbrodala]

The new version is now online and I can confirm that now Composer 1.x is used; there are no breaking changes in the suggested update anymore.

I still see some unnecessary lock file changes however which can be remedied with a manual composer update --lock and amend in the Renovate update branch. I'd assume this is related to the --no-scripts used by Renovate.