Renovate: Private gem hosted on gemfury: lookup failure
What Renovate type are you using?
Self-hosted, configured for GitLab.com
Describe the bug
Dependency updates in all of our rails apps for our private gems hosted on fury.io (gemfury) fail with a 403 status code. The provided token was verified to work correctly using the fury CLI.
Updates for public gems do work.
Relevant debug logs
Renovate debug output:
DEBUG: RubyGems lookup failure: authentication failed (repository=Group/group/repository, dependency=gem_name)
"registry": "https://gem.fury.io/company_name/",
"err": {
"name": "HTTPError",
"hostname": "gem.fury.io",
"method": "GET",
"path": "/api/v1/gems/gem_name.json",
"protocol": "https:",
"url": "https://gem.fury.io/api/v1/gems/gem_name.json",
"gotOptions": {
"path": "/api/v1/gems/gem_name.json",
"protocol": "https:",
"hostname": "gem.fury.io",
"hash": "",
"search": "",
"pathname": "/api/v1/gems/gem_name.json",
"href": "https://gem.fury.io/api/v1/gems/gem_name.json",
"retry": {
"methods": {},
"statusCodes": {},
"errorCodes": {},
"maxRetryAfter": 60000
},
"headers": {
"user-agent": "https://github.com/renovatebot/renovate",
"hosttype": "rubygems",
"accept": "application/json",
"accept-encoding": "gzip, deflate"
},
"hooks": {
"beforeRequest": [],
"beforeRedirect": [null],
"beforeRetry": [],
"afterResponse": [],
"beforeError": [],
"init": []
},
"decompress": true,
"throwHttpErrors": true,
"followRedirect": true,
"stream": false,
"form": false,
"json": true,
"cache": false,
"useElectronNet": false,
"method": "GET",
"hostType": "rubygems",
"baseUrl": "https://gem.fury.io/company_name/",
"gotTimeout": {"request": 60000}
},
"statusCode": 403,
"statusMessage": "Forbidden",
"headers": {
"server": "Cowboy",
"connection": "close",
"content-type": "text/plain; charset=utf-8",
"x-content-type-options": "nosniff",
"date": "Fri, 24 Jul 2020 10:39:23 GMT",
"content-length": "10",
"via": "1.1 vegur"
},
"body": "Forbidden\n",
"message": "Response code 403 (Forbidden)",
"stack": "HTTPError: Response code 403 (Forbidden)\n at EventEmitter.<anonymous> (/usr/src/app/node_modules/got/source/as-promise.js:74:19)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:97:5)"
},
"statusCode": 403,
"token": "***********"
DEBUG: Failed to look up dependency gem_name (repository=Group/Group/repository, packageFile=Gemfile, dependency=gem_name)
Gemfile of the rails app:
source 'https://gem.fury.io/company_name/'
gem 'gem_name', '~> 0.1.4'
To Reproduce
Additional context
We configure bundler by setting the environment variable BUNDLE_GEM__FURY__IO to a gemfury deploy token. Setting it to a full-access API token also does not work. trustLevel in renovate is set to "high".
NikolaiGulatz
All 18 comments
Is the URL in the error correct? Have you added the token also via hostRules?
rarkins
on 24 Jul 2020
@rarkins the URL is correct. My hostRules look like this:
[
{"hostName": "https://gem.fury.io/company_name/", "hostType": "bundler", "token": "**********"}
]
I've also tried
[
{"hostName": "https://gem.fury.io/company_name/", "hostType": "bundler", "username": "", "password": "**********"}
]
Unfortunately I could not find out how exactly the request looks like which renovate sends, like how it passes the token (basic auth?).
NikolaiGulatz
on 24 Jul 2020
If a token is provided in hostRules then Bearer authentication is used, otherwise Basic is used. Can you verify which curl command works for that URL? i.e. verify then paste in the exact curl command here, other than token being masked.
rarkins
on 24 Jul 2020
In Gemfury you can use different types of token for authentication and authorization. I tried using both a deploy token and a full access token, both don't work with renovate. With the full access token, I can use the API to list all gems:
curl -H "Accept: application/vnd.fury.v1+json" https://[email protected]/gems
{...}
This however is using a different endpoint (api.fury.io) than renovate is using (gem.fury.io). Trying to get something from this endpoint always fails with a "Forbidden" response body, although the token has read permissions on everything:
curl -H "Authorization: $FURY_TOKEN" https://gem.fury.io/api/v1/gems/gem_name.json
Forbidden
When I use gem install however, the token works:
gem uninstall gem_name
...
gem install gem_name --source https://[email protected]/company_name
Successfully installed gem_name-0.1.4
Parsing documentation for gem_name-0.1.4
Installing ri documentation for gem_name-0.1.4
Done installing documentation for gem_name after 0 seconds
1 gem installed
also using the BUNDLE_GEM__FURY__IO environment variable works:
export BUNDLE_GEM__FURY__IO="$FURY_TOKEN"
gem uninstall gem_name
...
gem install --source https://gem.fury.io/company_name gem_name
Successfully installed gem_name-0.1.4
Parsing documentation for gem_name-0.1.4
Installing ri documentation for gem_name-0.1.4
Done installing documentation for gem_name after 0 seconds
1 gem installed
[1] [Gemfury documentation](https://gemfury.com/help/install-gems)
[2] [Rubygems API documentation](https://guides.rubygems.org/rubygems-org-api/)
NikolaiGulatz
on 24 Jul 2020
So it would seem that the URL is the problem, not the token? We need the URL that lists all versions for a package, not the URL that lists all packages.
rarkins
on 24 Jul 2020
Frankly this won't be possible to progress unless you or someone else is able to dig into the APIs to work out what we are doing wrong or gem fury is doing differently. Once that's worked out, it's hopefully minimal in implementation.
rarkins
on 24 Jul 2020
It seems like gemfury does not provide that URL (or at least the authentication mechanism seems to be different, since it does not 404 but 403), unlike rubygems.org. Unfortunately, their documentation is not helpful here. I've created a support issue on their side.
I can keep you updated here once I hear from them, or you can close it and I'll open another issue once I have more information. Anyway thanks for the fast response and have a nice weekend!
NikolaiGulatz
on 24 Jul 2020
Let's keep this issue open. We generally don't close issues unless we don't want to add or fix something, which is rare
rarkins
on 24 Jul 2020
Michael here from Gemfury. I'm looking through the specs for Bundler source, and it seems that the following may be the right configuration. I'm not familiar with this tool, so can someone please try this.
[
{"baseUrl": "https://gem.fury.io/company_name/", "hostType": "bundler", "username": "TOKEN", "password": "NOPASS"}
]
Spec:
https://github.com/renovatebot/renovate/blob/master/lib/manager/bundler/host-rules.spec.ts
rykov
on 27 Jul 2020
@rykov does not work with this configuration. However, as you pointed out via email, one can obtain the list of available versions using this endpoint:
https://[email protected]/USERNAME/info/GEMNAME
which returns the available versions in the following format:
curl --silent https://[email protected]/company_name/info/gem_name
---
0.1.0 activesupport:>= 0|checksum:f822315e5522481a3e5ff7da5641a47c9b313db53f87bd3da87b7a5a8615af98
0.1.1 activesupport:>= 0|checksum:be4d39e966bf9d1ab841f81a2c9a2f9f53515f769ba3d0a35b3e3da375e89ac5
0.1.2 activesupport:>= 0|checksum:0e7680cb585c83433235f09becd6a712bb66ab788f632f48f578623c1bbd7e63
0.1.3 activesupport:>= 0|checksum:ab116f1ba09578a5520b16297db19e1fc99feefcc0402221c745d22a4865a6b3
0.1.4 activesupport:>= 0|checksum:6c027c675a1a198927bf052720863466ab5b15648c6965fa195f5f7c9a5f6509
I could probably provide a PR which adds a new module lib/datasource/rubygems/get-fury-io.ts and add a switch to datasource/rubygems/releases.ts the parsing part is already mostly done in https://github.com/renovatebot/renovate/blob/master/lib/datasource/rubygems/get-rubygems-org.ts
@rarkins
NikolaiGulatz
on 27 Jul 2020
It seems we're not sure if it's an authentication problem or a URL problem?
rarkins
on 27 Jul 2020
It is an URL problem since gemfury does not support the RubyGems API
NikolaiGulatz
on 27 Jul 2020
I've made it working with some small changes, going to clean it up a bit and provide a PR within the next few days.
NikolaiGulatz
on 27 Jul 2020
As initially reported in #7677, I'm having a similar issue. I can't seem to access gemfury's private package from renovate.
The config
I have the following config:
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:base"
],
"enabledManagers": [
"pip_requirements"
],
"python": {
"registryUrls": [
"https://pypi.fury.io/<org>/",
"https://pypi.org/simple/"
]
},
"hostRules": [
{
"platform": "pypi",
"hostName": "pypi.fury.io",
"password": "",
"encrypted": {
"username": "..."
}
}
]
}
The issues
Multiple registries issue
sample run
If this is set:
"registryUrls": [
"https://pypi.fury.io/<org>/",
"https://pypi.org/simple/"
]
I can see in the logs, it will only look on the standard repository when it should look on both:
{
"depName": "lib-service-infra",
"currentValue": "==1.3.0",
"datasource": "pypi",
"fromVersion": "1.3.0",
"depIndex": 5,
"updates": [],
"warnings": [
{
"depName": "lib-service-infra",
"message": "Failed to look up dependency lib-service-infra"
}
],
"fixedVersion": "1.3.0"
},
And if I only add the private registry, it will only look on that one for all packages including public ones.
Authentication issue
Even if I make it select only the private registry like that:
"registryUrls": [
"https://pypi.fury.io/<org>/"
]
It still doesn't work because there's an authentication issue:
DEBUG: Failed to look up dependency requests (requests)(packageFile="requirements.txt", dependency="requests")
DEBUG: Datasource unauthorized
{
"datasource": "pypi",
"lookupName": "lib-service-infra",
"url": "https://pypi.fury.io/<org>/lib-service-infra"
}
Although the same token does work fine with curl:
$ curl -u <token>: https://pypi.fury.io/<org>/ -s |grep lib-service-infra
<a href="https://pypi.fury.io/<org>/lib-service-infra">lib-service-infra</a><br/>
fclairamb
on 8 Nov 2020
@fclairamb If registryUrls aren't working as you'd expect then it's an independent topic, best reproduced in a public repo so it can be raised and debugged in a new issue. If you think Renovate isn't even trying the private registry then should be fine to use any URL in a public reproduction.
rarkins
on 8 Nov 2020
Ok, I think we can focus on the authentication and once it's solved I'll file another issue around this.
fclairamb
on 9 Nov 2020
@fclairamb I took another look and your host rule has an error. Change from:
"hostRules": [
{
"platform": "pypi",
"hostName": "pypi.fury.io",
"password": "",
"encrypted": {
"username": "..."
}
}
]
to:
"hostRules": [
{
"hostName": "pypi.fury.io",
"password": "",
"encrypted": {
"username": "..."
}
}
]
I don't think it should make a difference to the result, but best you fix it first and check.
rarkins
on 9 Nov 2020
Sorry for the late answer, this indeed doesn't solve it:
Config:
"hostRules": [
{
"hostName": "pypi.fury.io",
"password": "",
"encrypted": {
"username": "***"
}
}
],
Result from these logs:
DEBUG: Datasource unauthorized
{
"datasource": "pypi",
"lookupName": "lib-service-infra",
"url": "https://pypi.fury.io/<org>/lib-service-infra"
}
DEBUG: Failed to look up dependency lib-service-infra (lib-service-infra)(packageFile="requirements.txt", dependency="lib-service-infra")
Related issues
kogai
路
3Comments
Arcanemagus
路
4Comments
ChristianMurphy
路
4Comments
OmgImAlexis
路
4Comments
ikatyang
路
4Comments
Most helpful comment
I've made it working with some small changes, going to clean it up a bit and provide a PR within the next few days.