Renovate: Bazel: support github tags for http_archive

[cpsauer]

Hey @rarkins,

Sorry to bother you again, but I'm really running into walls trying to get this off the ground with Bazel.

I've made it through through the Onboarding PR and configuration, but am still stuck trying to get renovate to recognize updates to dependencies.

Taking a single, arbitrary WORKSPACE dependency as an example:
(I've backed off its version as a test, and changed it to a .tar.gz URL to work around https://github.com/renovatebot/renovate/issues/7851. Happy to help on that once I get things working.)

http_archive(
    name = "GBDeviceInfo",
    build_file = "@//Bazel/ThirdPartyBuild:GBDeviceInfo.BUILD",
    url = "https://github.com/lmirosevic/GBDeviceInfo/archive/6.3.0.tar.gz",
    sha256 = "d7666275dff039407ea467c3083b83e24934101777c8b55b6b1b3b7e9a9e220b",
    strip_prefix = "GBDeviceInfo-6.3.0/GBDeviceInfo"
)

But despite newer tagged releases, I'm seeing an empty list of updates in the Renovate log. And after banging around for a bit, I'm still at a loss on what's going wrong here. Could I ask for your help?

DEBUG: Found no results from datasource that look like a version (GBDeviceInfo)(dependency="GBDeviceInfo")
{
  "result": {
    "sourceUrl": "https://github.com/lmirosevic/GBDeviceInfo",
    "releases": []
  }
}
<later>
{
            "depType": "http_archive",
            "managerData": {
              "def": "http_archive(\n    name = \"GBDeviceInfo\",\n    build_file = \"@//Bazel/ThirdPartyBuild:GBDeviceInfo.BUILD\",\n    url = \"https://github.com/lmirosevic/GBDeviceInfo/archive/6.3.0.tar.gz\",\n    sha256 = \"d7666275dff039407ea467c3083b83e24934101777c8b55b6b1b3b7e9a9e220b\",\n    strip_prefix = \"GBDeviceInfo-6.3.0/GBDeviceInfo\"\n)"
            },
            "depName": "GBDeviceInfo",
            "repo": "lmirosevic/GBDeviceInfo",
            "currentValue": "6.3.0",
            "datasource": "github-releases",
            "lookupName": "lmirosevic/GBDeviceInfo",
            "depIndex": 4,
            "updates": [],
            "warnings": [],
            "fixedVersion": "6.3.0",
            "sourceUrl": "https://github.com/lmirosevic/GBDeviceInfo"
},

Thanks so much,
Chris

[rarkins]

Please create a minimal reproduction in a public repo so we can debug

[cpsauer]

Sure thing, but it's literally just that snippet. Here's a one-file repo. Renovate added--you can see the log with the pattern in question in job #268089631. (...and probably thereafter. Merging Renovate in a moment.)

[rarkins]

Taking a look at the upstream repo, the problem is that there are no releases in that repo - only tags (confirmed with this API call: https://api.github.com/repos/lmirosevic/GBDeviceInfo/releases).

If Bazel is happy to use tags-only then we should be too, but we'd need to understand how the logic works. Transferring into the main repo as it will require code changes.

[rarkins]

It's strange that the GitHub web UI does seem to treat these tags as releases while the REST API does not. It would be useful to confirm that there's nothing else we've misunderstood about how GitHub handles this.

Here is how a release normally looks like:

Here's on that repo:

Does GitHub create an archive for every pushed tag, or only if it meets certain conditions?

[rarkins]

Another point, auto-generated assets don't appear to be included in the assets endpoint for a release, e.g. https://api.github.com/repos/renovatebot/renovate/releases/34553677/assets returns [] even though in the above screenshot it has two "Source code" assets.

[cpsauer]

Oh, yikes, I see. From the Renovate docs it looks like the tag-as-release functionality already exists in Renovate for the Bazel git_repository rule? Perhaps http_archive could just fall back to the same behavior? Looks like I inadvertently chose a hard case for my test.

Bazel is definitely happy to take tags as releases in a http_archive--and recommends doing so--but it's totally disregarding the git structure underneath. It just cares that there's an archive at that URL, but it's pretty common--and often recommended by official Google projects--to use http_archive instead of git_repository. I'd seen that there's already Renovate support for commit-based updating (useful for Google live-at-HEAD projects and definitely useful to us.)

And yes, I believe that GitHub shows a release for every tag in the absence of manually created releases. Just checked in a private repo with a non-version-looking tag name.

[rarkins]

Also related to this issue: there are bazel source libraries where the http archive you want bazel to download is generated manually. See https://github.com/bazelbuild/rules_nodejs/releases/tag/2.3.1 for an example.

Let's see if @alexeagle minds me calling him with the bat signal (eagle signal?) for this topic and can advise on how Renovate should best handle this (release downloads vs source code archives).

[cpsauer]

Haha :)

Definitely didn't mean to imply that people didn't use manual releases--just that Bazel is abstracted away from the git structure in the (common) http_archive->github case.

But shouldn't it be just be determined from the url? Upgrade Release->Release. Archive/tag->Archive/tag. Archive/commit->Archive/commit. My understanding is that the first and third are current behavior.

Goodnight/morning for now, @rarkins, and thank you. (It's super late over here.)

[rarkins]

You might be right that we can determine if it's a tag or a release based on the url field

[cpsauer]

In the background: Because dependency management is so manual with Bazel, there's an extra burning need for Renovate. We're way excited to have found it. Was reading a bit about the Renovate/Whitesource history on my way to bed :)

[alexeagle]

Hi there! I don't think there's anything special about Bazel here. It's just some client trying to fetch archive files from Github via a URL. So like you observed, tags that aren't releases are still downloadable. I'm sure the same issue exists in other Renovate manager types, like a python requirements file that points to a github URL

[cpsauer]

Whoa, the eagle call worked! (@alexeagle, hey fellow Eagle Scout and ex-Googler! [Saw your personal website.])

[cpsauer]

@rarkins, as a friendly heads, it's looking to me like commit-based updating, the other archive case besides tags, is brittle/broken enough that it might make sense to unify and fix both together.

Details:
Taking boost as an example and massaging their recommended form until it's close as is runnable to the commit-based update example still isn't picked up.

WORKSPACE snippet:

http_archive(
    name = "com_github_nelhage_rules_boost",
    url = "https://github.com/nelhage/rules_boost/archive/135d46b4c9423ee7d494c78a21ff621bc73c12f3.tar.gz",
    sha256 = "3651f5dda0f7296e4cecafacc7f9d1f274be0fd64e30bebd74e28ffba28fe77f",
    strip_prefix = "rules_boost-135d46b4c9423ee7d494c78a21ff621bc73c12f3",
)
load("@com_github_nelhage_rules_boost//:boost/boost.bzl", "boost_deps")
boost_deps()

Log snippet:

{
  "depType": "http_archive",
  "managerData": {
    "def": "http_archive(\n    name = \"com_github_nelhage_rules_boost\",\n    url = \"https://github.com/nelhage/rules_boost/archive/135d46b4c9423ee7d494c78a21ff621bc73c12f3.tar.gz\",\n    sha256 = \"3651f5dda0f7296e4cecafacc7f9d1f274be0fd64e30bebd74e28ffba28fe77f\",\n    strip_prefix = \"rules_boost-135d46b4c9423ee7d494c78a21ff621bc73c12f3\",\n)"
  },
  "depName": "com_github_nelhage_rules_boost",
  "repo": "nelhage/rules_boost",
  "currentDigest": "135d46b4c9423ee7d494c78a21ff621bc73c12f3",
  "datasource": "github-releases",
  "lookupName": "nelhage/rules_boost",
  "depIndex": 1,
  "updates": [],
  "warnings": [],
  "skipReason": "unsupported-value"
},

I'll push that example to the test repo above, in case it's useful. Sorry in advance if I'm missing something obvious here.

But if not, as excited as I'd be to use Renovate with Bazel (see above), I think I'd better not rely fully on Renovate just yet and wait to see if the Bazel support gets a bit more robust. And if not--and I get a little bandwidth--I'll see what I can build down the line. Given that, I think I should probably be transferring the test repo to you; I'll send a request.

-CS

[rarkins]

Yes, please add it to the test repo. It looks like the logic should be mostly overlapping.

[renovate-release]

:tada: This issue has been resolved in version 23.90.2 :tada:

The release is available on:

• GitHub release
• 23.90.2

Your semantic-release bot :package::rocket:

[rarkins]

@cpsauer see https://github.com/rarkins/bazel1/pulls?q=is%3Apr+is%3Aopen+sort%3Aupdated-desc

[cpsauer]

Wow! Thank you @rarkins! You're awesome and amazingly fast.

Can confirm: The fix unstuck all 4 my http_archives--both commit and tag-based--that were hung up this way, leading to Renovate PRs. Thank you again. This takes Renovate to a very usable, useful place for me in Bazel land :)

[cpsauer]

Hmm, and in case it's of interest to you, the only Renovate PR I'm holding off merging is kind of an interesting case:

I'd forked a repository (Tulsi) to submit a PR, and was pointing to my fork until the PR got merged. The Renovate PR was trying to move backwards to master within my fork, off the feature branch.

I'd planned to manually move back to master of the main repo once the PR was merged, at which point Renovate would manage tracking Tulsi/master, commit by commit (awesome!). And for now, I can totally just close the Renovate PR until mine lands, and then do as I'd planned. And will do so happily :)

Probably too much of an edge case to be worth handling. But I thought I'd raise it here for your consideration, because there are a few layers of things here: (1) it seems happy to move off the branch you're on and (2) there's this interesting return-to-main repo workflow--though probably well out of scope.

[rarkins]

Renovate submitted a PR to your forked repo? It should only run on forks in the app if you have manually selected the forked repo using "Select repositories". If you select "All repositories" then it won't run on forks.

Anyway best to have any new feature request or bug in a new issue.

[cpsauer]

Ah, sorry, no. Rather, it updated the version of a dependency in unexpected ways, where the dependency itself is a repository I'd forked to submit a PR. Some ambiguity and confusion on my part, I realize. I've revised the above to clarify--in place, for anyone who might read this later.

I thought I should put it here because (1, above) seemed like a problematic edge case for this issue within the commit/archive URL logic that we hadn't considered. Maybe it's not an important enough one or too separate in implementation, but I figured "updating" backwards was unintended behavior you'd want to hear about.

And for (2), I thought, before scoping/filing, I should gauge whether you're interested in handling the case where people point to forks while they submit PRs to their dependencies.

If you still want me to file them separately, happy to. If you think they don't matter enough, also fine by me.

[rarkins]

Ideally we shouldn't be changing the repo that a dependency points to unexpectedly. I think I'd need to see the diff to fully understand though.