Renovate: Go PRs require extra tidying

[tbpg]

Which Renovate are you using?

WhiteSource Renovate App (the forking version)

Which platform are you using?

GitHub.com

Have you checked the logs? Don't forget to include them if relevant

No.

What would you like to do?

Have go mod tidy run when making dependency updates. For example, https://github.com/GoogleCloudPlatform/golang-samples/pull/1556 required me updating the PR manually to run go mod tidy on all modules.

I do have postUpdateOptions.gomodTidy configured, but maybe it's not configured correctly? https://github.com/GoogleCloudPlatform/golang-samples/blob/2c581f8b444a3b4cf790d4350ea212d1d9c0f8d0/.github/renovate.json#L7-L9

[rarkins]

Here's the command that was run: docker run --rm --name=renovate_go --label=renovate_child -v \"/mnt/renovate/gh/GoogleCloudPlatform/golang-samples\":\"/mnt/renovate/gh/GoogleCloudPlatform/golang-samples\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/tmp/renovate-cache/others/go\":\"/tmp/renovate-cache/others/go\" -e GOPATH -e CGO_ENABLED -w \"/mnt/renovate/gh/GoogleCloudPlatform/golang-samples/functions/firebase/upper\" renovate/go:latest bash -l -c \"git config --global url.\\\"https://x-access-token:**redacted**@github.com/\\\".insteadOf \\\"https://github.com/\\\" && go get -d ./...\"

This confirms that go mod tidy isn't being run. I'll dig into it more now.

[rarkins]

• go mod tidy was run, as configured
• Strangely, go.sum wasn't updated at all, despite go get and go mod tidy being run
• This PR is old/closed so I can't really do much now

Best if you raise a new issue here or @ me in the repo the next time you have a PR that looks wrong, and before you close it.

[tbpg]

Thank you for the speedy response and for looking into it!

Got it. I'll close this and let you know the next time it happens (probably next Monday, as that's when updates are configured to happen).

[tbpg]

Reopening to track the current PR with this issue: https://github.com/GoogleCloudPlatform/golang-samples/pull/1573

Thank you for the help with this! Let me know if there are any adjustments or anything I should try.

[rarkins]

Noting a few things:

This repo doesn't just run go mod tidy, it also runs goimports -w. I'm not sure yet what that part does or if it matters.

Here's how it's run:

Another point: the repo is running Go 1.14 on all go.mod files while the go.mod files mostly specify go1.11 or go1.13, which is what we run. Go has changed quite a lot around modules in 2 years so there can definitely be a difference between go1.11 and go1.14 tidying, but that may not be the cause.

Most likely solution: run go mod tidy again.

I'm also wondering: why should we run go get at all - maybe go mod tidy on its own does what we need?

[tbpg]

Thank you for investigating!

goimports -w . only modifies *.go files -- not go.sum. The go.sum changes only come from go mod tidy.

Go 1.14 vs other versions could cause inconsistencies in go.sum files. Would it be possible to upgrade the Go version used? Go 1.15 will be out in ~1 month, too.

Yes, you should run go get. go get can also modify the version of other dependencies to meet new requirements. So, it's the best way to change the version of any dependency. So, no, go mod tidy doesn't do everything needed.

[rarkins]

Would it be possible to upgrade the Go version used?

Renovate follows whatever Go version that is specified in the go.mod files. For example: https://github.com/GoogleCloudPlatform/golang-samples/blob/5520efc715afa2e11ed8f86d0e851a899372f716/go.mod#L3

I'm thinking though that Renovate should ideally have a feature to "upgrade" that value anyway.

Thanks for the clarification re go get and go mod tidy.

[tbpg]

Renovate follows whatever Go version that is specified in the go.mod files.

Gotcha. I missed that. Often, the development version and the compatibility version are different. Is there any way we could specify the version of Go to use as part of the config?

I'm not sure "upgrading" it is a good idea. That value indicates the minimum Go version needed to use the module. So, it's reasonable for it to stay the same for a long time.

[rarkins]

I wasn't aware that it means the minimum version - thank you for clarifying. In such case we may be better using the latest version of Go if that is also compatible.

We also plan to allow configurable versions in the future but it's not supported yet

[tbpg]

Yes, I think latest would make the most sense.

Sounds good with configuring. If it always used the latest, I don't think we'd need the ability to configure.

[rarkins]

Waiting on #6909

[tbpg]

Friendly nudge -- looks like #6909 is fixed.

[rarkins]

Yes, I guess it's as simple as adding one more go mod tidy to our command sequence to see if that fixes it.

[renovate-release]

:tada: This issue has been resolved in version 22.19.1 :tada:

The release is available on:

• GitHub release
• 22.19.1

Your semantic-release bot :package::rocket:

[rarkins]

I'll put this live tomorrow

[tbpg]

Thank you!

[rarkins]

Can you tick the rebase checkbox to try your existing PR again?

[tbpg]

Unfortunately, it looks like they still aren't tidy. Perhaps the new release wasn't fully rolled out yet?

[rarkins]

It's rolled out. Here's the command that was run: docker run --rm --name=renovate_go --label=renovate_child -v "/mnt/renovate/gh/GoogleCloudPlatform/golang-samples":"/mnt/renovate/gh/GoogleCloudPlatform/golang-samples" -v "/tmp/renovate-cache":"/tmp/renovate-cache" -v "/tmp/renovate-cache/others/go":"/tmp/renovate-cache/others/go" -e GOPATH -e CGO_ENABLED -w "/mnt/renovate/gh/GoogleCloudPlatform/golang-samples/run/logging-manual" renovate/go:1.12.17 bash -l -c "git config --global url.\"https://**redacted**@github.com/\".insteadOf \"https://github.com/\" && go get -d ./... && go mod tidy && go mod tidy"","msg":"Executing command","time":"2020-08-18T13:30:55.764Z"} {"level":20,"branch":"renovate/all","cmd":"docker run --rm --name=renovate_go --label=renovate_child -v "/mnt/renovate/gh/GoogleCloudPlatform/golang-samples":"/mnt/renovate/gh/GoogleCloudPlatform/golang-samples" -v "/tmp/renovate-cache":"/tmp/renovate-cache" -v "/tmp/renovate-cache/others/go":"/tmp/renovate-cache/others/go" -e GOPATH -e CGO_ENABLED -w "/mnt/renovate/gh/GoogleCloudPlatform/golang-samples/run/logging-manual" renovate/go:1.12.17 bash -l -c "git config --global url.\"https://**redacted**@github.com/\".insteadOf \"https://github.com/\" && go get -d ./... && go mod tidy && go mod tidy"

[rarkins]

When I check out that PR and run go mod tidy I get this:

diff --git a/go.mod b/go.mod
index e60a1d4..0d6a58f 100644
--- a/go.mod
+++ b/go.mod
@@ -22,7 +22,6 @@ require (
        github.com/fluent/fluent-logger-golang v1.5.0
        github.com/go-sql-driver/mysql v1.5.0
        github.com/gofrs/uuid v3.3.0+incompatible
-       github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
        github.com/golang/protobuf v1.4.2
        github.com/gomodule/redigo v2.0.0+incompatible
        github.com/google/go-cmp v0.5.1
diff --git a/go.sum b/go.sum
index 7ea2449..8566c1c 100644
--- a/go.sum
+++ b/go.sum
@@ -203,6 +203,7 @@ github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORN
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/lib/pq v1.8.0 h1:9xohqzkUwzR4Ga4ivdTcawVS89YSDVxXMa3xJX3cGzg=
 github.com/lib/pq v1.8.0/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
 github.com/linkedin/goavro/v2 v2.9.8 h1:jN50elxBsGBDGVDEKqUlDuU1cFwJ11K/yrJCBMe/7Wg=
 github.com/linkedin/goavro/v2 v2.9.8/go.mod h1:UgQUb2N/pmueQYH9bfqFioWxzYCZXSfF8Jw03O5sjqA=

Surely we don't need another tidy?

[rarkins]

Something else that is perhaps contributing is that we're not actually running the go commands once, we're running them 35 times for the 35 go.mod files. Maybe they interfere with each other somehow?

[tbpg]

That command looks OK to me. I was concerned about the working directory because go commands work on the module in the current directory, but that seems correct with the -w flag to docker run. I don't think another go mod tidy would make a difference... I would think one would do the trick.

Each go.mod should be completely independent from the others. When you first add a "nested" go.mod it may lead to changes in the go.mod in the higher directory because the higher one doesn't have to track as many deps. But, I don't think that's the issue here.

I tried running the docker command locally (modifying it to mount pwd instead of /mnt/renovate/gh/GoogleCloudPlatform/golang-samples), but I ran into a volume write permission error.

I don't think we need the go get -d ./.... That doesn't affect go mod tidy (tidy will download what it needs). But, again, I don't think that would cause the problem.

[sluongng]

Im also facing a similar issue

ERROR: Error updating branch: Command failed: go mod tidy
       go: k8s.io/klog/v2/[email protected]: reading https://proxy.golang.org/k8s.io/klog/v2/v2/@v/v2.3.0.mod: 410 Gone
        server response: not found: k8s.io/klog/v2/[email protected]: invalid version: unknown revision v2/v2.3.0
        (repository=redacted/my-repo, branch=renovate/major-2-k8s-modules)
       "err": {
         "killed": false,
         "code": 1,
         "signal": null,
         "cmd": "go mod tidy",
         "stdout": "",
         "stderr": "go: k8s.io/klog/v2/[email protected]: reading https://**redacted**@v/v2.3.0.mod: 410 Gone\n\tserver response: not found: k8s.io/klog/v2/[email protected]: invalid version: unknown revision v2/v2.3.0\n",
         "message": "Command failed: go mod tidy\ngo: k8s.io/klog/v2/[email protected]: reading https://**redacted**@v/v2.3.0.mod: 410 Gone\n\tserver response: not found: k8s.io/klog/v2/[email protected]: invalid version: unknown revision v2/v2.3.0\n",
         "stack": "Error: Command failed: go mod tidy\ngo: k8s.io/klog/v2/[email protected]: reading https://**redacted**@v/v2.3.0.mod: 410 Gone\n\tserver response: not found: k8s.io/klog/v2/[email protected]: invalid version: unknown revision v2/v2.3.0\n\
n    at ChildProcess.exithandler (child_process.js:303:12)\n    at ChildProcess.emit (events.js:315:20)\n    at ChildProcess.EventEmitter.emit (domain.js:483:12)\n    at maybeClose (internal/child_process.js:1021:16)\n    at Process.ChildPr
ocess._handle.onexit (internal/child_process.js:286:5)"
       }

Running go mod tidy inside /tmp/renovate/repo/.../my-repo works fine.

$ renovate --version
23.5.0

[rarkins]

Not the same issue, please see if you can reproduce it in a public repo and raise a separate bug report.

[sluongng]

@rarkins thanks, raised https://github.com/renovatebot/renovate/issues/7121

[HonkingGoose]

Closing this issue as it's a duplicate of #6213.