Hugo: Mage failing with "undefined: mg.Context" error

[vassudanagunta]

All mage commands (except mage -h) in the hugo source directory fail with the exact same error output:

mage hugo
# command-line-arguments
./mage_output_file.go:330:17: undefined: mg.Context
Error: error compiling magefiles

mage install
# command-line-arguments
./mage_output_file.go:330:17: undefined: mg.Context
Error: error compiling magefiles

mage -l
# command-line-arguments
./mage_output_file.go:330:17: undefined: mg.Context
Error: error compiling magefiles

I even tried with a clean GO environment, following all the steps in CONTRIBUTING.md exactly, with the same result.

I ran mage -keep to diagnose the problem. The resulting mage_output_file.go is attached. Line 330 is ctx, cancel := mg.Context(), indicating that for some reason the Context func in the mg package can't be linked even though I confirmed it was there.

I can't tell if this is a flaw with mage or the hugo source. I was able to run mage -l and mage build from the ~/go/src/github.com/magefile/ directory, so I'm pretty sure my mage installation is sound.

[vassudanagunta]

@natefinch Any ideas on how to isolate?

[moorereason]

You likely need to install a new mage (using ... in your go get command).

go get -u github.com/magefile/mage/...

[vassudanagunta]

Trust me, I tried that (and just did again) and everything else outside of what a go/mage expert my try. Has anyone tried doing building hugo from a completely clean (from scratch) go environment?

[natefinch]

do you have a vendor directory under hugo's root directory? It's probably caching the wrong version there. That version will override anything in your gopath.

[KevinGimbel]

I just tried to run mage hugo before updating mage but _after_ pulling the latest changes from master.

Without updating mage it works, after running go get -u github.com/magefile/mage/... I am getting the same error as @vassudanagunta.

Before Updating mage (with latest source from master branch):

~GOPATH/src/github.com/gohugoio/hugo master
λ mage hugo

~GOPATH/src/github.com/gohugoio/hugo master
λ mage -version
Mage Build Tool
Build Date: <not set>
Commit: <not set>

~GOPATH/src/github.com/gohugoio/hugo master
λ ./hugo version
Hugo Static Site Generator v0.31-DEV-B6F2E3B4 darwin/amd64 BuildDate: 2017-10-24T16:20:26+02:00

After Updating mage with go get -u github.com/magefile/mage/...:

~GOPATH/src/github.com/gohugoio/hugo master
λ go get -u github.com/magefile/mage/...

~GOPATH/src/github.com/gohugoio/hugo master
λ which mage
/Users/kevingimbel/Development/go/bin/mage

~GOPATH/src/github.com/gohugoio/hugo master
λ ls -l /Users/kevingimbel/Development/go/bin/mage
-rwxr-xr-x  1 kevingimbel  staff  6343840 Oct 24 16:22 /Users/kevingimbel/Development/go/bin/mage

~GOPATH/src/github.com/gohugoio/hugo master
λ mage hugo
# command-line-arguments
./mage_output_file.go:330:17: undefined: mg.Context
Error: error compiling magefiles

[natefinch]

Ahh, this is the problem: https://github.com/gohugoio/hugo/blob/master/Gopkg.toml#L29

The vendor directory is pinned at v1, but the versions of mage that are running are generating code that expects to be able to reference libraries from v2.

This is kind of a backwards compatibility problem. We may need a version number in the magefile so mage can know what version of the main file to generate.

[natefinch]

For now, if you delete the magefile directory under hugo's vendor directory, it'll fall back to using the one in your gopath, which should work.

[natefinch]

This is another form of this problem: https://github.com/magefile/mage/issues/79

[vassudanagunta]

Yup, that fixes it. But building hugo with mage puts the V1 mage right back in the vendor directory. Because mage uses the cached compiled magefile in ~/.magegfile, you won't notice the problem unless you mage -f or a change to the hugo magefile triggers a new compile.

[galopin]

@vassudanagunta Sadly I was able to replicate this behavior too 😕

mage -l
# command-line-arguments
.\mage_output_file.go:330:17: undefined: mg.Context
Error: error compiling magefiles

Fear not, I found a way to replicate consistently the dreaded undefined: mg.Context error.

TL;DR: do not mess up the MAGEFILE_CACHE directory contents…

I cleaned the Hugo slate once again today with go clean -i -x github.com/gohugoio/hugo..., followed the updated steps (https://github.com/gohugoio/hugo/commit/30e471b8c66f43b2505ba4a303b5cacb4a5f3af3) described in CONTRIBUTING.md and installed Hugo successfully with Mage:

Hugo Static Site Generator v0.31-DEV-B6F2E3B windows/386 BuildDate: 2017-10-24T16:08:46+02:00

Then listing Mage targets would work as expected 🏁

mage -l
Targets:
  check            Run tests and linters
  checkVendor      Verify that vendored packages match git HEAD
  docker           Build hugo Docker container
  fmt              Run gofmt linter
  hugo             Build hugo binary
  hugoNoGitInfo    Build hugo without git info
  hugoRace         Build hugo binary with race detector enabled
  install          Install hugo binary
  lint             Run golint linter
  test             Run tests
  test386          Run tests in 32-bit mode
  testCoverHTML    Generate test coverage report
  testRace         Run tests with race detector
  vendor           Install Go Dep and sync Hugo's vendored dependencies
  vet              Run go vet linter

But deleting — or sending to the trash — the corresponding compiled magefile binary in MAGEFILE_CACHE directory will trigger the undefined: mg.Context error. So is there a reliable way to (re)generate the compiled magefile binary if it has been deleted unwittingly? mage -fdoesn't work as expected ⁉️

The magefiles are hashed so that if they remain unchanged, the same compiled binary will be reused next time, to avoid the generation overhead.

mage -f
# command-line-arguments
.\mage_output_file.go:330:17: undefined: mg.Context
Error: error compiling magefiles

As I said previously, this is a Mage-only problem so we should probably open a new issue on the Mage project.

Hope this helps!

[galopin]

@natefinch @vassudanagunta Oops, sorry guys I took way too much time to summarize my findings 😉

[galopin]

@moorereason @natefinch This TOML constraint setting was committed yesterday (https://github.com/gohugoio/hugo/commit/45ad8ac381ab71aec6f62bc0842934b8bdecd9d7) after my initial bug report.

[natefinch]

This is now fixed in master of mage. I'll get a new binary built with the fix ASAP.

Note that this does not fix the fact that if you reference libraries in your magefile then you need to have those libraries on your local machine. But it at least fixes the backwards compatibility problem introduced recently that creates the mg.Context error. i.e. using the mage v2 binary with mage v1 libraries vendored.

Please retest with current master. I tested locally with hugo and could repro the problem before / not repro the problem after, but I would like independent confirmation.

[vassudanagunta]

Fix confirmed.

[natefinch]

The v2.0.1 binary release incorporates this fix:
https://github.com/magefile/mage/releases/tag/v2.0.1

[natefinch]

Btw, just for sake of completeness, there's a new fix to mage's own build script to run go install with the -a option so that it won't no-op if mage was already built with go get (this was confusing when you used go get and then tried to mage build), and there's new and better install instructions:

go get -u -d github.com/magefile/mage
cd $GOPATH/src/github.com/magefile/mage
go run bootstrap.go

(of course go get and then mage build also works)