Caddy: Build error regarding h2quic

[wallacyyy]

When I try to build the server I'm receiving:

package github.com/lucas-clemente/quic-go/h2quic: 
cannot find package "github.com/lucas-clemente/quic-go/h2quic"

I noticed there was a recent change on the library renaming the directory from h2quic to http3:
Example here.

[francislavoie]

@marten-seemann what do you think should be done here?

[marten-seemann]

Nothing needs to be done here. Caddy has its vendored version of quic-go, so the current state of the quic-go repository is irrelevant.

[francislavoie]

Alright, thanks. Closing this then.

[MOZGIII]

I have the same issue when building from source. I don't see the vendored code in caddy.
Could you please re-open this issue?

I'm following the build instructions from the README.md

[JoanMartinezRipoll]

Same here, I am getting the exact same issue

[mholt]

We no longer use vendor, starting with 1.0 beta 1.

Are you in a GOPATH? Try setting export GO111MODULE=on and try again.

[elcore]

We no longer use vendor, starting with 1.0 beta 1.

Are you in a GOPATH? Try setting export GO111MODULE=on and try again.

Yes, this should fix this.

[wP99DhYrMMTYbfAvgiWm]

Good evening,

I am running into this issue as well. Setting GO111MODULE to on does allow the go get call to continue, but I then run into the error shown below:

# github.com/mholt/caddy/caddyhttp/markdown/summary
../pkg/mod/github.com/mholt/[email protected]/caddyhttp/markdown/summary/render.go:24:5: cannot use (*renderer)(nil) (type *renderer) as type blackfriday.Renderer in assignment:
    *renderer does not implement blackfriday.Renderer (missing RenderFooter method)
../pkg/mod/github.com/mholt/[email protected]/caddyhttp/markdown/summary/summary.go:26:44: too many arguments to conversion to blackfriday.Markdown: blackfriday.Markdown(input, renderer literal, 0)
# github.com/mholt/caddy/caddytls
../pkg/mod/github.com/mholt/[email protected]/caddytls/setup.go:174:28: cannot use value (type "github.com/xenolf/lego/certcrypto".KeyType) as type "github.com/go-acme/lego/certcrypto".KeyType in assignment
../pkg/mod/github.com/mholt/[email protected]/caddytls/setup.go:354:4: cannot use config.Manager.KeyType (type "github.com/go-acme/lego/certcrypto".KeyType) as type "github.com/xenolf/lego/certcrypto".KeyType in field value

Is there something else I should be doing or setting?

[mholt]

@wP99DhYrMMTYbfAvgiWm v0.11.5 does not use Go modules, so you'll need to use v1.0.0-beta1.

[wP99DhYrMMTYbfAvgiWm]

Thanks for the quick reply!

The strange thing is I am getting the code using go get -u github.com/mholt/caddy/caddy, which, as far as I know, fetches the latest version. In addition, if I try to run this without the environment variable mentioned above, I run into issues with the h2quic library, which should not exist in 0.11.5.

Disclaimer: As you can no doubt tell, I am fairly new to Go. Please forgive me for any stupid things I might say :D

[mholt]

Don't worry, I'm also new to this (Go modules, anyway) hence the beta period. :)

What does go list -u -m all show?

[mholt]

Huh, apparently the docs on modules are missing some critical information: https://github.com/golang/go/wiki/Modules#how-to-upgrade-and-downgrade-dependencies -- says

go get -u – Update all direct and indirect dependencies to latest minor or patch upgrades

But actually it ignores beta releases.

Edit: I just updated the wiki page so it at least says it now...

[wP99DhYrMMTYbfAvgiWm]

Huh, I would not have guessed that. Features, I suppose :stuck_out_tongue:.

I'm fine with running 0.11.5 for now, but unfortunately this still does not build due to the errors mentioned above (when using GO111MODULE=on) or due to the error shown below when not using this variable. Running go list -u -m all did not return anything. I should probably also mention then I'm attempting to build this for Docker (as part of an image for something else, so I can't use one of the existing Caddy images on Docker Hub).

package github.com/lucas-clemente/quic-go/h2quic: cannot find package "github.com/lucas-clemente/quic-go/h2quic" in any of:
    /usr/lib/go-1.12/src/github.com/lucas-clemente/quic-go/h2quic (from $GOROOT)
    /go/src/github.com/lucas-clemente/quic-go/h2quic (from $GOPATH)

Edit: I guess this makes sense. Even though I'm using an older version of Caddy, the h2quic library has still been updated. I guess this means we can't really build 0.11.5 from source anymore (for now?).

One other possible solution I came up with was to clone the entire Caddy repo, replace all instances of h2quick with http3 using sed and then attempting to build. Unfortunately I ran into issues when running go get -d ./... (before attempting to build) this way:

# cd /go/src/github.com/marten-seemann/qpack; git submodule update --init --recursive
Cloning into 'integrationtests/interop/qifs'...
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:qpackers/qifs.git' into submodule path 'integrationtests/interop/qifs' failed
package github.com/marten-seemann/qpack: exit status 128

(I am ignoring the possibility that it would still not build with the updated h2quic code for now... :angel:)

[mholt]

If you check out v0.11.5, it should give you the vendor folder and use it, and it should build OK as far as I know.

[wP99DhYrMMTYbfAvgiWm]

I see. A final question then: how would I go about doing this?

I have attempted this using both go get -tags=v0.11.5 github.com/mholt/caddy/caddy and go get github.com/mholt/caddy/caddy@80dfb8b2a7f89b120a627bc4d866a1dc5ed3d92f, but both run into either the h2quic issue or the issues mentioned here.

Update: It is possible to get the beta version using the commit hash (using -tags does not work): go get -u github.com/mholt/caddy/caddy@657780bcdf1dae53269e4b024336bea6fb380e40:

go: finding github.com/mholt/caddy/caddy 657780bcdf1dae53269e4b024336bea6fb380e40
go: finding github.com/mholt/caddy 657780bcdf1dae53269e4b024336bea6fb380e40
go: downloading github.com/mholt/caddy v1.0.0-beta1

This still results in an error around markdown, but at least we now have the beta version!

go: extracting golang.org/x/text v0.3.0
# github.com/mholt/caddy/caddyhttp/markdown/summary
../pkg/mod/github.com/mholt/[email protected]/caddyhttp/markdown/summary/render.go:24:5: cannot use (*renderer)(nil) (type *renderer) as type blackfriday.Renderer in assignment:
    *renderer does not implement blackfriday.Renderer (missing RenderFooter method)
../pkg/mod/github.com/mholt/[email protected]/caddyhttp/markdown/summary/summary.go:26:44: too many arguments to conversion to blackfriday.Markdown: blackfriday.Markdown(input, renderer literal, 0)
# github.com/mholt/caddy/caddyhttp/httpserver
../pkg/mod/github.com/mholt/[email protected]/caddyhttp/httpserver/tplcontext.go:278:14: undefined: blackfriday.HtmlRenderer
../pkg/mod/github.com/mholt/[email protected]/caddyhttp/httpserver/tplcontext.go:280:11: undefined: blackfriday.EXTENSION_TABLES
../pkg/mod/github.com/mholt/[email protected]/caddyhttp/httpserver/tplcontext.go:281:11: undefined: blackfriday.EXTENSION_FENCED_CODE
../pkg/mod/github.com/mholt/[email protected]/caddyhttp/httpserver/tplcontext.go:282:11: undefined: blackfriday.EXTENSION_STRIKETHROUGH
../pkg/mod/github.com/mholt/[email protected]/caddyhttp/httpserver/tplcontext.go:283:11: undefined: blackfriday.EXTENSION_DEFINITION_LISTS
../pkg/mod/github.com/mholt/[email protected]/caddyhttp/httpserver/tplcontext.go:284:34: too many arguments to conversion to blackfriday.Markdown: blackfriday.Markdown([]byte(body), renderer, extns)

[elcore]

Hi,

I hope I can help you. Caddy v0.11.5 does not support modules. Because of this, these problems occur.

If you want to build Caddy v0.11.5, you will have to use these instructions. (GO111MODULES should be set to auto - export GO111MODULES=auto)

1. go get github.com/mholt/caddy
2. go get github.com/caddyserver/builds
3. cd $GOPATH/src/github.com/mholt/caddy/caddy
4. git checkout tags/v0.11.5
5. go run build.go

If you want to build Caddy v1.0.0-beta1, you will have to use these instructions.

1. export GO111MODULES=on This will not be necessary with Go 1.13 (it should be the default)
2. go get github.com/mholt/caddy
3. go get github.com/caddyserver/builds
4. cd $GOPATH/src/github.com/mholt/caddy/caddy
5. go run build.go

or

1. git clone https://github.com/mholt/caddy
2. cd caddy/caddy
3. go run build.go

[wP99DhYrMMTYbfAvgiWm]

Hi @elcore,

Thank you for the clear instructions, this helped me get it working! Much appreciated.

[MOZGIII]

I think this should be reflacted in the readme.

I ended up with the following:

export GO111MODULES=off

mkdir -p "$(go env GOPATH)/src/github.com/mholt/caddy"
cd "$(go env GOPATH)/src/github.com/mholt/caddy"
git clone https://github.com/mholt/caddy .
git checkout -f -b build 80dfb8b2a7f89b120a627bc4d866a1dc5ed3d92f

mkdir -p "$(go env GOPATH)/src/github.com/caddyserver/builds"
cd "$(go env GOPATH)/src/github.com/caddyserver/builds"
git clone https://github.com/caddyserver/builds .
git checkout -f -b build c62e2219460a8828970dad09212c3a4cec40b56c

cd "$(go env GOPATH)/src/github.com/mholt/caddy/caddy"
go run build.go

Why do we need to use this github.com/caddyserver/builds by the way?

[elcore]

We will upgrade the instructions, you just need to run export GO111MODULES=on beforehand, and the build will succeed. This is just temporary.

Why do we need to use this github.com/caddyserver/builds by the way?

To enable proper versioning of Caddy binaries.

[elcore]

Additionally, modules have the advantage that you can build them outside your GOPATH.

1. git clone https://github.com/mholt/caddy
2. cd caddy/caddy
3. go run build.go

[MOZGIII]

To enable proper versioning of Caddy binaries.

My question is: why would we want this code to be in a separate repo (github.com/caddyserver/builds)?
I makes no sense to keep it separate - it would've worked just as good if it was in a subdirectory in the same repo.

I'm not using modules because I need a very particular version of the caddy (commit hashes are for an older version in my real use case) - and used the vendoring setup back then.

I'm also donig git clone instead of a go get -d (or even go get as is the example), because it doesn't work if the latest version is "broken" (which is the case after switch to go modules for all the older Go versions or disabled GO111MODULES).

Of course, moving to the latest version could solve some of the issues - but it has other problems (with dependencies) that are blocking me from doing just that. I'll look into it later.

I'm just saying that for now I've came up with a workaround that's good enough for us - but this was not trivial. Please reconsider making caddy simply go-get-able.

[elcore]

I'm not using modules because I need a very particular version of the caddy (commit hashes are for an older version in my real use case) - and used the vendoring setup back then.

That is okay, you can use an older version of Caddy.

I'm also donig git clone instead of a go get -d (or even go get as is the example), because it doesn't work if the latest version is "broken" (which is the case after switch to go modules for all the older Go versions or disabled GO111MODULES).

It is not broken, go get github.com/mholt/caddy works

Of course, moving to the latest version could solve some of the issues - but it has other problems (with dependencies) that are blocking me from doing just that. I'll look into it later.

What is the issue?

I'm just saying that for now I've came up with a workaround that's good enough for us - but this was not trivial. Please reconsider making caddy simply go-get-able.

Caddy is go-get-able...

[smlx]

Hi,

None of the instructions in the thread above are working for me in the current (go 1.12) docker container.

The reason is that go run build.go in the README doesn't work, as go modules are not actually git repositories, and are stored in $GOPATH/pkg/mod.

Here's the procedure that is actually working for v1.0.0-beta1, starting from an empty GOPATH:

export GO111MODULE=on
go get -v -ldflags "-X github.com/mholt/caddy/caddy/caddymain.appVersion=1.0.0-beta1" github.com/mholt/caddy/[email protected]
$GOPATH/bin/caddy -version

Not sure what the correct fix is for the documentation @mholt ?

[elcore]

You should run it in $GOPATH/src/ not $GOPATH/pkg/mod

Take a look at https://github.com/abiosoft/caddy-docker

[deanishe]

I can't get v1.0.0-beta1 (or master) to build.

With GO111MODULE off, I get the above h2quic error. With GO111MODULE=on, I get this error with the ipfilter plugin:

go: extracting github.com/pyed/ipfilter v1.1.1
-> unzip /Users/username/pkg/mod/cache/download/github.com/pyed/ipfilter/@v/v1.1.1.zip: malformed file path "testdata/blacklist/1234/abcd/1234:abcd::1": invalid char ':'
build github.com/mholt/caddy/caddy: cannot load github.com/pyed/ipfilter: unzip /Users/username/pkg/mod/cache/download/github.com/pyed/ipfilter/@v/v1.1.1.zip: malformed file path "testdata/blacklist/1234/abcd/1234:abcd::1": invalid char ':'
2019/04/16 18:26:12 exit status 1
exit status 1

I'm trying to build on macOS High Sierra, though I understand that : is blacklisted as it's a reserved character for filenames on Windows.

Disabling the ipfilter plugin leads to this error in both v1.0.0-beta1 and master:

# github.com/mholt/caddy/caddyhttp/markdown/summary
../caddyhttp/markdown/summary/render.go:24:5: cannot use (*renderer)(nil) (type *renderer) as type blackfriday.Renderer in assignment:
    *renderer does not implement blackfriday.Renderer (missing RenderFooter method)
../caddyhttp/markdown/summary/summary.go:26:44: too many arguments to conversion to blackfriday.Markdown: blackfriday.Markdown(input, renderer literal, 0)
# github.com/mholt/caddy/caddyhttp/httpserver
../caddyhttp/httpserver/tplcontext.go:278:14: undefined: blackfriday.HtmlRenderer
../caddyhttp/httpserver/tplcontext.go:280:11: undefined: blackfriday.EXTENSION_TABLES
../caddyhttp/httpserver/tplcontext.go:281:11: undefined: blackfriday.EXTENSION_FENCED_CODE
../caddyhttp/httpserver/tplcontext.go:282:11: undefined: blackfriday.EXTENSION_STRIKETHROUGH
../caddyhttp/httpserver/tplcontext.go:283:11: undefined: blackfriday.EXTENSION_DEFINITION_LISTS
../caddyhttp/httpserver/tplcontext.go:284:34: too many arguments to conversion to blackfriday.Markdown: blackfriday.Markdown([]byte(body), renderer, extns)
2019/04/16 18:30:38 exit status 2
exit status 1

[elcore]

With GO111MODULE off, I get the above h2quic error

Caddy v1.0.0-beta1 requires modules to be enabled.

With GO111MODULE=on, I get this error with the ipfilter plugin

I kindly refer you to https://github.com/pyed/ipfilter.

Disabling the ipfilter plugin leads to this error in both v1.0.0-beta1 and master:

Are you sure you are on master/v1.0.0-beta1? This error is usually seen with Caddy v0.11.X with modules enabled.

I mean, it's successful in the new Azure Pipelines 🤷

[deanishe]

Caddy v1.0.0-beta1 requires modules to be enabled.

Yeah. I tried it the old way because ipfilter works that way. It doesn't get along with go mod.

I kindly refer you to https://github.com/pyed/ipfilter.

I've already created an issue.

Are you sure you are on master/v1.0.0-beta1? This error is usually seen with Caddy v0.11.X with modules enabled.

I thought I was:

in ~/src/github.com/mholt/caddy/caddy on tags/v1.0.0-beta1 ●
❯ GO111MODULE=on go run build.go -goarch amd64 -goos linux
# github.com/mholt/caddy/caddyhttp/markdown/summary
../caddyhttp/markdown/summary/render.go:24:5: cannot use (*renderer)(nil) (type *renderer) as type blackfriday.Renderer in assignment:
    *renderer does not implement blackfriday.Renderer (missing RenderFooter method)
../caddyhttp/markdown/summary/summary.go:26:44: too many arguments to conversion to blackfriday.Markdown: blackfriday.Markdown(input, renderer literal, 0)
# github.com/mholt/caddy/caddyhttp/httpserver
../caddyhttp/httpserver/tplcontext.go:278:14: undefined: blackfriday.HtmlRenderer
../caddyhttp/httpserver/tplcontext.go:280:11: undefined: blackfriday.EXTENSION_TABLES
../caddyhttp/httpserver/tplcontext.go:281:11: undefined: blackfriday.EXTENSION_FENCED_CODE
../caddyhttp/httpserver/tplcontext.go:282:11: undefined: blackfriday.EXTENSION_STRIKETHROUGH
../caddyhttp/httpserver/tplcontext.go:283:11: undefined: blackfriday.EXTENSION_DEFINITION_LISTS
../caddyhttp/httpserver/tplcontext.go:284:34: too many arguments to conversion to blackfriday.Markdown: blackfriday.Markdown([]byte(body), renderer, extns)
2019/04/16 18:31:12 exit status 2
exit status 1

I don't know what went wrong there, but after checking out v0.11.5 to build and then going back to v1.0.0-beta1 and master, both now compile perfectly.

Sorry for the erroneous error report and thanks for your help.

[elcore]

I am here to help :smile:. I wish you a wonderful day.

[smlx]

You should run it in $GOPATH/src/ not $GOPATH/pkg/mod

Take a look at https://github.com/abiosoft/caddy-docker

Sure, but that script uses git clone, not go get, which means that the instructions in the README are wrong.

Here's a repository with a few Dockerfiles showing the issue: https://github.com/smlx/caddy-build.

Using git clone as in the abiosoft/caddy-docker build script, the build succeeds for both 0.11.5 and 1.0.0-beta1:

docker build --pull --no-cache ./git-clone-0.11.5
docker build --pull --no-cache ./git-clone-1.0.0-beta1

Using go get as in the caddy README fails with:

docker build --pull --no-cache  ./go-get-0.11.5

...

# github.com/mholt/caddy/caddyhttp/markdown/summary
pkg/mod/github.com/mholt/[email protected]/caddyhttp/markdown/summary/render.go:24:5: cannot use (*renderer)(nil) (type *renderer) as type blackfriday.Renderer in assignment:
    *renderer does not implement blackfriday.Renderer (missing RenderFooter method)
pkg/mod/github.com/mholt/[email protected]/caddyhttp/markdown/summary/summary.go:26:44: too many arguments to conversion to blackfriday.Markdown: blackfriday.Markdown(input, renderer literal, 0)
# github.com/mholt/caddy/caddytls
pkg/mod/github.com/mholt/[email protected]/caddytls/setup.go:174:28: cannot use value (type "github.com/xenolf/lego/certcrypto".KeyType) as type "github.com/go-acme/lego/certcrypto".KeyType in assignment
pkg/mod/github.com/mholt/[email protected]/caddytls/setup.go:354:4: cannot use config.Manager.KeyType (type "github.com/go-acme/lego/certcrypto".KeyType) as type "github.com/xenolf/lego/certcrypto".KeyType in field value

Using go get as in the caddy README and specifying the 1.0.0-beta1 module version fails differently, as go get doesn't put anything in $GOPATH/src.

docker build --pull --no-cache  ./go-get-1.0.0-beta1

...

/bin/sh: cd: line 1: can't cd to /go/src/github.com/mholt/caddy/caddy: No such file or directory

It appears to be a documentation issue, and I'm happy to work on a fix if you could clarify which procedure is supposed to work - go get, or git clone.

[elcore]

Considering that modules will be the default, and that $GOPATH will be deprecated, we could change the documentation to something along those lines.

1. git clone https://github.com/mholt/caddy
2. cd caddy/caddy
3. go run build.go

[elcore]

What are your thoughts @mholt?

[mholt]

I've been playing with modules + Caddy all day to try to understand what is going on.

A few things that are not obvious:

• Caddy has a tagged release with modules. However, it's a -beta release, and go tools don't honor pre-releases unless they're the only tag. This is buried here in the go tool docs:

By default, get chooses the latest tagged release version, such as v0.4.5 or v1.2.3. If there are no tagged release versions, get chooses the latest tagged prerelease version, such as v0.0.1-pre1. If there are no tagged versions at all, get chooses the latest known commit.

• Because of that design decision, go get github.com/mholt/caddy/caddy pulls down v0.11.5, the latest non-prerelease tag, which is not a module.

• Because Caddy v0.11.5 is not a module, running go get -u has no constraints on which version of russross/blackfriday is pulled down, so it gets the latest, which is v2, which is not compatible with Caddy's current implementation which uses v1. Hence the errors undefined: blackfriday.HtmlRenderer etc.

• go get will always pull into a GOPATH:

When using modules, GOPATH is no longer used for resolving imports. However, it is still used to store downloaded source code (in GOPATH/pkg/mod) and compiled commands (in GOPATH/bin).

I haven't yet figured out why everyone saw the errors they described here, but I suspect it has to do
with a combination of these things, and lots of hidden magic in the Go tooling:

• The removal of our vendor folder for the beta release
• The presence of a prerelease value in the latest tag
• The value of the GO111MODULE env variable
• The state/version of the existing Caddy code in the GOPATH, if any
• Whether commands are run in or out of GOPATH
• Whether git clone was used first

In my testing, I've seen wildly different, difficult-to-explain behavior when any and all of these variables change the slightest bit.

It's very sensitive.

However, it appears that starting from scratch, and once we have a stable 1.0 release, we can finally test the transition to Go modules... in production! AFTER tagging the stable release. (Edit: Someone suggested to me that using replace in go.mod might help.)

Sigh.

[elcore]

In my testing, I've seen wildly different, difficult-to-explain behavior when any and all of these variables change the slightest bit.

True!!!!

It's very sensitive.

Unfortunately!

This is a great summary by you!

[MOZGIII]

In my case the behavior is pretty obvious. Remember, I need an older version of caddy. With modules enabled usual go get will pull v0.11.5 that has vendor dir, which will, however, be ignored, and go get will try to download the dependencies from the respective sources. And those won't be pinned by the mod file (and not from vendor dir). This obviously fails. The behavior is the same even if I do go get -d.
That's why I ended up doing git clone instead of go get -d, and then the good old git checkout <rev-i-want> to enforce proper version. This works like a charm because older caddy version still provides vendor directory - and everything that's needed for it to work is actually vendored, as opposed to just version-pinned. Personally I like vendoring more than version pinning, and mostly because it's much more easier to build some very old app version from just the app git repo. With pinning a lot of external services have to be up and provide old package versions - which is currently problematic for go.

By the way, it would be great if somebody notify go team about the issues we encounter here - as this is the very feedback they look on the new mod system.

[elcore]

With modules enabled usual go get will pull v0.11.5 that has vendor dir, which will, however, be ignored

That's the whole point, if you enable modules, vendor will be ignored.

[elcore]

So, if you want v0.11.5, you will have to disable modules or leave it in auto mode (see above)

[MOZGIII]

In my case the behavior is pretty obvious.

[MOZGIII]

So, if you want v0.11.5, you will have to disable modules or leave it in auto mode

Disabling modules will not help. You still have to do git clone, as go get (even with -d) will fail.

:/
The whole point of me writing all that text is to explain the behavior there. It's pretty easy to grasp if you know the rules of old vendoring and new modules system, however not everyone reads the release notes thoroughly.

[mholt]

The whole point of me writing all that text is to explain the behavior there. It's pretty easy to grasp if you know the rules of old vendoring and new modules system, however not everyone reads the release notes thoroughly.

Thank you for the explanation, which I find useful, as I am still learning modules here! 😇

To be fair, though, I've been referencing these resources for on and off for about a week:

• https://blog.golang.org/using-go-modules
• https://golang.org/cmd/go/
• https://github.com/golang/go/wiki/Modules

And not only are all those pages (the last 2 especially) huge walls of text, their docs are not totally consistent. For example, the wiki doesn't mention that prerelease tags are ignored (until I added a blurb to it myself). The go cmd docs say that go get will pull down the src folder, but neglects to mention that this only happens if GO111MODULE != on. That fact is instead very implicitly conveyed pages and pages away in a different section.

So it's all trial and error for me at this point. The docs have been mediocre and the transition has been frustrating for everyone. I appreciate everyone's patience.

[elcore]

Disabling modules will not help. You still have to do git clone

This works pretty well, you don't have to use git clone, just git checkout

If you want to build Caddy v0.11.5, you will have to use these instructions. (GO111MODULE should be set to auto - export GO111MODULE=auto)

1. go get github.com/mholt/caddy
2. go get github.com/caddyserver/builds
3. cd $GOPATH/src/github.com/mholt/caddy/caddy
4. git checkout tags/v0.11.5
5. go run build.go

[MOZGIII]

If that worked then our CI would not break, which it did. At the very step 1. @elcore, if you still insist that it works after reading my comment above - I find it hard to follow your reasoning.

[deanishe]

In my testing, I've seen wildly different, difficult-to-explain behavior when any and all of these variables change the slightest bit.

I feel much less stupid now.

[elcore]

@MOZGIII I hope this can help you.

Edit: Here I explicitly disabled modules, just to be sure.

[elcore]

I do not know, how your build environment (CI) is configured. I am using Ubuntu 18.10, with Go 1.12.4. If I can help you in any way, feel free to message me eldinhadzic [at] protonmail.com

[MOZGIII]

Are you starting from scratch? Empty GOPATH etc? We are building docker image with caddy, using golang:1.12-alpine.

[elcore]

Are you starting from scratch? Empty GOPATH etc? We are building docker image with caddy, using golang:1.12-alpine.

Yes, I am starting from scratch. On a side note, I would recommend you to update to Go 1.12.4

[MOZGIII]

Ok, let's go through what go get does currently with GO111MODULE=off. First of all, it doesn't crash - that's good, but it means something have changed since I last lloked into it. Nontheless.
It does load extra dependencies from the thirdparty repos, which is undesireable for the following reasons:

1. The versions of those dependencies are not pinned.
2. It's a waste of reources, since they're not gonig to be used (after we checkout to a tag with vendored versions).
3. They can break the build. In our current case it's not going to happen, but in another other it will due to stricter netwrok access rules and vendoring policy.

The worst this is what exactly is being attemped to be downloaded depends on the state of the current master branch of the caddy repo. And there was the case of this setup breaking the build, and there's no evidence that there are any measures taken to prevent that in the future.
Speaking of evidence, I don't think just providing and example of the command running successfully now proves that it's the right approach. At this point it looks like some kind of trolling from you, @elcore. I know you have something that works for you, but have to try and explain why it works and what are the potential downsides of the solution.
For us it still won't work, as we tried that, and now we know better, and will be using much more strightforward git cloning.

Regarding Go 1.12.4 - definetly a good idea. We already got our toolchain updated after we encountered this issue.

[francislavoie]

depends on the state of the cyrrent master branch of the caddy repo. And there was the case of this setup breaking the build, and there's no evidence that there are any measures taken to prevent that in the future.

Frankly, you shouldn't be relying on master for any kind of stability. That's effectively the development trunk of this repo. If you want stability you should always be using a tagged release or specific commits. It feels like you're asking for the Caddy repo to stifle progress in the name of stability in the master branch, but that's unrealistic.

I can definitely vouch for @elcore and that he's doing what he can to help you. He's not trolling.

We've also learned a lot here when it comes to go get and go mod. See the issue that @mholt opened on the Go upstream describing some of your complaints: https://github.com/golang/go/issues/31529. go mod is still a new tool, and everyone's just trying to figure out how to use it properly.

With all that said, your feedback about this is definitely appreciated, it's helping uncover edge cases of the build tooling. Thanks for that.

[MOZGIII]

Ok, once again.

@francislavoie, go get with GO111MODULES=off relies on master (not latest tag as it does with modules on).
Therefore, the (@elcore's proposed) go get will load stuff from master before letting you checkout to the older commit you want.

My proposal for installing older version does not have that issue:

export GO111MODULES=off

mkdir -p "$(go env GOPATH)/src/github.com/mholt/caddy"
cd "$(go env GOPATH)/src/github.com/mholt/caddy"
git clone https://github.com/mholt/caddy .
git checkout -f -b build 80dfb8b2a7f89b120a627bc4d866a1dc5ed3d92f

mkdir -p "$(go env GOPATH)/src/github.com/caddyserver/builds"
cd "$(go env GOPATH)/src/github.com/caddyserver/builds"
git clone https://github.com/caddyserver/builds .
git checkout -f -b build c62e2219460a8828970dad09212c3a4cec40b56c

cd "$(go env GOPATH)/src/github.com/mholt/caddy/caddy"
go run build.go

That said, I'm unsubbing from this issue as I feel there's enough context by now for people to see my point of view on the issue.

One last note: maybe it's a good idea to re-add vendor dir, but with go mod vedoring instead of the old go dep. This would definetly be a good option for the transitioning period, while the non-suffixed tag is there. @mholt

[elcore]

Ok, let's go through what go get does currently with GO111MODULE=off. First of all, it doesn't crash - that's good

Great to hear that from you.

It does load extra dependencies from the thirdparty repos, which is undesireable for the following reasons

Your reasons are totally valid, it does pull some thirdparty repos over the network.

Speaking of evidence, I don't think just providing and example of the command running successfully now proves that it's the right approach.

I was trying to show you, that the build will be successful, if you follow instructions I have provided.

At this point it looks like some kind of trolling from you, @elcore. I know you have something that works for you [...].

I am very sorry, if you felt like I was trolling you! I was just trying to be helpful, the transition to modules is not as easy as it should have been. I do not expect anything in return, I am using my resources available to help this project and its community.

I even created a simple Docker container as a proof of concept a few minutes ago. https://github.com/elcore/caddy-docker

For us it still won't work, as we tried that, and now we know better, and will be using much more strightforward git cloning.

It is great that you have found a solution that works for you.

[elcore]

While we are at it, I will follow Matt´s advice, and give you some space to breath.

[ntgeralt]

same problem when docker build.
cannot find package "github.com/lucas-clemente/quic-go/h2quic"

Is there any quick solution?

[pascalandy]

Here is the answer and it's a hacky one:
https://github.com/swarmstack/caddy/blob/master/Dockerfile-no-stats#L50