Hugo: Use Go Modules for Theme Component dependency management

[bep]

This has been mentioned before (I will track down and close that issue), but I'll start fresh.

So, Hugo Themes and Theme Components are very powerful constructs, but the current manual installation/dependency management leaves a lot to be desired. I have tinkered with the idea of using some of the ideas from Go to create a hugo get type of command that could do this stuff, but I imagined lots of work, reinventing the wheel, so to speak.

Today I did some quick tests with Go Modules, and it works surprisingly well with non-Go type of modules. We would need to add some small layer of logic in Hugo to support it, but we would then get really, really good dependency management in Hugo.

To add a new dependency to a theme or a project would then look like this:

1. go get -u github.com/bep/my-shortcodes
2. Add github.com/bep/my-shortcodes to the theme slice in config.toml (and yes, @kaushalmodi, we should add an alias for that config key). Note that we of course still will support the /themes/my-shortcodes variant.

When you want to update your dependencies, you do go get -u to get the latest minor or patch release etc.

Note that there can only be one version of my-shortcodes in a Hugo project at a given time, which is how Minimal Version Selection comes into play. We need to investigate github.com/bep/my-shortcodes/v2 vs namespace vs names etc., but that can probably wait.

Note that the above will require that you have Go (>= v1.11) installed, but we should also probably add a hugo mod vendor command that writes all the dependencies to a folder in the project which could be used without needing Go.

Shout if you think this is a terrible idea, @kaushalmodi @onedrawingperday @digitalcraftsman @budparr @regisphilibert @RickCogley, and gang.

[budparr]

I Love this idea for getting us away from submodules.

If I understand this correctly, everything would be packaged up _before_ our site hits a deployment service/script. It's not clear in my mind exactly: Would we be sending a binary to deploy, so instead of building with Hugo, Netlify—for instance—is building the site with Hugo_site_somethingspecific or whatever?

[kaushalmodi]

So this would need the users of themes with "theme components" to have go >= v1.11 installed, right?

Update: Or may be, for the folks with go >= v1.11 installed, they can use the go get approach, others can fall back to using git sub modules?

I am just trying to understand how this will pan out on Netlify.

[regisphilibert]

I find it very appealing.
If there's a Go fallback for users using < v1.11, it's great, but the current solution could also suffice as a fallback.

I am just trying to understand how this will pan out on Netlify.

Same here, worst case scenario we have special build command to run for when a dep needs to be updated/added and it's totally fine with me.
Best case: Netlify gets/updates the dep as it builds.

[bep]

@budparr there is no "binary"; the dependencies (shortcodes, themes etc.) are purely files and directories.

So, Netlify (at least their latest build image) has Go 1.12 installed, so that should not be a problem. And this is kind of analogous to JS projects needing NPM.

We can possibly improve this in the future if Go exports these commands as an API somewhat.

So for Netlify (given that it has Go 1.12 installed) you would not do anything special on the build side.

Running:

hugo

Will pull down any (new) dependency to the Go Module cache (which I assume is or could be easily cached between builds) and build.

For the situations where you want to create "standalone Hugo" distribution, you would run:

hugo mod vendor

The above would write the dependencies to /vendor (or something) below your Hugo project, and it will build fine even on servers without Go installed.

[budparr]

Yes, of course. I wasn't thinking straight. Makes perfect sense; I was just trying to work out deployment in my head. I like the stand alone solution as well.

[regisphilibert]

Very exciting!

[bep]

Or may be, for the folks with go >= v1.11 installed, they can use the go get approach, others can fall back to using git sub modules?

Yes, submodules etc. will still work. Also see the hugo mod vendor thing.

[onedrawingperday]

This looks great. Particularly the planned hugo mod vendor command, since it will make this feature host agnostic.

[larzza]

If I understand this correctly, it means that this could be used to download any git repository to use as a dependency, for example Bootstrap (which I today add as a git submodule in the assets directory for sass customizing)?

[bep]

@larzza this is, in general, true, but there is a caveat: It will be added as a Hugo Theme Component (which has a fixed folder structure: layouts, static, assets ...), so anything outside those will currently have no effect. We could improve on that, but that is outside of the scope of this particular issue.

[bep]

That said, there are some exploration left to do here -- thinking about it, I'm not totally sure how a submodule inside a dependency would work, etc.

[bep]

Google just announced that they are hosting a proxy server for these modules which should speed things up, and it is supposed to make multi-modules easier; yet to be tested.

[Jos512]

I'm concerned that this makes it harder for new people to use Hugo. It introduces an additional dependency (Go), which needs to be installed and updated. Plus that can generate its own error messages.

Every so often on the forum we already get people who get stuck with the well-documented quick start. I imagine that having to install Go and run go commands leads to much more confusion and people giving up on Hugo.

[bep]

@Jos512 what alternative would you suggest?

[regisphilibert]

I thought you had to install Go to use Hugo in the first place...

If not, couldn't we wrap a default go get into the Hugo CLI? hugo get -u github.com/bep/my-shortcodes

[bep]

If not, couldn't we wrap a default go get into the Hugo CLI? hugo get -u github.com/bep/my-shortcodes

Yes, and we will probably do that (to be able to wrap it with the Hugo stuff we need), but you will even then currently need to have Go installed. I have not looked closely into this, but it looks like the Go Modules stuff isn't available as methods/functions from the outside, so you need to interact with the binary. I do, however, suspect that this will change in the future. I think they have kept this API close to their chest until it is carved in stone.

And no, you don't need Go installed to use Hugo. But I would say that it would feel more natural to me to install Go to get package management than having to install Node/NPM which feels more foreign in this context.

[regisphilibert]

But I would say that it would feel more natural to me to install Go to get package management than having to install Node/NPM which feels more foreign in this context.

I agree. It seemed no one ever complained about having to install node in order to build a website.I don't know of any framework which will install node + npm for you.

[larzza]

It seems kind of strange (but pretty cool) to package (a part of) Go in an executable built by Go. It’s also kind of strange to have a dependency on Go from a stand-alone executable, i.e. Hugo. (I think docker would be of good use to bundle things up, then of course the user has to install docker, but not git and hugo and go and....)

Is it possible to elaborate on the usecase? Is a theme supposed to be a ”Go module”? Or just components in a theme... shortcodes, partials etc? Or is the scenario that it will be possible to mix.

If I develop a theme I guess I would like to have it as a git submodule (or just an ordinary git repo) in the themes directory in my hugo site and then add module dependencies to public available shortcodes and partials I find useful.

[bep]

Is it possible to elaborate on the usecase?

When I start a new site, I want to do add something like this:

imports:
- github.com/bep/mynicetheme
- github.com/bep/common-shortcodes
- github.com/budparr/hugo-components/useful-partials

And then, when I run:

All of the dependencies (including any transitive dependencies) is set up for me and cached as effectively as possible. Then I want easy ways to update dependencies on change and many of the other benefits that come with a proper package manager.

Most of the above may be possible using Git submodules, but it isn't practical (your head will get dizzy if you start to think about transitive dependencies) -- which is why Hugo Theme Components are so criminally underused.

[larzza]

Thanks, got it!

[gcushen]

Can we statically link Go/Go Modules so that there's no need for users to install Go?

[bep]

@gcushen what/how?

[bep]

I have been doing a quick implementation today. I have some challenges left, but it fits surprisingly well into the existing theme code.

See https://github.com/bep/my-modular-site

For a very lean site ...

A positive side to this that I didn't know before is that you can have submodules in subfolders that can have its own version, which is super useful, see

https://github.com/bep/hugotestmods

[moorereason]

@bep, this is an interesting idea.

I'm not super-excited about requiring Go for theme development. I'm inclined to add this as a "preview feature" similar to who Go handled modules initially (check for HUGO056MODULE=on in env).

I agree that we're all hoping for the day when the Go team exports the modules packages. Can we keep the module feature in preview until they do? I wonder how long that will be. Perhaps we should ask Steve what the team is thinking.

[larzza]

Cool!

I'm not super-excited about requiring Go for theme development.

Isn't the idea that the way we do it today will still exist?

A positive side to this that I didn't know before is that you can have submodules in subfolders that can have its own version, which is super useful, see

https://github.com/bep/hugotestmods

About multi module repos:
https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories

I note that the recommendation is not to use multi module repos, but maybe this is a case of a perfect fit. The possibility to tag a subdirectory in git with its own separate tag was new to me.

[bep]

Isn't the idea that the way we do it today will still exist?

Yes.

[bep]

Just tested this with the replace directive for local development:

https://github.com/bep/my-modular-site/commit/9825c51b42ce329d119c1cdaab22ed2b2fd0832d

And it just works with live-reload and all. In short: Everything I have thrown at it has just worked really smoothly. I will test some more during the weekend with transitive dependencies, but this is a feature I really, really want. I would have installed the complete Microsoft Office package if that was a prerequisite.

As described above, there are several ways to opt out of using this. I will listen to arguments against, but I think you need to do better than "I don't want to install Go", and ideally it would be good if you could sketch out a realistic alternative.

/cc @spf13

[spf13]

I love this idea. It's a novel approach to an old wart.

What I don't love is adding a new dependency for using Hugo.

I'd love to see a solution where the old way still works just like it always has, but there's a new easier way that we encourage people to use. It's better in every way, but requires a new dependency.

[spf13]

If not, couldn't we wrap a default go get into the Hugo CLI? hugo get -u github.com/bep/my-shortcodes

Yes, and we will probably do that (to be able to wrap it with the Hugo stuff we need), but you will even then currently need to have Go installed. I have not looked closely into this, but it looks like the Go Modules stuff isn't available as methods/functions from the outside, so you need to interact with the binary. I do, however, suspect that this will change in the future. I think they have kept this API close to their chest until it is carved in stone.

I really like the idea of Hugo wrapping the Go command.

All of the tools we are building internally are just calling the Go command directly. I think that is the API for modules.

I think it would be a nice solution to have Hugo wrap the Go command and if it isn't present have an error that tells you how to manually install the theme and a note that if you want to have Hugo install the theme for you, install Go and link to the Go install page.

[bep]

All of the tools we are building internally are just calling the Go command directly. I think that is the API for modules.

Yes, I studied some of the relevant tools in the tools package below golang/ and all of them integrate via os/exec which makes them loosely coupled. @spf13 I appreciate your input on this, and will repeat myself about what I have said earlier:

• You can use Hugo happily as before with or without themes (by cloning, submodule etc. into /themes)
• People using the "module approach" would need Go during development, but I plan to add a hugo mod vendor command that writes these dependencies to a /vendor folder which can then be used without any Go installed.

Also, I appreciate that Hugo users have vastly different requirements regarding this. For power users like @budparr and @regisphilibert I imagine this to be really important, for the common blogger not so much.

[regisphilibert]

For power users like @budparr and @regisphilibert I imagine this to be really important

💯

[spf13]

I think this the exact right approach. Excited to have it.

On Fri, May 3, 2019 at 6:24 PM Regis Philibert [e-mail]
wrote:

For power users like @budparr https://github.com/budparr and
@regisphilibert https://github.com/regisphilibert I imagine this to be
really important

💯

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gohugoio/hugo/issues/5911#issuecomment-489258856, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABKKZAIYLN63Z6ATYIC6WLPTS3TPANCNFSM4HJD4OCQ
.

[larzza]

I plan to add a hugo mod vendor command that writes these dependencies to a /vendor folder which can them be used without any Go installed.

@bep What is the workflow? Commit /vendor to git so that the user without Go installed can download the Hugo project and build the site? Or distribute a zip?

[bep]

@bep What is the workflow?

How would I know, but the Git thing is probably most common. And I also suspect this to be more attractive for theme builders -- to make their themes as portable as possible, esp. when the themes are not changing too often. Not sure. But there are plenty of people who would want a site without dependencies, so to speak.

[larzza]

Ok! Just wondered if i missed some magic... anyway, this will be a really nice feature!

[bep]

I must say that, playing around with this, adding theme components / changing theme etc. while the server is running (with autodownload of any missing pieces) feels kind of magical.

[bep]

... even changing versions of components works while the server is running. I got to create some kind of video demoing this.

[larzza]

@bep I've built your hugo-modules branch to see if I can take the feature for a test drive. I get various errors, most probably due to that I don't know what I'm doing...

If there is an easy way to get the feature going (or if it isn't possible) I would appreciate a hint, otherwise I'll just keep trying...

I've also reviewed the code trying to get a grip of things, with the distant goal to be able to use hugo/go module with SCSS/SASS include paths. Shouldn't be impossible or what do you think?

I'm using config in the style of this:

theme = [
    #"github.com/bep/hugotestmods/mypartials",
    "github.com/bep/hugotestmods/myshortcodes",
    "github.com/bep/hugo-fresh",
    #"github.com/jeblister/kube",
    #"github.com/spf13/hyde",
]

Some of the things I've tried:

hugo mod get github.com/bep/hugo-fresh
can't load package: package github.com/bep/hugo-fresh: no Go files in  github.com/bep/hugo-fresh: no Go files in <GOPATH>/src/github.com/bep/hugo-fresh 

I've read somewhere that there has to be go source files in the module for Go to think it's a valid module, so in that sense I guess the error makes sense.

$ hugo mod get --config config.toml,dev.toml 
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/gohugoio/hugo/commands.newModCmd.func1(0xc000568f00, 0xc000290240, 0x0, 0x2, 0x0, 0x0)
        <GOPATH>/src/github.com/bep/hugo/commands/mod.go:49 +0x83

[larzza]

@bep I got it working with https://github.com/bep/my-modular-site.git

First I happened to run hugo serve with the --gc-flag and got (I know this feature isn't ready so sorry if I'm getting ahead of things...):

Error: failed to prune cache "assets": remove my-modular-site/resources/_gen/assets/sass/style.sass_cf66e63debe6917c04534d6c7b66f623.content: no such file or directory

[larzza]

• I got it working with my custom theme. One of the struggles was to get download of a module (theme) in a private git repo on gitlab to work. But that has nothing to do with this feature.
• Another issue was that I have an import of themes/myhugotheme/style/theme_style.scss from hugo-site/assets/sass/custom_style.scss. For now I hardcoded the path to the module location. I think this makes it even more important to make SCSS/SASS include paths to work with modules (as I suggested in a previous comment).

[bep]

The implementation is very much work in progress.

[larzza]

Yes, I'm aware of that.

[bep]

@larzza @regisphilibert @budparr @RickCogley @digitalcraftsman @onedrawingperday and gang; this branch is now back to a working state and I will wrap it up in a few days. This is a classic example where the first 90% was easy, but the final 10 ... With the introduction of the very flexible disk mounts (now including content) required me to take a step back and redesign parts of the filesystem handling.

The config below is from https://github.com/bep/my-modular-site and shows the syntax.

In short, the goal is a module definition replaces most of the "dir config" in Hugo (configDir, staticDir etc.). I have created adapters to handle the old setup, but this should be much easier to grasp, esp. for bigger projects.

Summary points:

• Every Hugo Component (main project, shortcode collection etc.) can have a module configuration in their config.toml (or similar).
• All of these can define a set of mounts (there are defaults if not sets).
• A mount is:
  
  
  
  • source: Directory path to the source. For the main project, this can be absolute.
  
  
  • target: The mount point in the project. This must start with the component folder, e.g. content/blog.
  
  
  • lang: Language code, only relevant for content mounts and static mounts when in multihost mode.
  
  
  • You can also define mounts when you import a component; if so, any mounts definition in the imported component will be ignored. This can be especially useful when importing non-Hugo-projects (e.g. Bootstrap SCSS) or if you only need a small part of a theme.
  
  

All of the above can be used with components inside the old themes folder or as components managed (version, dependency management) by the new hugo mod commands -- which is a whole another cool story.

The above may look technical, but is in my head super duper cool.

[module]
[[module.mounts]]
source="content"
target="content"
[[module.imports]]
path="github.com/bep/hugotestmods/mymounts"
[[module.imports.mounts]]
source="myassets/subfolder"
target="assets/images"
[[module.imports.mounts]]
source="mydata/subfolder"
target="data/datakey"
[[module.imports]]
path="github.com/bep/hugotestmods/mypartials"
[[module.imports]]
path="github.com/bep/hugo-fresh"