Stack: Pull stack templates from hackage packages

[mgsloan]

In https://github.com/commercialhaskell/stack-templates/issues/55 , https://github.com/lethjakman proposes that we allow hackage packages to come with their templates. This might look something like stack init yesod:basic

Doing just stack init yesod might enumerate the list of available templates, or, if there is just one, default to that template. So, here's how this would work:

1) If there is a colon in the name, treat it as a pkg-name:template-name, telling exactly which package to pull the template from

2) Otherwise, check if the template is available from our templates repo. If it isn't check if it is a hackage package name, download that package, and pull out the list of templates from the source distribution.

3) If the requested package template is missing, or otherwise ambiguous, give the user the list of templates found in the package.

What should the directory convention be in the source distribution? I'm thinking that it should be templates/template-name/.... This way, we can also enumerate the dirs under templates

[mgsloan]

I would also like to propose that this new templating system use a different mechanism than .hsfiles. Instead:

Each template would have a template.yaml file in its directory, with info looking like this:

description: A template for reducing boilerplate!
renames:
    "my-template.cabal": "{name}.cabal"

Why is this better? Because now my text editor can understand these files better! Also, we can have descriptions. We already have a way of having descriptions, but it isn't used by anything yet - https://github.com/commercialhaskell/stack/issues/1571

This would also be a great place for resolving https://github.com/commercialhaskell/stack/issues/1390, by having compiler: ghcjs be valid in the template.

[tolysz]

From my perspective if we could specify relevant bits of "stack.yaml" including resolver, and setup. It would be quite easy to add template for ghcjs:lts6, ghcjs:lts, and possibly ghcjs:nightly (once aeson is fixed, https://github.com/bos/aeson/issues/471)

[mgsloan]

That's a good point, @tolysz ! It may well make sense to allow a subset of stack.yaml be processed by the init system.

[devlinzed]

I think it makes more sense to have the templates be separate packages in the format of stack-template-yesod. This is the approach taken by Leiningen in the Clojure community.

[mgsloan]

Perhaps it does make sense to have them be packages directly. One issue is that we will need some per-template metadata, for the file renaming. Perhaps we just auto-rename the cabal file, and hope that no other files need to be renamed?

[typedrat]

It's not as pretty, but I think that something more like github-username:template-name (where it would look for a repository named stack-templates owned by that user) might be worth considering, so that they're kept separate from the package code and would allow it to be used more generally than just for Hackage packages.

[dpwiz]

github-username:template-name

Please make that at least github:gh-username/template-name. I understand that GitHub is the most massive code warehouse, but it can be unavailable or even undesirable for a plenty of purposes.

[typedrat]

That makes sense. Both specifically due to the issues with GH as well as the way that it will make it easy to add more sources of templates as desired. Could also have a hackage:package/template format for the originally proposed solution.

[lethjakman]

@SASinestro That's a really interesting idea. When I proposed the idea it was intended for packages that were already downloaded/installed...it'd be cool if there were a convention to reference. It'd be even cooler of *: were something that you could add new protocols to as well. For example, you could add a gitlabs: github-enterprise: bitbucket: etc...based off of a package name convention.

For example:

$ stack install stack-protocol-gitlabs
$ stack new gitlabs:lethjakman/yesod:api-application

Network requests would make future tab completion potentially a bit slow though, at least on the first hit. Perhaps without a protocol it defaults to locally downloaded packages?

P.S. For commercial success I think github enterprise would be crucial, I have seen several places require it.

[lethjakman]

@mgsloan I agree completely on the new templating system. Maybe it's a possibility already, but I would really like to see the templates transform into a directory that reflects what a real project would look like, then you would use variables to trigger different paths to be used to set up the project differently. For example, rather than having postgres-yesod postgres-mysql postgres-minimal, you really only have one template with a bunch of flags that switch out key files in the file system. There are several benefits to this:

1. You can reference the project during upgrades/if you mess something up because the template more closely represents your current project, even the file system.
2. Less code to upgrade each time you upgrade Yesod
3. Each "seed" project looks the same.

Example:

$ stack new yesod --database=postgres --auth=email --email=ses

This would also allow more details to be moved out into configuration files which allow for fast failure, and less modification of project files like Foundation.

[kadoban]

Could we unify everything by just allowing packages in general, instead of just from hackage?

What I mean is: stack already has concepts of referring to packages from:

• git URLs of any sort (including github)
• locally from the hard drive
• hackage
• user-specified hackage replacements/mirrors
• as .tar.gz locally and over the internet I think?

Those encompass every place you can currently get a template from, and more. What if we just reuse the normal package resolution process used when stack needs to find a package to use as a dependency, that is:
1) Is it specified as an extra-dep?
2) Is it in the resolver?
3) etc.

That would allow a huge amount of flexibility and configurability without adding much cognitive load, since stackers already need to understand most all of how that works (where stack gets a package from and how you override it).

And then we could dump a lot of the specialized stuff we currently use for 'stack new' without losing any abilities.

For stack template to work, we could then just have a configurable list of package names, and use the normal resolution process to fetch each one when they're needed.

To be clear, what I am proposing is:

• The [TEMPLATE_NAME] that stack new accepts now only accepts a package name (no local files, no URLs)
• To override templates (or to allow ones not in the resolver), you'd specify in an extra-dep/package in ~/.stack/global-project/stack.yaml where to find it, which can be a local directory, git repo, hackage, other package index, etc.

[ocramz]

Seeing as the latest reference to this issue is from last summer, I figured I'd revive it.

I have a few questions:

• Is this redesign still being considered?
• Will the .hsfiles format be phased out, and if so, when?

I think stack templates are a valuable tool for beginners so I've been considering to build a few domain-specific ones; my only doubt is on how much to invest on it if the format is going to change.

Thanks!

[bitemyapp]

I think stack new supports pulling templates from URLs now.

TEMPLATE_NAME            Name of a template or a local template in a file or a
                           URL. For example: foo or foo.hsfiles or ~/foo or
                           https://example.com/foo.hsfiles

[Tehnix]

With stack new supporting URLs, would it make sense if we turned stack-templates into a repository that has a list of names pointing to URLs?

These would then typically point to a .hsfiles maintained in the project it belongs to (like yesod in the yesod repositories etc), making it easier for maintainers to update them, while keeping it easy for users to find them via their short names.

[dbaynard]

Related: #4103 #4168

stack new now allows template names of the form username/foo to
download from a user other than commercialstack on Github, and can be prefixed
with the service github:, gitlab:, or bitbucket:.

[silky]

hmm, even though the --help of stack new _says_ it allows for local filenames, when i get it one, it fails:

09:42 AM noon ∈ Double-Slit (master*) ♫ stack new Double-Slit --bare ../stack-template/fashion.hsfiles
Expected a template like: foo or foo.hsfiles or https://example.com/foo.hsfiles or github:user/foo

Usage: stack new PACKAGE_NAME [--bare] [TEMPLATE_NAME] [-p|--param KEY:VALUE]
                 [DIR] [--solver] [--omit-packages] [--force] [--ignore-subdirs]
                 [--help]
  Create a new project from a template. Run `stack templates' to see available
  templates. Note: you can also specify a local file or a remote URL as a
  template.

09:42 AM noon ∈ Double-Slit (master*) ♫ stack --version
Version 1.8.0, Git revision 94d302c40d3d08dcaa0213f3f1ffdc1384842714 x86_64 hpack-0.29.0

[silky]

ah; turns out it's just that it can't do it from a relative directory!

crazy.

running with the full path works.

[dbaynard]

The help text should be accurate in the new release, at least.

[mihaimaruseac]

I think that the fact that it doesn't support relative paths is a regression. Afaik it used to work a few years ago.

[Sir4ur0n]

Hi,

Stack templates repo README mentions that templates are no longer maintained because the way Stack handles templates will change in this issue. But I see no update for a few months now.
Moreover, the current templates do not work, some stuff like {name}.cabal are not correctly replaced, resulting in invalid cabal files, etc.

I think this is a major setback for new Haskell users. When learning a new language, scaffolding is a nice way to start (e.g. with tests, for a ReST API, database access, etc.) and quickly prototype something. But the templates have been broken for some time, and we see no move here.

What's the status please? It looks like this issue was flagged for 1.9.1 release but 1.9.1 is released and this issue has not been updated... I'm getting worried this gets forgotten 😢

[bradrn]

The changelog for the recent 1.9.1 release says that #4039 was implemented, allowing 'namespaced templates' like github:username/template, which allows templates to be found on GitHub. It's not quite pulling from Hackage, but it's a step in this direction.

[dbaynard]

@Sir4ur0n {name}.cabal is in #4381 which will be part of the next minor release. The relevance to 1.9.1 was an issue with the help text — that said, it doesn't look like it has been fixed.

[m-renaud]

Hey, first off, thanks everyone who's worked on this in the past, I've found templates are super helpful to get up and running on a new project :)

One thing I was wondering was if this project initialization code could be exposed as a library. Looking through examples there doesn't seem to be anything Stack specific, so if it was possible to expose the functionality so that other tools could use it that would be awesome. There's several tools that support project initialization (stack, cabal, summoner, probably more), and if they could all use the same underlying mechanism that would be great and give users a more consistent experience, regardless of what their preferred build tool is.

[snoyberg]

We ended up in a different direction with namespaced templates in #4039, closing.