Marko: Allow alternate directory for compiled templates

[patrick-steele-idem]

Currently, compiled templates are written next to the original file. It might be nice to offer the ability to place compiled templates in an alternate root directory. For example:

• my-app/.marko-compiled/src/components/foo/template.marko.js
• my-app/node_modules/some-installed-package/.marko-compiled/src/components/bar/template.marko.js

[adammcarth]

I think a folder within node_modules is a very sound place to compile templates to _by default_. Reference the full path to a compiled file if an error occurs, and the person can easily navigate their way to the destination for further inspection.

I know it's awesome and cool to see the Javascript that is compiled from .marko syntax files, but I stand by my opinion that they serve quite literally no real purpose besides debugging and, for a lack of a better term, pure self-interest in how Marko works. In either case, at no point should these files need to be visible in a project. You don't edit or make changes to compiled Marko files, and they shouldn't ever need to be included in version control.

The node_modules folder ticks as a solution to all these boxes. PLUS the fact it's already checked out of version control in 99% of projects means compiled files are hidden straight away, instead of realising down the track and needing to add line(s) to your .gitignore file.

[patrick-steele-idem]

@adammcarth I don't think putting compiled templates directly under the node_modules is the best choice since it I think it is an abuse of that folder and would not be expected by end users. I think putting compiled templates under a .marko-compiled directory at the root directory of the package associated with the template is the right thing to do. Yes, you would have to make sure that directory is included in .gitignore but I think that is okay.

NOTE: I adjusted the original text because there was a typo.

As a side note, one of the potential benefits of compiling a template and keeping it right next to the original file is that as a package prepublish step you can pre-compile all Marko templates, delete the original Marko template files (temporarily) and all of the requires would work and return the precompiled template. That is, if you you have a require('./template.marko') call in your code, Node.js will first look for template.marko and not find it and then it will look for template.marko.js and find that. That way, you can use the Node.js require extension for Marko during development, but when you publish your package you would only publish the pre-compiled templates (not the original Marko source files) and it would not be required for a user of the package to install Marko's Node.js require extension.

Thoughts?

[adammcarth]

Hey @patrick-steele-idem! Sorry I missed your reply to this :(. Better now than never though, I guess.

Hmmmm. That's a tricky one. In regards to your second use case, I still stand by the fact that even if you did decide to delete your original Marko templates as some kind of release process - you'd probably be doing more harm than good. Unless you have some incredibly unique situation where storage space is an issue, what would deleting the originals and keeping the compiled achieve? You'd have compiled Marko templates that would serve literally no purpose to anyone except the process workers, right? I guess if it was part of some kind of testing environment I could see the use for that though. That kind of makes sense, but I don't see any real advantage of that over the files being in a seperate directory.

As for not using the node_modules folder - I think that's a fair enough reservation to have. Something like .mark-compiled would work well, if you were going to go down the whole, _compiling-to-a-different-directory_ path.

Let me know what you decide on anyway. Glad to see v3 is gradually getting rolled out, keep up the good work mate :+1:

[jsumners]

It would be very nice to be able to supply an alternate directory. It could be the case that the user running the application doesn't have write permissions to node_modules, but does have write permissions elsewhere.

[patrick-steele-idem]

The current thinking is that we can make this an option by introducing a new configuration option in marko.json similar to the following:

{
  "compiled-output-dir": "./.marko-compiled",
   ...
}

After updating the compiler for Marko v3 and working on this task it became clear that we won't actually be able to directly load the template from the alternate directory. We would merely be writing it disk at the alternate location for reference purposes (and possibly to avoid recompiling in production). The stack trace would still need to match up where the original .marko file is on disk in order for any require() calls to work as expected.

It's also a little unclear if this setting would be applied globally. Currently, we only search up to the root of the package to find a package.json file so installed packages would currently not share the option if it comes from a marko.json in the root package. We might need to consider other options if we want the configuration option to also apply to all installed packages.

[albertogasparin]

+1 for .marko-compiled (or .marko-cache) on the root, in a similar way as Sass does.

[mlrawlings]

This is not on our roadmap, but we're not opposed to it if someone from the community would like to tackle it. If you want to take a stab, let us know and we'll help if necessary.

As far as directory naming, I think a sensible directory would be .cache/marko since Lasso and other modules already use .cache/ to store cached files so this way we're not cluttering the top level directory.

[Hesulan]

The compiled template location is handled on lines 38 and 92 of node-require.js: . Right now they just append '.js' to the end of the source template's filename. I've hacked on my local copy a bit and added new compiler options for defining the source and cache directories, and it works perfectly under ideal circumstances.

The problems I'm looking at now are ensuring that the requested template is actually _below_ the given source directory (which I suppose could be handled by throwing an error if path.relative(<src-dir>, <filename>).startsWith('..')) and that the target cache directory exists before trying to write to it (ideally without too much extra code or pulling in another dependency like mkdirp). I'll try to poke at this a bit more tomorrow if I get the chance, but I'm running out of time for today.

EDIT:
The reason for worrying about the path relative to the source directory is that otherwise it just dumps everything in the same place, which is fine until you start having name collisions. The only other option I can think of would be naming the files by some kind of unique hash, but that would make debugging a nightmare.

[Hesulan]

It seems this isn't quite as simple as I'd thought. After posting I noticed the comment above about it only storing in the alternate location for reference purposes, and it seems that is what's happening. It also doesn't seem to affect layouts. I'll still try to poke at this a bit when I get the chance, but disregard my previous post as premature optimism.

[Hesulan]

I did some more digging and found that node-require.js isn't the only place that is hard-coded to simply append '.js' to the source filename. I'm seeing 3 more such lines in runtime/loader/index.js. Fortunately the line always seems to be basically the same thing, var targetFile = templatePath + '.js';, sometimes with different variable names. I'll keep poking at this whenever I get the chance.

In the meantime, regardless of the solution, we'll still need to ensure that the target directory exists. My first idea was to use the mkdirp module, but that would mean adding another dependency. Any thoughts or suggestions?

[Hesulan]

I tested every scenario I could think of, and it _appears_ to work perfectly except for two things:
1.) I got a "Cannot find module" error when trying to use a component, though that's _probably_ the result of my being lazy and hard-coding some of it just to see if this was even viable.
2.) I _think_ with the current setup, if the source template has been deleted, moved, or renamed, it falls back to require()'s built-in behavior of attempting to append '.js' to the end of the filename. This is probably an easy fix.

To anyone who would like to put my research into practice:

Step 1: Edit lines 38 and 92 of node-require.js, and lines 55, 82, and 92 of runtime/loader/index.js(marko v3.10.1); They should all look something like var targetFile = templatePath + '.js';. You'll want to modify them to point to wherever you want the compiled templates to be cached (+1 to the previous suggestion of using .cache/marko by default). You could probably implement a new compiler option for users to specify the source and cache directory paths, both modules already have access to markoCompiler.defaultOptions.

Step 2: Ensure valid file paths; All sources should be below src/, all compiled templates should be below .cache/marko/, and the target directory should exist before attempting to write to it - for example, .cache/marko/layouts/main/ must exist before attempting to write .cache/marko/layouts/main/layout.marko.js - ideally without adding too much extra code or another dependency.

Step 3: Make sure components work correctly. The error I saw may just have been a fluke, or it may be a quirk in how components are loaded as opposed to layouts (which did work correctly for me).

Step 4 (optional): Modify the require() extension to check for an existing compiled version of the requested template even if it can't find the source file.

I apologize for the length and number of my posts. I'm not sure if or when I'll get around to forking or initiating a pull request, so I want to make sure that if anyone wants to finish what I've started they'll have as much information as possible.

[BigFax]

Is it on the roadmap now or in the future ? Would be great to be able to define our own directory for compiled templates.