Chapel: Enable Local Mason repositories.

[buddha314]

With maven we can have local maven repositories, thereby having access to all the maveny goodness. It would be nice to have local Mason repos so we don't have to release our code to the whole world.

Mason should support local repos! Shall I say it again?

[buddha314]

PyPi Precedent: https://gist.github.com/Jaza/fcea493dd0ba6ebf09d3

[SpireFlux]

This could also support 'offline' use cases, where people are not connected to the internet.

[bradcray]

I believe that this has been part of Mason's design goals from the start, and was waiting for someone working more closely to Mason to comment here, but apparently that hasn't happened. @ben-albrecht? @Spartee? @benharsh?

[Spartee]

This has been in the design goals for Mason, but there are some other features that I think are more important see #7106. I will add this to the feature list on #7106 though as it looks like it has been left off.

[daviditen]

Here's a proposal for how mason can be extended to handle local registries and multiple registries.

A local registry mirrors the format of the main Mason registry, but is entirely local and describes local/private Projects. It should be a local git repository created with 'git init' that has registry toml files 'git add' / 'git commit'ted to it.

The environment variable MASON_REGISTRY can be set to a comma separated list of 'name:location' pairs where 'name' is the local name for a registry, and 'location' is a path or url pointing at a registry. The registries will be searched for dependencies in the order they appear in the MASON_REGISTRY variable. If MASON_REGISTRY is unset it defaults to "registry:https://github.com/chapel-lang/mason-registry", which is the current, unchanged behavior.

To use a local registry located at "/path/to/local/registry" and fall back to the one at "github.com/chapel-lang/mason-registry" for dependencies not in the local registry, the MASON_REGISTRY variable can be set to:

export MASON_REGISTRY="local_registry:/path/to/local/registry,registry:https://github.com/chapel-lang/mason-registry"

If MASON_CACHED_REGISTRY is set, it is a comma separated list of directories and must have the same number of elements as MASON_REGISTRY.

Mason will clone the local registry in the same way it does for the main registry, to "$MASON_HOME/name". The toml files follow the same format as in the main registry except the "source" field will be a local filesystem path instead of a url. It could potentially also be a remote path on some centralized machine, e.g. "user@host:/path/to/repository".

When a local project is used it is cached in "$MASON_HOME/src", in the same way the main registry caches itself.

[ben-albrecht]

@daviditen - Thanks for putting together this proposal. Could you elaborate more on what MASON_CACHED_REGISTRY achieves? I would assume this could be dropped as part of this change.

[ben-albrecht]

Separately, could the MASON_REGISTRY keys be inferred from the repository name if not provided?

e.g.

# These are all the same

export MASON_REGISTRY=\
"/path/to/local/registry,https://github.com/chapel-lang/mason-registry"

export MASON_REGISTRY=\
"/path/to/local/registry,mason-registry:https://github.com/chapel-lang/mason-registry"

export MASON_REGISTRY=\
"registry:/path/to/local/registry,mason-registry:https://github.com/chapel-lang/mason-registry"

I imagine users will only need to specify the key when a collision takes place (which we could catch and throw an error for).

[daviditen]

Could you elaborate more on what MASON_CACHED_REGISTRY achieves?

MASON_CACHED_REGISTRY allows you to specify the directory where each registry is cloned to in case you want them somewhere other than $MASON_HOME/registry_name. I'd be fine with just removing this functionality.

Separately, could the MASON_REGISTRY keys be inferred from the repository name if not provided?

I think we'd have to change the separator to something other than ':' in name:location if the name is optional, but otherwise that seems reasonable.

[ben-albrecht]

I'd be fine with just removing this functionality.

Same here. I'm not sure how valuable that functionality is. What do you think @benharsh?

I think we'd have to change the separator to something other than ':' in name:location if the name is optional, but otherwise that seems reasonable.

Ah, the URLs would make that tricky, unless we special-cased checking for http*://. What would be a safe delimiter? blank space?

[benharsh]

I'd be fine with just removing this functionality.

Same here. I'm not sure how valuable that functionality is. What do you think @benharsh?

Sounds good to me!

[daviditen]

Ah, the URLs would make that tricky, unless we special-cased checking for http*://. What would be a safe delimiter? blank space?

How about commas between pairs and vertical bars between name|location? Commas are technically allowed in both URLs and file paths, but I think they are rare. Including a vertical bar in a file path would be pretty strange, though legal. Spaces are more common in file paths even if most *nix users tend to avoid them.

# These are all the same

export MASON_REGISTRY=\
"/path/to/local/registry,https://github.com/chapel-lang/mason-registry"

export MASON_REGISTRY=\
"/path/to/local/registry,mason-registry|https://github.com/chapel-lang/mason-registry"

export MASON_REGISTRY=\
"registry|/path/to/local/registry,mason-registry|https://github.com/chapel-lang/mason-registry"

[daviditen]

Support for multiple registries including local registries is merged into mason now. It uses MASON_REGISTRY as described above.

This process can surely be improved upon, but here are the steps I followed to test a local registry:

Example:

To create a local registry starting with Bricks for ProjectA and ProjectB which were created with 'mason new ProjectA' and 'mason new ProjectB', and are located at "/path/to/my/projects/Project[AB]":

ProjectA and ProjectB are the local projects your local registry will point to.
The libraries must have their files committed and have a version tag added:

cd /path/to/my/projects
mason new ProjectA
cd ProjectA
git add Mason.toml src/ProjectA.chpl
git commit
git tag -a v0.1.0 -m "Tag version 0.1.0"
cd ..
mason new ProjectB
cd ProjectB
git add Mason.toml src/ProjectB.chpl
git commit
git tag -a v0.1.0 -m "Tag version 0.1.0"

Create local registry

mkdir /path/to/local/registry
cd /path/to/local/registry
mkdir -p Bricks/ProjectA Bricks/ProjectB
cp /path/to/my/projects/ProjectA/Mason.toml Bricks/ProjectA/0.1.0.toml
cp /path/to/my/projects/ProjectB/Mason.toml Bricks/ProjectB/0.1.0.toml

<edit Bricks/ProjectA/0.1.0.toml to add>
source = "/path/to/my/projects/ProjectA"

<edit Bricks/ProjectB/0.1.0.toml to add>
source = "/path/to/my/projects/ProjectB"

git init
git add Bricks/ProjectA/0.1.0.toml Bricks/ProjectB/0.1.0.toml
git commit

Set MASON_REGISTRY to point at both the local registry and the default registry:

export MASON_REGISTRY="local-registry|/path/to/local/registry,mason-registry|https://github.com/chapel-lang/mason-registry"

ProjectA and ProjectB can now be listed as dependencies for a new ProjectC.

[ben-albrecht]

@daviditen - we ought to get that content into https://chapel-lang.org/docs/master/tools/mason/mason.html

[buddha314]

Yeah David, why didn't you think of that like a minute ago on Gitter?

[daviditen]

Thanks for catching that @ben-albrecht. I edited it to use | instead of :.

[bradcray]

@daviditen - we ought to get that content into https://chapel-lang.org/docs/master/tools/mason/mason.html

Yeah David, why didn't you think of that like a minute ago on Gitter?

You guys are slow... I mentioned this to David a whole 2 days ago... :D

[buddha314]

Pics or it didn't happen...

[bradcray]

Straight from the mail client of champions: :D

[buddha314]

SNAP! There it is ladies and gentlemen.

Does it strike the rest of you as odd that somebody put this man in charge of a computer department?