Chapel: Mason Improvements

[Spartee]

This issue is to gather all of the ideas to improve chapel's package manager Mason. If there is something you would like to add please comment it below.

Commands to add

• [x] add (add dependency) (#10258)
• [x] rm (remove dependency) (#10258)
• [x] init (create project in existing directory) (#14820)
• [x] publish (upload toml to mason-registry) (#13198)
• [ ] bench (run benchmarks)
• [x] test (run tests) (#8714)
• [x] examples (run examples) (#10498)

Features (general, others listed by command)

• [ ] Use of other git platforms(bitbucket, gitlab, etc..)
• [x] Allow for commands within project sub-directories
• [x] More descriptive error messages (#8868)
• [ ] Developer dependencies
• [x] exit with an error code (ideally by returning from main) when the command failed - this is important for integration with scripts and Makefiles
• [x] Bash completion for mason commands
• [x] Environment variables that change the default value of certain mode-like flags
  
  
  
  • e.g. Someone without internet access might want to set MASON_NO_UPDATE instead of always throwing the flag --no-update.
  
  

Mason New

• [ ] mason packages can have files with same name #8470
• [x] Support mason new --moduleName flag (https://github.com/chapel-lang/chapel/issues/14955)

Mason Update

• [ ] Resolution strategies for major versions
• [ ] Lazy versioning (e.g. 2.3.x)
• [ ] give a reasonable error message when a requested tag version is missing from the source

Mason Build

• [x] Build on last modified (#9843)
• [ ] Build Chapel programs by specifying Chapel files rather than directories using -M
• [x] support mason build --example <example_name> (#10498)
• [ ] support mason build --test <test_name>
• [x] pass --no-update-registry flag for offline use.

Mason Run

• [ ] Fix ambiguous behavior listed in #8852
• [ ] implement -- for runtime arguments (#10622)
• [x] support mason run --example <example_name> (#10498)
• [ ] support mason run --test <test_name>

Mason Modify(add/rm)

• [ ] Remove packages by version
• [ ] Check for package existence upon user calling mason add <package>

Mason External (Spack integration)

• [x] External dependencies (#10289)
  
  
  
  • [x] C Dependencies: at least allow .c files and .h files to be included in the package
  
  
  • [x] Allow for users to install and uninstall external dependencies through separate backend package manager. (#10488)
  
  

Mason System (pkg-config integration)

• [ ] Install Pkg-config if old version or doesn't exist
• [ ] Make search for .pc file recursive

Mason Env

• [x] show new mason registry directory when creating it and/or allow querying its path (mason env)

Mason Test

• [x] Remove user input required for mason test.
• [ ] Compile tests in parallel (Issues with cygwin)
• [ ] Specify tests and configurations in [test] table
• [ ] Allow for -- syntax for compile/runtime arguments
• [ ] Allow tests to ignore compopts for src files
• [ ] Allow excluding tests like .notest files from start_test. (#13786)

Mason Doc

• [ ] Tests for mason doc

Mason Search

• [ ] Fix bug in search behavior when encountering bad file (#8770)

Mason Example

• [ ] Allow for examples to ignore compopts for src files.
• [ ] Allow for -- syntax for compile/runtime arguments

Mason Help

• [ ] Add examples for each of the commands
  
  
  
  • e.g. mason external search openblas
  
  
• [ ] Help commands outside project directory (#10640)

Mason-Registry

• [ ] Support package encryption
• [ ] More test directories with more versions
• [ ] Reduce barriers to publishing a package
  
  
  
  • e.g. explore moving away from hosting registry on GitHub, which requires PR model
  
  
• [ ] Cache packages on a centralized host
• [ ] Blessed vs. open packages
• [ ] Checksumming
• [ ] Binary packages(applications)
• [ ] CI testing (#9846)

Documentation

• [ ] show commonly encountered problems and fixes (#8712)
• [ ] describe how to tag, how to push a tag to GitHub (git push --tags)

Tests

• [ ] More multi-dependency testing
• [ ] Major version conflict testing (IVRS) (#9844)
• [ ] Offline testing
• [x] valid/invalid package names

Build

• [ ] anonymize directories listed in "mason --about"

Release

• [ ] Separate mason into its own repository for independent releases

Interaction with TOML library (See #7104)

• [ ] Use of array of tables for dependencies once that functionality is put in

Extras

• [ ] [Loading bars!](https://gist.github.com/ben-albrecht/33978abe6449e7b8089322a003cbd98c)
• [ ] [Colors!](https://gist.github.com/ben-albrecht/d876460d8dacdaa16e3d336036d80022)
• [ ] View directory function (draws diagram of mason project outline in terminal)

Miscellaneous

• [ ] Do not use soon-to-be-deprecated 'msg' field of Error

[Spartee]

In reference to one of the features listed in this issue, there has been discussion of how mason will handle multiple versions of chapel. So far, the idea that is the most favorable amoung team members who have discussed this is to add a field to that project mason.toml that indicates chapel version.

[brick]
name = "MyPackage"
version = "0.1.0"
chplVersion = "1.16.0"

We have also proposed the idea of using chapel range syntax to indicate multiple compatible versions as follows

[brick]
name = "MyPackage"
version = "0.1.0"
chplVersion = "1.13.0..1.16.0"

We are welcoming all ideas and thoughts on this topic.

[ben-albrecht]

Extend mason doc to document dependencies

I'm wondering if this is really that useful of a feature to support..

[benharsh]

There should be an item for building mason with an appropriate chplenv. For example, CHPL_COMM should be set to 'none' so that users don't need to run mason -nl 1 to do something. CHPL_LAUNCHER should also be set to 'none' because mason shouldn't be running on an actual node. We ought to verify that the module build supports such a configuration.

Edit: we probably don't want to build with hugepages either, but let's try the others first.

[nspark]

I've only tried Mason a bit, but here are some comments on / suggestions for new features:

1. Build on last modified

   Total 👍 Mason should be able to replace a basic Makefile w.r.t. rebuilding on changes.

2. Related to (1), I think that mason run --build should not run the executable if the build fails.

3. Currently, Mason discovers the name of the executable based on the project directory. I was surprised it did not use the name of the brick itself. I discovered this when I created a project and renamed the directory; mason run wouldn't run the proper binary it would build, but also wouldn't report an error that the subprocess spawned didn't properly execute.

4. From (3), if mason run fails to run the executable (for some reason), it should report an error.

[buddha314]

To help with the discussion around #8771 I thought I would point to how we are using Make and Makefiles instead of mason build right now. Here is one from Numsuch and one from Relch You can see we have some boilerplate. Our internal projects are a bit more complicated, but face some of the same issues.

[ben-albrecht]

I've added:

Allow excluding tests like .notest files from start_test.

in response to a user inquiry on the mailing list (@tjstavenger-pnnl)

[tjstavenger-pnnl]

I have an email on the developer mailing list, but it looks like "exit with an error code (ideally by returning from main) when the command failed - this is important for integration with scripts and Makefiles" doesn't happen in 1.19.0 on mason test (I'm assuming that the checkmark means it is done?).

$mason test
...
--- Summary:  23 tests run ---
-----> 15 Passed
-----> 8 Failed
$ echo $?
0

Also, it would be nice to see the working directory used on mason test standardized too. You get the working directory of wherever you execute mason from, so if you do a mason test from the root of your project, then that is your working directory, but it is different if you run mason test from your tests directory. If your tests use relative paths, they can break.

[ankingcodes]

Move registry off of github (centralized registry with cached packages)

I think having the registry on github is good because we can enable CI tests and would make it easier for package moderators to assess a package. However, I did think that Mason should have its own website, where the data regarding the packages can be displayed using the Github API. Also there are other ways to accomplish this.
A website could have functionalities like :

• A list of most popular packages on the main page.
• Nav bar which contains tabs for : Documentation(regarding Mason cli & the registry) , search bar to search for packages, option for creating account (if the registry is taken off of github), forum for discussion and community members.
• If a user searches and opens a link to a package, he is redirected to a new page showing contents of a package such as, Name of the package, information of the Mason.toml file parsed and showed in a systematic way, number of downloads, date of last update, also the content of README file of that project can be displayed.
• Also, all these would allow us to incorporate a ranking system in the future for ranking relevant packages on the basis of some metrics.

@ben-albrecht What are your views on this ?

[ben-albrecht]

Move registry off of github (centralized registry with cached packages)

I think having the registry on github is good because we can enable CI tests and would make it easier for package moderators to assess a package.

I think the real goal is to centralize the packages, such that a package author cannot break / remove an existing working version, breaking anyone's code who relies on it, ultimately avoiding problems like the npm left-pad incident. I think we can consider this a separate feature request from where the registry is maintained. I will update the list in the description to reflect that.

The goal to move away from GitHub eventually is to reduce the burden for users to publish packages. That said, this feature request is still an open discussion. Maybe the GitHub-hosted main registry model will work well for us in the long term.

I did think that Mason should have its own website, where the data regarding the packages can be displayed using the Github API.

Agreed. I think your proposes features sound like a good first step. I encourage you to split off a new issue for us to explore the specifics further.