Berry: Implement `yarn pack` and `yarn publish`

[arcanis]

Describe the user story

I'm a package author, and I'd like to publish a package on the npm registry.

Describe the solution you'd like

This is a perfect example for a plugin. The best solution would be to implement yarn pack and yarn publish into their own plugins - this would allow us to iterate much faster on this, for example by adding validation checks on the submitted packages (one could imagine a check to make sure that licenses are correct for example).

Describe the drawbacks of your solution

This is a rare case of a feature with no drawback. Our users simply expect to have yarn publish, so there will be no surprise.

Describe alternatives you've considered

Similarly, not many alternatives available.

[LaRuaNa]

Hey @arcanis tried to give that one a go. Followed this instruction but unfortunately couldn't get nothing else than bunch of errors. Is that document up to date?

Seems like the regex here doesn't match 'plugin-hello' at all. see -> https://regex101.com/r/hT1J2J/2

Do you have a hint for me?

❯ yarn @berry-build-plugin Error: Invalid plugin name "plugin-hello" at getNormalizedName (/Users/laruana/Workspace/forks/berry/packages/berry-builder/sources/build-plugin.js:24:11) at Command.concierge.command.action [as run] (/Users/laruana/Workspace/forks/berry/packages/berry-builder/sources/build-plugin.js:33:24) at /Users/laruana/Workspace/forks/berry/.yarn/cache/@manaflair-concierge-npm-0.10.2-725b73770737cec6aeabe85ac343c02757c83a6fdf4dc321680d9346f31bba17.zip/node_modules/@manaflair/concierge/core/Concierge.js:1207:48 at _loop (/Users/laruana/Workspace/forks/berry/.yarn/cache/@manaflair-concierge-npm-0.10.2-725b73770737cec6aeabe85ac343c02757c83a6fdf4dc321680d9346f31bba17.zip/node_modules/@manaflair/concierge/core/Concierge.js:69:39) at runMaybePromises (/Users/laruana/Workspace/forks/berry/.yarn/cache/@manaflair-concierge-npm-0.10.2-725b73770737cec6aeabe85ac343c02757c83a6fdf4dc321680d9346f31bba17.zip/node_modules/@manaflair/concierge/core/Concierge.js:80:9) at Concierge.run (/Users/laruana/Workspace/forks/berry/.yarn/cache/@manaflair-concierge-npm-0.10.2-725b73770737cec6aeabe85ac343c02757c83a6fdf4dc321680d9346f31bba17.zip/node_modules/@manaflair/concierge/core/Concierge.js:1202:34) at Concierge.runExit (/Users/laruana/Workspace/forks/berry/.yarn/cache/@manaflair-concierge-npm-0.10.2-725b73770737cec6aeabe85ac343c02757c83a6fdf4dc321680d9346f31bba17.zip/node_modules/@manaflair/concierge/core/Concierge.js:1262:41) at Object.<anonymous> (/Users/laruana/Workspace/forks/berry/packages/berry-builder/sources/build-plugin.js:107:4) at Module._compile (internal/modules/cjs/loader.js:702:30) at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10) Error: Command failed: /usr/local/Cellar/node/10.5.0_1/bin/node /Users/laruana/Workspace/forks/berry/packages/berry-builder/sources/build-plugin.js null null at makeError (/Users/laruana/Workspace/forks/berry/packages/berry-cli/bin/berry.js:66632:9) at Promise.all.then.arr (/Users/laruana/Workspace/forks/berry/packages/berry-cli/bin/berry.js:66736:16) at process._tickCallback (internal/process/next_tick.js:68:7)

[arcanis]

At the moment plugins should be named either @berry/plugin-something or berry-plugin-something (in the second case any scope will be fine). I forgot to add a settings in the project yarnrc to define the scope when running yarn init, so the name is likely wrong! Try to rename your new plugin to berry-plugin-hello.

The error message should also make this much clearer - ideally by throwing a UsageError when running with an invalid package name 🤔

[LaRuaNa]

Hmm. Somehow didn't help much. Get the following error with both options.

ERROR in Entry module not found: Error: Can't resolve '.' in '/Users/laruana/Workspace/forks/berry/packages/berry-plugin-hello'
Error: Command failed: /usr/local/Cellar/node/10.5.0_1/bin/node /Users/laruana/Workspace/forks/berry/packages/berry-builder/sources/build-plugin.js
null
null
    at makeError (/Users/laruana/Workspace/forks/berry/packages/berry-cli/bin/berry.js:66632:9)
    at Promise.all.then.arr (/Users/laruana/Workspace/forks/berry/packages/berry-cli/bin/berry.js:66736:16)
    at process._tickCallback (internal/process/next_tick.js:68:7)

[arcanis]

Two things:

• Can you pull? I've just noticed a bug in how plugins were loaded, should be fixed now
• You either need to rename your sources/index.ts into index.ts, or add a main field in your package.json file to tell the bundler where is your entry file located in. I've made this clearer in the documentation! 😃

[LaRuaNa]

Lovely! Thanks :]

[LaRuaNa]

I played a bit around on plugins to get a better understanding of the plugin system. But honestly, I'm a bit lost in the code base and not sure if my approach is the best. Currently, I determine the manifest root via Configuration.find and Project.find. And that’s more or less all I could get. Should I archive files via tgzUtils.makeArchive? What about writing them on fs is there already some logic or should I handle streams manually? Ignoring dot files? Is there more docs than in /docs? Need a bit your introduction :]

[arcanis]

The pack command is relatively decoupled from the rest of Yarn in that it doesn't really need to play a lot with the dependency graph - the main thing needed are:

• Run the right scripts within the workspace that is getting packed (cf yarn run - you should just have to use scriptUtils.executeWorkspaceScript to do this)
• Filter the list of files that will need to be packed (that will require to read the .gitignore / .npmignore from the workspace being built, plus the files field from the manifest)
  
  
  
  • The Manifest class doesn't yet hydrate the files field, so it has to be added in Manifest.ts
  
  
• Generate a tgz archive from this file list. Somewhat ironically the utilities in tgzUtils actually generate zip archives, so there's some work that would be needed here.

I think the best is to have an iterative approach - rather than implementing everything at once, the task should be split into smaller sub-pieces:

1. A PR that adds a @berry/plugin-pack plugin with yarn pack (which doesn't do anything yet)
2. A PR that modify the Manifest class to add support for the files entry.
3. A PR that adds function in the plugin-publish plugin that, given a workspace, returns the list of files that should be packed if yarn pack was actually implemented. This function can be exposed through the API via yarn pack -n,--dry-run.
4. A PR that adds a bunch of tests in our acceptance testsuite
   
   
   
   • "it should list the files from the current workspace directory"
   
   
   • "it shouldn't list the files from other directories"
   
   
   • "it should recurse into the nested directories"
   
   
   • "it shouldn't include the files from the .gitignore"
   
   
   • "it shouldn't include the files from the .npmignore"
   
   
   • "it should ignore the .gitignore if there's also a .npmignore"
   
   
   • "it shouldn't include the files listed in the files field"
   
   
   • "it shouldn't ever include yarn.lock"
   
   
5. A PR that creates a .tgz archive from the list of files we've generated in the previous step
   
   
   
   • I'd recommend to rename tgzUtils.ts into archiveUtils.ts, to rename the functions it contains to add "zip" somewhere (for example makeZipArchiveFromDirectory), and to add a new makeTgzFromDirectory function.
   
   

This way we'll be able to iterate on each one of those different sub-steps and discuss their design independently - and you'll have less pressure of doing a "all or nothing" feature.

And once we're done with this, we'll just do a similar plan to figure out how to implement the yarn publish command 😃 How does that sound?

Also feel free to pop on Yarn's Discord channel to ask implementation questions 👍

[olingern]

@arcanis Sorry to ghost on the yarn publish. I have something basic together right now ... okay if I claim this?

[LaRuaNa]

Hey @olingern since I already started working my way into that, I wish to keep that one. If it's okay for you :]

[olingern]

Sure, I was late getting to opening something meaningful up ... so all yours :slightly_smiling_face:

[arcanis]

Hey @olingern - copying it here since I'm not sure you visit Discord everyday 😃

@olingern @LaRuaNa hey sorry about the duplicated work :open_mouth: I'm pretty sure we can figure out a way to parallelize things though, don't you think?

@olingern can you submit a PR? Depending how advanced it is we'll figure out the next steps
Don't worry, there is no shortage of things to do :stuck_out_tongue:

[arcanis]

Implemented via #115 and #124 🙂