Apm: Use release archives instead of release tags

[steelbrain]

The way releases are handled at the moment are great, but it has a lot of issues and problems

The problems

1. We can not have a .apmignore file in the repo
2. If we want to put transpiled/compiled files into the package, we have to commit them into the repo and upgrade them on each change
3. We can not support mono repo architectures with this

The solution
Create tar archive (or zip if you prefer) respecting ignore files and counting in files ignored by git repo from the the local version of the publisher. Add a release with the newly created tag and attach that archive with that release.

The release tag should be prefixed by package name, like linter-ruby-v1.1. The archive name attached to the release should be named package-name.tgz or package-name.zip.

When installing a package, check for the latest tag in APM database for that package, if the tag starts with package-name, try to find release for that tag and download the package file. Otherwise, just download the zip archive of commit as we do now. If we follow this pattern, we would be backward-compatible with the already released packages.

I know the problems I mentioned above do not sound very important, but they are very annoying and very difficult to workaround.

For example @basarat, @blakeembrey and the rest of the atom-typescript team transpiles the typescript files and keeps them in their repo.

@nmote @ssorallen @bolinfest and the rest of the nuclide team had to create a new github org facebooknuclideapm and create a repo in it for each package they wanted to publish.

@david-driscoll and the rest of the omnisharp-atom team also has to transpile typescript files and keep them in the repo.

@devoncarew and the rest of dart-lang team has to keep transpiled dart files in their repo.

@steelbrain @Arcanemagus and the rest of linter-eslint team has to commit transpiled babel files into the repo, because we spawn a child process that requires our files and child processes don't support babel requires

[thedaniel]

This is good feedback - thank you. These are definitely edge cases, as the current deployment strategy works for the majority of the thousands of packages on atom.io, but as we want to embrace more comprehensive IDE features helping people keep their repos clean would be helpful.

As you note, backwards-compatibility is important. Another concern of mine is that releases be consistent - that is to say, we want to avoid someone shipping a release and then changing the downloaded archive later - version 1.1 (or whatever) should always be the exact same code.

This probably won't make it onto our roadmap in the very near future, but I think it's interesting to have a discussion here about what we can do to address the various pain points - the one most interesting to me is transpilation - are there other ways this could work to prevent check-in of build products without adding a second place to put products for download? e.g. a transpilation step on the server side for the most popular targets like typescript, etc?

[ssorallen]

e.g. a transpilation step on the server side for the most popular targets like typescript, etc?

I think Atom should generalize its involvement in transpilers rather than have first class support for a few (looking at you, CoffeeScript and Babel), and enable developers to set up easily with whatever transpiler they choose. VSCode has generalized transpilers/compilers by hooking them into the package's config and into VSCode's task runner for refresh/reload development. While all the transpiler examples talk about and show TypeScript, they demonstrate they are just shelling out to the TS Compiler, tsc. I think it'd be great if this lead to Atom bundling no transpilers by default.

Ideally developers shouldn't have to commit transpiled code in order to release it, and they should be able to exclude the source files from the released package. An '.apmignore' file would accomplish the second part, like @steelbrain mentioned, and that is the direction that VSCode went with its '.vscodeignore'. Their Yeoman project generators are set up for TypeScript, but VSCode only runs JavaScript. The projects add **/*.ts to their '.vscodeigore' files on generation.

[Alhadis]

I recently learned it's possible to exclude resources from package distributions by using .gitattributes:

/docs/ export-ignore
/1.5mbs-animated-banner-for-your-readme.gif export-ignore
/other-rubbish-that-shreds-bandwidth/ export-ignore

This attribute relates to the git archive command, which is clearly responsible for generating the tarballs GitHub serves to those downloading zipped repos. Examining the output of apm install --verbose revealed this is how packages are being downloaded:

    via: '1.1 vegur' }
REQUEST end event https://atom.io/api/packages/jss-atom-snippets
REQUEST has body https://atom.io/api/packages/jss-atom-snippets 3034
REQUEST emitting complete https://atom.io/api/packages/jss-atom-snippets
REQUEST { url: 'https://www.atom.io/api/packages/jss-atom-snippets/versions/1.3.0/tarball',
    strictSSL: true,
    headers: { 'User-Agent': 'npm/3.10.5 node/v4.4.5 darwin x64' },
    method: 'GET',
    callback: undefined }
REQUEST make request https://www.atom.io/api/packages/jss-atom-snippets/versions/1.3.0/tarball
REQUEST onRequestResponse https://www.atom.io/api/packages/jss-atom-snippets/versions/1.3.0/tarball 302 { server: 'Cowboy',
    connection: 'close',
    date: 'Sun, 13 Nov 2016 00:59:29 GMT',
    status: '302 Found',
    'x-frame-options': 'DENY',
    'x-xss-protection': '1; mode=block',
    'x-content-type-options': 'nosniff',
    'strict-transport-security': 'max-age=631152000',
    'content-security-policy': 'default-src \'self\'; connect-src \'self\'; font-src https://github-atom-io-herokuapp-com.global.ssl.fastly.net; frame-src \'self\' https://www.youtube.com; img-src https://* \'self\' https://github-atom-io-herokuapp-com.global.ssl.fastly.net data:; media-src \'none\'; object-src \'self\' https://github-atom-io-herokuapp-com.global.ssl.fastly.net; script-src \'self\' \'unsafe-inline\' https://ssl.google-analytics.com https://www.google-analytics.com https://platform.twitter.com https://github-atom-io-herokuapp-com.global.ssl.fastly.net; style-src \'self\' \'unsafe-inline\' https://github-atom-io-herokuapp-com.global.ssl.fastly.net;',
    location: 'https://codeload.github.com/march213/jss-atom-snippets/legacy.tar.gz/v1.3.0',
    'content-type': 'text/html; charset=utf-8',
    'cache-control': 'no-cache',
    'x-request-id': 'e867ee1d-4778-469f-a255-8925c8c5f385',
    'x-runtime': '0.063747',
    'x-rack-cache': 'miss',
    vary: 'Origin',
    via: '1.1 vegur' }
REQUEST redirect https://codeload.github.com/march213/jss-atom-snippets/legacy.tar.gz/v1.3.0
REQUEST redirect to https://codeload.github.com/march213/jss-atom-snippets/legacy.tar.gz/v1.3.0

This seems to be a more logical approach than using an .apmignore or .npmignore file... and the fact that it's drawn from Git itself makes it feel like a "canonical" answer to the issue of auxiliary resource exclusion.

However, the fact this works based solely on a redirect to GitHub makes me hesitant to embrace it as a long-term solution. How likely is it that Atom's package repository will continue serving tarballs this way? Infrastructure changes are inevitable, and packages might one day be served using a strategy that doesn't involve a call to git archive.

The existence of export-ignore makes me wonder why NPM feels .npmignore files are necessary... the world needs more vendor-specific manifests like it needs another Vietnam War.

[rictic]

For myself, I have the opposite problem @alhadis. There are files which I want excluded from git (because they are build artifacts, and so diffs etc aren't useful) but included in my atom plugin. .npmignore allows this because npm ignores the .gitignore file when it is present.

[Alhadis]

Diffs can easily be suppressed using .gitattributes too, although not on GitHub (which is a known issue; just check Linguist's backlog). Still, you're right... though I don't understand why NPM requires ignored paths to be specified in an external file, rather than package.json.

[victorhqc]

This is good feedback - thank you. These are definitely edge cases, as the current deployment strategy works for the majority of the thousands of packages on atom.io, but as we want to embrace more comprehensive IDE features helping people keep their repos clean would be helpful.

Is this still considered an edge case? The JS ecosystem has changed since 2016, now almost everybody uses Babel, and Typescript is getting a lot of traction. It feels bad to have the transpiled JS in the repo just to include it in the package.

[idleberg]

_Just to give some insights on modern package development, I would like to share my workflow, hoping that .apmignore will be supported in the future._

For my own package development, I'm following Microsoft's suggestion for VSCode to use bundling.

Visual Studio Code extensions often grow quickly in size. They are authored in multiple source files and depend on modules from npm. Decomposition and reuse are development best practices but they come at a cost when installing and running extensions. Loading 100 small files is much slower than loading one large file. That's why we recommend bundling. Bundling is the process of combining multiple small source files into a single file.

In practice, I'm using Webpack to create a bundle from all files in /src – no matter if my packages are written in JavaScript, CoffeeScript and TypeScript. While this approach gives me excellent performance, it comes with redundancy. The end-user neither needs the /src folder nor node_modules. To get rid off the latter, I guess I could declare all dependencies as devDependencies, but that doesn't _feel_ right.

Ultimately, I think that if many package developers adopt bundling strategies, not only their packages will benefit, but Atom as a whole. Calling this feature request an edge-case is a bit short-sighted, don't you think?

[aminya]

I made a solution to this. We can use build-commit:

• Add the folder that has the built files (e.g. dist) to .gitgnore
• Add build-commit to the repository
• Add this script to the scripts entry of package.json

    "build-commit": "build-commit -o dist",

• Then run npm run build-commit before doing apm publish.

[DeeDeeG]

To get rid off the latter, I guess I could declare all dependencies as devDependencies, but that doesn't _feel_ right.

@idleberg I think it _is_ right, though for your case.

For the record, if you are shipping a transpiled copy of all of your required modules or functions, and loading them from the transpiled output at runtime, then _end-users don't need your dependencies anymore_, and they _genuinely are_ development-only dependencies, when referring to their non-transpiled, module form.

Specifying a dependency is requesting that your package manager "please download these modules during the install of my package, thanks." The devDependencies field seems totally correct for your use-case.

[DeeDeeG]

Summarizing what has been said earlier in the thread: some of these capabilities folks are asking for here are already available.

• Ignoring files can be done with export-ignore in .gitattributes. https://github.com/atom/apm/issues/498#issuecomment-260161290
• And for anything you want the user to receive as tarballed content (such as precompiled files), you can "bite the bullet" and commit those files to the git repo. @aminya wrote an automated solution for this so that a release bot precompiles and commits the files in CI for every release: https://github.com/atom/apm/issues/498#issuecomment-607471283
  
  
  
  • Or alternatively, publish the precompiled/tarballed files as a package to the npm package registry and depend on that package.
  
  

That leaves some room for improvement, so yeah, I agree being able to actually directly upload tarballs of Atom packages would be an improvement. I just thought it would be nice to restate that info simply and put it in one place, in case folks are in need of a workaround _today_.

Solutions available today, in more detail

1. We can not have a .apmignore file in the repo

As @Alhadis mentioned, the same effect can be achieved by making a .gitattributes file and putting export-ignore next to the file(s), folder(s), and/or glob pattern(s) you want to exclude from the tarball GitHub makes for your repo.

1. If we want to put transpiled/compiled files into the package, we have to commit them into the repo and upgrade them on each change

As a compromise, you can update the built files only on every release, which can be automated if desired, like what @aminya proposed and already does for some of the Atom IDE packages.

1. We can not support mono repo architectures with this

If you are willing to commit every file in the monorepo, then monorepo is perfectly doable. (Which is a lot like how the npm package manager repository is maintained, for example. Note that even some of the node_modules content is checked in.)

---

While being able to upload tarballs would be an improvement in capabilities, there are some downsides as well:

• The downside for GitHub/Atom, as I see it, would be the need to spin up some infrastructure to store and serve these tarballs, and the need to run some API server(s) so people can upload/download/tag/deprecate/unpublish packages and so on. Basically a separate npm-style package registry. Which is not unheard-of, and the npm tooling is all meant to work with arbitrary package registries, so this isn't a "rewrite npm" problem, it's strictly the trouble of setting up and supporting the infrastructure.
  
  
  
  • GitHub already serves tarballs, so getting tarballs from GitHub like Atom currently does is sort of a freebie.
  
  
• There is the downside for end-users that it's a bit harder to verify where the files came from, since they may or may not have anything to do with the GitHub repo's contents. Just like with the npm package registry, the uploader can include arbitrary files. But the npm ecosystem already has to deal with this problem, and so does the Atom package ecosystem, since npm packages are already allowed as dependencies.