Bower: Discussion: NPM Migration Path

[FredKSchott]

npm WARN deprecated [email protected]: ..psst! While Bower is maintained, we recommend Yarn 
and Webpack for *new* front-end projects! Yarn's advantage is security and reliability, and 
Webpack's is support for both CommonJS and AMD projects. Currently there's no migration 
path but we hope you'll help us figure out one.

I want to take a stab at the deprecation message's invitation to discuss a migration path.

I've tried moving a few projects from Bower to NPM, and the obvious problem that I always get stuck on is "what do I do when a package doesn't exist on npm?" Managing dependencies on both Bower and NPM at the same time isn't feasible, so I'm pretty much hosed.

So, here's a rough idea for how to allow all interested package/application authors migrate to NPM:

1. Each package in the Bower registry is automatically re-published to npm under the @bower/ organization scope, with source code/files untouched.
2. This automatic publisher generates a package.json for each Bower package based on the bower.json, correctly pointing to the other npm @bower/X dependencies.
3. If all Bower packages exist on NPM, application authors could then easily make their move from Bower packages to NPM's @bower/X packages without needing to change any/much source.
   
   
   
   • Because everything would be installed in the @bower/ node_modules/ subdirectory, any hardcoded ../ dependency paths would continue to work as written.
   
   
   • Flat installs would be supported by yarn install --flat.
   
   
4. Once squarely inside the npm ecosystem, applications could start moving away from @bower/X dependencies gradually.

If the "mirror Bower on NPM" part of the idea sounds crazy, there's actually already some successful precedent for this. DefinitelyTyped, for example, mirrors their big github repo of type definitions under the @types/ scope on npm. Obviously the npm team would need to be looped in to confirm/approve this for us, but it's possible.

Also worth highlighting again that this plan is focused on making the Bower->NPM migration easier. Bower's future would continue to exist outside the scope of this plan.

What does everyone think? Crazy? Feasible?

[sheerun]

Hey! Thank you for opening this discussion.

I had similar idea as you some time ago (i.e. register all bower packages on npm), I'm even an owner of @bower organization on npm but I figured out it would be inappropriate to publish non-commonjs packages on npm's registry by non-authors in bulk.

What I decided instead is to improve yarn so it's able to install bower's packages directly from github. I've taken steps to implement it, but not 100% successfully. I hope in the near future you'll be able to yarn add dbrekalo/lang and it'll be installed in node_modules/lang.

Then there's ability of bower to install github dependencies by semver, and yarn seems to allow to accept this feature as well, if someone implements it.

Then, there's a need to write bower-migrate tool that is able to convert bower.json to package.json. It would resolve names from Bower's registry and set appropriate entries in npm's package.json, so for example "angular": "1.x" would become "angular": "angular/bower-angular#1.x".

In my opinion here are steps needed for migration:

• [x] Finish and test Allow for installing dependencies without manifest feature in Yarn
• [ ] Implement and test Branch/tag resolution for git dependencies feature in Yarn
• [ ] Implement bower migrate tool that is able to convert bower.json to package.json
• [ ] Figure out how to configure Webpack so it works for Global/AMD/CommonJS mixed codebase.

Of course ideally developers would gradually would switch to Webpack and CommonJS. Thanks to Yarn, most bower packages would already be installed in CommonJS form, so for example switching to CJS jQuery would be as simple as putting const $ = require('jquery') at the top of the file. This can be done thank's to Webpack's support for both CommonJS and AMD projects.

[FredKSchott]

Ha! I didn't even check to see if anyone owned "@bower/", glad to hear this isn't a new idea then.

I agree that publishing non-CJS to npm is strange. It's intentionally something that can not be require'd by Node. However @types/ is a good example of a complementary ecosystem built into npm that helps npm developers, even if they can't be used as actual CJS modules (all in TS, would fail if they were ever required). Maybe the expectations of what an NPM package is have changed since you last considered it?

Those three steps sound great, really excited to see those PRs being actively worked on. I have some work I did for an auto bower.json -> package.json tool that could be useful for bower migrate, let me try to get that up somewhere for people to check out.

One concern is that those 3 steps together still don't solve the original show-stopper problem I brought up, where a Bower dependency that doesn't exist on NPM blocks the entire migration of my project. I like that if that dep doesn't have any bower.json dependencies then the git: url in your package.json would work. But if it did have bower.json dependencies then those wouldn't come through, and your install would still fail.

In my experience, that's still the biggest problem any user will have migrating to NPM/webpack/CJS/etc.

[sheerun]

@FredKSchott Bower's package ecosystem is lacking modularity. It means that most packages don't list much dependencies except maybe frameworks like angular or jquery.

I think the solution for your problem is to write bower migrate tool, so it flattens all bower dependencies, and puts all of them into package.json. e.g. if angular-router depends on angular in bower.json, it would put both in package.json. Maybe it could also generate bower_components.js file with require statements in proper order, so you could do something like require('./bower_components') to load all legacy bower components into global.

Ultimately, most packages would maintain proper package.json, so you can start requiring it in your code (Yarn would install dependencies even of bower packages, if they have proper package.json).

[FredKSchott]

So bower migrate could look at your full bower dependency tree, and generate a package.json with a github URL for each direct and indirect bower dependency listed as an npm dependency. Then the fact that it doesn't handle indirect dependencies wouldn't matter. That's really interesting...

One extra thing to think of in any solution / automatic migration: Some packages have gotten away with describing one entrypoint & set of dependencies in their bower.json for the frontend, and another entrypoint and set of dependencies in their package.json for the backend. (Ex: web-component-tester bower.json & package.json). The same git url ultimately describes two different packages (functionally).

[sheerun]

I think the best we can do is to politely ask such authors to move development tools from dependencies to devDependencies and write proper dependencies. Until then yarn would install all their "dev" dependencies, as I'm not aware of switch that prevents it any of npm and yarn.

And if project needs separate dependencies for front-end and back-end, I guess for now the best advice is to create separate directory for front-end (or back-end) and put 2nd package.json there.

It can be even bower_components/package.json with postinstall hook but it seems hacky.

[souyunku]

root@iZsfb1h2cwbm97Z:~# npm install -g bower
npm WARN deprecated [email protected]: ..psst! While Bower is maintained, we recommend Yarn and Webpack for new front-end projects! Yarn's advantage is security and reliability, and Webpack's is support for both CommonJS and AMD projects. Currently there's no migration path, but please help to create it: https://github.com/bower/bower/issues/2467
/usr/local/bin/bower -> /usr/local/lib/node_modules/bower/bin/bower
/usr/local/lib
└── [email protected]

[chase2981]

before: (bower.json)

add:
npm install script-loader

after: (vendor.ts)
/* Bower imports */ import "script-loader!../../../bower_components/jquery/dist/jquery.js"; import "script-loader!../../../bower_components/angular/angular.js"; import "script-loader!../../../bower_components/angular-animate/angular-animate.js"; import "script-loader!../../../bower_components/angular-sanitize/angular-sanitize.js"; import "script-loader!../../../bower_components/angular-resource/angular-resource.js"; import "script-loader!../../../bower_components/angular-route/angular-route.js";

or better yet..

/* Node imports */ import "script-loader!../../../node_modules/jquery/dist/jquery.js"; import "script-loader!../../../node_modules/angular/angular.js"; import "script-loader!../../../node_modules/angular-animate/angular-animate.js"; import "script-loader!../../../node_modules/angular-sanitize/angular-sanitize.js"; import "script-loader!../../../node_modules/angular-resource/angular-resource.js"; import "script-loader!../../../node_modules/angular-route/angular-route.js"; import "script-loader!../../../node_modules/angular-aria/angular-aria.js"; import "script-loader!../../../node_modules/angular-messages/angular-messages.js";

[evenstensberg]

I'd like to help out on the bower migrate feature if you need help with that :)

[sheerun]

I've already started a prototype, I'll post the link today so maybe you'd
like to take over :)

On Sat, 8 Jul 2017 at 20:35, Even Stensberg [e-mail]
wrote:

I'd like to help out on the bower migrate feature if you need help with
that :)

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/bower/bower/issues/2467#issuecomment-313873627, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAR2Des702v9nSFp6jzXbIAw9JuruZALks5sL8wBgaJpZM4OIRQk
.

[sheerun]

@ev1stensberg I've pushed https://github.com/sheerun/bower2yarn with prototype, I've added you as a collaborator so you can edit directly. If anyone else would like to help, please fork and send PR (or create an issue to discuss some feature).

[evenstensberg]

Awesome

[HaykoKoryun]

this is definitely needed as it would mean and end to things like this in my projects (due to incomplete migrations to npm/yarn):

fs.symlinkSync
(
  __dirname + "/node_modules/web-animations-js",
  __dirname + "/node_modules/@polymer/web-animations-js",
  "dir"
);

[sheerun]

@HaykoKoryun Yarn will be able to install Bower dependencies 1:1, for example in node_modules/@bower_components directory. Then you need only one symlink to migrate.

[sheerun]

Yarn 0.28.1 includes my feature to install git repositories without package.json (i.e. any Bower package). To test it you can do following:

npm install -g [email protected]
mkdir test
cd test
yarn init --yes
yarn add IndexOverflow/orange-notification#2.0.x
ls node_modules/orange-notification

What's left is this PR to support semver for git tags, finish bower2yarn migration script, and maybe write a blogposts how to a) migrate bower to yarn b) introduce webpack into legacy bower project

@chase2981 maybe you'd like to write blogpost b) ?

[FredKSchott]

Would love to get involved as well, I'll jump in to the issues on bower2yarn.

Awesome work so far @sheerun + @ev1stensberg + everyone! Love to see this progressing 🏆

[evenstensberg]

@FredKSchott Actually I've not done anything yet, this is all on @sheerun :)

[publicocean0]

for powering bower in lacking modularity i use bindep tool

[graingert]

Figure out how to configure Webpack so it works for Global/AMD/CommonJS mixed codebase.

AMD/CommonJS/UMD is supported out of the box in webpack.

You can support modules that export via Globals with legacy-loader, imports-loader and exports-loader.

[waw3ru]

I am still an active user of bower but I use it for purposes of getting mostly jquery ui libraries in my projects then I use webpack to bundle them with my codebase.

My question is will there be any issue because of this need to migrate in terms of changing already written codebase ?

I honestly view bower as an awesome tool specifically for getting small ui libraries that depend on jquery mostly without having to manually download or use npm which may require to depend on any external tool like webpack or even rollup. It's more of an efficient library provider/downloader that is configurable and allows maintainability for my small projects.

My suggestion is to identify those modules that don't need to be got via bower e.g. angular, jquery and the like and let the users of bower get a notification or warning to suggest to use yarn instead.

Thanks.

[Aghassi]

I'd just like to add here that this will need to handle certain edge cases. Specifically, my team is working on moving to React 16.0. However, the bower version of React flattens the packages so that React-Dom is included in the resolution. When migrating to NPM, you will end up with react, react-dom, and prop-types as separate packages you must install. I've made a very poor, and terrible, script to handle this internally, but wanted to share it here in case it helps with this effort:

/*
 * THIS IS A DIRTY HACK
 * This *will* be removed after the migration to Yarn/NPM 3+
 */
var glob = require('glob');
var fs = require('fs-extra');
var path = require('path');
var packageJson = require('../../package.json');

/**
 * Finds and copies .js and .min.js files from specific node modules to dist/modules folder.
 * Exceptions made for react-dom since legacy architecture requires it be in the same place at React
 * @param  {[type]} dep    Name of dependency
 * @param  {[type]} folder Folder to search through using the globbing pattern we have
 * @param  {[type]} file   Glob pattern
 * @return {[type]}        Promises
 */
var findAndCopy = function(dep, folder, file) {
  return fs.ensureDir('dist/modules/' + dep)
    .then(function() {
      return new Promise(function(resolve, reject) {
        glob('node_modules/' + dep + '/' + folder + '/' + file, function(error, files) {
          if (error) reject(error);
          resolve(files);
        });
      });
    })
    .then(function(files) {
      var promises = [];

      files.forEach(function(file) {
        if (dep === 'react-dom') {
          // This needs to be in the same folder due to outdated legacy architecture
          promises.push(fs.copy(file, 'dist/modules/react/' + path.basename(file)));
          fs.remove('dist/modules/react-dom');
        } else {
          promises.push(fs.copy(file, 'dist/modules/' + dep + '/' + path.basename(file)));
        }
      });

      return Promise.all(promises);
    })
    .catch(function(error) {
      console.error('Failed to copy artifact ' + dep + ':', error);
    });
}

// Ensures existing dist modules and create folder if non existent
fs.ensureDir('dist/modules')
  .then(function() {
    var promises = [];

    // Iterate over package.json dependencies
    for (var dep in packageJson.dependencies) {
      console.log('Currently copying artifact: ' + dep);
      // Copy React (which outputs to dist directory)
      if (dep === 'react' || dep === 'react-dom') {
        promises.push(findAndCopy(dep, 'dist', '*.js'));
      }

      // Copy moment
      if (dep === 'moment') {
        promises.push(findAndCopy(dep, 'min', '*.js'));
        promises.push(findAndCopy(dep, 'locale', '*.js'));
      }

      // Copy prop-types
      if (dep === 'prop-types') {
        promises.push(findAndCopy(dep, '', 'prop-types.*'));
      }
    }

    return Promise.all(promises);
  })
  .catch(function(error) {
    console.error('Failed to copy artifacts:', error);
  });

This script is because, up until now, we have been consuming a package that resolves with bower via NPM by moving the bower resolutions to a dist folder and then publishing that as a "package" to our internal NPM repo. Once resolved, we have the projects map to that dist folder so as to "point" them to where it needs look. Obviously, this is not ideal at all and we would like to move away from this. However, I still wanted to share this with the community as it may aid in the effort to have this feature truly flushed out.

[benmann]

Hey @Aghassi :-)
I kinda feel like this does not necessarily need to be handled. I do understand the frustration of writing your own script to fix this but I'd suggest you open an issue with the react team if you really want to know why they differentiate between the bower and npm version of the package.

Also I think it's important to note that those are different packages. One registered with the Bower registry, containing react-dom and the other one registered with the npm registry, not containing it. So what should also ultimately work, is requiring the Bower version with yarn, giving you the same results as with Bower.

[sheerun]

@trendy-weshy Switching Yarn with Bower won't require any changes in codebase. You could re-write your code to use require() instead of <script> and globals / AMD, but that's another task.

We're waiting for next Yarn version to be released that is able to handle both repositories without package.json and semver tags of git repositories (that's what is needed to install most of Bower packages).

[sheerun]

@Aghassi It seems your team is switching from globals to CommonJS and that more involved task than just switching from Bower to Yarn. In particular it requires finding CommonJS-enabled equivalents of all dependencies, replacing them in package.json, and changing codebase to use require().

The solution in your case is to put "react": "*" and "react-dom": "*"in package.json and something like following in your code:

const React = require('react');
const { PropTypes } = React;
const ReactDOM = require('react-dom');

[sheerun]

@Aghassi As for the bower2yarn script, it'll list all bower dependencies flattened in package.json (even sub-dependencies), so prop-types and react-dom will be there.

[Aghassi]

@sheerun Appreciate the feedback. I will be sure to take this back to my team. We do have a few moving parts at the moment ;).

@benmann Thanks. I just wanted to throw my two cents in. If nothing else, someone stumbles across this and it helps them so they don't feel like they are the first person to come across this issue. Though, I will go open an issue against the react bower package and ask why.

[sheerun]

@arcanis Hope you manage to release current master Yarn version as it blocks this issue :)

[arcanis]

@sheerun We're currently on a sprint to smooth everything we can before releasing the 1.0, which is expected to come early september! 😃 In the meantime, know that we have publicly documented nightly builds that should be relatively stable (we make sure all our tests pass before merging PRs).

[sheerun]

Yarn 1.x can install bower modules :)

Here's current blocker: https://github.com/yarnpkg/yarn/issues/4604

[graingert]

@sheerun looks like it's not a blocker anymore! Maybe time to deprecate bower a bit more?

[graingert]

@sheerun or maybe another blog post

[sheerun]

I'm working on https://github.com/sheerun/bower2yarn script. Going to write blogpost when ready

[bennycode]

Hello everyone! 👋

I would like to share insights on how we at Wire migrated from Bower to npm.

For our web application we use Grunt as a task runner. To make use of Bower components, we installed grunt-bower-task. The special thing about the grunt-bower-task plugin is that it reads a property called exportsOverride from the bower.json file.

The exportsOverride property makes it possible (using glob patterns) to copy Bower components into different target directories. We had to make sure that the information stored in exportsOverride is not lost when migrating from bower.json to package.json. We also had to make sure that Bower components (which are defined in a package.json file using the @bower_components prefix) are copied to their specific target directories.

To cover our requirements, we created a Grunt plugin: grunt-npm-bower.

The grunt-npm-bower plugin provides a Grunt task which copies all exportsOverride patterns from bower.json to package.json. Afterwards it copies all Bower components (found in node_modules) into their specific target directories (the ones defined in exportsOverride). If there is no exportsOverride pattern, then it will copy all contents of matched Bower components into the root of a specified folder.

Here is our entire migration workflow:

1. Running bower-away to migrate dependencies from bower.json to package.json (without deletion of bower.json!)
2. Exchanging grunt-bower-task with grunt-npm-bower
3. Deleting bower.json and removing grunt-bower-task

The only corner case we experienced was resolving a Zip archive. Zipped files are not supported by Yarn. Luckily Google Code Archive supported exporting projects to GitHub. This allowed us to extract the zip files and upload them to GitHub.

Our complete migration can be viewed here:

• https://github.com/wireapp/wire-webapp/pull/1857

Our Grunt task is available here:

• https://github.com/wireapp/grunt-npm-bower

Best,

Benny

[aimad-majdou]

@bennyn Thanks for sharing your approach, unfortunately our project uses gulp instead of grunt so the grunt-npm-bower plugin won't work with gulp, it would be great if someone created the same plugin for gulp.

[bennycode]

@aimad-majdou Can you post your Gulp task definition which you are using to install your Bower components? Then I can have a look if it's possible to resuse the business logic from grunt-npm-bower for Gulp. 🙂

[aimad-majdou]

Hi,

This is how I moved from using npm to yarn.

In our application we use Gulp for automating tasks in our development workflow, the first step was migrating to package.json from bower.json using bower-away, this will add the @bower_component scope to each package entry defined in the dependencies section, so after running the Yarn installation, a sub-folder called @bower_components is created inside the node_modules folder, the only problem we had is resolving zipped files which are not supported by Yarn.

The next step was resolving zipped files issue, as a solution, I extracted and then I published them into our private npm registry with the scope @bower_components, now they can be installed by their name in our npm private registry.

When we were using bower, we defined inside .bowerrc the path to bower_components, but unfortunately this can't be done using Yarn, so I created a Gulp task which copies the installed components inside node_modules/@bower_components to another target directory, something like this:

gulp.task('move-dependencies', function () {
    return gulp.src('node_modules/@bower_components/**')
               .pipe( gulp.dest(destPath + '/dependencies') );
});

The next step, is injecting the installed dependencies into our index.html page, when we were using bower, we used the main-bower-files which will read our bower.json, iterate through our dependencies and returns an array of files defined in the main property of the packages bower.json, to cover this, I created a plugin main-yarn-files forked from main-bower-files which will l read our package.json, iterate through our dependencies and returns an array of files defined in the main property of the packages package.json, some package has no package.json, so the plugin will check for bower.json, .bower.json and finally component.json, this is how this plugin is used:

var yarnFiles = require('main-yarn-files');
…
yarnFiles({
    paths: {
        modulesFolder: destPath + '/dependencies',
        jsonFile: './package.json',
        scope: '@bower'

    }
})

• modulesFolder : path for our dependencies (if not defined it will throw an error).
• jsonFile : path for our package.json (default value: package.json).
• scope : the scope used for dependencies declared in the package.json dependencies section (default value: @bower_components).

Finally, running our gulp file will :

1. Install dependencies using Yarn (inside node_modules/@bower_components) and devDependencies inside (node_modules),
2. Move dependencies inside (node_modules/@bower_components) into a target directory where the index.html page is located.
3. Run the main-yarn-files plugin, and inject all the returned dependencies into index.html.

Hope this will work in your case.

The main-yarn-files gulp task : https://github.com/aimad-majdou/main-yarn-files

Cheers.

[andreyvolokitin]

I'm investigating a current state of things with "bower -> npm" transition. Specifically I'm trying to find a way to auto-wire newly installed npm dependencies (similar to wiredep). By "auto-wire" I mean not necessarily auto-inserting script/style tags in html, but more like automatically getting all dependencies paths of installed package and then doing with them anything I want (i.e. auto-inserting import statements in js/css files, or same old auto-inserting script/style tags in html). And I don't quite understand how non-js/other dependencies can be handled in the following scenario:

• we install some frontend package with npm install --save
• the main field in package.json usually contains main js entry point which we can use as we want
• but usually there is no notion about other files: css, other js files if present, etc.
• the notion about these resources usually resides in the main field inside bower.json, i.e.: https://github.com/dimsemenov/PhotoSwipe/blob/master/bower.json
  https://github.com/dimsemenov/PhotoSwipe/blob/master/package.json

The current situation is that existing packages have all info we need inside bower.json but bower is now deprecated. And at the same time there is no sufficient info inside package.json. So it's like we forced to still use bower.json to get info we need, and in the future it'll likely remain the same because I highly doubt that package authors will update packages accordingly. I understand that with webpack we can require any module we want, but still we need to manually get and insert paths for dependencies. Could anybody share their thoughts/experience regarding this problem?

[graingert]

I actually prefer it the other way around: https://github.com/webpack-contrib/npm-install-webpack-plugin

Adding the import to the file cause the package to be installed

[andreyvolokitin]

Very interesting, will definitely consider this approach. Though for arbitrary npm frontend package, there is a random set of dependency files so we can't always know ahead of time the names we need to import and even if we could — we still should manually import them. With bower and wiredep all this process was automatic... I suppose that this is sort of an edge case though, and by now we still can use bower.json if/while we can

[graingert]

I think this is probably a solution looking for a problem, I've been using webpack and npm for a few years now and have never needed this sort of thing

[andreyvolokitin]

How would you handle using i.e. PhotoSwipe? You import photoswipe, webpack installs npm package automatically. Then you need to research what dependencies PhotoSwipe has (i.e. listed here https://github.com/dimsemenov/PhotoSwipe/blob/master/bower.json), and then you copy these dependencies paths and import additional js and css files manually. Is it more or less correct or I'm missing something?

[graingert]

I think the community favours more composable components these days

On Fri, 24 Aug 2018, 18:17 andreyvolokitin, [e-mail]
wrote:

How would you handle using i.e. PhotoSwipe? You import photoswipe,
webpack installs npm package automatically. Then you need to research what
dependencies PhotoSwipe has (i.e. listed here
https://github.com/dimsemenov/PhotoSwipe/blob/master/bower.json), and
then you copy these dependencies paths and import additional js and css
files manually. Is it more or less correct or I'm missing something?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/bower/bower/issues/2467#issuecomment-415824002, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZQTOVE_D_xqV8-jeUO-ibYAZAfrw5pks5uUDUMgaJpZM4OIRQk
.

[sheerun]

Someone suggested https://github.com/johnnyasantoss/wiredep-away on top of https://github.com/sheerun/bower-away as a way to migrate wiredep dependencies, hope it helps

[sheerun]

I think this topic has been exhausted here, we recommend to migrate to Yarn with bower-away