Js-ipfs: Getting IPFS to work with ES Modules and Rollup

[AlbertoElias]

• Version: 0.34.0
• Platform: Chrome

Type: Bug

Severity: Medium

Description:

So I spent quite a lot of time yesterday trying to compile js-ipfs with Rollup yesterday and here are my findings:

• Rollup doesn't support readable-stream, so I use this solution: https://github.com/rollup/rollup-plugin-alias/issues/33#issuecomment-339724994
• Many sub dependencies have circular dependencies which break Rollup, most of these, or all that I've seen, are unmaintained modules. Basically, all those made by https://github.com/indutny like asn1.js, elliptic and hash.js (these are the ones I've bumped into before giving up). There are forks of those that do work:
  
  
  
  • https://github.com/lordvlad/asn1.js
  
  
  • https://github.com/xg-wang/elliptic
  
  
  • https://github.com/AlbertoElias/hash.js
  
  
• Also circular dependency issues with zcash-bitcore-lib right here https://github.com/BitMEX/zcash-bitcore-lib/blob/master/lib/util/preconditions.js#L21. If I change that to `require('Buffer'), it works. It's actually deprecated (https://github.com/ipld/js-ipld-zcash/issues/24).
• Even if I manually change all these libraries, they themselves add their own copies of these dependencies, and you can't sustainably change package.lock.json to address that. Like zcash-bitcore-lib adds its own hash.js
• Many dependencies require core node.js APIs which need to be polyfilled and add up a lot to the bundle

So, to sum up, js-ipfs ends up depending on old unmaintained dependencies that don't work well at all on the Web. They depend too much on Node, Common.JS and circular dependencies. It would be great to see js-ipfs using a more cleaned up dependency tree that's friendlier to browsers and makes it easy to use ES6 modules

Steps to reproduce the error:

index.js

export {* as IPFS} from 'ipfs'

rollup.config.js

import resolve from 'rollup-plugin-node-resolve'
import commonjs from 'rollup-plugin-commonjs'
import json from 'rollup-plugin-json'
import builtins from 'rollup-plugin-node-builtins'
import globals from 'rollup-plugins-node-globals'
import replace from 'rollup-plugins-replace'

const config = {
    input: 'index.js',
    output: [{
        file: 'dist/index.js',
        format: 'es'
    }],
    plugins: [
        replace({
            [`readable-stream`]: `require('stream')`,
            delimiters: ['require(\'', '\')']
        }),
        resolve({
            preferBuiltins: false,
            browser: true
        }),
        json(),
        globals(),
        builtins(),
        commonjs({
            ignoreGlobal: true
        })
    ]
}

export default [config]

[hugomrdias]

hi @AlbertoElias thank you for this research, im trying to fix some of these problems the first step is this https://github.com/ipfs/js-ipfs/pull/1795 to streamline dependencies and reduce bundle size.

but there still much to do, and some of it is related to this work you did, the end game will be to change our current webpack config with rollup and have a esm version.

did you manage to understand why rollup doesn't support readable-stream ?

[AlbertoElias]

Wow, that PR sure seems promising :)

change our current webpack config with rollup and have a esm version.

That's really good to hear!

did you manage to understand why rollup doesn't support readable-stream ?

sadly, no, this is the response I got a long time ago: https://github.com/calvinmetcalf/rollup-plugin-node-builtins/issues/35#issuecomment-339358963 which isn't very useful

[mikeal]

Related: I just chatted with Myles Borins who is working on ESM in Node.js and there are some big changes/improvements that will be landing soon. In particular, my complaints about variances between Node’s dynamic import method and webpack’s are going to be addressed, so it should get easier to maintain a future ESM code base for Node.js and browsers.

-Mikeal

---

From: Alberto Elias [e-mail]
Sent: Wednesday, March 13, 2019 10:24 AM
To: ipfs/js-ipfs
Cc: Subscribed
Subject: Re: [ipfs/js-ipfs] Getting IPFS to work with ES Modules and Rollup (#1927)

Wow, that PR sure seems promising :)

change our current webpack config with rollup and have a esm version.

That's really good to hear!

did you manage to understand why rollup doesn't support readable-stream ?

sadly, no, this is the response I got a long time ago: calvinmetcalf/rollup-plugin-node-builtins#35 (comment)https://github.com/calvinmetcalf/rollup-plugin-node-builtins/issues/35#issuecomment-339358963 which isn't very useful

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/ipfs/js-ipfs/issues/1927#issuecomment-472523799, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAACQxIneQK2KSaClzNToxqtUch6wgdOks5vWTRigaJpZM4btRIg.

[lastmjs]

Any timeline for this?

[mitra42]

I'm trying to figure if this would help a problem I've got ... someone wants to use ES6 modules (in the browser) to import our https://github.com/dweb-transports library, which currently includes IPFS via require. If I understand it correctly I'm going to need to replace all the require with import, and the hard part will be something like IPFS with its own complex web of dependencies.

Am I correct that successfully concluding this would allow include IPFS via e.g. import ipfs from ... or am I misreading this project ?

[mikeal]

@mitra42 that used to be the case, but it looks like people have implemented proper plugins for node modules and require for rollup https://stackoverflow.com/questions/50081548/how-to-make-rollup-expand-require-statements?answertab=active#tab-top

You probably won’t get tree shaking within the required modules though, since the way rollup wrote that feature isn’t all that smart and just excludes code you didn’t include in import statements.

[mitra42]

Does anyone know if this (using rollup) has been done successfully for IPFS. I think that if IPFS has a successfull rollup then I can either import it in my own rollup OR copy whatever workarounds the IPFS rollup uses into one layer higher up. If IPFS itself can't rollup then the chances of doing it one more level up seems unlikely ...