Google-cloud-node: webpack & google-cloud

[brandonmp]

Environment details

• OS: ubuntu 16.10
• Node.js version: 4.3.2
• npm version: 2.14.12
• google-cloud-node version: 0.44

Steps to reproduce

I'm using a webpack setup to build a nodejs project, the dependencies of which include google-cloud

Webpack sort of loses its mind w/ google-cloud, though, so hoping someone has some insight on how to fix.

The first error, ironically, appeared to be a deeply nested aws-sdk dependency, b/c I guess aws-sdk doesn't play nicely w/ webpack. I was able to fix that by including a pre-packaged AWS in my webpack-config.js, which is here:

// webpack.config.js
var fs = require('fs');
var path = require("path");
var webpack = require('webpack')
var prePackagedAwsPath = path.join('aws-sdk', 'dist', 'aws-sdk');

var nodeModules = {};
fs.readdirSync('node_modules')
    .filter(function (x) {
        return ['.bin'].indexOf(x) === -1;
    })
    .forEach(function (mod) {
        nodeModules[mod] = 'commonjs ' + mod;
    });

module.exports = {
    entry: path.join(__dirname, '/src/index.js'),
    output: {
        path: path.join(__dirname, "/dist/"),
        filename: "[name].js"
    },
    resolve: {
        alias: {
            'aws-sdk': prePackagedAwsPath
        }
    },
    target: "node",
    externals: nodeModules,
    module: {
        loaders: [
            {
                test: /\.node/,
                loader: 'node'
            },
            {
                test: require.resolve(prePackagedAwsPath),
                loader: 'imports?window=>null'
            },
            {
                test: /\.js$/,
                exclude: /node_modules/,
                loader: 'babel'
            },
            {
                test: /\.json$/,
                loader: 'json'
            }
        ]
    }
};

Now, however, the problem lies with the JSONStream dependency. The file apparently includes a # character at the top of its index.js, and that's breaking the build process.

Here's the error:

ERROR in ./~/@wayhome/wayhome-utils/~/google-cloud/~/JSONStream/index.js
Module parse failed: /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/JSONStream/index.js Unexpected character '#' (1:0
)
You may need an appropriate loader to handle this file type.
SyntaxError: Unexpected character '#' (1:0)
    at Parser.pp$4.raise (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2221:15)
    at Parser.pp$7.getTokenFromCode (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2756:10)
    at Parser.pp$7.readToken (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2477:17)
    at Parser.pp$7.nextToken (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2468:15)
    at Parser.parse (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:515:10)
    at Object.parse (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:3098:39)
    at Parser.parse (/home/bmp/code/scraping/node_modules/webpack/lib/Parser.js:902:15)
    at DependenciesBlock.<anonymous> (/home/bmp/code/scraping/node_modules/webpack/lib/NormalModule.js:104:16)
    at DependenciesBlock.onModuleBuild (/home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:310:10)
    at nextLoader (/home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:275:25)
    at /home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:259:5
    at Storage.finished (/home/bmp/code/scraping/node_modules/webpack/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:38:16)
    at /home/bmp/code/scraping/node_modules/graceful-fs/graceful-fs.js:78:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:380:3)
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/@google-cloud/prediction/src/model.js 25:17-38

I'd normally call this a JSONStream issue and file it there, but google-cloud is throwing warnings everywhere (below) as well, so I assume there's some overarching issue w/ webpack.

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/hash-stream-validation/index.js
Module not found: Error: Cannot resolve module 'fast-crc32c' in /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/hash-s
tream-validation
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/hash-stream-validation/index.js 5:8-30

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/src/node/src/grpc_extension.js
Critical dependencies:
38:14-35 the request of a dependency is an expression
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/src/node/src/grpc_extension.js 38:14-35

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/bytebuffer/dist/bytebuffer-node.js
Module not found: Error: Cannot resolve module 'memcpy' in /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/bytebuffer/
dist
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/bytebuffer/dist/bytebuffer-node.js 29:38-55

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/pre-binding.js
Critical dependencies:
19:22-48 the request of a dependency is an expression
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/pre-binding.js 19:22-48
WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/versioning.js
Critical dependencies:
15:20-67 the request of a dependency is an expression
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/versioning.js 15:20-67

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/nw-pre-gyp/index.html
Module parse failed: /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/grpc/node_modules/node-pre-gyp/lib/util/nw-pre-gy
p/index.html Unexpected token (1:0)
You may need an appropriate loader to handle this file type.
SyntaxError: Unexpected token (1:0)
    at Parser.pp$4.raise (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2221:15)
    at Parser.pp.unexpected (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:603:10)
    at Parser.pp$3.parseExprAtom (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1822:12)
    at Parser.pp$3.parseExprSubscripts (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1715:21)
    at Parser.pp$3.parseMaybeUnary (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1692:19)
    at Parser.pp$3.parseExprOps (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1637:21)
    at Parser.pp$3.parseMaybeConditional (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1620:21)
    at Parser.pp$3.parseMaybeAssign (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1597:21)
    at Parser.pp$3.parseExpression (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1573:21)
    at Parser.pp$1.parseStatement (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:727:47)
    at Parser.pp$1.parseTopLevel (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:638:25)
    at Parser.parse (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:516:17)
    at Object.parse (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:3098:39)
    at Parser.parse (/home/bmp/code/scraping/node_modules/webpack/lib/Parser.js:902:15)
    at DependenciesBlock.<anonymous> (/home/bmp/code/scraping/node_modules/webpack/lib/NormalModule.js:104:16)
    at DependenciesBlock.onModuleBuild (/home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:310:10)
    at nextLoader (/home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:275:25)
    at /home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:259:5
    at Storage.finished (/home/bmp/code/scraping/node_modules/webpack/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:38:16)
    at /home/bmp/code/scraping/node_modules/graceful-fs/graceful-fs.js:78:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:380:3)
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib ^\.\/.*$

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/compile.js
Module not found: Error: Cannot resolve module 'node-gyp' in /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/grpc/node
_modules/node-pre-gyp/lib/util
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/compile.js 29:28-55

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/compile.js
Module not found: Error: Cannot resolve module 'npm' in /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/grpc/node_modu
les/node-pre-gyp/lib/util
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/compile.js 39:27-49

[stephenplusplus]

We haven't been testing against webpack, so I'm quite confident it would be pure luck if all of our dependencies would be compatible. I'm going to play around with it as time allows, but feel free to dig into it and google around for issues if you want to help out. After a quick search, I found this issue which might get you past the JSONStream error: https://github.com/webpack/webpack/issues/2168

If you do dig into this, please keep me updated!

[brandonmp]

thanks @stephenplusplus ! that helped get around the !# error, though i've hit a few more I'll detail here for posterity. if anyone has ideas on how to address the unsolved problem, would be grateful :) note that this sort of stuff is a little outta my league, so some of these fixes may be suboptimal

!# error

error: wepback won't parse .js files that start with a !# arg
solution: install a special loader to handle

$ npm install shebang-loader

// in webpack.config.js, add:
module: {
  loaders: [
...
     {
      // several gcloud dependencies have !#, so we'll apply loader broadly
      test: /node_modules\/.*\.js$/,
      loaders: ['shebang', 'babel']
     }
...
  ]
}

source: https://github.com/webpack/webpack/issues/2168

node-forge

error: node-forge throws Invalid left-hand side in assignment
solution: this one feels a bit janky. we'll minimize and sideload it into our webpack.config.js:

npm i node-forge
cd node_modules/node-forge
npm i
node node_modules\requirejs\bin\r.js -o minify.js optimize=none out=js/forge.bundle.js
// copy forge.bundle.js elsewhere if you like. mine is in src/webpack/

// webpack.config.js:
var forge = require('node-forge')

//then modify your babel-loader module loader to add forge.bundle.js, and add the file to `noParse`:
module: {
    loaders: [
...
          {
                test: /\.js$/,
                exclude: /(node_modules|forge\.bundle\.js)/,
                loader: 'babel-loader'
            }
...
    ],
     noParse: [
            path.resolve(__dirname, './whatever/path/you/chose/for/forge.bundle.js')
        ]

source: https://github.com/digitalbazaar/forge/issues/198

node-pre-gyp

error: haven't researched this one yet. here's error text:

ERROR in ./~/@google-cloud/storage/~/@google-cloud/common/~/grpc/~/node-pre-gyp/~/tar/tar.js
Module build failed: SyntaxError: Invalid number (108:10)

  106 | // values for the mode field
  107 | var modes =
> 108 |   { suid: 04000 // set uid on extraction
      |           ^
  109 |   , sgid: 02000 // set gid on extraction
  110 |   , svtx: 01000 // set restricted deletion flag on dirs on extraction
  111 |   , uread:  0400

 @ ./~/@google-cloud/storage/~/@google-cloud/common/~/grpc/~/node-pre-gyp/lib/install.js 56:24-38

[stephenplusplus]

I'm going to close this issue, since we don't have any plans to support being run through webpack at this time. Please subscribe to https://github.com/GoogleCloudPlatform/google-cloud-node/issues/531 and if we end up carving out some time to spend towards this goal, you can follow along. Sorry I can't be of more help at this time.

[brandonmp]

thanks @stephenplusplus , though my interests were more on the-server-side--specifically for aws-lambda/google-cloud functions. webpack is pretty clutch in that workflow, so I'm hopeful the plan to support it changes in the near future.

[mikelambert]

@brandonmp , I believe the node-pre-gyp code would require a change:
{ suid: parseInt("4000", 8)
and so on.
(Or if node-pre-gyp is relying on ES6, which I'm not sure is safe...one could write 0o4000)

Seems webpack is enforcing the fact that 0400 is a non-standard way to write an octal number, which I'd classify as a bug, except that maybe node-pre-gyp is assuming it only ever runs in a node environment.

[jonwoodring]

Here's a webpack loader to get around the octal number issue:
https://www.npmjs.com/package/octal-number-loader
https://github.com/jonwoodring/octal-number-loader

[olegccc]

based on the comments above I assume the only safe way is to exclude this module from bundling using externals webpack option or webpack-node-externals module.

[slootzky]

any news with this issue? trying to use Google's pubsub from Azure functions & Firebase functions so we're using webpack which fails (currently at node-pre-gyp / pre-binding.js in this part :
if (!existsSync(package_json_path)) { throw new Error("package.json does not exist at " + package_json_path); }

[gvillenave]

I'm seeing the same error when building a React website with Firebase using Gatsby. Webpack is THE bundling standard nowadays, it seems quite odd that the Google Cloud Platform SDK would not be able to work with it...
Why was that issue closed in the first place?

[ksnyde]

I've tried bundling with webpack, rollup, and browserify (all targeting node) and they all have problems with the @google-cloud/* libraries. Everything else works flawlessly.

[dundas]

I dont imagine anyone has come up with an answer for this. this bug is making me crazy right now. any work around?

[ohabash]

@dundas did you find a solution?

How do use firebase-admin in angular unversal server?

[odedniv]

Took me a while but I finally got it. In Google Cloud Functions you don't need webpack because you don't upload the function artifact with the dependencies, you upload it with package.json and GCF installs the dependencies itself.

It won't help AWS/Azure users, but at least I can finally use GCP resources from GCF...

[tettoffensive]

@brandonmp @dundas any solutions or workarounds?

I've been building an internal content management system with vue + webpack. I am using the firebase libraries for the database.

I just want to be able to upload an image to the storage. firebase.storage() isn't available in node. so then I read that I'm supposed to use google-cloud/storage, but that doesn't work either.

For simply adding const { Storage } = require('@google-cloud/storage');

index.js?1c59:45 Uncaught TypeError: Expected `input` to be a `Function` or `Object`, got `undefined`
    at module.exports (index.js?1c59:45)
    at eval (index.js?838e:22)
    at Object../node_modules/gtoken/build/src/index.js (app.js:3626)
    at __webpack_require__ (app.js:679)
    at fn (app.js:89)
    at eval (jwtclient.js?e0ab:26)
    at Object../node_modules/google-auth-library/build/src/auth/jwtclient.js (app.js:3153)
    at __webpack_require__ (app.js:679)
    at fn (app.js:89)
    at eval (googleauth.js?983a:37)

[ksnyde]

I have added the following to externals:

module.exports = {
  // ...
  externals: [ 'aws-sdk', /@google-cloud/ ]
}

seems to do the trick.