Clasp: npm install is inconsistent

[grant]

I'm not able to consistently install clasp. I sometimes get these errors:

sudo npm i -g @google/clasp
/usr/local/bin/clasp -> /usr/local/lib/node_modules/@google/clasp/index.js

> [email protected] install /usr/local/lib/node_modules/@google/clasp/node_modules/grpc
> node-pre-gyp install --fallback-to-build --library=static_library

node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v57 ABI, unknown) (falling back to source compile with node-gyp)
node-pre-gyp WARN Hit error EACCES: permission denied, mkdir '/usr/local/lib/node_modules/@google/clasp/node_modules/grpc/src/node'
gyp WARN EACCES user "root" does not have permission to access the dev dir "/var/root/.node-gyp/8.11.2"
gyp WARN EACCES attempting to reinstall using temporary dev dir "/usr/local/lib/node_modules/@google/clasp/node_modules/grpc/.node-gyp"
gyp WARN EACCES user "root" does not have permission to access the dev dir "/usr/local/lib/node_modules/@google/clasp/node_modules/grpc/.node-gyp/8.11.2"
...

Configs

npm -v
5.6.0
node -v
v8.11.2

Command

sudo npm i -g @google/clasp
/usr/local/bin/clasp -> /usr/local/lib/node_modules/@google/clasp/index.js
+ @google/[email protected]
updated 1 package in 11.211s

sudo npm uninstall -g @google/clasp

Possible Solutions

This seems to not give the errors:
sudo npm i -g @google/clasp --allow-root

Another option would be to remove grpc, but I believe it is needed with a different npm module.

sudo npm i -g @google/clasp --unsafe-perm
seems to always work too.

However, running clasp gives this error still:

/usr/local/lib/node_modules/@google/clasp/node_modules/grpc/src/grpc_extension.js:45
    throw error;
    ^

Error: The gRPC binary module was not installed. This may be fixed by running "npm rebuild"
Original error: Cannot find module '/usr/local/lib/node_modules/@google/clasp/node_modules/grpc/src/node/extension_binary/node-v57-darwin-x64-unknown/grpc_node.node'
    at Object.<anonymous> (/usr/local/lib/node_modules/@google/clasp/node_modules/grpc/src/grpc_extension.js:43:17)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/usr/local/lib/node_modules/@google/clasp/node_modules/grpc/src/client_interceptors.js:145:12)
    at Module._compile (module.js:652:30)

[grant]

grpc is connect to protobufjs (which is probably connected to google APIs)

[grant]

Running sudo npm rebuild in the clasp folder fixes the CLI. Maybe it should be added as a postinstall step.

[grant]

I think grpc is used for StackDriver logs (@google-cloud/logging):

Error: Cannot find module 'grpc'
    at Function.Module._resolveFilename (module.js:547:15)
    at Function.Module._load (module.js:474:25)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/usr/local/lib/node_modules/@google/clasp/node_modules/@google-cloud/common-grpc/src/service.js:26:12)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)

If this is the case, we should remove @google-cloud/logging in favor of googleapis.

[ajlai]

For better installation consistency we should also look into using npm-shrinkwrap as a way to enforce dependencies more strictly.

The recommended use-case for npm-shrinkwrap.json is applications deployed through the publishing process on the registry: for example, daemons and command-line tools intended as global installs or devDependencies.

[campionfellin]

Looks like if we want to ditch @google-cloud/logging we have a couple of options. We can either use Bunyan or Winston, neither of which appear to use grpc just by glancing at their package.json. I don't have much experience with either, so I have no preference.

(Info from here)

[campionfellin]

Ok, I think that I have figured this out:

TLDR; We will continue to use @google-cloud/logging but NOT grpc.

1. Bunyan vs Winston
   Those these both look like great logging libraries, it is not clear that you can use them to read StackDriver logs, just to format and send log entries to StackDriver. Thus we will not use them.

2. Looking at @google-cloud/logging's dependencies, we see that it doesn't need grpc directly, but there are a couple packages it depends on that need grpc. These are:

3. @google-cloud/common-grpc see here

4. google-gaxsee here

5. Now let's take a look at where @google-cloud/logging uses these two packages.

@google-cloud/common-grpc:
logging uses common-grpc for creating and serializing entries.

Neither of which we do (since we just read logs).

google-gax: logging uses google-gax pretty extensively in this file, this file, and this file. Additionally, it looks like google-gax uses grpc pretty extensively in each of these files.

1. So, as it turns out, though @google-cloud/common-grpc depends on grpc, we don't really depend on @google-cloud/common-grpc because we are not writing or serializing logs. However, it seems like logging pretty heavily depends on google-gax, which definitely depends on grpc

So what does this mean?

1. It seems like if we want to use @google-cloud/logging, we need grpc because we need google-gax which depends and uses grpc.

However, what if we don't actually need `grpc? Let's find out.

• mkdir remove-grpc && cd remove-grpc
• git clone https://github.com/google/clasp && cd clasp
• cat package.json | grep grpc
"grpc": "^1.10.0",
• sudo npm uninstall --save grpc
• cat package.json | grep grpc
  (no output)
• code . (to open VSCode, alternatively use whatever editor you like).
• open integrated terminal
• sudo npm install (installs all packages, notably grpc is not one of them)
• open package-lock.json and ctrl+f grpc (there should be 9 occurrences.
• Remove line 41 ("grpc": "^1.10.0",) which is listed as a required module for @google-cloud/common-grpc
• Remove line 2012 ("grpc": "^1.10.0",) which is listed as a required module for google-gax
• Remove lines 2127-2558. This is grpc itself, which was put in the package-lock.json because it was depended on by other modules.
• There should now be 4 occurrences of grpc, one in line 32, two in line 34, and one in line 91.
• Now back to your integrated terminal, run sudo npm run build to build clasp (without grpc)
• Open up another terminal window and navigate to some temporary working directory.
• Run clasp login to login and then clasp clone <scriptId>. Make sure that this script has logs (if not, then make a new script and run it a few times, and log to Stackdriver with console.log
• Run clasp logs and it'll tell you to set it up with your projectId. Do that.
• Now run clasp logs and wait a minute or so...
  You should see your logs, mine look like:

DEBUG      Sat May 26 2018 10:56:33 GMT-0700 (PDT) myFunction      HELLO WORLD
DEBUG      Sat May 26 2018 10:56:32 GMT-0700 (PDT) myFunction      HELLO WORLD
DEBUG      Sat May 26 2018 10:56:32 GMT-0700 (PDT) myFunction      HELLO WORLD
DEBUG      Sat May 26 2018 10:56:31 GMT-0700 (PDT) myFunction      HELLO WORLD
DEBUG      Sat May 26 2018 10:56:30 GMT-0700 (PDT) myFunction      HELLO WORLD

Cool, it works! Now let's double check to make sure that it doesn't affect anything else.

• Go back to your IDE, and open up /src/test.ts file and find and replace all describe.skip with describe.
• Now rebuild sudo npm run build
• And test npm run test
  Note: a few tests should fail (I think they are in the wrong order, and they've been failing for a while). These tests are one of the clasp push, clasp deployments, clasp deploy, clasp version, and clasp versions Other than that they should all pass.

So now what? Well, it doesn't seem like we need grpc, but it is kind of an awful process to remove. It'd be nice if there was a better way...

@grant

[grant]

Thanks for investigating!
I think we shouldn't use any @google-cloud/ libraries. It brings another set of dependencies and is duplicative of the googleapis library. I think those libraries are better for a server environment and not a quick command line interface.

I'd want to explore a solution that uses googleapis and doesn't use @google-cloud/*. If that doesn't work, we can follow the above progress.

https://github.com/google/google-api-nodejs-client/blob/82920f4731eb16140f8bb7fabc9e06a3777a1d65/src/apis/logging/v2.ts

[campionfellin]

Ah, I did not realize googleapis has a logging client. I will check it out and see what we can do. Thanks!