Clasp: .claspignore fails **/node_modules/**

[nmccready]

Expected Behavior

**/node_modules/**
**/**

Ignores all files in node_modules .

Actual Behavior

 follow-redirects options { protocol: 'https:',
  maxRedirects: 21,
  maxBodyLength: 10485760,
  path: '/v1/projects/SCRUBBED/content',
  method: 'put',
  headers:
   { Accept: 'application/json, text/plain, */*',
     'Content-Type': 'application/json;charset=utf-8',
     Authorization: 'Bearer SCRUBBED',
     'User-Agent': 'google-api-nodejs-client/1.3.1',
     'Content-Length': 3879 },
  agent: undefined,
  auth: undefined,
  hostname: 'script.googleapis.com',
  port: null,
  nativeProtocols:
   { 'http:':
      { _connectionListener: [Function: connectionListener],
        METHODS: [Array],
        STATUS_CODES: [Object],
        Agent: [Object],
        ClientRequest: [Object],
        globalAgent: [Object],
        IncomingMessage: [Object],
        OutgoingMessage: [Object],
        Server: [Object],
        ServerResponse: [Object],
        createServer: [Function: createServer],
        get: [Function: get],
        request: [Function: request] },
     'https:':
      { Server: [Object],
        createServer: [Function: createServer],
        globalAgent: [Object],
        Agent: [Object],
        request: [Function: request],
        get: [Function: get] } } } +0ms
Push failed. Errors:
Invalid value at 'files[2].type' (TYPE_ENUM), "D"

node_modules/fsevents/build/Release/.deps/Release/.node.d Is getting picked up with the following error.

Added logging to clasp produces:

{ name: 'node_modules/fsevents/build/Release/.deps/Release/.node.d' }
{ name: 'node_modules/fsevents/build/Release/.deps/Release/.node',
  type: 'D',
....}

from adding:

if (getAPIFileType(name) && !anymatch(ignorePatterns, name)) {
                                                nonIgnoredFilePaths.push(name);
                                                var file = {
                                                    name: formattedName,
                                                    type: getAPIFileType(name),
                                                    source: contents[i] //the file contents
                                                };
                                                console.log({name});
                                                console.log(file);
                                                return file;
                                            }

Only ignore which did work:

node_modules/**
node_modules/fsevents/build/Release/.deps/Release/.node.d
**/**

Specifications

• Node version (node -v) : v8.9.4
• Version (npm list | grep clasp): "@google/[email protected]"
• OS (Mac/Linux/Windows): MAC

Maybe you guys should update or drop anymatch?

[tdooner]

I'm seeing this too. It looks like an upstream bug with anymatch having to do with dotfiles:

> anymatch('**/node_modules/**', '/foo/bar/node_modules/foo')
true
> anymatch('**/node_modules/**', '/foo/bar/node_modules/foo/.dotfile')
false

It's broken in the most recent release of anymatch.

I found a workaround, by adding these claspignore entries, but I imagine YMMV:

node_modules/**/.*
node_modules/**/.*/**

[grant]

Pardon the claspignore bug. I think it's an upstream bug in anymatch which uses micromatch.

It looks like we use 1.3.2, when there's a new version now 2.0.0. The new version probably fixes this.

Does someone want to send a PR that bumps the version and test that it works?

Other notes

• I think @arjun-rao is working on #73 (clasp status) to make debugging this easier.
• Usage of anymatch: https://github.com/google/clasp/blob/master/index.ts#L650
• Also see #14, #58.

[tdooner]

It looks like we use 1.3.2, when there's a new version now 2.0.0. The new version probably fixes this.

It doesn't, I tried the same thing with the latest version of minimatch and it still is broken.

[grant]

So is micromatch broken?
It looks like it has a better API, maybe we can just use that instead.

[nmccready]

What's wrong with glob it uses minimatch also maintined by the same author .

[grant]

glob would also work.

[ajlai]

micromatch/glob/anymatch treat dotfiles differently because default bash glob behavior is not to have the dotglob option enabled: https://www.gnu.org/software/bash/manual/html_node/Filename-Expansion.html

I think though that claspignore behavior should match gitignore behavior instead of bash otherwise we'll run into more situations like this.

@grant, what are your thoughts on using a matching library that matches gitignore behavior (ie: https://www.npmjs.com/package/globby#globbygitignoreoptions) instead?

[grant]

@ajlai globby looks like it could be better than the other options.

I honestly don't know these libraries very well.

Now that we have TravisCI testing working, I think a good first stab at this would to be to create a test that shows that the current ignoring is broken (test clasp status), and than swap the current code with one of these libraries so it's fixed.

[ajlai]

@grant actually just saw that globby is using https://www.npmjs.com/package/ignore internally. We can just use that instead.

TBH I don't know these libraries very well either, was just curious about this bug so I decided to dig in a bit 😄.

Awesome, good to hear that TravisCI is up and running! I'll see if I can take a stab at getting the test case/swap done when I have some time.

[nmccready]

oh look you can experiment globby to your heats content easily now https://github.com/jamiebuilds/globby-cli

[invmatt]

In addition to .claspignore failing to ignore node_modules correctly it's also generating errors with random .md files. I've tried matching against .md */*/.md and a whole bunch of other wildcards and the error:

Invalid value at 'files[9].type' (TYPE_ENUM), "MD"
Invalid value at 'files[10].type' (TYPE_ENUM), "MD"

Still happens no matter what you attempt to ignore.

[eskemojoe007]

Has anyone found a useful workaround for the time being? This is stopping some of my work.

[wildeyes]

clasp still doesn't ignore node_modules (also got me very pissed at clasp, since it was getting stuck on "pushing files" without me knowing it's trying to upload a (as usual) massive node_modules (simply containing clasp and typescript, since I like keeping things local), my workaround was to put my files in a src folder so it can be used with typescript.

[grant]

PRs welcome.
We should at least warn if we're going to try to push 100+ files.

[erickoledadevrel]

For the node_modules case I found success by adding the following three lines to .claspignore:

node_modules/**
node_modules/**/.*/**
node_modules/**/.*

That said, the .claspignore syntax should be made compatible with .gitignore as much as possible.

[grant]

So this issues has been hard to fix since getProjectFiles is a 130 line beast. 👹

https://github.com/google/clasp/blob/9a390c6/src/files.ts#L73

I'll try to refactor the code to make it more maintainable and some more progress here.

[istvandesign]

I can confirm that .claspignore doesn't work at all when trying to ignore .js files.

[SpeedoPasanen]

Looks like I can't get push to ignore anything after updating clasp to latest. It's pushing ./node_modules, ./src which contains a horde of .ts files, etc.

I'm using Webpack to transpile & bundle my files so my .claspignore should ignore everything except dist/** and appsscript.json:

**/**
!dist/**
!appsscript.json

[dustinmichels]

For the node_modules case I found success by adding the following three lines to .claspignore:

node_modules/**
node_modules/**/.*/**
node_modules/**/.*

That said, the .claspignore syntax should be made compatible with .gitignore as much as possible.

@erickoledadevrel 's solution doesn't work for me. It still tries to push my node_modules. Does this work for others? I'm on Windows.

[SpeedoPasanen]

@erickoledadevrel does not work for me. Also on Windows. Tried all combinations, nothing gets ignored since updating clasp.

[dustinmichels]

I deleted the node modules, did a clasp push, npm install-ed them again, and now they seem to be properly ignored. Does that make sense? Does clasp push retain some kind of memory of what it tried/failed to push last time?

[SpeedoPasanen]

I deleted the node modules, did a clasp push, npm install-ed them again, and now they seem to be properly ignored. Does that make sense? Does clasp push retain some kind of memory of what it tried/failed to push last time?

Didn't permanently fix it for me. Did one push while node_modules, src, etc. moved elsewhere, moved them back and tried to push again - it's still not ignoring any files.

Oh well, I guess I'll add a step in my pipeline that copies only the files I want to push, to a temp dir in the parent dir and pushes from there.

[dustinmichels]

Also, you can put all the files you do want to push in a folder (eg, src/ or GAS/) and then add a line in your .clasp.json file, specifying which folder to push.

{
  "scriptId": "1upj...vkY5",
  "rootDir": "GAS"
}

That works pretty well for me, as an alternative to using .claspignore.