Clasp: `clasp push` crashes with "JavaScript heap out of memory"

[DominikPalo]

Expected Behavior

Files are pushed to the server

Actual Behavior

clasp push crashes with the error "JavaScript heap out of memory".

The only way how to push files is by increasing the Node.js "max old space" before pushing (using the env variable NODE_OPTIONS=--max_old_space_size=4096), but it still takes a lot of time and resources (100% CPU for ~5 minutes) although I'm pushing only 4 files - 2 HTML files, appsscript.json and app.ts (apps script file). Sometimes even my Chrome browser or Slack app crashes during the push phase.

Steps to Reproduce the Problem

1. Execute clasp push (with default Nodejs settings)
2. Wait for a few seconds

Specifications

• Node version (node -v): 8.15.0, 10.13.0
• Version (clasp -v): 2.0.1
• OS (Mac/Linux/Windows): macOS, Windows

Error stack trace:

\ Pushing files...
<--- Last few GCs --->

[24799:0x103000000]    23791 ms: Mark-sweep 1389.4 (1430.1) -> 1389.3 (1432.6) MB, 45.8 / 0.3 ms  allocation failure GC in old space requested
[24799:0x103000000]    23855 ms: Mark-sweep 1389.3 (1432.6) -> 1389.3 (1417.1) MB, 63.6 / 0.1 ms  last resort GC in old space requested
[24799:0x103000000]    23908 ms: Mark-sweep 1389.3 (1417.1) -> 1389.3 (1417.1) MB, 52.8 / 0.1 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1364749a58b9 <JSObject>
    1: stringSlice(aka stringSlice) [buffer.js:560] [bytecode=0x1364f07ed4f9 offset=94](this=0x1364f56822d1 <undefined>,buf=0x136467284301 <Uint8Array map = 0x1364db3c51d1>,encoding=0x1364749b5b89 <String[4]: utf8>,start=0,end=1118012)
    2: oncomplete(aka readFileAfterClose) [fs.js:~489] [pc=0x384d35db50ad](this=0x1364fc077821 <FSReqWrap map = 0x13645f063ba9>,err=0x1364f5682201 <null>)

==== D...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/macbook/.nvm/versions/node/v8.15.0/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/macbook/.nvm/versions/node/v8.15.0/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/macbook/.nvm/versions/node/v8.15.0/bin/node]
 4: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/Users/macbook/.nvm/versions/node/v8.15.0/bin/node]
 5: v8::internal::Factory::NewStringFromUtf8(v8::internal::Vector<char const>, v8::internal::PretenureFlag) [/Users/macbook/.nvm/versions/node/v8.15.0/bin/node]
 6: v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [/Users/macbook/.nvm/versions/node/v8.15.0/bin/node]
 7: node::StringBytes::Encode(v8::Isolate*, char const*, unsigned long, node::encoding, v8::Local<v8::Value>*) [/Users/macbook/.nvm/versions/node/v8.15.0/bin/node]
 8: void node::Buffer::(anonymous namespace)::StringSlice<(node::encoding)1>(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/macbook/.nvm/versions/node/v8.15.0/bin/node]
 9: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [/Users/macbook/.nvm/versions/node/v8.15.0/bin/node]
10: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/Users/macbook/.nvm/versions/node/v8.15.0/bin/node]
11: v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/Users/macbook/.nvm/versions/node/v8.15.0/bin/node]
12: 0x384d35c042fd
13: 0x384d35cbedb6
Abort trap: 6

[grant]

What's in your clasp status? Are you pushing node_modules?

[DominikPalo]

clasp status returns a huge list of files (including all files from node_modules and all my .ts/.tsx files from the project). 😮 But it seemsclasp push actually pushes only 4 files - here is the output from clasp push:

└─ app.ts
└─ appsscript.json
└─ build/index.html
└─ build/tutorial.html
Pushed 4 files.

My .claspignore looks

**/**
!app.ts
!build/*.html
!appsscript.json

so if I understand correctly, only app.ts, appsscript.json and all HTML files from the build directory should be pushed.

BTW - also the clasp status command is crashing with "JavaScript heap out of memory" (when using default Nodejs max old space size).

[takanakahiko]

In currently clasp code, process of get files call forEach to all files of rootDir(default: ./).
So, clasp status and clasp push is very heavy even if you setting claspignore.

I will devise a process to avoid this, and create a PullRequest.

[grant]

Yes, there are some performance issues that could be fixed and are especially prevalent with npm/node_modules/.

Some folks use the rootDir feature to avoid this.

PRs are welcome.