Incubator-superset: "JavaScript heap out of memory" when "npm run build"

Created on 5 Jan 2018 · 5Comments · Source: apache/incubator-superset

Make sure these boxes are checked before submitting your issue - thank you!

[yes] I have checked the superset logs for python stacktraces and included it here as text if any
[yes] I have reproduced the issue with at least the latest released version of superset
[yes] I have checked the issue tracker for the same issue and I haven't found one similar

Superset version

latest master

Expected results

When I used version 0.20.1, it took about 5 minutes to finish the command "npm run build". However, after merge code from the latest master, it happens to take 20-30 minutes and finally raise the error "FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory"

Actual results

<--- Last few GCs --->

[116141:0x28f8a20] 772351 ms: Mark-sweep 1400.2 (1560.8) -> 1400.2 (1560.8) MB, 2083.1 / 0.0 ms allocation failure GC in old space requested
[116141:0x28f8a20] 774446 ms: Mark-sweep 1400.2 (1560.8) -> 1400.2 (1559.8) MB, 2095.4 / 0.0 ms last resort gc
[116141:0x28f8a20] 776611 ms: Mark-sweep 1400.2 (1559.8) -> 1400.2 (1559.8) MB, 2164.3 / 0.0 ms last resort gc

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x163814c0d39
2: doit(aka doit) [0x16381404311 :4358] [pc=0xcee7a99e4fd](this=0x16381404311 )
3: print [0x16381404311 :~4352] [pc=0xcee7a96a40c](this=0x2299bbcf5351 ,stream=0x177a7a3a03b9 ,force_parens=0x16381404311 )
4: arguments adaptor frame: 1->2
5: /* anonymous /(aka / an...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x126389c [node]
3: v8::Utils::ReportOOMFailure(char const, bool) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const, bool) [node]
5: v8::internal::Factory::NewOneByteInternalizedString(v8::internal::Vector, unsigned int) [node]
6: v8::internal::AstRawStringInternalizationKey::AsHandle(v8::internal::Isolate) [node]
7: v8::internal::StringTable::LookupKey(v8::internal::Isolate, v8::internal::HashTableKey) [node]
8: v8::internal::AstValueFactory::GetString(unsigned int, bool, v8::internal::Vector) [node]
9: v8::internal::AstValueFactory::GetOneByteStringInternal(v8::internal::Vector) [node]
10: v8::internal::Scanner::NextSymbol(v8::internal::AstValueFactory) [node]
11: v8::internal::ParserBase::ParsePrimaryExpression(v8::internal::ExpressionClassifier >, bool, bool) [node]
12: v8::internal::ParserBase::ParseMemberExpression(v8::internal::ExpressionClassifier >, bool, bool) [node]
13: v8::internal::ParserBase::ParseMemberWithNewPrefixesExpression(v8::internal::ExpressionClassifier >, bool, bool) [node]
14: 0xff8edd [node]
15: v8::internal::ParserBase::ParseLeftHandSideExpression(v8::internal::ExpressionClassifier >, bool) [node]
16: v8::internal::ParserBase::ParsePostfixExpression(v8::internal::ExpressionClassifier >, bool) [node]
17: v8::internal::ParserBase::ParseUnaryExpression(v8::internal::ExpressionClassifier >, bool) [node]
18: v8::internal::ParserBase::ParseBinaryExpression(int, bool, v8::internal::ExpressionClassifier >, bool) [node]
19: v8::internal::ParserBase::ParseConditionalExpression(bool, v8::internal::ExpressionClassifier >, bool) [node]
20: v8::internal::ParserBase::ParseAssignmentExpression(bool, v8::internal::ExpressionClassifier >, bool) [node]
21: v8::internal::ParserBase::ParseArguments(v8::internal::Scanner::Location, bool, v8::internal::ExpressionClassifier >, bool) [node]
22: 0xff9384 [node]
23: v8::internal::ParserBase::ParseLeftHandSideExpression(v8::internal::ExpressionClassifier >, bool) [node]
24: v8::internal::ParserBase::ParsePostfixExpression(v8::internal::ExpressionClassifier >, bool) [node]
25: v8::internal::ParserBase::ParseUnaryExpression(v8::internal::ExpressionClassifier >, bool) [node]
26: v8::internal::ParserBase::ParseBinaryExpression(int, bool, v8::internal::ExpressionClassifier >, bool) [node]
27: v8::internal::ParserBase::ParseConditionalExpression(bool, v8::internal::ExpressionClassifier >, bool) [node]
28: v8::internal::ParserBase::ParseAssignmentExpression(bool, v8::internal::ExpressionClassifier >, bool) [node]
29: v8::internal::ParserBase::ParseArguments(v8::internal::Scanner::Location, bool, v8::internal::ExpressionClassifier >, bool) [node]
30: 0xff9384 [node]
31: v8::internal::ParserBase::ParseLeftHandSideExpression(v8::internal::ExpressionClassifier >, bool) [node]
32: v8::internal::ParserBase::ParsePostfixExpression(v8::internal::ExpressionClassifier >, bool) [node]
33: v8::internal::ParserBase::ParseUnaryExpression(v8::internal::ExpressionClassifier >, bool) [node]
34: v8::internal::ParserBase::ParseBinaryExpression(int, bool, v8::internal::ExpressionClassifier >, bool) [node]
35: v8::internal::ParserBase::ParseConditionalExpression(bool, v8::internal::ExpressionClassifier >, bool) [node]
36: v8::internal::ParserBase::ParseAssignmentExpression(bool, v8::internal::ExpressionClassifier >, bool) [node]
37: v8::internal::ParserBase::ParseExpression(bool, v8::internal::ExpressionClassifier >, bool) [node]
38: v8::internal::ParserBase::ParseExpression(bool, bool) [node]
39: v8::internal::Parser::ParseExpressionOrLabelledStatement(v8::internal::ZoneList, v8::internal::AllowLabelledFunctionStatement, bool) [node]
40: v8::internal::Parser::ParseSubStatement(v8::internal::ZoneList, v8::internal::AllowLabelledFunctionStatement, bool) [node]
41: v8::internal::Parser::ParseBlock(v8::internal::ZoneList, bool) [node]
42: v8::internal::Parser::ParseSubStatement(v8::internal::ZoneList, v8::internal::AllowLabelledFunctionStatement, bool) [node]
43: v8::internal::Parser::ParseScopedStatement(v8::internal::ZoneList, bool, bool) [node]
44: v8::internal::Parser::ParseIfStatement(v8::internal::ZoneList, bool) [node]
45: v8::internal::Parser::ParseSubStatement(v8::internal::ZoneList, v8::internal::AllowLabelledFunctionStatement, bool) [node]
46: v8::internal::Parser::ParseBlock(v8::internal::ZoneList, bool) [node]
47: v8::internal::Parser::ParseSubStatement(v8::internal::ZoneList, v8::internal::AllowLabelledFunctionStatement, bool) [node]
48: v8::internal::Parser::ParseScopedStatement(v8::internal::ZoneList, bool, bool) [node]
49: v8::internal::Parser::ParseIfStatement(v8::internal::ZoneList, bool) [node]
50: v8::internal::Parser::ParseSubStatement(v8::internal::ZoneList, v8::internal::AllowLabelledFunctionStatement, bool) [node]
51: v8::internal::Parser::ParseBlock(v8::internal::ZoneList, bool) [node]
52: v8::internal::Parser::ParseSubStatement(v8::internal::ZoneList, v8::internal::AllowLabelledFunctionStatement, bool) [node]
53: v8::internal::Parser::ParseScopedStatement(v8::internal::ZoneList, bool, bool) [node]
54: v8::internal::Parser::ParseIfStatement(v8::internal::ZoneList, bool) [node]
55: v8::internal::Parser::ParseSubStatement(v8::internal::ZoneList, v8::internal::AllowLabelledFunctionStatement, bool) [node]
56: v8::internal::Parser::ParseStatementList(v8::internal::ZoneList, int, bool) [node]
57: v8::internal::Parser::ParseEagerFunctionBody(v8::internal::AstRawString const, int, v8::internal::ParserFormalParameters const&, v8::internal::FunctionKind, v8::internal::FunctionLiteral::FunctionType, bool) [node]
58: v8::internal::Parser::ParseFunctionLiteral(v8::internal::AstRawString const, v8::internal::Scanner::Location, v8::internal::FunctionNameValidity, v8::internal::FunctionKind, int, v8::internal::FunctionLiteral::FunctionType, v8::internal::LanguageMode, bool) [node]
59: v8::internal::Parser::DoParseLazy(v8::internal::ParseInfo, v8::internal::AstRawString const, v8::internal::Utf16CharacterStream) [node]
60: v8::internal::Parser::ParseLazy(v8::internal::Isolate, v8::internal::ParseInfo) [node]
61: v8::internal::Parser::Parse(v8::internal::ParseInfo) [node]
62: v8::internal::Parser::ParseStatic(v8::internal::ParseInfo) [node]
63: 0xd0084d [node]
64: 0xd0733f [node]
65: v8::internal::Compiler::Compile(v8::internal::Handle, v8::internal::Compiler::ClearExceptionFlag) [node]
66: v8::internal::Runtime_CompileLazy(int, v8::internal::Object, v8::internal::Isolate) [node]
67: 0xcee765063a7
Aborted (core dumped)

npm ERR! Linux 3.13.0-32-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "build"
npm ERR! node v7.4.0
npm ERR! npm v4.1.1
npm ERR! code ELIFECYCLE
npm ERR! [email protected] build: NODE_ENV=production webpack --colors --progress
npm ERR! Exit status 134
npm ERR!
npm ERR! Failed at the [email protected] build script 'NODE_ENV=production webpack --colors --progress'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the superset package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! NODE_ENV=production webpack --colors --progress
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs superset
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls superset
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! /home/yayi.duan/incubator-superset/superset/assets/npm-debug.log

Steps to reproduce

npm run build

Source

JessicaDuan

All 5 comments

Maybe there are some problems with "deck.gl"?
Everything goes smoothly after "//import deckglFactory from './deckgl/factory';"

JessicaDuan on 5 Jan 2018

You should use yarn to build superset as documented. Works fine for me here:

yarn run build
yarn run v1.3.2
$ NODE_ENV=production webpack --colors --progress
Hash: ca925a504a067dea7f36                                                                
Version: webpack 3.10.0
Time: 104775ms

xrmx on 5 Jan 2018

Could you show more details about your build?

We use yarn to install and npm to run build commands, and it works fine. Webpack may consume lots of memory, so if you build in a virtual machine or inside a docker container with less than 1 GB memory, there maybe an out-of-memory problem.