Ncc: error when building firebase-admin

[markozxuu]

ncc version: 0.16.1

Steps to reproduce

git clone https://github.com/MAPESO/ncc-bug-firebase
cd ncc-bug-firebase
yarn install  # dependency installation
ncc build index.js   # build in single file
node dist/index.js  # note the error output

logs of build

ncc: Module directory "/Users/mark/ncc-bug-firebase/node_modules/write-file-atomic" attempted to require "worker _threads" but could not be resolved, assuming external

ncc: Module directory "/Users/mark/ncc-bug-firebase/node_modules/hash-stream-validation" attempted to require "fast-crc32c" but could not be resolved, assuming external.

ncc: Module directory "/Users/mark/ncc-bug-firebase/node_modules/retry-request" attempted to require "request" but could not be resolved, assuming external.

ncc: Module directory "/Users/mark/ncc-bug-firebase/node_modules/write-file-atomic" attempted to require "worker _threads" but could not be resolved, assuming external.

ncc: Module directory "/Users/mark/ncc-bug-firebase/node_modules/debug/src" attempted to require "supports color" but could not be resolved, assuming external.

ncc: Module directory "/Users/mark/ncc-bug-firebase/node_modules/node-fetch/lib" attempted to require "encoding" but could not be resolved, assuming external.

ncc: Module directory "/Users/mark/ncc-bug-firebase/node_modules/bytebuffer/dist" attempted to require "memcpy" but could not be resolved, assuming external.

(node:41366) UnhandledPromiseRejectionWarning: Error [ERR_UNHANDLED_ERROR]: Unhandled error. (/Users/mark/ncc-bu
g-firebase/google/api)
    at EventEmitter.emit (events.js:171:17)
    at EventEmitter.<anonymous> (/Users/mark/ncc-bug-firebase/dist/index.js:187929:15)
    at Object.onceWrapper (events.js:273:13)
    at EventEmitter.emit (events.js:182:13)
    at fn (/Users/mark/ncc-bug-firebase/dist/index.js:187764:17)
    at statter (/Users/mark/ncc-bug-firebase/dist/index.js:187802:7)
    at walkdir (/Users/mark/ncc-bug-firebase/dist/index.js:187933:3)
    at Function.module.exports.walkdir.sync (/Users/mark/ncc-bug-firebase/dist/index.js:187679:10)
    at module.exports.COMMON_PROTO_DIRS.map.dir (/Users/mark/ncc-bug-firebase/dist/index.js:51861:17)
    at Array.map (<anonymous>)
(node:41366) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by thro
wing inside of an async function without a catch block, or by rejecting a promise which was not handled with .ca
tch(). (rejection id: 1)
(node:41366) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise r
ejections that are not handled will terminate the Node.js process with a non-zero exit code.

we can extract as something interesting is this phrase "but could not be resolved, assuming external". ncc does not offer much detail but I feel that these modules are causing the problem in some way

this is curious because that route exists but not in that order

(node:41366) UnhandledPromiseRejectionWarning: Error [ERR_UNHANDLED_ERROR]: Unhandled error. (/Users/mark/ncc-bug-firebase/google/api)

the correct order would be Users/mark/ncc-bug-firebase/dist/protos/google/api

result of index.js 51861

return walk.sync(dir)

result of index.js 187679

  return walkdir(path,options,eventHandler);

result of index.js 187764

emitter.emit('fail',path,err);

result of index.js 187802

statAction(ex,stat);

result of index.js 187929

emitter.emit('error',path,err);

result of index.js 187933

statter(path,1);

[guybedford]

The underlying error here is in google-gax, in this file - https://unpkg.com/[email protected]/build/src/grpc.js.

The code that is causing the problem is:

// INCLUDE_DIRS is passed to @grpc/proto-loader
const INCLUDE_DIRS = [];
INCLUDE_DIRS.push(googleProtoFilesDir);
// COMMON_PROTO_FILES logic is here for protobufjs loads (see
// GoogleProtoFilesRoot below)
const COMMON_PROTO_DIRS = [
    // This list of directories is defined here:
    // https://github.com/googleapis/googleapis/blob/master/gapic/packaging/common_protos.yaml
    'api',
    path.join('iam', 'v1'),
    path.join('logging', 'type'),
    path.join('monitoring', 'v3'),
    'longrunning',
    'protobuf',
    'rpc',
    'type',
].map(dir => path.join(googleProtoFilesDir, 'google', dir));
INCLUDE_DIRS.push(...COMMON_PROTO_DIRS);
const COMMON_PROTO_FILES = COMMON_PROTO_DIRS
    .map(dir => {
    return walk.sync(dir)
        .filter(f => path.extname(f) === '.proto')
        .map(f => path.normalize(f).substring(googleProtoFilesDir.length + 1));
})
    .reduce((a, c) => a.concat(c), []);

where walk.sync is trying to do a dynamic directory walk on a dynamically known directory.

There is no general analysis fix unfortunately here, so like the express templates case, we need to work out what is a specific fix we can inline for this exact package to make things work out ok.

If I run this on my machine, I get COMMON_PROTO_FILES containing:

[ 'google/api/annotations.proto',
  'google/api/auth.proto',
  'google/api/backend.proto',
  'google/api/billing.proto',
  'google/api/client.proto',
  'google/api/config_change.proto',
  'google/api/consumer.proto',
  'google/api/context.proto',
  'google/api/control.proto',
  'google/api/distribution.proto',
  'google/api/documentation.proto',
  'google/api/endpoint.proto',
  'google/api/experimental/authorization_config.proto',
  'google/api/experimental/experimental.proto',
  'google/api/expr/v1alpha1/cel_service.proto',
  'google/api/expr/v1alpha1/checked.proto',
  'google/api/expr/v1alpha1/conformance_service.proto',
  'google/api/expr/v1alpha1/eval.proto',
  'google/api/expr/v1alpha1/explain.proto',
  'google/api/expr/v1alpha1/syntax.proto',
  'google/api/expr/v1alpha1/value.proto',
  'google/api/expr/v1beta1/decl.proto',
  'google/api/expr/v1beta1/eval.proto',
  'google/api/expr/v1beta1/expr.proto',
  'google/api/expr/v1beta1/source.proto',
  'google/api/expr/v1beta1/value.proto',
  'google/api/field_behavior.proto',
  'google/api/http.proto',
  'google/api/httpbody.proto',
  'google/api/label.proto',
  'google/api/launch_stage.proto',
  'google/api/log.proto',
  'google/api/logging.proto',
  'google/api/metric.proto',
  'google/api/monitored_resource.proto',
  'google/api/monitoring.proto',
  'google/api/quota.proto',
  'google/api/resource.proto',
  'google/api/service.proto',
  'google/api/servicecontrol/v1/check_error.proto',
  'google/api/servicecontrol/v1/distribution.proto',
  'google/api/servicecontrol/v1/log_entry.proto',
  'google/api/servicecontrol/v1/metric_value.proto',
  'google/api/servicecontrol/v1/operation.proto',
  'google/api/servicecontrol/v1/quota_controller.proto',
  'google/api/servicecontrol/v1/service_controller.proto',
  'google/api/servicemanagement/v1/resources.proto',
  'google/api/servicemanagement/v1/servicemanager.proto',
  'google/api/source_info.proto',
  'google/api/system_parameter.proto',
  'google/api/usage.proto',
  'google/iam/v1/iam_policy.proto',
  'google/iam/v1/logging/audit_data.proto',
  'google/iam/v1/policy.proto',
  'google/logging/type/http_request.proto',
  'google/logging/type/log_severity.proto',
  'google/monitoring/v3/alert.proto',
  'google/monitoring/v3/alert_service.proto',
  'google/monitoring/v3/common.proto',
  'google/monitoring/v3/dropped_labels.proto',
  'google/monitoring/v3/group.proto',
  'google/monitoring/v3/group_service.proto',
  'google/monitoring/v3/metric.proto',
  'google/monitoring/v3/metric_service.proto',
  'google/monitoring/v3/mutation_record.proto',
  'google/monitoring/v3/notification.proto',
  'google/monitoring/v3/notification_service.proto',
  'google/monitoring/v3/span_context.proto',
  'google/monitoring/v3/uptime.proto',
  'google/monitoring/v3/uptime_service.proto',
  'google/longrunning/operations.proto',
  'google/protobuf/any.proto',
  'google/protobuf/api.proto',
  'google/protobuf/descriptor.proto',
  'google/protobuf/duration.proto',
  'google/protobuf/empty.proto',
  'google/protobuf/field_mask.proto',
  'google/protobuf/source_context.proto',
  'google/protobuf/struct.proto',
  'google/protobuf/timestamp.proto',
  'google/protobuf/type.proto',
  'google/protobuf/util/json_format.proto',
  'google/protobuf/util/json_format_proto3.proto',
  'google/protobuf/wrappers.proto',
  'google/rpc/code.proto',
  'google/rpc/error_details.proto',
  'google/rpc/status.proto',
  'google/type/calendar_period.proto',
  'google/type/color.proto',
  'google/type/date.proto',
  'google/type/dayofweek.proto',
  'google/type/expr.proto',
  'google/type/fraction.proto',
  'google/type/latlng.proto',
  'google/type/money.proto',
  'google/type/postal_address.proto',
  'google/type/quaternion.proto',
  'google/type/timeofday.proto' ]

Reading https://github.com/googleapis/googleapis/blob/master/gapic/packaging/common_protos.yaml, it seems the list above is actually fixed by package version of the google-proto-files package, so that one fix could simply be to do the following transformation:

• When processing google-gax/build/src/grpc.js, replace the definition of COMMON_PROTO_FILES to simply be the inlined list above, provided the version of google-gax matches our current major.
• We then need to make sure we keep the above list in sync with what is defined in google-proto-files. One option might be to actually build this list in the ncc build itself with this package as a dependency to make sure it stays up-to-date.

I just tested out the above code replacement manually but unfortunately that then gives me a new error:

Error: no such type: Document
    at Service.lookupType (/node_modules/protobufjs/src/namespace.js:382:1)
    at Method.resolve (/node_modules/protobufjs/src/method.js:148:1)
    at Service.resolveAll (/node_modules/protobufjs/src/service.js:111:1)
    at getProtobufServiceAttrs (/node_modules/grpc/src/protobuf_js_6_common.js:106:1)
    at loadObject (/node_modules/grpc/src/protobuf_js_6_common.js:134:1)
    at values.forEach.nested (/node_modules/grpc/src/protobuf_js_6_common.js:143:1)
    at Array.forEach (<anonymous>)
    at loadObject (/node_modules/grpc/src/protobuf_js_6_common.js:142:1)
    at values.forEach.nested (/node_modules/grpc/src/protobuf_js_6_common.js:143:1)
    at Array.forEach (<anonymous>)

which unfortunately doesn't give a good stack trace but seems like it would be a dynamic require issue in dynamically requesting a .proto file, since the standard dynamic require analysis issues would apply there.

Unfortunately we still haven't made much progress on dynamic require approaches though, and would need to start with a much simpler case to work up to this sort of a problem.

So for now, unfortunately, solving this firebase-admin case remains elusive.

[markozxuu]

@guybedford I really appreciate the research you just did, thank you very much. I'll have to resort to mongoDB Atlas as serverless db.

[markozxuu]

@rauchg That now and now-dev are not compatible with firestore will be a problem. Users will not have many options when using DB serverless :|

[guybedford]

I finally managed to find a fix for this in https://github.com/zeit/webpack-asset-relocator-loader/pull/12.

[markozxuu]

@guybedford firebase-admin works perfectly usingncc but using now-dev I get the following error Error: package.json does not exist at /private

this error is related to this issue #306

[styfle]

If it works with ncc, then this is not the appropriate place for the issue.

Please discuss with TooTallNate.

[Yuripetusko]

I am using ncc to build my ts api project which i later upload to AWS serverless lambda. I do use firebase-admin and custom node10 runtime on lambda. After executing my function I get following error:

"errorMessage": "/var/task/lib/src/node/extension_binary/node-v64-darwin-x64-unknown/grpc_node1.node: invalid ELF header",

is it related to ncc? Let me know if I need to create a separate issue

[styfle]

@Yuripetusko This error looks like you compiled on Mac and then ran on Linux. Perhaps you should try deploying to Now so that your build environment matches your run-time environment.

[Yuripetusko]

Thanks @styfle

[diogoviannaaraujo]

I still get this error when deploying on now, when using now-cli everything goes ok, but on now i get:

ERROR   Unhandled rejection: { Error: ENOENT: no such file or directory, open 'google/protobuf/api.proto'
    at Object.openSync (fs.js:443:3)
    at Object.readFileSync (fs.js:343:35)
    at fetch (/var/task/page.js:32228:34)
    at Root.load (/var/task/page.js:32262:13)
    at Root.loadSync (/var/task/page.js:32303:17)
    at Object.loadSync (/var/task/page.js:46295:17)
    at Object.8ZNE (/var/task/page.js:34417:37)
    at __webpack_require__ (/var/task/page.js:23:31)
    at Object.<anonymous> (/var/task/page.js:128795:25)
    at Object.b1/w (/var/task/page.js:129054:30)
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: 'google/protobuf/api.proto' }

[styfle]

Hi @diogoviannaaraujo

The @now/node builder no longer uses ncc.

Can you make sure you using the latest @now/node in your now.json and update the CLI to the latest version by running npm i -g now@latest?

[diogoviannaaraujo]

Hi,

I was using @now/next and got to work by using next@canary dependency
Thanks

On Aug 19, 2019, at 9:58 AM, Steven [e-mail] wrote:

Hi @diogoviannaaraujo

The @now/node builder no longer uses ncc.

Can you make sure you using the latest @now/node in your now.json and update the CLI to the latest version by running npm i -g now@latest?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.