Google-cloud-node: Datastore 503 error "connect failed"

Created on 31 Jan 2017  ·  32Comments  ·  Source: googleapis/google-cloud-node

With the latest 1.1.0 version datastore save returns an 503 error "connect failed"

Downgrading to grpc 1.0.1 in my project seems to fix the issue.

https://github.com/grpc/grpc/issues/9518

Environment details

  • OS: ubuntu / appengine nodejs
  • Node.js version: 6.9.4
  • npm version: 3.10.10
  • google-cloud-node version:
    "@google-cloud/datastore": "^0.6.1",
    "@google-cloud/storage": "^0.6.1",

Steps to reproduce

datastore save method returns error

bug core
Source
picture dskdas
👍7

Most helpful comment

@dzoba grpc-0.14.1 is way too old. I believe your project is using some older version of @google-cloud/compute -- consider upgrade it to the latest release.

picture jmuk on 1 Feb 2017
🎉2 👍2

All 32 comments

I haven't reproduced the 503, however, I noticed the request hanging in 1.1.0 and coming back instantly in 1.0.1.

To anyone affected by this, npm install google-cloud and access your sub-services through there.

@murgatroid99 -- any ideas what could have caused this? Currently, it looks like we'd have to roll back and force 1.0.1.

Here's repro code:

$ npm init -y
$ npm install --save @google-cloud/datastore
$ touch index.js

'use strict'

var datastore = require('@google-cloud/datastore')({
  projectId: 'nth-circlet-705'
})

datastore.save({
  key: datastore.key(['Kind', 1]),
  data: { test: true }
}, console.log) // never called in [email protected]
stephenplusplus picture stephenplusplus on 31 Jan 2017
👍1

@dskdas thank you for analyzing and reporting this problem. I can confirm that I have the same problem and solution.

I have a question for the developers/maintainers; I have noticed that shrinkwrap is being used for the main google-cloud package, but not for the @google-cloud packages. Is there a specific reason for this? These kind of problems can be prevented if the @google-cloud packages are also shrinkwrapped.

mdebruijne picture mdebruijne on 31 Jan 2017

We shrinkwrap the bundled package so that we can minimize duplicated dependencies for users of non-dedupe-ing npm versions (pre [email protected]). We'd like to not have to shrinkwrap at all, however, it's clear it would have prevented this situation we're in now. The cost is delayed bug fixes from our dependencies and a more complicated release process, which means a slower delivery to the end user. This type of issue is rare, and can be addressed in a short matter of time by us. And users can fix by downgrading or following other specific advice that we share here.

I don't want to squash the idea, but I would prefer we have it in another issue so we can keep this clear for users who might hit the same bug and benefit from related advice. If anyone feels strongly about it, please open a new issue.

stephenplusplus picture stephenplusplus on 31 Jan 2017

I can reproduce this problem. The root cause is a TCP connection failure, though I can't tell yet why it's happening.

murgatroid99 picture murgatroid99 on 31 Jan 2017

I have tracked down the issue, and I will publish [email protected] as soon as our tests pass and we can build the new packages.

murgatroid99 picture murgatroid99 on 31 Jan 2017
🎉3

@stephenplusplus I had a problem in the past with the main google-cloud package due to a bug in node-forge that was resolved upstream but still shrinkwrapped in google-cloud so I would prefer no shrinkwrap as well. I was just wondering if the decision to differentiate between the main google-cloud package and @google-cloud packages was taken deliberately. Thanks for clearing that up.

mdebruijne picture mdebruijne on 31 Jan 2017
👍1

Gotcha, it's only because google-cloud wraps so many modules, and they all share so many common dependencies. npm install google-cloud was downloading 800MB+ :(

stephenplusplus picture stephenplusplus on 31 Jan 2017

I have encountered what I think is a related error when deploying to Heroku. It has killed my build & deploy process.

```

[email protected] install /tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/node_modules/@google-cloud/compute/node_modules/grpc
node-pre-gyp install --fallback-to-build

   node-pre-gyp ERR! Tried to download: https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v0.14.1/node-v51-linux-x64.tar.gz
   node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v51 ABI) (falling back to source compile with node-gyp)
   make: Entering directory `/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/node_modules/@google-cloud/compute/node_modules/grpc/build'
   CC(target) Release/obj.target/grpc/src/core/lib/surface/init.o
   CC(target) Release/obj.target/grpc/src/core/lib/channel/channel_args.o
   CC(target) Release/obj.target/grpc/src/core/lib/channel/channel_stack.o
   CC(target) Release/obj.target/grpc/src/core/lib/channel/channel_stack_builder.o
   CC(target) Release/obj.target/grpc/src/core/lib/channel/compress_filter.o
   CC(target) Release/obj.target/grpc/src/core/lib/channel/connected_channel.o
   CC(target) Release/obj.target/grpc/src/core/lib/channel/http_client_filter.o
   CC(target) Release/obj.target/grpc/src/core/lib/channel/http_server_filter.o
   CC(target) Release/obj.target/grpc/src/core/lib/compression/compression_algorithm.o
   CC(target) Release/obj.target/grpc/src/core/lib/compression/message_compress.o
   CC(target) Release/obj.target/grpc/src/core/lib/debug/trace.o
   CC(target) Release/obj.target/grpc/src/core/lib/http/format_request.o
   CC(target) Release/obj.target/grpc/src/core/lib/http/httpcli.o
   CC(target) Release/obj.target/grpc/src/core/lib/http/parser.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/closure.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/endpoint.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/endpoint_pair_posix.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/endpoint_pair_windows.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/ev_poll_and_epoll_posix.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/ev_posix.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/exec_ctx.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/executor.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/iocp_windows.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/iomgr.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/iomgr_posix.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/iomgr_windows.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/pollset_set_windows.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/pollset_windows.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/resolve_address_posix.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/resolve_address_windows.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/sockaddr_utils.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/socket_utils_common_posix.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/socket_utils_linux.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/socket_utils_posix.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/socket_windows.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/tcp_client_posix.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/tcp_client_windows.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/tcp_posix.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/tcp_server_posix.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/tcp_server_windows.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/tcp_windows.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/time_averaged_stats.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/timer.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/timer_heap.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/udp_server.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/unix_sockets_posix.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/unix_sockets_posix_noop.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/wakeup_fd_eventfd.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/wakeup_fd_nospecial.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/wakeup_fd_pipe.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/wakeup_fd_posix.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/workqueue_posix.o
   CC(target) Release/obj.target/grpc/src/core/lib/iomgr/workqueue_windows.o
   CC(target) Release/obj.target/grpc/src/core/lib/json/json.o
   CC(target) Release/obj.target/grpc/src/core/lib/json/json_reader.o
   CC(target) Release/obj.target/grpc/src/core/lib/json/json_string.o
   CC(target) Release/obj.target/grpc/src/core/lib/json/json_writer.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/alarm.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/api_trace.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/byte_buffer.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/byte_buffer_reader.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/call.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/call_details.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/call_log_batch.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/channel.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/channel_init.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/channel_ping.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/channel_stack_type.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/completion_queue.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/event_string.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/lame_client.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/metadata_array.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/server.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/validate_metadata.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/version.o
   CC(target) Release/obj.target/grpc/src/core/lib/transport/byte_stream.o
   CC(target) Release/obj.target/grpc/src/core/lib/transport/connectivity_state.o
   CC(target) Release/obj.target/grpc/src/core/lib/transport/metadata.o
   CC(target) Release/obj.target/grpc/src/core/lib/transport/metadata_batch.o
   CC(target) Release/obj.target/grpc/src/core/lib/transport/static_metadata.o
   CC(target) Release/obj.target/grpc/src/core/lib/transport/transport.o
   CC(target) Release/obj.target/grpc/src/core/lib/transport/transport_op_string.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/bin_encoder.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/chttp2_plugin.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/frame_data.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/frame_goaway.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/frame_ping.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/frame_rst_stream.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/frame_settings.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/frame_window_update.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/hpack_encoder.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/hpack_parser.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/hpack_table.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/huffsyms.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/incoming_metadata.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/parsing.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/status_conversion.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/stream_lists.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/stream_map.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/timeout_encoding.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/varint.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/transport/writing.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/alpn/alpn.o
   CC(target) Release/obj.target/grpc/src/core/lib/http/httpcli_security_connector.o
   CC(target) Release/obj.target/grpc/src/core/lib/security/b64.o
   CC(target) Release/obj.target/grpc/src/core/lib/security/client_auth_filter.o
   CC(target) Release/obj.target/grpc/src/core/lib/security/credentials.o
   CC(target) Release/obj.target/grpc/src/core/lib/security/credentials_metadata.o
   CC(target) Release/obj.target/grpc/src/core/lib/security/credentials_posix.o
   CC(target) Release/obj.target/grpc/src/core/lib/security/credentials_win32.o
   CC(target) Release/obj.target/grpc/src/core/lib/security/google_default_credentials.o
   CC(target) Release/obj.target/grpc/src/core/lib/security/handshake.o
   CC(target) Release/obj.target/grpc/src/core/lib/security/json_token.o
   CC(target) Release/obj.target/grpc/src/core/lib/security/jwt_verifier.o
   CC(target) Release/obj.target/grpc/src/core/lib/security/secure_endpoint.o
   CC(target) Release/obj.target/grpc/src/core/lib/security/security_connector.o
   CC(target) Release/obj.target/grpc/src/core/lib/security/security_context.o
   CC(target) Release/obj.target/grpc/src/core/lib/security/server_auth_filter.o
   CC(target) Release/obj.target/grpc/src/core/lib/surface/init_secure.o
   CC(target) Release/obj.target/grpc/src/core/lib/tsi/fake_transport_security.o
   CC(target) Release/obj.target/grpc/src/core/lib/tsi/ssl_transport_security.o
   CC(target) Release/obj.target/grpc/src/core/lib/tsi/transport_security.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/client/secure/secure_channel_create.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/channel_connectivity.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/client_channel.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/client_channel_factory.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/client_config.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/client_config_plugin.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/connector.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/default_initial_connect_string.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/initial_connect_string.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/lb_policy.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/lb_policy_factory.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/lb_policy_registry.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/parse_address.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/resolver.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/resolver_factory.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/resolver_registry.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/subchannel.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/subchannel_call_holder.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/subchannel_index.o
   CC(target) Release/obj.target/grpc/src/core/ext/client_config/uri_parser.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/server/insecure/server_chttp2.o
   CC(target) Release/obj.target/grpc/src/core/ext/transport/chttp2/client/insecure/channel_create.o
   CC(target) Release/obj.target/grpc/src/core/ext/lb_policy/grpclb/load_balancer_api.o
   CC(target) Release/obj.target/grpc/src/core/ext/lb_policy/grpclb/proto/grpc/lb/v0/load_balancer.pb.o
   CC(target) Release/obj.target/grpc/third_party/nanopb/pb_common.o
   CC(target) Release/obj.target/grpc/third_party/nanopb/pb_decode.o
   CC(target) Release/obj.target/grpc/third_party/nanopb/pb_encode.o
   CC(target) Release/obj.target/grpc/src/core/ext/lb_policy/pick_first/pick_first.o
   CC(target) Release/obj.target/grpc/src/core/ext/lb_policy/round_robin/round_robin.o
   CC(target) Release/obj.target/grpc/src/core/ext/resolver/dns/native/dns_resolver.o
   CC(target) Release/obj.target/grpc/src/core/ext/resolver/sockaddr/sockaddr_resolver.o
   CC(target) Release/obj.target/grpc/src/core/ext/census/context.o
   CC(target) Release/obj.target/grpc/src/core/ext/census/grpc_context.o
   CC(target) Release/obj.target/grpc/src/core/ext/census/grpc_filter.o
   CC(target) Release/obj.target/grpc/src/core/ext/census/grpc_plugin.o
   CC(target) Release/obj.target/grpc/src/core/ext/census/initialize.o
   CC(target) Release/obj.target/grpc/src/core/ext/census/mlog.o
   CC(target) Release/obj.target/grpc/src/core/ext/census/operation.o
   CC(target) Release/obj.target/grpc/src/core/ext/census/placeholders.o
   CC(target) Release/obj.target/grpc/src/core/ext/census/tracing.o
   CC(target) Release/obj.target/grpc/src/core/plugin_registry/grpc_plugin_registry.o
   AR(target) Release/obj.target/libgrpc.a
   COPY Release/libgrpc.a
   CC(target) Release/obj.target/gpr/src/core/lib/profiling/basic_timers.o
   CC(target) Release/obj.target/gpr/src/core/lib/profiling/stap_timers.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/alloc.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/avl.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/backoff.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/cmdline.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/cpu_iphone.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/cpu_linux.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/cpu_posix.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/cpu_windows.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/env_linux.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/env_posix.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/env_win32.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/histogram.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/host_port.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/load_file.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/log.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/log_android.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/log_linux.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/log_posix.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/log_win32.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/murmur_hash.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/slice.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/slice_buffer.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/stack_lockfree.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/string.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/string_posix.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/string_util_win32.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/string_win32.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/subprocess_posix.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/subprocess_windows.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/sync.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/sync_posix.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/sync_win32.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/thd.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/thd_posix.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/thd_win32.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/time.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/time_posix.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/time_precise.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/time_win32.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/tls_pthread.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/tmpfile_msys.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/tmpfile_posix.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/tmpfile_win32.o
   CC(target) Release/obj.target/gpr/src/core/lib/support/wrap_memcpy.o
   AR(target) Release/obj.target/libgpr.a
   COPY Release/libgpr.a
   CXX(target) Release/obj.target/grpc_node/src/node/ext/byte_buffer.o
   ../src/node/ext/byte_buffer.cc: In function ‘v8::Local<v8::Value> grpc::node::MakeFastBuffer(v8::Local<v8::Value>)’:
   ../src/node/ext/byte_buffer.cc:100:72: warning: ‘v8::Local<v8::Object> v8::Function::NewInstance(int, v8::Local<v8::Value>*) const’ is deprecated (declared at /app/.node-gyp/7.4.0/include/node/v8.h:3287): Use maybe version [-Wdeprecated-declarations]
   Local<Object> fastBuffer = bufferConstructor->NewInstance(3, consArgs);
   ^
   CXX(target) Release/obj.target/grpc_node/src/node/ext/call.o
   ../src/node/ext/call.cc: In static member function ‘static Nan::NAN_METHOD_RETURN_TYPE grpc::node::Call::New(Nan::NAN_METHOD_ARGS_TYPE)’:
   ../src/node/ext/call.cc:616:20: error: ‘class v8::Object’ has no member named ‘SetHiddenValue’
   info.This()->SetHiddenValue(Nan::New("channel_").ToLocalChecked(),
   ^
   ../src/node/ext/call.cc:625:19: warning: ‘v8::Local<v8::Object> v8::Function::NewInstance(int, v8::Local<v8::Value>*) const’ is deprecated (declared at /app/.node-gyp/7.4.0/include/node/v8.h:3287): Use maybe version [-Wdeprecated-declarations]
   argc, argv);
   ^
   make: *** [Release/obj.target/grpc_node/src/node/ext/call.o] Error 1
   make: Leaving directory `/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/node_modules/@google-cloud/compute/node_modules/grpc/build'
   gyp ERR! build error
   gyp ERR! stack Error: `make` failed with exit code: 2
   gyp ERR! stack     at ChildProcess.onExit (/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/.heroku/node/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:276:23)
   gyp ERR! stack     at emitTwo (events.js:106:13)
   gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
   gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
   gyp ERR! System Linux 3.13.0-105-generic
   gyp ERR! command "/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/.heroku/node/bin/node" "/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/.heroku/node/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--module=/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/node_modules/@google-cloud/compute/node_modules/grpc/src/node/extension_binary/grpc_node.node" "--module_name=grpc_node" "--module_path=/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/node_modules/@google-cloud/compute/node_modules/grpc/src/node/extension_binary"
   gyp ERR! cwd /tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/node_modules/@google-cloud/compute/node_modules/grpc
   gyp ERR! node -v v7.4.0
   gyp ERR! node-gyp -v v3.4.0
   gyp ERR! not ok
   node-pre-gyp ERR! build error
   node-pre-gyp ERR! stack Error: Failed to execute '/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/.heroku/node/bin/node /tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/.heroku/node/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/node_modules/@google-cloud/compute/node_modules/grpc/src/node/extension_binary/grpc_node.node --module_name=grpc_node --module_path=/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/node_modules/@google-cloud/compute/node_modules/grpc/src/node/extension_binary' (1)
   node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/node_modules/@google-cloud/compute/node_modules/grpc/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
   node-pre-gyp ERR! stack     at emitTwo (events.js:106:13)
   node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
   node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:885:16)
   node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)
   node-pre-gyp ERR! System Linux 3.13.0-105-generic
   node-pre-gyp ERR! command "/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/.heroku/node/bin/node" "/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/node_modules/@google-cloud/compute/node_modules/grpc/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
   node-pre-gyp ERR! cwd /tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/node_modules/@google-cloud/compute/node_modules/grpc
   node-pre-gyp ERR! node -v v7.4.0
   node-pre-gyp ERR! node-pre-gyp -v v0.6.28
   node-pre-gyp ERR! not ok
   Failed to execute '/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/.heroku/node/bin/node /tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/.heroku/node/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/node_modules/@google-cloud/compute/node_modules/grpc/src/node/extension_binary/grpc_node.node --module_name=grpc_node --module_path=/tmp/build_0a211b45cbd6681b8d427f6cfaaeb0d3/node_modules/@google-cloud/compute/node_modules/grpc/src/node/extension_binary' (1)```
dzoba picture dzoba on 1 Feb 2017

@dzoba grpc-0.14.1 is way too old. I believe your project is using some older version of @google-cloud/compute -- consider upgrade it to the latest release.

jmuk picture jmuk on 1 Feb 2017
🎉2 👍2

Our v1.1.x branch now contains the fix to the bug, and the package will be published shortly (probably later today, definitely by early tomorrow). I have verified that it works with the provided reproduction case, but it would be very helpful if you guys could verify that it works more generally.

murgatroid99 picture murgatroid99 on 1 Feb 2017

I have now published [email protected]. This problem should now be fixed.

murgatroid99 picture murgatroid99 on 1 Feb 2017

thank @murgatroid99 Just to confirm that it works for me on my machine, but it is not working on app engine

sebastianovide picture sebastianovide on 1 Feb 2017

Please try upgrading to the latest version of google-cloud or @google-cloud/datastore.

stephenplusplus picture stephenplusplus on 2 Feb 2017

I tested that datastore save works fine with the latest 1.1.1 grpc version (pulled in by @google-cloud/datastore).

Thanks

dskdas picture dskdas on 2 Feb 2017

@stephenplusplus, I just need some clarification. I upgraded locally to @google-cloud/datastore and it seems to be working now.

However, whenever I deploy with gcloud app deploy, it seems to be having the same issue as before with the datastore timing out. Is my deployed version caching the old package somehow?

kbyatnal picture kbyatnal on 2 Feb 2017

That could definitely be the case. Is it possible to clear out the cache?

stephenplusplus picture stephenplusplus on 2 Feb 2017

I am also having the same issue on app engine. Investigating...

dskdas picture dskdas on 2 Feb 2017
👍3

same here... just to confirm that it is still broken on app engine flex.

sebastianovide picture sebastianovide on 3 Feb 2017
👍1

We found that [email protected] fixed the issues when running locally on Mac OS X and in a Fedora 25 virtual machine, but the issue still seems to occur when running in an ubuntu based docker image.

We're using:

When running our app with GRPC_VERBOSITY=DEBUG npm start we continuously get these errors printed out at one per second:

I0203 16:19:44.762300262     631 dns_resolver.c:192]         dns resolution failed (will retry): {"created":"@1486138784.762281118","description":"getaddrinfo failed","file":"../src/core/lib/iomgr/resolve_address_uv.c","file_line":66,"os_error":"service not available for socket type"}

And our subscriber callback is never called:

  pubsub.subscribe(topicName, subscriptionName, {
    ackDeadlineSeconds: 60
  }, (err, subscription, apiResponse) => {
    // THIS NEVER RUNS
    if (err) return callback(err)
    subscription.on('message', event => {
      // do stuff with event
    })
  })
djbeaumont picture djbeaumont on 3 Feb 2017
👍2

I would like to investigate that DNS error. Do you have a small complete script that reproduces this error?

murgatroid99 picture murgatroid99 on 3 Feb 2017

@murgatroid99 https://github.com/GoogleCloudPlatform/google-cloud-node/issues/1946#issuecomment-276398600

@djbeaumont I've just deployed on Heroku which is based on Ubuntu and it works..... (and it deploys in a fraction of the time...)

sebastianovide picture sebastianovide on 3 Feb 2017

I tried running the reproduction case in that comment in a Fedora 25 docker image, and I successfully got a response with no DNS errors using both Node 6 and Node 7.

murgatroid99 picture murgatroid99 on 3 Feb 2017

I just managed to reproduce it in a clean repository set aside from my company's libraries. Just writing up some instructions now.

djbeaumont picture djbeaumont on 3 Feb 2017

@murgatroid99, I've put an example up here: https://github.com/djbeaumont/grpc-issue
Please let me know if you've got any questions.

djbeaumont picture djbeaumont on 3 Feb 2017

OK, when I run that, I see exactly the errors that you described.

murgatroid99 picture murgatroid99 on 4 Feb 2017

I believe I have found a solution (or at least a workaround) for the DNS error you are experiencing in the Ubuntu docker image: in the Dockerfile, add the line RUN apt-get install netbase. Among other things, netbase includes the file /etc/services, which is required to resolve the https service to the port 443.

murgatroid99 picture murgatroid99 on 6 Feb 2017
👍1

Adding that to our docker image works around the problem for me too 👍

djbeaumont picture djbeaumont on 6 Feb 2017

I also noticed that all of my (remote) deployed google cloud functions stopped being able to store data to google cloud datastore about a week ago. At that time, cloud console was logging a grpc network error. After the grpc 1.1.1 update, the network error is no longer logged, but data is still not being stored to cloud datastore for remotely deployed functions. (Note Locally deployed cloud functions do store data to the remote cloud datastore.) Bundling the node_modules directory with the remotely deployed functions doesn't solve the problem either. ((With package.json dependencies: "@google-cloud/datastore": "^0.6.1" and "grpc": "^1.1.1") When the node_modules directory is included with the remote deployed function, the cloud console log shows a "function timeout" error. When the node_modules directory is not included, the cloud function fails (times-out) silently.

It kinda seems like the google cloud functions network is not able to see the google datastore network. Since my local dev server can see both networks, it doesn't have the same problem.

Any fix update or insight would be greatly appreciated. / Thanks.

tobeach picture tobeach on 6 Feb 2017
👍2

I had the same problem then @tobeach.

ert78gb picture ert78gb on 7 Feb 2017

@ofrobots:
Hmmm... there is no issue when the cloud function is run on a local server with the same source tree. (i.e no netbase issue.) But, when the exact same function and node_modules bundle is deployed to the remote google cloud functions environment, it can no longer save data to cloud datastore. - Note that the same remotely deployed function worked perfectly up until a week ago.

tobeach picture tobeach on 7 Feb 2017

@tobeach The above linked workaround in the Node.js docker image should resolve the problem. Cloud Functions is set to pick up the docker image with the fix sometime this week. /cc swalkowski

ofrobots picture ofrobots on 7 Feb 2017

I believe this has either already been resolved, or will be when you update to google-cloud 0.46.2 which was just released. I'm going to close, but just let me know if this isn't resolved.

stephenplusplus picture stephenplusplus on 8 Feb 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

stephenplusplus picture stephenplusplus  ·  4Comments
dsimmons picture dsimmons  ·  4Comments
sporkd picture sporkd  ·  5Comments
charly37 picture charly37  ·  3Comments
positlabs picture positlabs  ·  3Comments