Google-cloud-ruby: DataStore protocol error?

[timanovsky]

Got this exception (Ruby) when reading from DataStore, as far as I understand it is while deserializing server response:

TypeError: wrong argument type false (expected String)
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0.pre1-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:62:in `decode'
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0.pre1-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:62:in `call'
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0.pre1-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:62:in `block in unmarshal_proc'
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0.pre1-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:231:in `remote_read'
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0.pre1-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:309:in `request_response'
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0.pre1-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:167:in `request_response'
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0.pre1-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:184:in `block (3 levels) in rpc_stub_class'
/app/vendor/bundle/ruby/2.3.0/gems/gcloud-0.12.1/lib/gcloud/datastore/service.rb:76:in `block in lookup'
/app/vendor/bundle/ruby/2.3.0/gems/gcloud-0.12.1/lib/gcloud/datastore/service.rb:144:in `block in execute'
/app/vendor/bundle/ruby/2.3.0/gems/gcloud-0.12.1/lib/gcloud/backoff.rb:112:in `block in execute_grpc'
/app/vendor/bundle/ruby/2.3.0/gems/gcloud-0.12.1/lib/gcloud/backoff.rb:110:in `loop'
/app/vendor/bundle/ruby/2.3.0/gems/gcloud-0.12.1/lib/gcloud/backoff.rb:110:in `execute_grpc'
/app/vendor/bundle/ruby/2.3.0/gems/gcloud-0.12.1/lib/gcloud/datastore/service.rb:143:in `execute'
/app/vendor/bundle/ruby/2.3.0/gems/gcloud-0.12.1/lib/gcloud/datastore/service.rb:76:in `lookup'
/app/vendor/bundle/ruby/2.3.0/gems/gcloud-0.12.1/lib/gcloud/datastore/dataset.rb:337:in `find_all'
/app/vendor/bundle/ruby/2.3.0/gems/gcloud-0.12.1/lib/gcloud/datastore/dataset.rb:306:in `find'

Probably it is more gcloud bug than grpc's one (https://github.com/grpc/grpc/issues/7661)

[nicolasnoble]

I have created a full bug report on protobuf, since this is the relevant code base for this one: https://github.com/google/protobuf/issues/2004

[blowmage]

Thanks @nicolasnoble!

[blowmage]

@timanovsky Can you retry this using the recent 0.20.0 release? That version is the first to use the grpc 1.0 and google-protobuf 3.0 releases.

[bmclean]

@blowmage We just pushed 0.20.0 to our staging environment today and have not seen it yet. We did also upgrade to Ruby 2.3.1, which seems to make it less likely to occur.
https://github.com/google/protobuf/issues/2004 already has a pull request which is fantastic!

[timanovsky]

ok, will try in next couple of days and will report

Best regards,
Alexey Timanovsky.

On 29 August 2016 at 17:52, Mike Moore [email protected] wrote:

@timanovsky https://github.com/timanovsky Can you retry this using the
recent 0.20.0 release? That version is the first to use the grpc 1.0 and
google-protobuf 3.0 releases.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-ruby/issues/817#issuecomment-243147830,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABK0FsTbx_4jOqJeC5Ooofv9SzngG5Flks5qkvI8gaJpZM4JfQ5y
.

[timanovsky]

@blowmage It has been running on staging for last three days, and I have not seen it yet. We will be pushing this to production this week and I'll let you know how it behaves.

BTW, we are running with built-in retries: 0 now to let our higher level logic to deal with it, and I'm a bit surprised with a rate of Google::Cloud::UnavailableError caused by GRPC::BadStatus – it is roughly one exception every 5 minutes, and it always happens during socket read:

GRPC::BadStatus: 14:{"created":"@1473617781.545426997","description":"Secure read failed","file":"src/core/lib/security/transport/secure_endpoint.c","file_line":157,"grpc_status":14,"referenced_errors":[{"created":"@1473617781.545386622","description":"EOF","file":"src/core/lib/iomgr/tcp_posix.c","file_line":235}]} (code[Fixnum]=14) (details[String]="{\"created\":\"@1473617781.545426997\",\"description\":\"Secure read failed\",\"file\":\"src/core/lib/security/transport/secure_endpoint.c\",\"file_line\":157,\"grpc_status\":14,\"referenced_errors\":[{\"created\":\"@1473617781.545386622\",\"description\":\"EOF\",\"file\":\"src/core/lib/iomgr/tcp_posix.c\",\"file_line\":235}]}") (metadata[Hash]={})

[blowmage]

@murgatroid99 Any thoughts on this error?

[murgatroid99]

Status code 14 just means "Unavailable", and it looks like that information is already surfaced by the Google Cloud exception type. That kind of error is transient, and you should be able to fix it by retrying.

[blowmage]

Thanks @murgatroid99. Is that status code coming from the server? Or is that from the client because it can't connect to the server?

[murgatroid99]

It looks like it can either come from the server directly, or it can be translated from a 503 error.

[timanovsky]

@blowmage It looks like protocol error has been fixed.

[blowmage]

Great! Glad this is now working!

[timanovsky]

Just got this error:
TypeError: wrong argument type false (expected String)

stacktrace:

/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:62:in `decode' /
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:62:in `call' /
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:62:in `block in unmarshal_proc' /
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:231:in `remote_read' /
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:309:in `request_response' /
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:167:in `request_response' /
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:184:in `block (3 levels) in rpc_stub_class' /
/app/vendor/bundle/ruby/2.3.0/gems/google-cloud-datastore-0.20.1/lib/google/cloud/datastore/service.rb:83:in `block in lookup' /
/app/vendor/bundle/ruby/2.3.0/gems/google-cloud-datastore-0.20.1/lib/google/cloud/datastore/service.rb:151:in `block in execute' /
/app/vendor/bundle/ruby/2.3.0/gems/google-cloud-core-0.20.1/lib/google/cloud/core/grpc_backoff.rb:77:in `block in execute' /
/app/vendor/bundle/ruby/2.3.0/gems/google-cloud-core-0.20.1/lib/google/cloud/core/grpc_backoff.rb:75:in `loop' /
/app/vendor/bundle/ruby/2.3.0/gems/google-cloud-core-0.20.1/lib/google/cloud/core/grpc_backoff.rb:75:in `execute' /
/app/vendor/bundle/ruby/2.3.0/gems/google-cloud-datastore-0.20.1/lib/google/cloud/datastore/service.rb:150:in `execute' /
/app/vendor/bundle/ruby/2.3.0/gems/google-cloud-datastore-0.20.1/lib/google/cloud/datastore/service.rb:83:in `lookup' /
/app/vendor/bundle/ruby/2.3.0/gems/google-cloud-datastore-0.20.1/lib/google/cloud/datastore/dataset.rb:339:in `find_all' /
/app/vendor/bundle/ruby/2.3.0/gems/google-cloud-datastore-0.20.1/lib/google/cloud/datastore/dataset.rb:308:in `find' /

[timanovsky]

Seems it is back :(

[timanovsky]

@blowmage would you mind reopening the ticket? :(

[bmclean]

The discussion around TypeError: wrong argument type false (expected String) on grpc#7761 was closed and moved to google/protobuf#2004. The fix is in a merged pull request but doesn't look like it has been released just yet.

[timanovsky]

@bmclean looks like they did a release, but not gem publish

[quartzmo]

@timanovsky OK then to keep this issue closed and focus discussion in google/protobuf#2004?

[timanovsky]

ok

Best regards,
Alexey Timanovsky.

On 13 September 2016 at 21:15, Chris Smith [email protected] wrote:

@timanovsky https://github.com/timanovsky OK then to keep this issue
closed and focus discussion in google/protobuf#2004
https://github.com/google/protobuf/issues/2004?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-ruby/issues/817#issuecomment-246772914,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABK0FnytOEm9-QLQ79U4OPwA0SCFmGUkks5qpugqgaJpZM4JfQ5y
.

[timanovsky]

@quartzmo @bmclean Do you guys have good contacts with protobuf team? I tried to connect, but they are not responding. I wonder why they did 3.0.2 source release, but did not publish updated gem, and if they could do it.

[quartzmo]

@timanovsky I have no special relationship, but I asked as well on google/protobuf#2004.

[timanovsky]

@quartzmo Thanks, looks like we may expect updated gem today.

[blowmage]

cc @nicolasnoble