Google-cloud-ruby: Firestore listen eventually fails

Created on 14 Mar 2021 · 17Comments · Source: googleapis/google-cloud-ruby

Environment details

OS: Heroku
Ruby version: 2.7.0p0
Gem name and version: google-cloud-firestore (2.4.1)

Steps to reproduce

Running this on Heroku as a separate dyno via Procfile (firestore: bundle exec ruby lib/listen_to_firestore.rb):

# Load Rails environment
require File.expand_path('../../config/environment', __FILE__)

firestore = Google::Cloud::Firestore.new(
  project_id: '...',
  credentials: { ... },
)

firestore.collection_group('thread').order('createdAt', 'desc').limit(25).listen do |snapshot|
  snapshot.changes.each do |change|
    # ActiveRecord stuff
  end
end

puts "Listening..."
while true
  sleep 0.05
end

After a while, the thread crashes with No status received.

Full backtrace

2021-03-14T16:54:20.962157+00:00 app[firestore.1]: #<Thread:0x000055df0b291450 /app/vendor/bundle/ruby/2.7.0/gems/google-cloud-firestore-2.4.1/lib/google/cloud/firestore/watch/listener.rb:129 run> terminated with exception (report_on_exception is true):
2021-03-14T16:54:20.965255+00:00 app[firestore.1]: /app/vendor/bundle/ruby/2.7.0/gems/grpc-1.36.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status': 2:No status received. debug_error_string:{"created":"@1615740860.961643731","description":"No status received","file":"src/core/lib/surface/call.cc","file_line":1091,"grpc_status":2} (GRPC::Unknown)
2021-03-14T16:54:20.965302+00:00 app[firestore.1]: from /app/vendor/bundle/ruby/2.7.0/gems/grpc-1.36.0-x86_64-linux/src/ruby/lib/grpc/generic/bidi_call.rb:209:in `block in read_loop'
2021-03-14T16:54:20.965330+00:00 app[firestore.1]: from /app/vendor/bundle/ruby/2.7.0/gems/grpc-1.36.0-x86_64-linux/src/ruby/lib/grpc/generic/bidi_call.rb:195:in `loop'
2021-03-14T16:54:20.965357+00:00 app[firestore.1]: from /app/vendor/bundle/ruby/2.7.0/gems/grpc-1.36.0-x86_64-linux/src/ruby/lib/grpc/generic/bidi_call.rb:195:in `read_loop'
2021-03-14T16:54:20.965382+00:00 app[firestore.1]: from lib/listen_to_firestore.rb:in `each'

firestore question

Source

gilbert

All 17 comments

FYI our current workaround is to restart the listening process every 10 minutes.

gilbert on 1 Apr 2021

@gilbert Thanks for opening this issue and for the update about restarting.

Possibly related:

https://github.com/grpc/grpc-node/issues/1569
https://github.com/grpc/grpc/issues/22120

I'll try to reproduce this myself soon.

quartzmo on 1 Apr 2021

Can you provide the dependency versions as well? Most importantly:

google-cloud-firestore-v1
gapic-common
google-protobuf
googleapis-common-protos
googleapis-common-protos-types
googleauth
grpc
google-protobuf

quartzmo on 1 Apr 2021

google-cloud-firestore-v1 - 0.4.0
gapic-common - 0.4.0
google-protobuf - 3.15.6
googleapis-common-protos - 1.3.11
googleapis-common-protos-types - 1.0.6
googleauth - 0.16.0
grpc - 1.36.0
google-protobuf - 3.15.6

gilbert on 2 Apr 2021

After a while, the thread crashes with No status received.

I ran the example for 20 min but that did not reproduce the error. Should I have let it run longer?

OS X

ruby 2.7.3p183 (2021-04-05 revision 6847ee089d) [x86_64-darwin19]

gapic-common 0.4.0
google-protobuf 3.15.8 (universal-darwin)
googleapis-common-protos 1.3.11
googleapis-common-protos-types 1.0.6
grpc 1.37.0 (universal-darwin)
googleauth 0.16.1

Did the grpc issues that I linked above lead you to any ideas?

quartzmo on 9 Apr 2021

Yes, our issue happened closer to an hour running, though sometimes it was as short at 20m.

I saw the grpc issues, but I'm not sure what to make of them as we're only doing pure ActiveRecord queries, as well as an ActiveJob perform_later.

I will turn off the restart process today to confirm it's still behaving this way.

gilbert on 9 Apr 2021

OK, I'll try again. Thanks!

quartzmo on 9 Apr 2021

Are you able to reproduce this in a different environment, outside of Heroku and Rails, using your example above?

quartzmo on 9 Apr 2021

I am not, good question. I can run this file on my local machine indefinitely and doesn't reproduce the error.

gilbert on 9 Apr 2021

When you turn off the restart process, can you also try to enable grpc debug logging (assuming doing so is not a security problem)?

https://github.com/grpc/grpc/blob/master/TROUBLESHOOTING.md

quartzmo on 9 Apr 2021

👍1

So it's still happening. Here are the logs, from the last message it handled successfully, to the point where it crashed (looks to be 7 seconds).

Heroku logs

2021-04-09T20:54:34.325412+00:00 app[firestore.1]: added ..../....
2021-04-09T20:54:34.325413+00:00 app[firestore.1]: D, [2021-04-09T20:54:34.304608 #4] DEBUG -- :   Kv Exists? (1.3ms)  SELECT 1 AS one FROM "kvs" WHERE "kvs"."ns" = $1 AND "kvs"."subject" IS NULL AND "kvs"."key" = $2 LIMIT $3  [["ns", 100], ["key", "..../...."], ["LIMIT", 1]]
2021-04-09T20:54:34.325413+00:00 app[firestore.1]: > Notify
2021-04-09T20:54:34.325414+00:00 app[firestore.1]: {:type=>"text", :content=>"....", :inReplyToItem=>nil, :senderUid=>"....", :readUserIDs=>["...."], :url=>nil, :createdAt=>2021-04-09 20:54:34.225 +0000}
2021-04-09T20:54:34.325414+00:00 app[firestore.1]: D, [2021-04-09T20:54:34.306456 #4] DEBUG -- :   ChatChannel Load (0.9ms)  SELECT "chat_channels".* FROM "chat_channels" WHERE "chat_channels"."uuid" = $1 LIMIT $2  [["uuid", "...."], ["LIMIT", 1]]
2021-04-09T20:54:34.325415+00:00 app[firestore.1]: D, [2021-04-09T20:54:34.310041 #4] DEBUG -- :   TRANSACTION (0.8ms)  BEGIN
2021-04-09T20:54:34.325415+00:00 app[firestore.1]: D, [2021-04-09T20:54:34.311752 #4] DEBUG -- :   ChatChannel Update (1.0ms)  UPDATE "chat_channels" SET "updated_at" = $1, "last_chat_item" = $2, "last_chat_item_id" = $3, "last_chat_item_at" = $4, "chat_count" = $5 WHERE "chat_channels"."id" = $6  [["updated_at", "2021-04-09 20:54:34.308048"], ["last_chat_item", "{\"type\":\"text\",\"content\":\"....\",\"inReplyToItem\":null,\"senderUid\":\"....\",\"readUserIDs\":[\"....\"],\"url\":null,\"createdAt\":\"2021-04-09T20:54:34.225+00:00\"}"], ["last_chat_item_id", "...."], ["last_chat_item_at", "2021-04-09 20:54:34.307086"], ["chat_count", 19], ["id", 1]]
2021-04-09T20:54:34.325416+00:00 app[firestore.1]: D, [2021-04-09T20:54:34.314904 #4] DEBUG -- :   TRANSACTION (2.4ms)  COMMIT
2021-04-09T20:54:34.325420+00:00 app[firestore.1]: D, [2021-04-09T20:54:34.319297 #4] DEBUG -- :   ChatChannelMember Update All (2.6ms)  UPDATE "chat_channel_members" SET chat_count = chat_count + 1 WHERE "chat_channel_members"."id" IN (SELECT "chat_channel_members"."id" FROM "chat_channel_members" INNER JOIN "mobile_users" ON "mobile_users"."id" = "chat_channel_members"."mobile_user_id" WHERE "chat_channel_members"."chat_channel_id" = $1 AND "mobile_users"."firebase_user_id" = $2)  [["chat_channel_id", 1], ["firebase_user_id", "...."]]
2021-04-09T20:54:34.325420+00:00 app[firestore.1]: D, [2021-04-09T20:54:34.322794 #4] DEBUG -- : [ActiveJob]   TRANSACTION (0.8ms)  BEGIN
2021-04-09T20:54:34.325421+00:00 app[firestore.1]: D, [2021-04-09T20:54:34.324714 #4] DEBUG -- : [ActiveJob]   GoodJob::Job Create (1.0ms)  INSERT INTO "good_jobs" ("queue_name", "priority", "serialized_params", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"  [["queue_name", "default"], ["priority", 0], ["serialized_params", "{\"job_class\":\"NotifyNewChatJob\",\"job_id\":\"ca6e7c78-b048-4e68-b710-4f89f82436d3\",\"provider_job_id\":null,\"queue_name\":\"default\",\"priority\":null,\"arguments\":[{\"_aj_globalid\":\"gid://gumdrop/ChatChannel/1\"},\"....\",{\"type\":\"text\",\"content\":\".... \",\"inReplyToItem\":null,\"senderUid\":\"....\",\"readUserIDs\":[\"....\"],\"url\":null,\"createdAt\":{\"_aj_serialized\":\"ActiveJob::Serializers::TimeSerializer\",\"value\":\"2021-04-09T20:54:34.225000000+00:00\"},\"_aj_symbol_keys\":[\"type\",\"content\",\"inReplyToItem\",\"senderUid\",\"readUserIDs\",\"url\",\"createdAt\"]}],\"executions\":0,\"exception_executions\":{},\"locale\":\"en\",\"timezone\":\"UTC\",\"enqueued_at\":\"2021-04-09T20:54:34Z\"}"], ["created_at", "2021-04-09 20:54:34.320527"], ["updated_at", "2021-04-09 20:54:34.320527"]]
2021-04-09T20:54:34.330574+00:00 app[firestore.1]: I0409 20:54:34.330527923       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e68)
2021-04-09T20:54:34.330628+00:00 app[firestore.1]: I0409 20:54:34.330602975       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e80)
2021-04-09T20:54:34.330663+00:00 app[firestore.1]: I0409 20:54:34.330641057       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0ea0)
2021-04-09T20:54:34.330695+00:00 app[firestore.1]: I0409 20:54:34.330674755       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0eb8)
2021-04-09T20:54:34.330751+00:00 app[firestore.1]: I0409 20:54:34.330712637       8 call.cc:1971]               grpc_call_start_batch(call=0x7f72602d7da0, ops=0x55fc285e0be0, nops=1, tag=0x7f7272d0ed80, reserved=(nil))
2021-04-09T20:54:34.330804+00:00 app[firestore.1]: I0409 20:54:34.330780853       8 call.cc:1569]               ops[0]: RECV_MESSAGE ptr=0x55fc285e0e98
2021-04-09T20:54:34.330855+00:00 app[firestore.1]: I0409 20:54:34.330832212       8 chttp2_transport.cc:1605]   perform_stream_op[s=0x7f72603d2e90]:  RECV_MESSAGE
2021-04-09T20:54:34.330891+00:00 app[firestore.1]: I0409 20:54:34.330870747       8 chttp2_transport.cc:1347]   perform_stream_op_locked:  RECV_MESSAGE; on_complete = (nil)
2021-04-09T20:55:19.303615+00:00 app[firestore.1]: I0409 20:55:19.303551996      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T20:55:19.303637+00:00 app[firestore.1]: I0409 20:55:19.303590556      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T20:55:19.303639+00:00 app[firestore.1]: I0409 20:55:19.303607896      14 tcp_posix.cc:685]           TCP:0x7f72603ac250 call_cb 0x7f72603bdad8 0x7f7265b60100:0x7f72603bdaa0
2021-04-09T20:55:19.303653+00:00 app[firestore.1]: I0409 20:55:19.303616221      14 tcp_posix.cc:688]           READ 0x7f72603ac250 (peer=ipv4:172.217.13.234:443) error="No Error"
2021-04-09T20:55:19.303656+00:00 app[firestore.1]: D0409 20:55:19.303627361      14 tcp_posix.cc:695]           DATA: 17 03 03 00 3c 06 8d 82 57 01 f6 5d c9 b3 4e 39 16 9b 58 24 9c 59 55 05 ed 32 9a b6 33 63 23 1b 87 97 5a ec c3 f1 a5 4f 72 e2 cf 80 67 f5 61 09 55 06 17 15 26 e1 03 35 7d e4 fc 6a 25 36 f0 ec 17 '....<...W..]..N9..X$.YU..2..3c#...Z....Or...g.a.U...&..5}..j%6...'
2021-04-09T20:55:19.303683+00:00 app[firestore.1]: I0409 20:55:19.303664661      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T20:55:19.303687+00:00 app[firestore.1]: I0409 20:55:19.303680402      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T20:55:19.303715+00:00 app[firestore.1]: I0409 20:55:19.303689838      14 tcp_posix.cc:548]           TCP:0x7f72603ac250 notify_on_read
2021-04-09T20:55:19.303720+00:00 app[firestore.1]: I0409 20:55:19.303710421      14 completion_queue.cc:786]    cq_end_op_for_pluck(cq=0x7f72602d7bd0, tag=0x7f7272d0ed80, error="No Error", done=0x7f72659ca2b0, done_arg=0x7f72602d9cf0, storage=0x7f72602d9d38)
2021-04-09T20:55:19.303810+00:00 app[firestore.1]: I0409 20:55:19.303772470       8 completion_queue.cc:1320]   RETURN_EVENT[0x7f72602d7bd0]: OP_COMPLETE: tag:0x7f7272d0ed80 OK
2021-04-09T20:55:19.303841+00:00 app[firestore.1]: I0409 20:55:19.303819040       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e68)
2021-04-09T20:55:19.303843+00:00 app[firestore.1]: I0409 20:55:19.303834185       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e80)
2021-04-09T20:55:19.303859+00:00 app[firestore.1]: I0409 20:55:19.303841951       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0ea0)
2021-04-09T20:55:19.303861+00:00 app[firestore.1]: I0409 20:55:19.303853700       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0eb8)
2021-04-09T20:55:19.304324+00:00 app[firestore.1]: I0409 20:55:19.304292949       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e68)
2021-04-09T20:55:19.304325+00:00 app[firestore.1]: I0409 20:55:19.304314759       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e80)
2021-04-09T20:55:19.304327+00:00 app[firestore.1]: I0409 20:55:19.304321828       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0ea0)
2021-04-09T20:55:19.304345+00:00 app[firestore.1]: I0409 20:55:19.304327773       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0eb8)
2021-04-09T20:55:19.304364+00:00 app[firestore.1]: I0409 20:55:19.304344508       8 call.cc:1971]               grpc_call_start_batch(call=0x7f72602d7da0, ops=0x55fc285e0be0, nops=1, tag=0x7f7272d0ed80, reserved=(nil))
2021-04-09T20:55:19.304366+00:00 app[firestore.1]: I0409 20:55:19.304359549       8 call.cc:1569]               ops[0]: RECV_MESSAGE ptr=0x55fc285e0e98
2021-04-09T20:55:19.304400+00:00 app[firestore.1]: I0409 20:55:19.304378934       8 chttp2_transport.cc:1605]   perform_stream_op[s=0x7f72603d2e90]:  RECV_MESSAGE
2021-04-09T20:55:19.304402+00:00 app[firestore.1]: I0409 20:55:19.304395398       8 chttp2_transport.cc:1347]   perform_stream_op_locked:  RECV_MESSAGE; on_complete = (nil)
2021-04-09T20:56:04.325841+00:00 app[firestore.1]: I0409 20:56:04.325764178      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T20:56:04.325860+00:00 app[firestore.1]: I0409 20:56:04.325812837      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T20:56:04.325862+00:00 app[firestore.1]: I0409 20:56:04.325836339      14 tcp_posix.cc:685]           TCP:0x7f72603ac250 call_cb 0x7f72603bdad8 0x7f7265b60100:0x7f72603bdaa0
2021-04-09T20:56:04.325863+00:00 app[firestore.1]: I0409 20:56:04.325850498      14 tcp_posix.cc:688]           READ 0x7f72603ac250 (peer=ipv4:172.217.13.234:443) error="No Error"
2021-04-09T20:56:04.326049+00:00 app[firestore.1]: D0409 20:56:04.325988322      14 tcp_posix.cc:695]           DATA: 17 03 03 00 3c e1 45 3b b3 7f 19 c7 e3 6b 65 72 51 18 50 4c 4f ac c3 f1 7a b0 66 b4 e4 65 b5 aa 98 9e fe 9e 6e 6c 93 81 5d 9f 9d 2a c8 6c 5d 65 57 dc 5e c3 11 87 8b 19 92 cd 94 98 82 3c 0f af b5 '....<.E;.....kerQ.PLO...z.f..e......nl..]..*.l]eW.^..........<...'
2021-04-09T20:56:04.326109+00:00 app[firestore.1]: I0409 20:56:04.326064543      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T20:56:04.326173+00:00 app[firestore.1]: I0409 20:56:04.326117821      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T20:56:04.326177+00:00 app[firestore.1]: I0409 20:56:04.326156630      14 tcp_posix.cc:548]           TCP:0x7f72603ac250 notify_on_read
2021-04-09T20:56:04.326212+00:00 app[firestore.1]: I0409 20:56:04.326179193      14 completion_queue.cc:786]    cq_end_op_for_pluck(cq=0x7f72602d7bd0, tag=0x7f7272d0ed80, error="No Error", done=0x7f72659ca2b0, done_arg=0x7f72602d9cf0, storage=0x7f72602d9d38)
2021-04-09T20:56:04.326293+00:00 app[firestore.1]: I0409 20:56:04.326259602       8 completion_queue.cc:1320]   RETURN_EVENT[0x7f72602d7bd0]: OP_COMPLETE: tag:0x7f7272d0ed80 OK
2021-04-09T20:56:04.326336+00:00 app[firestore.1]: I0409 20:56:04.326308678       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e68)
2021-04-09T20:56:04.326340+00:00 app[firestore.1]: I0409 20:56:04.326329325       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e80)
2021-04-09T20:56:04.326361+00:00 app[firestore.1]: I0409 20:56:04.326341239       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0ea0)
2021-04-09T20:56:04.326386+00:00 app[firestore.1]: I0409 20:56:04.326360048       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0eb8)
2021-04-09T20:56:04.326928+00:00 app[firestore.1]: I0409 20:56:04.326883528       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e68)
2021-04-09T20:56:04.326929+00:00 app[firestore.1]: I0409 20:56:04.326907733       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e80)
2021-04-09T20:56:04.326933+00:00 app[firestore.1]: I0409 20:56:04.326919175       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0ea0)
2021-04-09T20:56:04.326950+00:00 app[firestore.1]: I0409 20:56:04.326928786       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0eb8)
2021-04-09T20:56:04.327111+00:00 app[firestore.1]: I0409 20:56:04.327066070       8 call.cc:1971]               grpc_call_start_batch(call=0x7f72602d7da0, ops=0x55fc285e0be0, nops=1, tag=0x7f7272d0ed80, reserved=(nil))
2021-04-09T20:56:04.327112+00:00 app[firestore.1]: I0409 20:56:04.327091676       8 call.cc:1569]               ops[0]: RECV_MESSAGE ptr=0x55fc285e0e98
2021-04-09T20:56:04.327149+00:00 app[firestore.1]: I0409 20:56:04.327119389       8 chttp2_transport.cc:1605]   perform_stream_op[s=0x7f72603d2e90]:  RECV_MESSAGE
2021-04-09T20:56:04.327178+00:00 app[firestore.1]: I0409 20:56:04.327145200       8 chttp2_transport.cc:1347]   perform_stream_op_locked:  RECV_MESSAGE; on_complete = (nil)
2021-04-09T20:56:49.348319+00:00 app[firestore.1]: I0409 20:56:49.348234207      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T20:56:49.348360+00:00 app[firestore.1]: I0409 20:56:49.348294649      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T20:56:49.348362+00:00 app[firestore.1]: I0409 20:56:49.348323711      14 tcp_posix.cc:685]           TCP:0x7f72603ac250 call_cb 0x7f72603bdad8 0x7f7265b60100:0x7f72603bdaa0
2021-04-09T20:56:49.348362+00:00 app[firestore.1]: I0409 20:56:49.348340127      14 tcp_posix.cc:688]           READ 0x7f72603ac250 (peer=ipv4:172.217.13.234:443) error="No Error"
2021-04-09T20:56:49.348449+00:00 app[firestore.1]: D0409 20:56:49.348360737      14 tcp_posix.cc:695]           DATA: 17 03 03 00 3c 58 1d 15 e2 50 44 ca 71 c7 80 cd 36 70 d3 f0 ec 17 48 14 eb 24 a0 ac 2b 3e 37 19 08 55 95 0c 2f c8 74 ce 43 31 79 08 c3 89 d9 bf 14 ea fa d9 35 01 dd 91 c9 68 78 13 13 22 96 c4 d3 '....<X...PD.q...6p....H..$..+>7..U../.t.C1y.........5....hx.."...'
2021-04-09T20:56:49.355351+00:00 app[firestore.1]: I0409 20:56:49.355278693      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T20:56:49.355399+00:00 app[firestore.1]: I0409 20:56:49.355370208      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T20:56:49.355448+00:00 app[firestore.1]: I0409 20:56:49.355422372      14 tcp_posix.cc:548]           TCP:0x7f72603ac250 notify_on_read
2021-04-09T20:56:49.355501+00:00 app[firestore.1]: I0409 20:56:49.355473459      14 completion_queue.cc:786]    cq_end_op_for_pluck(cq=0x7f72602d7bd0, tag=0x7f7272d0ed80, error="No Error", done=0x7f72659ca2b0, done_arg=0x7f72602d9cf0, storage=0x7f72602d9d38)
2021-04-09T20:56:49.355638+00:00 app[firestore.1]: I0409 20:56:49.355602329       8 completion_queue.cc:1320]   RETURN_EVENT[0x7f72602d7bd0]: OP_COMPLETE: tag:0x7f7272d0ed80 OK
2021-04-09T20:56:49.355751+00:00 app[firestore.1]: I0409 20:56:49.355686820       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e68)
2021-04-09T20:56:49.355930+00:00 app[firestore.1]: I0409 20:56:49.355902135       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e80)
2021-04-09T20:56:49.355972+00:00 app[firestore.1]: I0409 20:56:49.355945681       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0ea0)
2021-04-09T20:56:49.356011+00:00 app[firestore.1]: I0409 20:56:49.355987039       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0eb8)
2021-04-09T20:56:49.358742+00:00 app[firestore.1]: I0409 20:56:49.358686320       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e68)
2021-04-09T20:56:49.358782+00:00 app[firestore.1]: I0409 20:56:49.358757043       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e80)
2021-04-09T20:56:49.358833+00:00 app[firestore.1]: I0409 20:56:49.358795237       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0ea0)
2021-04-09T20:56:49.359155+00:00 app[firestore.1]: I0409 20:56:49.359115956       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0eb8)
2021-04-09T20:56:49.359233+00:00 app[firestore.1]: I0409 20:56:49.359204808       8 call.cc:1971]               grpc_call_start_batch(call=0x7f72602d7da0, ops=0x55fc285e0be0, nops=1, tag=0x7f7272d0ed80, reserved=(nil))
2021-04-09T20:56:49.359278+00:00 app[firestore.1]: I0409 20:56:49.359252362       8 call.cc:1569]               ops[0]: RECV_MESSAGE ptr=0x55fc285e0e98
2021-04-09T20:56:49.359349+00:00 app[firestore.1]: I0409 20:56:49.359310971       8 chttp2_transport.cc:1605]   perform_stream_op[s=0x7f72603d2e90]:  RECV_MESSAGE
2021-04-09T20:56:49.359576+00:00 app[firestore.1]: I0409 20:56:49.359548887       8 chttp2_transport.cc:1347]   perform_stream_op_locked:  RECV_MESSAGE; on_complete = (nil)
2021-04-09T20:57:34.374323+00:00 app[firestore.1]: I0409 20:57:34.374214833      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T20:57:34.374350+00:00 app[firestore.1]: I0409 20:57:34.374278490      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T20:57:34.374352+00:00 app[firestore.1]: I0409 20:57:34.374307310      14 tcp_posix.cc:685]           TCP:0x7f72603ac250 call_cb 0x7f72603bdad8 0x7f7265b60100:0x7f72603bdaa0
2021-04-09T20:57:34.374352+00:00 app[firestore.1]: I0409 20:57:34.374323126      14 tcp_posix.cc:688]           READ 0x7f72603ac250 (peer=ipv4:172.217.13.234:443) error="No Error"
2021-04-09T20:57:34.374356+00:00 app[firestore.1]: D0409 20:57:34.374341772      14 tcp_posix.cc:695]           DATA: 17 03 03 00 3c b6 28 50 af 1c 12 b9 b5 26 90 85 90 8c 69 8d 94 28 e1 55 0b 92 ea 53 ff 60 ea b8 9b ea b3 16 aa 2f 93 7b e9 e7 37 4b 9f d4 5b aa ff 95 4a 26 22 8e ca 1a da 97 21 93 fa 12 c4 f4 b2 '....<.(P.....&....i..(.U...S.`......./.{..7K..[...J&".....!......'
2021-04-09T20:57:34.374422+00:00 app[firestore.1]: I0409 20:57:34.374394038      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T20:57:34.378168+00:00 app[firestore.1]: I0409 20:57:34.378132661      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T20:57:34.378222+00:00 app[firestore.1]: I0409 20:57:34.378198730      14 tcp_posix.cc:548]           TCP:0x7f72603ac250 notify_on_read
2021-04-09T20:57:34.378272+00:00 app[firestore.1]: I0409 20:57:34.378247301      14 completion_queue.cc:786]    cq_end_op_for_pluck(cq=0x7f72602d7bd0, tag=0x7f7272d0ed80, error="No Error", done=0x7f72659ca2b0, done_arg=0x7f72602d9cf0, storage=0x7f72602d9d38)
2021-04-09T20:57:34.378390+00:00 app[firestore.1]: I0409 20:57:34.378364108       8 completion_queue.cc:1320]   RETURN_EVENT[0x7f72602d7bd0]: OP_COMPLETE: tag:0x7f7272d0ed80 OK
2021-04-09T20:57:34.378545+00:00 app[firestore.1]: I0409 20:57:34.378441231       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e68)
2021-04-09T20:57:34.378585+00:00 app[firestore.1]: I0409 20:57:34.378562368       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e80)
2021-04-09T20:57:34.378618+00:00 app[firestore.1]: I0409 20:57:34.378598353       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0ea0)
2021-04-09T20:57:34.378650+00:00 app[firestore.1]: I0409 20:57:34.378630266       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0eb8)
2021-04-09T20:57:34.379468+00:00 app[firestore.1]: I0409 20:57:34.379442329       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e68)
2021-04-09T20:57:34.379499+00:00 app[firestore.1]: I0409 20:57:34.379480031       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e80)
2021-04-09T20:57:34.379526+00:00 app[firestore.1]: I0409 20:57:34.379509331       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0ea0)
2021-04-09T20:57:34.379552+00:00 app[firestore.1]: I0409 20:57:34.379536051       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0eb8)
2021-04-09T20:57:34.379588+00:00 app[firestore.1]: I0409 20:57:34.379568054       8 call.cc:1971]               grpc_call_start_batch(call=0x7f72602d7da0, ops=0x55fc285e0be0, nops=1, tag=0x7f7272d0ed80, reserved=(nil))
2021-04-09T20:57:34.379620+00:00 app[firestore.1]: I0409 20:57:34.379601525       8 call.cc:1569]               ops[0]: RECV_MESSAGE ptr=0x55fc285e0e98
2021-04-09T20:57:34.379668+00:00 app[firestore.1]: I0409 20:57:34.379647141       8 chttp2_transport.cc:1605]   perform_stream_op[s=0x7f72603d2e90]:  RECV_MESSAGE
2021-04-09T20:57:34.379701+00:00 app[firestore.1]: I0409 20:57:34.379682415       8 chttp2_transport.cc:1347]   perform_stream_op_locked:  RECV_MESSAGE; on_complete = (nil)
2021-04-09T20:58:19.393861+00:00 app[firestore.1]: I0409 20:58:19.393785504      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T20:58:19.393972+00:00 app[firestore.1]: I0409 20:58:19.393941124      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T20:58:19.394870+00:00 app[firestore.1]: I0409 20:58:19.394820569      14 tcp_posix.cc:685]           TCP:0x7f72603ac250 call_cb 0x7f72603bdad8 0x7f7265b60100:0x7f72603bdaa0
2021-04-09T20:58:19.394975+00:00 app[firestore.1]: I0409 20:58:19.394916252      14 tcp_posix.cc:688]           READ 0x7f72603ac250 (peer=ipv4:172.217.13.234:443) error="No Error"
2021-04-09T20:58:19.395051+00:00 app[firestore.1]: D0409 20:58:19.395021580      14 tcp_posix.cc:695]           DATA: 17 03 03 00 3c bd 59 ec e4 b5 29 9d cf 66 71 89 84 2d 44 b3 70 68 8e 9e ef 95 5d 72 03 4c 8e de f1 e6 f9 57 ee 72 6f 0e 5a 19 29 10 04 53 81 81 3b bd ee 86 e5 fc 90 9c 7f 8d fb 2e 8e 8a da a5 18 '....<.Y...)..fq..-D.ph....]r.L.....W.ro.Z.)..S..;................'
2021-04-09T20:58:19.395390+00:00 app[firestore.1]: I0409 20:58:19.395346872      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T20:58:19.395617+00:00 app[firestore.1]: I0409 20:58:19.395586701      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T20:58:19.395708+00:00 app[firestore.1]: I0409 20:58:19.395647745      14 tcp_posix.cc:548]           TCP:0x7f72603ac250 notify_on_read
2021-04-09T20:58:19.395915+00:00 app[firestore.1]: I0409 20:58:19.395846691      14 completion_queue.cc:786]    cq_end_op_for_pluck(cq=0x7f72602d7bd0, tag=0x7f7272d0ed80, error="No Error", done=0x7f72659ca2b0, done_arg=0x7f72602d9cf0, storage=0x7f72602d9d38)
2021-04-09T20:58:19.400907+00:00 app[firestore.1]: I0409 20:58:19.400848751       8 completion_queue.cc:1320]   RETURN_EVENT[0x7f72602d7bd0]: OP_COMPLETE: tag:0x7f7272d0ed80 OK
2021-04-09T20:58:19.409355+00:00 app[firestore.1]: I0409 20:58:19.409276350       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e68)
2021-04-09T20:58:19.418238+00:00 app[firestore.1]: I0409 20:58:19.418148367       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e80)
2021-04-09T20:58:19.418319+00:00 app[firestore.1]: I0409 20:58:19.418288094       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0ea0)
2021-04-09T20:58:19.418362+00:00 app[firestore.1]: I0409 20:58:19.418335279       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0eb8)
2021-04-09T20:58:19.421330+00:00 app[firestore.1]: I0409 20:58:19.421279320       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e68)
2021-04-09T20:58:19.429268+00:00 app[firestore.1]: I0409 20:58:19.429093253       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e80)
2021-04-09T20:58:19.433309+00:00 app[firestore.1]: I0409 20:58:19.433165552       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0ea0)
2021-04-09T20:58:19.433470+00:00 app[firestore.1]: I0409 20:58:19.433439159       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0eb8)
2021-04-09T20:58:19.433541+00:00 app[firestore.1]: I0409 20:58:19.433502660       8 call.cc:1971]               grpc_call_start_batch(call=0x7f72602d7da0, ops=0x55fc285e0be0, nops=1, tag=0x7f7272d0ed80, reserved=(nil))
2021-04-09T20:58:19.433768+00:00 app[firestore.1]: I0409 20:58:19.433562600       8 call.cc:1569]               ops[0]: RECV_MESSAGE ptr=0x55fc285e0e98
2021-04-09T20:58:19.434382+00:00 app[firestore.1]: I0409 20:58:19.434334680       8 chttp2_transport.cc:1605]   perform_stream_op[s=0x7f72603d2e90]:  RECV_MESSAGE
2021-04-09T20:58:19.434832+00:00 app[firestore.1]: I0409 20:58:19.434416625       8 chttp2_transport.cc:1347]   perform_stream_op_locked:  RECV_MESSAGE; on_complete = (nil)
2021-04-09T20:59:04.419801+00:00 app[firestore.1]: I0409 20:59:04.419725032      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T20:59:04.419894+00:00 app[firestore.1]: I0409 20:59:04.419869581      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T20:59:04.419994+00:00 app[firestore.1]: I0409 20:59:04.419965430      14 tcp_posix.cc:685]           TCP:0x7f72603ac250 call_cb 0x7f72603bdad8 0x7f7265b60100:0x7f72603bdaa0
2021-04-09T20:59:04.420072+00:00 app[firestore.1]: I0409 20:59:04.420045712      14 tcp_posix.cc:688]           READ 0x7f72603ac250 (peer=ipv4:172.217.13.234:443) error="No Error"
2021-04-09T20:59:04.420285+00:00 app[firestore.1]: D0409 20:59:04.420242783      14 tcp_posix.cc:695]           DATA: 17 03 03 00 3c ea b3 c7 6d cf 21 44 b9 7e a2 90 e8 7f a7 da a6 b9 42 97 ef bd 3b ec 01 6e 61 61 51 16 cc 43 1c d7 d5 7e de ea 75 e9 96 4a 8e c1 49 e8 2e e4 3b dc b0 fd 57 86 9d 8a d2 6e 7f d1 c6 '....<...m.!D.~........B...;..naaQ..C...~..u..J..I...;...W....n...'
2021-04-09T20:59:04.420380+00:00 app[firestore.1]: I0409 20:59:04.420339628      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T20:59:04.420620+00:00 app[firestore.1]: I0409 20:59:04.420595027      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T20:59:04.420656+00:00 app[firestore.1]: I0409 20:59:04.420636968      14 tcp_posix.cc:548]           TCP:0x7f72603ac250 notify_on_read
2021-04-09T20:59:04.420699+00:00 app[firestore.1]: I0409 20:59:04.420677748      14 completion_queue.cc:786]    cq_end_op_for_pluck(cq=0x7f72602d7bd0, tag=0x7f7272d0ed80, error="No Error", done=0x7f72659ca2b0, done_arg=0x7f72602d9cf0, storage=0x7f72602d9d38)
2021-04-09T20:59:04.426295+00:00 app[firestore.1]: I0409 20:59:04.426231470       8 completion_queue.cc:1320]   RETURN_EVENT[0x7f72602d7bd0]: OP_COMPLETE: tag:0x7f7272d0ed80 OK
2021-04-09T20:59:04.426373+00:00 app[firestore.1]: I0409 20:59:04.426344646       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e68)
2021-04-09T20:59:04.426411+00:00 app[firestore.1]: I0409 20:59:04.426387149       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e80)
2021-04-09T20:59:04.426450+00:00 app[firestore.1]: I0409 20:59:04.426426672       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0ea0)
2021-04-09T20:59:04.426488+00:00 app[firestore.1]: I0409 20:59:04.426464069       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0eb8)
2021-04-09T20:59:04.427223+00:00 app[firestore.1]: I0409 20:59:04.427196523       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e68)
2021-04-09T20:59:04.427256+00:00 app[firestore.1]: I0409 20:59:04.427235253       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e80)
2021-04-09T20:59:04.427290+00:00 app[firestore.1]: I0409 20:59:04.427267923       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0ea0)
2021-04-09T20:59:04.427319+00:00 app[firestore.1]: I0409 20:59:04.427301103       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0eb8)
2021-04-09T20:59:04.427362+00:00 app[firestore.1]: I0409 20:59:04.427337279       8 call.cc:1971]               grpc_call_start_batch(call=0x7f72602d7da0, ops=0x55fc285e0be0, nops=1, tag=0x7f7272d0ed80, reserved=(nil))
2021-04-09T20:59:04.427401+00:00 app[firestore.1]: I0409 20:59:04.427379116       8 call.cc:1569]               ops[0]: RECV_MESSAGE ptr=0x55fc285e0e98
2021-04-09T20:59:04.427455+00:00 app[firestore.1]: I0409 20:59:04.427430855       8 chttp2_transport.cc:1605]   perform_stream_op[s=0x7f72603d2e90]:  RECV_MESSAGE
2021-04-09T20:59:04.427496+00:00 app[firestore.1]: I0409 20:59:04.427472581       8 chttp2_transport.cc:1347]   perform_stream_op_locked:  RECV_MESSAGE; on_complete = (nil)
2021-04-09T20:59:49.439215+00:00 app[firestore.1]: I0409 20:59:49.439144046      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T20:59:49.439339+00:00 app[firestore.1]: I0409 20:59:49.439309495      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T20:59:49.439398+00:00 app[firestore.1]: I0409 20:59:49.439367153      14 tcp_posix.cc:685]           TCP:0x7f72603ac250 call_cb 0x7f72603bdad8 0x7f7265b60100:0x7f72603bdaa0
2021-04-09T20:59:49.439528+00:00 app[firestore.1]: I0409 20:59:49.439498062      14 tcp_posix.cc:688]           READ 0x7f72603ac250 (peer=ipv4:172.217.13.234:443) error="No Error"
2021-04-09T20:59:49.439755+00:00 app[firestore.1]: D0409 20:59:49.439722821      14 tcp_posix.cc:695]           DATA: 17 03 03 00 3c 26 33 5d 1c c0 9d 1e 0f e0 eb d8 3f d4 5b 85 e3 bf 1c fe 57 be 16 e3 c4 31 fc 2a 4d 83 b8 bd 4e 4e af a4 7b db 71 81 c8 4d 36 f1 b0 bc 1b 60 f0 b5 e9 df b7 f5 f6 a7 c9 66 c4 17 2a '....<&3]........?.[.....W....1.*M...NN..{.q..M6....`.........f..*'
2021-04-09T20:59:49.439832+00:00 app[firestore.1]: I0409 20:59:49.439806372      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T20:59:49.439914+00:00 app[firestore.1]: I0409 20:59:49.439889067      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T20:59:49.440098+00:00 app[firestore.1]: I0409 20:59:49.440070974      14 tcp_posix.cc:548]           TCP:0x7f72603ac250 notify_on_read
2021-04-09T20:59:49.440151+00:00 app[firestore.1]: I0409 20:59:49.440122697      14 completion_queue.cc:786]    cq_end_op_for_pluck(cq=0x7f72602d7bd0, tag=0x7f7272d0ed80, error="No Error", done=0x7f72659ca2b0, done_arg=0x7f72602d9cf0, storage=0x7f72602d9d38)
2021-04-09T20:59:49.440677+00:00 app[firestore.1]: I0409 20:59:49.440644016       8 completion_queue.cc:1320]   RETURN_EVENT[0x7f72602d7bd0]: OP_COMPLETE: tag:0x7f7272d0ed80 OK
2021-04-09T20:59:49.440801+00:00 app[firestore.1]: I0409 20:59:49.440727749       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e68)
2021-04-09T20:59:49.440991+00:00 app[firestore.1]: I0409 20:59:49.440959157       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e80)
2021-04-09T20:59:49.441036+00:00 app[firestore.1]: I0409 20:59:49.441011507       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0ea0)
2021-04-09T20:59:49.441072+00:00 app[firestore.1]: I0409 20:59:49.441049264       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0eb8)
2021-04-09T20:59:49.442078+00:00 app[firestore.1]: I0409 20:59:49.442045721       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e68)
2021-04-09T20:59:49.442206+00:00 app[firestore.1]: I0409 20:59:49.442176526       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e80)
2021-04-09T20:59:49.442250+00:00 app[firestore.1]: I0409 20:59:49.442227385       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0ea0)
2021-04-09T20:59:49.442280+00:00 app[firestore.1]: I0409 20:59:49.442261186       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0eb8)
2021-04-09T20:59:49.442364+00:00 app[firestore.1]: I0409 20:59:49.442298595       8 call.cc:1971]               grpc_call_start_batch(call=0x7f72602d7da0, ops=0x55fc285e0be0, nops=1, tag=0x7f7272d0ed80, reserved=(nil))
2021-04-09T20:59:49.442399+00:00 app[firestore.1]: I0409 20:59:49.442380193       8 call.cc:1569]               ops[0]: RECV_MESSAGE ptr=0x55fc285e0e98
2021-04-09T20:59:49.442450+00:00 app[firestore.1]: I0409 20:59:49.442427177       8 chttp2_transport.cc:1605]   perform_stream_op[s=0x7f72603d2e90]:  RECV_MESSAGE
2021-04-09T20:59:49.442502+00:00 app[firestore.1]: I0409 20:59:49.442464772       8 chttp2_transport.cc:1347]   perform_stream_op_locked:  RECV_MESSAGE; on_complete = (nil)
2021-04-09T21:00:34.466635+00:00 app[firestore.1]: I0409 21:00:34.466556672      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T21:00:34.466745+00:00 app[firestore.1]: I0409 21:00:34.466717093      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T21:00:34.466798+00:00 app[firestore.1]: I0409 21:00:34.466772997      14 tcp_posix.cc:685]           TCP:0x7f72603ac250 call_cb 0x7f72603bdad8 0x7f7265b60100:0x7f72603bdaa0
2021-04-09T21:00:34.466919+00:00 app[firestore.1]: I0409 21:00:34.466892291      14 tcp_posix.cc:688]           READ 0x7f72603ac250 (peer=ipv4:172.217.13.234:443) error="No Error"
2021-04-09T21:00:34.466965+00:00 app[firestore.1]: D0409 21:00:34.466939096      14 tcp_posix.cc:695]           DATA: 17 03 03 00 3c 4b af f2 26 23 97 bc 7e 60 6d c1 92 26 f1 32 e6 ea a1 c1 b8 cc 8f a2 2c 9f 08 ec 6f 73 6b e1 82 50 9d 9c ab d3 41 b6 93 82 1e 19 af 50 99 b2 58 98 c1 bb 18 d9 92 72 40 13 58 46 9e '....<K..&#..~`m..&.2........,[email protected].'
2021-04-09T21:00:34.467080+00:00 app[firestore.1]: I0409 21:00:34.467017049      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T21:00:34.467115+00:00 app[firestore.1]: I0409 21:00:34.467092325      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T21:00:34.467154+00:00 app[firestore.1]: I0409 21:00:34.467131462      14 tcp_posix.cc:548]           TCP:0x7f72603ac250 notify_on_read
2021-04-09T21:00:34.467219+00:00 app[firestore.1]: I0409 21:00:34.467178119      14 completion_queue.cc:786]    cq_end_op_for_pluck(cq=0x7f72602d7bd0, tag=0x7f7272d0ed80, error="No Error", done=0x7f72659ca2b0, done_arg=0x7f72602d9cf0, storage=0x7f72602d9d38)
2021-04-09T21:00:34.467341+00:00 app[firestore.1]: I0409 21:00:34.467316248       8 completion_queue.cc:1320]   RETURN_EVENT[0x7f72602d7bd0]: OP_COMPLETE: tag:0x7f7272d0ed80 OK
2021-04-09T21:00:34.467445+00:00 app[firestore.1]: I0409 21:00:34.467419743       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e68)
2021-04-09T21:00:34.467623+00:00 app[firestore.1]: I0409 21:00:34.467594605       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e80)
2021-04-09T21:00:34.467987+00:00 app[firestore.1]: I0409 21:00:34.467954860       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0ea0)
2021-04-09T21:00:34.468038+00:00 app[firestore.1]: I0409 21:00:34.468013373       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0eb8)
2021-04-09T21:00:34.469025+00:00 app[firestore.1]: I0409 21:00:34.468985663       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e68)
2021-04-09T21:00:34.470640+00:00 app[firestore.1]: I0409 21:00:34.470603116       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e80)
2021-04-09T21:00:34.470689+00:00 app[firestore.1]: I0409 21:00:34.470666061       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0ea0)
2021-04-09T21:00:34.470719+00:00 app[firestore.1]: I0409 21:00:34.470700228       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0eb8)
2021-04-09T21:00:34.470758+00:00 app[firestore.1]: I0409 21:00:34.470737746       8 call.cc:1971]               grpc_call_start_batch(call=0x7f72602d7da0, ops=0x55fc285e0be0, nops=1, tag=0x7f7272d0ed80, reserved=(nil))
2021-04-09T21:00:34.470792+00:00 app[firestore.1]: I0409 21:00:34.470773560       8 call.cc:1569]               ops[0]: RECV_MESSAGE ptr=0x55fc285e0e98
2021-04-09T21:00:34.470874+00:00 app[firestore.1]: I0409 21:00:34.470850009       8 chttp2_transport.cc:1605]   perform_stream_op[s=0x7f72603d2e90]:  RECV_MESSAGE
2021-04-09T21:00:34.470908+00:00 app[firestore.1]: I0409 21:00:34.470889228       8 chttp2_transport.cc:1347]   perform_stream_op_locked:  RECV_MESSAGE; on_complete = (nil)
2021-04-09T21:01:19.494233+00:00 app[firestore.1]: I0409 21:01:19.494156879      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T21:01:19.494343+00:00 app[firestore.1]: I0409 21:01:19.494310322      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T21:01:19.494405+00:00 app[firestore.1]: I0409 21:01:19.494374777      14 tcp_posix.cc:685]           TCP:0x7f72603ac250 call_cb 0x7f72603bdad8 0x7f7265b60100:0x7f72603bdaa0
2021-04-09T21:01:19.494510+00:00 app[firestore.1]: I0409 21:01:19.494481415      14 tcp_posix.cc:688]           READ 0x7f72603ac250 (peer=ipv4:172.217.13.234:443) error="No Error"
2021-04-09T21:01:19.494562+00:00 app[firestore.1]: D0409 21:01:19.494532330      14 tcp_posix.cc:695]           DATA: 17 03 03 00 3c 67 27 fb 43 eb e6 93 8e be 5c ae b1 85 8d 2b 34 37 a1 de b0 c7 00 7a aa cc 18 e3 5f 36 61 ac ec 89 f9 a9 2e e9 86 6e 11 f3 c9 f1 2c 40 2f b3 60 7e 2c 2a e1 6b 2b d6 88 b8 b9 e2 21 '....<g'.C.....\....+47.....z...._6a........n....,@/.`~,*.k+.....!'
2021-04-09T21:01:19.494689+00:00 app[firestore.1]: I0409 21:01:19.494658568      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T21:01:19.494862+00:00 app[firestore.1]: I0409 21:01:19.494833204      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T21:01:19.494910+00:00 app[firestore.1]: I0409 21:01:19.494883329      14 tcp_posix.cc:548]           TCP:0x7f72603ac250 notify_on_read
2021-04-09T21:01:19.494984+00:00 app[firestore.1]: I0409 21:01:19.494936768      14 completion_queue.cc:786]    cq_end_op_for_pluck(cq=0x7f72602d7bd0, tag=0x7f7272d0ed80, error="No Error", done=0x7f72659ca2b0, done_arg=0x7f72602d9cf0, storage=0x7f72602d9d38)
2021-04-09T21:01:19.498214+00:00 app[firestore.1]: I0409 21:01:19.498170117       8 completion_queue.cc:1320]   RETURN_EVENT[0x7f72602d7bd0]: OP_COMPLETE: tag:0x7f7272d0ed80 OK
2021-04-09T21:01:19.498302+00:00 app[firestore.1]: I0409 21:01:19.498274740       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e68)
2021-04-09T21:01:19.498335+00:00 app[firestore.1]: I0409 21:01:19.498314454       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e80)
2021-04-09T21:01:19.498367+00:00 app[firestore.1]: I0409 21:01:19.498346917       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0ea0)
2021-04-09T21:01:19.498608+00:00 app[firestore.1]: I0409 21:01:19.498560756       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0eb8)
2021-04-09T21:01:19.500196+00:00 app[firestore.1]: I0409 21:01:19.500164213       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e68)
2021-04-09T21:01:19.500247+00:00 app[firestore.1]: I0409 21:01:19.500224156       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e80)
2021-04-09T21:01:19.500277+00:00 app[firestore.1]: I0409 21:01:19.500258502       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0ea0)
2021-04-09T21:01:19.500355+00:00 app[firestore.1]: I0409 21:01:19.500287120       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0eb8)
2021-04-09T21:01:19.500425+00:00 app[firestore.1]: I0409 21:01:19.500400319       8 call.cc:1971]               grpc_call_start_batch(call=0x7f72602d7da0, ops=0x55fc285e0be0, nops=1, tag=0x7f7272d0ed80, reserved=(nil))
2021-04-09T21:01:19.500460+00:00 app[firestore.1]: I0409 21:01:19.500440176       8 call.cc:1569]               ops[0]: RECV_MESSAGE ptr=0x55fc285e0e98
2021-04-09T21:01:19.500505+00:00 app[firestore.1]: I0409 21:01:19.500484976       8 chttp2_transport.cc:1605]   perform_stream_op[s=0x7f72603d2e90]:  RECV_MESSAGE
2021-04-09T21:01:19.500569+00:00 app[firestore.1]: I0409 21:01:19.500520285       8 chttp2_transport.cc:1347]   perform_stream_op_locked:  RECV_MESSAGE; on_complete = (nil)
2021-04-09T21:01:24.844234+00:00 app[firestore.1]: I0409 21:01:24.844153944      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T21:01:24.844339+00:00 app[firestore.1]: I0409 21:01:24.844312423      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T21:01:24.844500+00:00 app[firestore.1]: I0409 21:01:24.844470398      14 tcp_posix.cc:685]           TCP:0x7f72603ac250 call_cb 0x7f72603bdad8 0x7f7265b60100:0x7f72603bdaa0
2021-04-09T21:01:24.844538+00:00 app[firestore.1]: I0409 21:01:24.844515950      14 tcp_posix.cc:688]           READ 0x7f72603ac250 (peer=ipv4:172.217.13.234:443) error="No Error"
2021-04-09T21:01:24.844616+00:00 app[firestore.1]: D0409 21:01:24.844575530      14 tcp_posix.cc:695]           DATA: 17 03 03 00 1a 92 c3 57 7e dc c3 95 55 e7 3f e3 6d 0a 7d 80 30 60 91 42 90 87 bc ca 7e 13 66 17 03 03 00 2f 41 09 c7 4b 4b de 90 17 1b f9 54 86 07 24 08 62 6a b7 3f 98 9c a0 3a c1 77 fd a1 45 33 20 f3 6e f9 71 1d bc f1 9c 52 f8 7c 8e b8 c1 61 52 e8 17 03 03 00 3a d3 e2 97 f7 2f 32 49 db d3 7e 22 66 a2 1b d8 aa 25 4b 0a 35 4a 29 50 63 eb 02 42 3d cc 6b ac 33 84 3e 21 1a e2 7f 57 5f b4 ac 69 79 7b ea 85 d1 05 27 08 6d 55 fc f1 7f 2f b2 '.......W~...U.?.m.}.0`.B....~.f..../A..KK.....T..$.bj.?...:.w..E3 .n.q....R.|...aR.....:..../2I..~"f....%K.5J)Pc..B=.k.3.>!...W_..iy{....'.mU.../.'
2021-04-09T21:01:24.844729+00:00 app[firestore.1]: I0409 21:01:24.844702821      14 chttp2_transport.cc:776]    W:0x7f72603d0b90 CLIENT [ipv4:172.217.13.234:443] state IDLE -> WRITING [PING_RESPONSE]
2021-04-09T21:01:24.844774+00:00 app[firestore.1]: I0409 21:01:24.844750722      14 chttp2_transport.cc:1064]   transport 0x7f72603d0b90 got goaway with last stream id 2147483647
2021-04-09T21:01:24.844843+00:00 app[firestore.1]: I0409 21:01:24.844819676      14 chttp2_transport.cc:2869]   transport 0x7f72603d0b90 set connectivity_state=3
2021-04-09T21:01:24.846514+00:00 app[firestore.1]: I0409 21:01:24.846458236      14 chttp2_transport.cc:776]    W:0x7f72603d0b90 CLIENT [ipv4:172.217.13.234:443] state WRITING -> WRITING+MORE [PING_RESPONSE]
2021-04-09T21:01:24.846567+00:00 app[firestore.1]: I0409 21:01:24.846543325      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T21:01:24.846601+00:00 app[firestore.1]: I0409 21:01:24.846579531      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T21:01:24.846640+00:00 app[firestore.1]: I0409 21:01:24.846618734      14 tcp_posix.cc:548]           TCP:0x7f72603ac250 notify_on_read
2021-04-09T21:01:24.846761+00:00 app[firestore.1]: I0409 21:01:24.846733172      14 completion_queue.cc:786]    cq_end_op_for_pluck(cq=0x7f72602d7bd0, tag=0x7f7272d0ed80, error="No Error", done=0x7f72659ca2b0, done_arg=0x7f72602d9cf0, storage=0x7f72602d9d38)
2021-04-09T21:01:24.846926+00:00 app[firestore.1]: I0409 21:01:24.846891744       8 completion_queue.cc:1320]   RETURN_EVENT[0x7f72602d7bd0]: OP_COMPLETE: tag:0x7f7272d0ed80 OK
2021-04-09T21:01:24.847107+00:00 app[firestore.1]: I0409 21:01:24.847081274       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e68)
2021-04-09T21:01:24.847143+00:00 app[firestore.1]: I0409 21:01:24.847121433       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e80)
2021-04-09T21:01:24.847175+00:00 app[firestore.1]: I0409 21:01:24.847155020       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0ea0)
2021-04-09T21:01:24.847243+00:00 app[firestore.1]: I0409 21:01:24.847188500       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0eb8)
2021-04-09T21:01:24.847362+00:00 app[firestore.1]: I0409 21:01:24.847336358       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e68)
2021-04-09T21:01:24.847431+00:00 app[firestore.1]: I0409 21:01:24.847407379       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0e80)
2021-04-09T21:01:24.847466+00:00 app[firestore.1]: I0409 21:01:24.847443072       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0ea0)
2021-04-09T21:01:24.847498+00:00 app[firestore.1]: I0409 21:01:24.847478974       8 metadata_array.cc:29]       grpc_metadata_array_init(array=0x55fc285e0eb8)
2021-04-09T21:01:24.847560+00:00 app[firestore.1]: I0409 21:01:24.847520778       8 call.cc:1971]               grpc_call_start_batch(call=0x7f72602d7da0, ops=0x55fc285e0be0, nops=1, tag=0x7f7272d0ecf0, reserved=(nil))
2021-04-09T21:01:24.847610+00:00 app[firestore.1]: I0409 21:01:24.847587713       8 call.cc:1569]               ops[0]: RECV_STATUS_ON_CLIENT metadata=0x55fc285e0eb8 status=0x55fc285e0ed4 details=0x55fc285e0ed8
2021-04-09T21:01:24.847662+00:00 app[firestore.1]: I0409 21:01:24.847638102       8 chttp2_transport.cc:1605]   perform_stream_op[s=0x7f72603d2e90]:  RECV_TRAILING_METADATA
2021-04-09T21:01:24.855054+00:00 app[firestore.1]: I0409 21:01:24.854999002      14 completion_queue.cc:702]    cq_end_op_for_next(cq=0x7f7260296c90, tag=0x7f72603ab3c0, error="No Error", done=0x7f726596f3c0, done_arg=0x7f72602d7260, storage=0x7f72602d7310)
2021-04-09T21:01:24.855318+00:00 app[firestore.1]: I0409 21:01:24.855287146      14 completion_queue.cc:1083]   RETURN_EVENT[0x7f7260296c90]: OP_COMPLETE: tag:0x7f72603ab3c0 OK
2021-04-09T21:01:24.855373+00:00 app[firestore.1]: I0409 21:01:24.855348930      14 channel_connectivity.cc:41] grpc_channel_check_connectivity_state(channel=0x7f72602d6a70, try_to_connect=0)
2021-04-09T21:01:24.855412+00:00 app[firestore.1]: I0409 21:01:24.855390553      14 channel_connectivity.cc:224] grpc_channel_watch_connectivity_state(channel=0x7f72602d6a70, last_observed_state=0, deadline=gpr_timespec { tv_sec: 9223372036854775807, tv_nsec: 0, clock_type: 1 }, cq=0x7f7260296c90, tag=0x7f726062a840)
2021-04-09T21:01:24.855456+00:00 app[firestore.1]: I0409 21:01:24.855433833      14 completion_queue.cc:974]    grpc_completion_queue_next(cq=0x7f7260296c90, deadline=gpr_timespec { tv_sec: 9223372036854775807, tv_nsec: 0, clock_type: 1 }, reserved=(nil))
2021-04-09T21:01:24.866495+00:00 app[firestore.1]: I0409 21:01:24.866407752      10 chttp2_transport.cc:1347]   perform_stream_op_locked:  RECV_TRAILING_METADATA; on_complete = (nil)
2021-04-09T21:01:24.866504+00:00 app[firestore.1]: D0409 21:01:24.866487584      10 call.cc:1087]               Received trailing metadata with no error and no status
2021-04-09T21:01:24.866570+00:00 app[firestore.1]: I0409 21:01:24.866531598      10 completion_queue.cc:786]    cq_end_op_for_pluck(cq=0x7f72602d7bd0, tag=0x7f7272d0ecf0, error="No Error", done=0x7f72659ca2b0, done_arg=0x55fc2539df10, storage=0x55fc2539df58)
2021-04-09T21:01:24.866620+00:00 app[firestore.1]: I0409 21:01:24.866582169      10 writing.cc:126]             CLIENT: Ping sent [ipv4:172.217.13.234:443]: 2/2
2021-04-09T21:01:24.866621+00:00 app[firestore.1]: I0409 21:01:24.866604936      10 chttp2_transport.cc:776]    W:0x7f72603d0b90 CLIENT [ipv4:172.217.13.234:443] state WRITING+MORE -> WRITING [begin write in current thread]
2021-04-09T21:01:24.866684+00:00 app[firestore.1]: I0409 21:01:24.866640272      10 tcp_posix.cc:1572]          WRITE 0x7f72603ac250 (peer=ipv4:172.217.13.234:443)
2021-04-09T21:01:24.866702+00:00 app[firestore.1]: D0409 21:01:24.866669946      10 tcp_posix.cc:1576]          DATA: 17 03 03 00 51 b9 15 06 37 dd b4 10 4e 7b 61 04 1e 11 ab 6e b6 45 99 80 02 1d 46 16 84 ad 0f ca a8 bc fa 60 f0 98 2b cf c0 74 90 77 40 44 1b e3 35 e1 4f 8f 5d 1c b4 f7 85 6d 49 e4 5c 11 99 1e 2a e1 8b f4 9f 31 cc 8e 41 59 c9 f4 f7 87 d7 87 6c 52 99 51 99 0e '....Q...7...N{a....n.E....F........`[email protected].]....mI.\...*....1..AY......lR.Q..'
2021-04-09T21:01:24.866739+00:00 app[firestore.1]: I0409 21:01:24.866649170       8 completion_queue.cc:1320]   RETURN_EVENT[0x7f72602d7bd0]: OP_COMPLETE: tag:0x7f7272d0ecf0 OK
2021-04-09T21:01:24.866805+00:00 app[firestore.1]: I0409 21:01:24.866778919       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e68)
2021-04-09T21:01:24.866871+00:00 app[firestore.1]: I0409 21:01:24.866796321      10 tcp_posix.cc:1622]          write: "No Error"
2021-04-09T21:01:24.866912+00:00 app[firestore.1]: I0409 21:01:24.866887986      10 chttp2_transport.cc:776]    W:0x7f72603d0b90 CLIENT [ipv4:172.217.13.234:443] state WRITING -> IDLE [finish writing]
2021-04-09T21:01:24.866959+00:00 app[firestore.1]: I0409 21:01:24.866931295      10 chttp2_transport.cc:2587]   ipv4:172.217.13.234:443: Start BDP ping err="No Error"
2021-04-09T21:01:24.867359+00:00 app[firestore.1]: I0409 21:01:24.867008343       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0e80)
2021-04-09T21:01:24.867421+00:00 app[firestore.1]: I0409 21:01:24.867391793       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0ea0)
2021-04-09T21:01:24.867467+00:00 app[firestore.1]: I0409 21:01:24.867438744       8 metadata_array.cc:34]       grpc_metadata_array_destroy(array=0x55fc285e0eb8)
2021-04-09T21:01:24.868756+00:00 app[firestore.1]: I0409 21:01:24.868718140      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T21:01:24.868818+00:00 app[firestore.1]: I0409 21:01:24.868788963      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T21:01:24.868874+00:00 app[firestore.1]: I0409 21:01:24.868846095      14 tcp_posix.cc:685]           TCP:0x7f72603ac250 call_cb 0x7f72603bdad8 0x7f7265b60100:0x7f72603bdaa0
2021-04-09T21:01:24.868936+00:00 app[firestore.1]: I0409 21:01:24.868891060      14 tcp_posix.cc:688]           READ 0x7f72603ac250 (peer=ipv4:172.217.13.234:443) error="No Error"
2021-04-09T21:01:24.869005+00:00 app[firestore.1]: D0409 21:01:24.868970648      14 tcp_posix.cc:695]           DATA: 17 03 03 00 3a f1 71 f5 ab ea ca 7c 3d 11 72 81 22 1a e2 53 e4 d8 d4 41 ab fc bc 5a 29 fe 7e e7 ee b2 08 db 35 ba ec 04 69 5c c5 58 c9 ef 19 0c e0 54 dc 18 e1 b1 60 37 b9 cc 28 ea de 73 a5 '....:.q....|=.r."..S...A...Z).~.....5...i\.X.....T....`7..(..s.'
2021-04-09T21:01:24.869081+00:00 app[firestore.1]: I0409 21:01:24.869052083      14 chttp2_transport.cc:1064]   transport 0x7f72603d0b90 got goaway with last stream id 1
2021-04-09T21:01:24.869175+00:00 app[firestore.1]: I0409 21:01:24.869146341      14 chttp2_transport.cc:2869]   transport 0x7f72603d0b90 set connectivity_state=3
2021-04-09T21:01:24.869224+00:00 app[firestore.1]: I0409 21:01:24.869195694      14 tcp_posix.cc:893]           TCP:0x7f72603ac250 got_read: "No Error"
2021-04-09T21:01:24.869268+00:00 app[firestore.1]: I0409 21:01:24.869240231      14 tcp_posix.cc:885]           TCP:0x7f72603ac250 do_read
2021-04-09T21:01:24.869363+00:00 app[firestore.1]: I0409 21:01:24.869334178      14 tcp_posix.cc:685]           TCP:0x7f72603ac250 call_cb 0x7f72603bdad8 0x7f7265b60100:0x7f72603bdaa0
2021-04-09T21:01:24.869423+00:00 app[firestore.1]: I0409 21:01:24.869394272      14 tcp_posix.cc:688]           READ 0x7f72603ac250 (peer=ipv4:172.217.13.234:443) error={"created":"@1618002084.869326817","description":"Socket closed","fd":9,"file":"src/core/lib/iomgr/tcp_posix.cc","file_line":787,"grpc_status":14,"target_address":"ipv4:172.217.13.234:443"}
2021-04-09T21:01:24.869533+00:00 app[firestore.1]: I0409 21:01:24.869505151      14 chttp2_transport.cc:2611]   ipv4:172.217.13.234:443: Complete BDP ping err="No Error"
2021-04-09T21:01:24.869677+00:00 app[firestore.1]: I0409 21:01:24.869648401      14 chttp2_transport.cc:2869]   transport 0x7f72603d0b90 set connectivity_state=4
2021-04-09T21:01:24.870449+00:00 app[firestore.1]: #<Thread:0x000055fc27c128d8 /app/vendor/bundle/ruby/2.7.0/gems/google-cloud-firestore-2.4.1/lib/google/cloud/firestore/watch/listener.rb:129 run> terminated with exception (report_on_exception is true):
2021-04-09T21:01:24.870635+00:00 app[firestore.1]: /app/vendor/bundle/ruby/2.7.0/gems/grpc-1.36.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status': 2:No status received. debug_error_string:{"created":"@1618002084.866503454","description":"No status received","file":"src/core/lib/surface/call.cc","file_line":1091,"grpc_status":2} (GRPC::Unknown)
2021-04-09T21:01:24.870675+00:00 app[firestore.1]: from /app/vendor/bundle/ruby/2.7.0/gems/grpc-1.36.0-x86_64-linux/src/ruby/lib/grpc/generic/bidi_call.rb:209:in `block in read_loop'
2021-04-09T21:01:24.870701+00:00 app[firestore.1]: from /app/vendor/bundle/ruby/2.7.0/gems/grpc-1.36.0-x86_64-linux/src/ruby/lib/grpc/generic/bidi_call.rb:195:in `loop'
2021-04-09T21:01:24.870721+00:00 app[firestore.1]: from /app/vendor/bundle/ruby/2.7.0/gems/grpc-1.36.0-x86_64-linux/src/ruby/lib/grpc/generic/bidi_call.rb:195:in `read_loop'
2021-04-09T21:01:24.870747+00:00 app[firestore.1]: from lib/listen_to_firestore.rb:in `each'

I ran using GRPC_VERBOSITY=debug GRPC_TRACE=tcp,http,api from your link. The instances of .... in the beginning are me omitting ids, etc.

gilbert on 9 Apr 2021

@gilbert Thank you!!!

quartzmo on 10 Apr 2021

@schmidt-sebastian Could this recurring 2:No status received error in google/cloud/firestore/watch/listener.rb be addressed by adding GRPC::Unknown to the listener's set of retryable errors?

quartzmo on 12 Apr 2021

Yes, you can retry on UNKNOWN.