Google-cloud-php: [Spanner]ErrorException: shm_attach(): failed for key 0x153ef2: No space left on device at shm_attach

[axot]

We are tracing this issue, which caused our Laravel's Job failed to run, I will update this when more information gets to know, Thank you.

ErrorException: shm_attach(): failed for key 0x153ef2: No space left on device
at shm_attach (/project/vendor/google/grpc-gcp/src/Config.php:154)
at Grpc\Gcp\Config->getShmem (/project/vendor/google/grpc-gcp/src/Config.php:154)
at Grpc\Gcp\Config->__construct (/project/vendor/google/grpc-gcp/src/Config.php:63)
at Google\Cloud\Spanner\V1\Gapic\SpannerGapicClient::initGrpcGcpConfig (/project/vendor/google/gax/src/ApiCore/GapicClientTrait.php:129)
at Google\Cloud\Spanner\V1\Gapic\SpannerGapicClient->buildClientOptions (/project/vendor/google/gax/src/ApiCore/GapicClientTrait.php:185)
at Google\Cloud\Spanner\V1\Gapic\SpannerGapicClient->__construct (/project/vendor/google/cloud-spanner/src/V1/Gapic/SpannerGapicClient.php:312)
at Google\Cloud\Spanner\Connection\Grpc->__construct (/project/vendor/google/cloud-spanner/src/Connection/Grpc.php:158)
at Google\Cloud\Spanner\SpannerClient->__construct (/project/vendor/google/cloud-spanner/src/SpannerClient.php:143)

$ ipcs -m | wc -l
4100

[jdpedrie]

If you run composer info | grep "google/grpc-gcp", do you have 0.1.3 installed, or an earlier version?

[axot]

Hi @jdpedrie, I confirmed grpc-gcp version, which 0.1.2 was used. Any related bug fixes between 0.1.2 and 0.1.3?

[jdpedrie]

No, 0.1.2 should be ok. If you were running an older version it may have been an indication of an issue.

We're looking into this further. We will keep you posted with what we find.

[jdpedrie]

Hi @axot,

We've been continuing to investigate this issue. Have you found anything else which might be of note?

How do you have PHP-FPM configured on the server in question? Are you using a dynamic worker pool?

[axot]

I restart to investigate this issue from today, once the same phenomena happens, I will update the thread.

How do you have PHP-FPM configured on the server in question? Are you using a dynamic worker pool?

This happened in standalone php process without php-fpm.

[jdpedrie]

This happened in standalone php process without php-fpm.

In what context was it called? Is the script in question running as a CLI server (i.e. $ php -S), or is there a cron invoking the PHP executable? Do you have the ENABLE_GCP_CHANNEL_MANAGEMENT environment variable set? (If not, please do not enable it.)

[jdpedrie]

@axot, please be advised that until this issue is identified and resolved, we'll be disabling the channel pooling feature in google/cloud-spanner. If you're continuing to test, be sure to use google/cloud-spanner v1.11.4 or google/cloud v0.90.0, not the soon to be released google/cloud-spanner v1.11.5 or google/cloud v0.91.0.

[axot]

@jdpedrie I found this issue happened in both php-fpm and php-cli. The result of ipcs -m will be increased one per process restart.

[jdpedrie]

@axot that corresponds with our finding as well. I'd advise that you upgrade to the latest version of google/cloud or google/cloud-spanner with this feature disabled while we work to get the issue resolved on our side. I'll be sure to keep you updated with the status of the work!

[dwsupplee]

This issue should be resolved now, thanks to https://github.com/GoogleCloudPlatform/grpc-gcp-php/pull/33.