Google-cloud-php: google/cloud-pubsub cURL error 35 gnutls_handshake

[DONSA]

The package google/cloud-core, which is used by google/cloud-pubsub, is throwing the exception referenced below from time to time.

I'm using a PHP flex environment:

• PHP 7.2.18
• curl 7.52.1

exception:  "Google\Cloud\Core\Exception\ServiceException: cURL error 35: gnutls_handshake() failed: The TLS connection was non-properly terminated. (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) in /app/symfony/vendor/google/cloud-core/src/RequestWrapper.php:263
Stack trace:
#0 /app/symfony/vendor/google/cloud-core/src/RequestWrapper.php(168): Google\Cloud\Core\RequestWrapper->convertToGoogleException(Object(GuzzleHttp\Exception\ConnectException))
#1 /app/symfony/vendor/google/cloud-core/src/RestTrait.php(95): Google\Cloud\Core\RequestWrapper->send(Object(GuzzleHttp\Psr7\Request), Array)
#2 /app/symfony/vendor/google/cloud-logging/src/Connection/Rest.php(60): Google\Cloud\Logging\Connection\Rest->send('entries', 'write', Array)
#3 /app/symfony/vendor/google/cloud-logging/src/Logger.php(389): Google\Cloud\Logging\Connection\Rest->writeEntries(Array)
#4 /app/symfony/vendor/google/cloud-logging/src/Logger.php(358): Google\Cloud\Logging\Logger->writeBatch(Array, Array)
#5 /app/symfony/vendor/google/cloud-logging/src/PsrLogger.php(528): Google\Cloud\Logging\Logger->write(Object(Google\Cloud\Logging\Entry))
#6 /app/symfony/vendor/google/cloud-logging/src/PsrLogger.php(413): Google\Cloud\Logging\PsrLogger->sendEntry(Object(Google\Cloud\Logging\Entry))
#7 /app/symfony/src/Services/StackDriverHandler.php(58): Google\Cloud\Logging\PsrLogger->log('critical', 'Uncaught PHP Ex...', Array)
#8 /app/symfony/vendor/monolog/monolog/src/Monolog/Logger.php(337): App\Services\StackDriverHandler->handle(Array)
#9 /app/symfony/vendor/monolog/monolog/src/Monolog/Logger.php(644): Monolog\Logger->addRecord(500, 'Uncaught PHP Ex...', Array)
#10 /app/symfony/vendor/symfony/http-kernel/EventListener/ExceptionListener.php(108): Monolog\Logger->critical('Uncaught PHP Ex...', Array)
#11 /app/symfony/vendor/symfony/http-kernel/EventListener/ExceptionListener.php(48): Symfony\Component\HttpKernel\EventListener\ExceptionListener->logException(Object(Google\Cloud\Core\Exception\ServiceException), 'Uncaught PHP Ex...')
#12 /app/symfony/vendor/symfony/event-dispatcher/EventDispatcher.php(212): Symfony\Component\HttpKernel\EventListener\ExceptionListener->logKernelException(Object(Symfony\Component\HttpKernel\Event\GetResponseForExceptionEvent), 'kernel.exceptio...', Object(Symfony\Component\EventDispatcher\EventDispatcher))
#13 /app/symfony/vendor/symfony/event-dispatcher/EventDispatcher.php(44): Symfony\Component\EventDispatcher\EventDispatcher->doDispatch(Array, 'kernel.exceptio...', Object(Symfony\Component\HttpKernel\Event\GetResponseForExceptionEvent))
#14 /app/symfony/vendor/symfony/http-kernel/HttpKernel.php(219): Symfony\Component\EventDispatcher\EventDispatcher->dispatch('kernel.exceptio...', Object(Symfony\Component\HttpKernel\Event\GetResponseForExceptionEvent))
#15 /app/symfony/vendor/symfony/http-kernel/HttpKernel.php(77): Symfony\Component\HttpKernel\HttpKernel->handleException(Object(Google\Cloud\Core\Exception\ServiceException), Object(Symfony\Component\HttpFoundation\Request), 1)
#16 /app/symfony/vendor/symfony/http-kernel/Kernel.php(188): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#17 /app/symfony/public/index.php(34): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request))
#18 {main}" 

[dwsupplee]

@DONSA this type of error comes from layers down below our client library. Here's a stackoverflow post which may provide some helpful guidance. Please feel free to re-open if there's anything we can do to help here.

[DONSA]

The problem is that my application is hosted in a GCP flex environment.
Do you think creating and using an SSL certificate will help?

[ipansensoy]

Experiencing the same issue.

[ipansensoy]

GoogleCloudCoreExceptionServiceException: cURL error 35: gnutls_handshake() failed: An unexpected TLS packet was received. (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) in /app/vendor/google/cloud/Core/src/RequestWrapper.php:362

[danielgsims]

@DONSA and @acidrainz -

So far I'm unable to reproduce this. Could you two please confirm the versions of PHP, Curl and google cloud packages you're using? Also, are you using the PHP runtime for App Engine, or a custom runtime? Thanks for your patience.

[DONSA]

PHP - PHP 7.2.21-1+0~20190807.25+debian9~1.gbp935ebf (cli) (built: Aug 7 2019 19:49:58) ( NTS )
Curl - curl 7.52.1 (x86_64-pc-linux-gnu) libcurl/7.52.1 OpenSSL/1.0.2s zlib/1.2.8 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) libssh2/1.7.0 nghttp2/1.18.1 librtmp/2.3
google/cloud-pubsub - ^1.6

Try to do multiple concurrent requests to reproduce it.

[dwsupplee]

@danielgsims do you have an update you'd be able to share?

[vrtt]

I'm experiencing this with StorageObject->exists() method.

Using google cloud app engine flex php7.2

Google\Cloud\Core\Exception\ServiceException: cURL error 35: gnutls_handshake() failed: The TLS connection was non-properly terminated. (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)

        at Google\Cloud\Core\RequestWrapper->send (/app/vendor/google/cloud/Core/src/RequestWrapper.php:362)
        at Google\Cloud\Storage\Connection\Rest->send (/app/vendor/google/cloud/Core/src/RestTrait.php:95)
        at Google\Cloud\Storage\Connection\Rest->getObject (/app/vendor/google/cloud/Storage/src/Connection/Rest.php:199)
        at Google\Cloud\Storage\StorageObject->exists (/app/vendor/google/cloud/Storage/src/StorageObject.php:147)
        at App\Controller\ManagerController->sendTranscription (/app/src/Controller/ManagerController.php:762)
        at App\Controller\ManagerController->sendWordfiles (/app/src/Controller/ManagerController.php:635)
        at Symfony\Component\HttpKernel\HttpKernel->handleRaw (/app/vendor/symfony/http-kernel/HttpKernel.php:157)
        at Symfony\Component\HttpKernel\HttpKernel->handle (/app/vendor/symfony/http-kernel/HttpKernel.php:79)
        at Symfony\Component\HttpKernel\Kernel->handle (/app/vendor/symfony/http-kernel/Kernel.php:191)
        at {main} (/app/public/index.php:25)

[brunogaspar]

Having this issue myself, for quite some time, but it's starting to get extremely annoying.

I'm on App Engine Flex PHP 7.3 (pretty much the official Google App Engine PHP Docker image..)

We have a Laravel Artisan command that runs once a day (at around 3am) and stores several files on Google Storage, therefore it performs several uploads, around 500, more or less.

The uploaded files are not that big, are gzipped and are on average around 5kb.

Noticed that while this still happens during the week, is not every single day and can go the entire week without problems sometimes, but during the weekends, it usually fails, a lot.

Here's the stack (similar as above)

Google\Cloud\Core\Exception\ServiceException cURL error 35: gnutls_handshake() failed: An unexpected TLS packet was received. (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) 
    vendor/google/cloud-core/src/RequestWrapper.php:368 Google\Cloud\Core\RequestWrapper::convertToGoogleException
    vendor/google/cloud-core/src/RequestWrapper.php:207 Google\Cloud\Core\RequestWrapper::send
    vendor/google/cloud-core/src/RestTrait.php:97 Google\Cloud\Storage\Connection\Rest::send
    vendor/google/cloud-storage/src/Connection/Rest.php:224 Google\Cloud\Storage\Connection\Rest::getObject
    vendor/google/cloud-storage/src/StorageObject.php:147 Google\Cloud\Storage\StorageObject::exists
    vendor/superbalist/flysystem-google-storage/src/GoogleStorageAdapter.php:326 Superbalist\Flysystem\GoogleStorage\GoogleStorageAdapter::has
    vendor/league/flysystem/src/Filesystem.php:57 League\Flysystem\Filesystem::has
    vendor/league/flysystem/src/Filesystem.php:98 League\Flysystem\Filesystem::put
    vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php:231 Illuminate\Filesystem\FilesystemAdapter::put
    app/Jobs/DailyEmail/CacheMentions.php:96 App\Jobs\DailyEmail\CacheMentions::handle
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}
    vendor/laravel/framework/src/Illuminate/Container/Util.php:37 Illuminate\Container\Util::unwrapIfClosure
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call
    vendor/laravel/framework/src/Illuminate/Container/Container.php:596 Illuminate\Container\Container::call
    vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php:94 Illuminate\Bus\Dispatcher::Illuminate\Bus\{closure}
    vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:128 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
    vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:103 Illuminate\Pipeline\Pipeline::then
    vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php:98 Illuminate\Bus\Dispatcher::dispatchNow
    vendor/laravel/framework/src/Illuminate/Foundation/Bus/Dispatchable.php:53 App\Jobs\AbstractJob::dispatchNow
    app/Jobs/DailyEmail/SendDailyEmail.php:146 App\Jobs\DailyEmail\SendDailyEmail::getDataFromStorage
    app/Jobs/DailyEmail/SendDailyEmail.php:160 App\Jobs\DailyEmail\SendDailyEmail::getData
    app/Jobs/DailyEmail/SendDailyEmail.php:73 App\Jobs\DailyEmail\SendDailyEmail::App\Jobs\DailyEmail\{closure}
    [internal] array_map
    vendor/laravel/framework/src/Illuminate/Support/Collection.php:638 Illuminate\Support\Collection::map
    vendor/laravel/framework/src/Illuminate/Database/Eloquent/Collection.php:276 Illuminate\Database\Eloquent\Collection::map
    app/Jobs/DailyEmail/SendDailyEmail.php:93 App\Jobs\DailyEmail\SendDailyEmail::handle
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}
    vendor/laravel/framework/src/Illuminate/Container/Util.php:37 Illuminate\Container\Util::unwrapIfClosure
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call
    vendor/laravel/framework/src/Illuminate/Container/Container.php:596 Illuminate\Container\Container::call
    vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php:94 Illuminate\Bus\Dispatcher::Illuminate\Bus\{closure}
    vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:128 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
    vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:103 Illuminate\Pipeline\Pipeline::then
    vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php:98 Illuminate\Bus\Dispatcher::dispatchNow
    vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php:83 Illuminate\Queue\CallQueuedHandler::Illuminate\Queue\{closure}
    vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:128 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
    vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:103 Illuminate\Pipeline\Pipeline::then
    vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php:85 Illuminate\Queue\CallQueuedHandler::dispatchThroughMiddleware
    vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php:59 Illuminate\Queue\CallQueuedHandler::call
    vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php:98 Illuminate\Queue\Jobs\Job::fire
    vendor/laravel/framework/src/Illuminate/Queue/Worker.php:356 Illuminate\Queue\Worker::process
    vendor/laravel/framework/src/Illuminate/Queue/Worker.php:306 Illuminate\Queue\Worker::runJob
    vendor/laravel/framework/src/Illuminate/Queue/Worker.php:132 Illuminate\Queue\Worker::daemon
    vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php:112 Illuminate\Queue\Console\WorkCommand::runWorker
    vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php:96 Illuminate\Queue\Console\WorkCommand::handle
    vendor/laravel/horizon/src/Console/WorkCommand.php:46 Laravel\Horizon\Console\WorkCommand::handle
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}
    vendor/laravel/framework/src/Illuminate/Container/Util.php:37 Illuminate\Container\Util::unwrapIfClosure
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call
    vendor/laravel/framework/src/Illuminate/Container/Container.php:596 Illuminate\Container\Container::call
    vendor/laravel/framework/src/Illuminate/Console/Command.php:134 Illuminate\Console\Command::execute
    vendor/symfony/console/Command/Command.php:258 Symfony\Component\Console\Command\Command::run
    vendor/laravel/framework/src/Illuminate/Console/Command.php:121 Illuminate\Console\Command::run
    vendor/symfony/console/Application.php:920 Symfony\Component\Console\Application::doRunCommand
    vendor/symfony/console/Application.php:266 Symfony\Component\Console\Application::doRun
    vendor/symfony/console/Application.php:142 Symfony\Component\Console\Application::run
    vendor/laravel/framework/src/Illuminate/Console/Application.php:93 Illuminate\Console\Application::run
    vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129 Illuminate\Foundation\Console\Kernel::handle
    artisan:37 [main]

Anything i can help to identify the culprit of this? It seems to be caused only (as far as i can tell) for the PHP libraries, the Java one for example, seems to works as expected without major disruptions.

[rubenflush]

@brunogaspar I am experiencing the same error with the same setup, assuming you are running this on Laravel Horizon. Did you find a solution? I think it has to do with some Curl 'leaking'

[brunogaspar]

Hey @rubenflush

Yes this was during some queued jobs on Laravel Horizon.

I was using the official Google App Engine PHP Docker image, i've built my own with updated PHP versions and some other settings, after doing that, i no longer received these errors, but i don't know what was causing it.

[rubenflush]

@brunogaspar Thanks for your quick response! Did you by any chance compile PHP curl with openssl instead of gnutls?

[brunogaspar]

Should be OpenSSL, but i've not manually compiled it though, just installed both curl and then the php extension and that's it.