Gsutil: All of my `gsutil ` commands cannot works and return `socket.timeout: timed out`

Created on 29 Jan 2018 · 15Comments · Source: GoogleCloudPlatform/gsutil

my version:

gcloud info
Google Cloud SDK [185.0.0]
...
Python Version: [2.7.12 (default, Nov 20 2017, 18:23:56)  [GCC 5.4.0 20160609]]
Python Location: [/usr/bin/python2]
Site Packages: [Disabled]

Installation Root: [/home/david/important/TS_P/google-cloud-sdk]
Installed Components:
  core: [2018.01.12]
  gsutil: [4.28]
  bq: [2.0.28]
...
Installation Properties: [/home/david/important/TS_P/google-cloud-sdk/properties]
User Config Directory: [/home/david/.config/gcloud]
Active Configuration Name: [default]
Active Configuration Path: [/home/david/.config/gcloud/configurations/config_default]
Logs Directory: [/home/david/.config/gcloud/logs]
Last Log File: [/home/david/.config/gcloud/logs/2018.01.29/08.58.57.179050.log]

git: [git version 2.7.4]
ssh: [OpenSSH_7.2p2 Ubuntu-4ubuntu2.4, OpenSSL 1.0.2g  1 Mar 2016]

I have a poxy(socks5) on my linux .And I can run gcloud init and connect successfully.But my all gsutil commands cannot works and have no request timeout,like these:

gsutil ls -d
INFO 0128 22:56:07.070065 util.py] Retrying request, attempt #1...
INFO 0128 22:57:09.526079 util.py] Retrying request, attempt #2...
INFO 0128 22:58:13.198094 util.py] Retrying request, attempt #3...
...
INFO 0128 23:26:39.959507 util.py] Retrying request, attempt #22...
Traceback (most recent call last):
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gsutil", line 22, in <module>
    gsutil.RunMain()
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gsutil.py", line 114, in RunMain
    sys.exit(gslib.__main__.main())
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 383, in main
    perf_trace_token=perf_trace_token)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 577, in _RunNamedCommandAndHandleExceptions
    collect_analytics=True)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 299, in RunNamedCommand
    return_code = command_inst.RunCommand()
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/commands/ls.py", line 464, in RunCommand
    bucket_fields=bucket_fields):
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 456, in IterBuckets
    for blr in self._ExpandBucketWildcards(bucket_fields=bucket_fields):
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 375, in _ExpandBucketWildcards
    provider=self.wildcard_url.scheme):
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 531, in ListBuckets
    global_params=global_params)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/third_party/storage_apitools/storage_v1_client.py", line 425, in List
    config, request, global_params=global_params)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/base_api.py", line 701, in _RunMethod
    http, http_request, **opts)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 341, in MakeRequest
    check_response_func=check_response_func)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 391, in _MakeRequestNoRetry
    redirections=redirections, connection_type=connection_type)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/oauth2client/oauth2client/client.py", line 616, in new_request
    self._refresh(request_orig)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/oauth2client/oauth2client/client.py", line 885, in _refresh
    self._do_refresh_request(http_request)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/oauth2client/oauth2client/client.py", line 905, in _do_refresh_request
    self.token_uri, method='POST', body=body, headers=headers)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/httplib2/python2/httplib2/__init__.py", line 1570, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/httplib2/python2/httplib2/__init__.py", line 1317, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/httplib2/python2/httplib2/__init__.py", line 1252, in _conn_request
    conn.connect()
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/httplib2/python2/httplib2/__init__.py", line 1018, in connect
    sock.connect((self.host, self.port))
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/httplib2/python2/httplib2/socks.py", line 424, in connect
    self.__negotiatehttp(destpair[0], destpair[1])
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/httplib2/python2/httplib2/socks.py", line 376, in __negotiatehttp
    resp = resp + self.recv(1)
socket.timeout: timed out

How to solve it?

Source

hhh151671

Most helpful comment

I'm having the same problem just using cp.
This utility is a janky way to copy files. Come on Google.

kaizen1 on 20 May 2018

👍4

All 15 comments

This issue isn't specific to gsutil; we've seen a slightly-higher-than-usual number of socket timeouts to GCS in general over the past week or so. The GCS team is looking into it. For Google developers wanting more information, the issue tracker number is 72221304 (sorry to external users; the visibility for that issue is internal-only).

Given that this isn't a gsutil bug, I'll go ahead and close this GitHub issue.

houglum on 1 Feb 2018

I'm having the same problem just using cp.
This utility is a janky way to copy files. Come on Google.

kaizen1 on 20 May 2018

👍4

I had the same problem, for me it was a firewall configuration issue. I had an egress deny on tcp 80,443, 8080, which explains the socket timeout. I changed this for my service account and now it works. Hope this helps

hollicksfeir on 1 Aug 2018

INFO 0306 14:44:04.069279 retry_util.py] Retrying request, attempt #6...

jvargh81 on 6 Mar 2019

Same issue, how do I even troubleshoot this?

gsutil ls
INFO 0417 16:07:04.280598 retry_util.py] Retrying request, attempt #4...
INFO 0417 16:07:16.602803 retry_util.py] Retrying request, attempt #5...
INFO 0417 16:07:44.378415 retry_util.py] Retrying request, attempt #6...

I was just following this tutorial:

https://cloud.google.com/vision/automl/docs/quickstart?refresh=1

denfromufa on 17 Apr 2019

BTW, I'm using the cloud shell, so supposedly no firewall issues should apply to me?

denfromufa on 17 Apr 2019

@denfromufa you can use the top-level -D flag to see the command being run, which request is failing, etc.

I can get this to happen as well -- I think this occurs because the "gsutil" executable on Cloud Shell instances points to the gcloud installation of gsutil, which assumes that if you're running on GCE, you're on an instance owned by your project. Normally, it will just use the GCE auth plugin to query the VM instance's metadata server and get credentials for your instance's default service account. However, since the instance is owned by the Cloud Shell project, it can't do that, and it fails when trying to fetch those credentials (I see a BadStatusLine error when running gsutil -D ls).

I confirmed this by creating a config file with my credentials at ~/.boto, then running the underlying gsutil executable (/google/google-cloud-sdk/platform/gsutil/gsutil) directly -- this works fine, whereas the gcloud gsutil executable fails because it proxies an extra option through to gsutil that causes it to try and fetch credentials from the GCE metadata server (the -D output shows the command as Command being run: /google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=myproject -o GoogleCompute:service_account=default -D ls).

This is an issue with the Cloud SDK -- it should detect if users are running on the Cloud Shell, and if so, not pass the GoogleCompute:service_account=default option through to gsutil. Please file an issue with the Cloud SDK team by clicking the "Create new Cloud SDK issue" link at https://cloud.google.com/support/docs/issue-trackers.

houglum on 18 Apr 2019

@houglum it works today without any changes ¯\_(ツ)_/¯

denfromufa on 18 Apr 2019

Still fails for me today. I'd still guess that the problem comes from trying to obtain the initial access token, but I'd have to add some import pdb; pdb.set_trace() statements down in the depths of the system's httplib module and poke around to be sure.

houglum on 18 Apr 2019

I'm facing the same issue from my compute engine. Yesterday the gsutil command was working fine. I didn't do any change over the night. If i run the command enabling debug mode, i'm getting following error :

DEBUG 1124 03:46:39.659342 http_wrapper.py] Caught server not found error, retrying: Unable to find the server at storage.googleapis.com
DEBUG 1124 03:46:39.659546 http_wrapper.py] Retrying request to url  after exception Unable to find the server at storage.googleapis.com

shihabuddinbuet on 24 Nov 2019

I'm facing the same issue from my compute engine. Yesterday the gsutil command was working fine. I didn't do any change over the night. If i run the command enabling debug mode, i'm getting following error :
DEBUG 1124 03:46:39.659342 http_wrapper.py] Caught server not found error, retrying: Unable to find the server at storage.googleapis.com
DEBUG 1124 03:46:39.659546 http_wrapper.py] Retrying request to url  after exception Unable to find the server at storage.googleapis.com

It's working now without any change. Don't know why. :(

shihabuddinbuet on 24 Nov 2019

I had the same problem. It was ipv6.

What I tried was: gsutil -D blabla
It has shown me the url. I copied it and went to wget as
wget "URL" (with quotes, as the URL has some &)

It got lost trying to resolve ipv6 name - Disabled ipv6 and all is well in the world.

surak on 7 Aug 2020

👍2

Been having this problem for the third day in a row. Disabling IPv6 did not help. I just cannot find out what is going on.

The weird part is that some other things break too and it looks like it's not deterministic... like sometimes go get, apk add, or apt-get install fail (mainly building Docker containers), sometimes docker pull... and everything from Cloud SDK that touches the internet. gcloud init takes forever to pass the networking test.

But nslookup, ping, dig are ALL fine (and working pretty fast). Browsing the internet via Firefox or Chrome too. I tried with DNS servers from Google, CloudFlare and my network provider (either set in my router or directly in my computer).

All of this happens in Fedora 32 (kernel Linux 5.8.13-200.fc32.x86_64) and Ubuntu 20.04 (didn't remember to take note of the kernel, but it was the latest available). Cloud SDK versions:

$ gcloud info
Google Cloud SDK [313.0.0]

Platform: [Linux, x86_64] uname_result(system='Linux', node='xps', release='5.8.13-200.fc32.x86_64', version='#1 SMP Thu Oct 1 21:49:42 UTC 2020', machine='x86_64', processor='x86_64')
Locale: ('en_GB', 'UTF-8')
Python Version: [3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0]]
Python Location: [/home/gbrl/miniconda3/envs/gcloud/bin/python3]
Site Packages: [Disabled]

Installation Root: [/home/gbrl/google-cloud-sdk]
Installed Components:
  core: [2020.10.02]
  gsutil: [4.53]
  bq: [2.0.61]
System PATH: [/home/gbrl/google-cloud-sdk/bin:/home/gbrl/miniconda3/envs/gcloud/bin:/home/gbrl/miniconda3/condabin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/gbrl/bin]
Python PATH: [/home/gbrl/google-cloud-sdk/lib/third_party:/home/gbrl/google-cloud-sdk/lib:/home/gbrl/miniconda3/envs/gcloud/lib/python38.zip:/home/gbrl/miniconda3/envs/gcloud/lib/python3.8:/home/gbrl/miniconda3/envs/gcloud/lib/python3.8/lib-dynload]
Cloud SDK on PATH: [True]
Kubectl on PATH: [False]

Installation Properties: [/home/gbrl/google-cloud-sdk/properties]
User Config Directory: [/home/gbrl/.config/gcloud]
Active Configuration Name: [default]
Active Configuration Path: [/home/gbrl/.config/gcloud/configurations/config_default]

Account: [<my-email>]
Project: [<my-project>]

Current Properties:
  [core]
    account: [<my-email>]
    disable_usage_reporting: [True]
    project: [<my-project>]

Logs Directory: [/home/gbrl/.config/gcloud/logs]
Last Log File: [/home/gbrl/.config/gcloud/logs/2020.10.08/10.40.47.488518.log]

git: [git version 2.26.2]
ssh: [OpenSSH_8.3p1, OpenSSL 1.1.1g FIPS  21 Apr 2020]

Feel free to ping me if someone needs more information.

ggabriel96 on 8 Oct 2020

Well... apparently the problems with Docker and Cloud SDK were unrelated, because all of a sudden gsutil and friends started working again for some reason that I can't explain. Then I could focus on Docker and fixed that later. I guess I'll never know what happened, because this behavior ocurred on 2 fresh-installs (Fedora 32 and Ubuntu 20.04). :man_shrugging:

ggabriel96 on 8 Oct 2020

Nice debugging @surak.
Thank you for saving my day