Python-slack-sdk: Issue: Running into `URLError <urlopen error The write operation timed out>` when uploading CSV file through API

[praveen2593]

Description

Hi Folks, I am running into URLError <urlopen error The write operation timed out> when uploading CSV file of large size (~200MB). The file is eventually uploaded into slack, yet the code breaks with that error. Can anyone explain why that's happening?

What type of issue is this? (place an x in one of the [ ])

• [ ] bug
• [ ] enhancement (feature request)
• [x] question
• [ ] documentation related
• [ ] testing related
• [ ] discussion

Requirements (place an x in each of the [ ])

• [x] I've read and understood the Contributing guidelines and have done my best effort to follow them.
• [x] I've read and agree to the Code of Conduct.
• [x] I've searched for any related issues and avoided creating a duplicate issue.

---

Bug Report

Filling out the following details about bugs will help us solve your issue sooner.

Reproducible in:

slackclient version: 2.7.1

python version: 3.6.5

OS version(s): macOS 10.14.6

Steps to reproduce:

1. Create a webclient with the necessary permissions
2. Create/Load a dataframe with size ~200MB as df (the one I tested with had a shape of (5400000, 6))
3. use the files_upload method to load files using the command as shown web_client.files_upload(channels = 'CXXXXXXXXXX', initial_comment="something", content=df.to_csv(index=False), filename="test.csv")

Expected result:

What you expected to happen: File sent with response 'ok':true

Actual result:

What actually happened:

timeout                                   Traceback (most recent call last)
~/anaconda3/lib/python3.6/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1317                 h.request(req.get_method(), req.selector, req.data, headers,
-> 1318                           encode_chunked=req.has_header('Transfer-encoding'))
   1319             except OSError as err: # timeout error

~/anaconda3/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1238         """Send a complete request to the server."""
-> 1239         self._send_request(method, url, body, headers, encode_chunked)
   1240

~/anaconda3/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1284             body = _encode(body, 'body')
-> 1285         self.endheaders(body, encode_chunked=encode_chunked)
   1286

~/anaconda3/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1233             raise CannotSendHeader()
-> 1234         self._send_output(message_body, encode_chunked=encode_chunked)
   1235

~/anaconda3/lib/python3.6/http/client.py in _send_output(self, message_body, encode_chunked)
   1064                         + b'\r\n'
-> 1065                 self.send(chunk)
   1066

~/anaconda3/lib/python3.6/http/client.py in send(self, data)
    985         try:
--> 986             self.sock.sendall(data)
    987         except TypeError:

~/anaconda3/lib/python3.6/ssl.py in sendall(self, data, flags)
    971                 while count < amount:
--> 972                     v = self.send(byte_view[count:])
    973                     count += v

~/anaconda3/lib/python3.6/ssl.py in send(self, data, flags)
    940                     self.__class__)
--> 941             return self._sslobj.write(data)
    942         else:

~/anaconda3/lib/python3.6/ssl.py in write(self, data)
    641         """
--> 642         return self._sslobj.write(data)
    643

timeout: The write operation timed out
During handling of the above exception, another exception occurred:                                                                                                                                                                                                     [74/639]

URLError                                  Traceback (most recent call last)
<ipython-input-16-e4c541ebf2f5> in <module>
----> 1 web_client.files_upload(channels = 'CXXXXXXXXXX', initial_comment="something", content=df.to_csv(index=False), filename="test.csv")

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/client.py in files_upload(self, file, content, **kwargs)
   1217         data = kwargs.copy()
   1218         data.update({"content": content})
-> 1219         return self.api_call("files.upload", data=data)
   1220
   1221     def groups_archive(self, *, channel: str, **kwargs) -> Union[Future, SlackResponse]:

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in api_call(self, api_method, http_verb, files, data, params, json, headers, auth)
    213                 return self._event_loop.run_until_complete(future)
    214         else:
--> 215             return self._sync_send(api_url=api_url, req_args=req_args)
    216
    217     def _get_url(self, api_method):

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _sync_send(self, api_url, req_args)
    346             files=files,
    347             json_body=_json,
--> 348             additional_headers=headers,
    349         )
    350

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _urllib_api_call(self, token, url, query_params, json_body, body_params, files, additional_headers)
    448                 url = f"{url}&{q}" if "?" in url else f"{url}?{q}"
    449
--> 450             response = self._perform_urllib_http_request(url=url, args=request_args)
    451             if response.get("body", None):
    452                 response_body_data: dict = json.loads(response["body"])

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _perform_urllib_http_request(self, url, args)
    579         except Exception as err:
    580             self._logger.error(f"Failed to send a request to Slack API server: {err}")
--> 581             raise err
    582
    583     def _build_urllib_request_headers(

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _perform_urllib_http_request(self, url, args)
    560
    561                 resp: HTTPResponse = urlopen(
--> 562                     req, context=self.ssl, timeout=self.timeout
    563                 )
    564                 charset = resp.headers.get_content_charset()

~/anaconda3/lib/python3.6/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    221     else:
    222         opener = _opener
--> 223     return opener.open(url, data, timeout)
    224
    225 def install_opener(opener):

~/anaconda3/lib/python3.6/urllib/request.py in open(self, fullurl, data, timeout)
    524             req = meth(req)
    525
--> 526         response = self._open(req, data)
    527
    528         # post-process response
~/anaconda3/lib/python3.6/urllib/request.py in _open(self, req, data)
    542         protocol = req.type
    543         result = self._call_chain(self.handle_open, protocol, protocol +
--> 544                                   '_open', req)
    545         if result:
    546             return result

~/anaconda3/lib/python3.6/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
    502         for handler in handlers:
    503             func = getattr(handler, meth_name)
--> 504             result = func(*args)
    505             if result is not None:
    506                 return result

~/anaconda3/lib/python3.6/urllib/request.py in https_open(self, req)
   1359         def https_open(self, req):
   1360             return self.do_open(http.client.HTTPSConnection, req,
-> 1361                 context=self._context, check_hostname=self._check_hostname)
   1362
   1363         https_request = AbstractHTTPHandler.do_request_

~/anaconda3/lib/python3.6/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1318                           encode_chunked=req.has_header('Transfer-encoding'))
   1319             except OSError as err: # timeout error
-> 1320                 raise URLError(err)
   1321             r = h.getresponse()
   1322         except:

URLError: <urlopen error The write operation timed out>

[seratch]

Seemingly the uploading request took more than 30 seconds. The default timeout is the period of time. You can customize the value by timeout arg in the constructor.

• https://github.com/slackapi/python-slackclient/blob/v2.7.2/slack/web/base_client.py#L41
• https://github.com/slackapi/python-slackclient/blob/v2.7.2/slack/web/base_client.py#L564

[praveen2593]

So now it throws a URLError: <urlopen error [Errno 32] Broken pipe>
Is there a limit on file size?

BrokenPipeError                           Traceback (most recent call last)
~/anaconda3/lib/python3.6/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1317                 h.request(req.get_method(), req.selector, req.data, headers,
-> 1318                           encode_chunked=req.has_header('Transfer-encoding'))
   1319             except OSError as err: # timeout error

~/anaconda3/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1238         """Send a complete request to the server."""
-> 1239         self._send_request(method, url, body, headers, encode_chunked)
   1240

~/anaconda3/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1284             body = _encode(body, 'body')
-> 1285         self.endheaders(body, encode_chunked=encode_chunked)
   1286

~/anaconda3/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1233             raise CannotSendHeader()
-> 1234         self._send_output(message_body, encode_chunked=encode_chunked)
   1235

~/anaconda3/lib/python3.6/http/client.py in _send_output(self, message_body, encode_chunked)
   1064                         + b'\r\n'
-> 1065                 self.send(chunk)
   1066

~/anaconda3/lib/python3.6/http/client.py in send(self, data)
    985         try:
--> 986             self.sock.sendall(data)
    987         except TypeError:

~/anaconda3/lib/python3.6/ssl.py in sendall(self, data, flags)
    971                 while count < amount:
--> 972                     v = self.send(byte_view[count:])
    973                     count += v

~/anaconda3/lib/python3.6/ssl.py in send(self, data, flags)
    940                     self.__class__)
--> 941             return self._sslobj.write(data)
    942         else:

~/anaconda3/lib/python3.6/ssl.py in write(self, data)
    641         """
--> 642         return self._sslobj.write(data)
    643

BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

URLError                                  Traceback (most recent call last)
<ipython-input-8-75bdb47149f2> in <module>
----> 1 test_slacker.upload_dataframe(data=df, filename="check_result.csv", msg="Adding Large DataFrame as CSV")

~/Documents/scratch_folder/slack_testing.py in upload_dataframe(self, data, msg, filename, filepath)
     98                     content=data.to_csv(index=False),
     99                     msg=msg,
--> 100                     filename=filename,
    101                 )
    102                 # TODO: Log this

~/Documents/scratch_folder/slack_testing.py in _upload_content(self, content, msg, filename)
     79         except Exception as err:
     80             # TODO: Log this error
---> 81             raise err
     82         # TODO: Log this
     83         print("File Uploaded")

~/Documents/scratch_folder/slack_testing.py in _upload_content(self, content, msg, filename)
     75                 initial_comment=msg,
     76                 content=content,
---> 77                 filename=filename,
     78             )
     79         except Exception as err:

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/client.py in files_upload(self, file, content, **kwargs)
   1217         data = kwargs.copy()
   1218         data.update({"content": content})
-> 1219         return self.api_call("files.upload", data=data)
   1220
   1221     def groups_archive(self, *, channel: str, **kwargs) -> Union[Future, SlackResponse]:

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in api_call(self, api_method, http_verb, files, data, params, json, headers, auth)
    213                 return self._event_loop.run_until_complete(future)
    214         else:
--> 215             return self._sync_send(api_url=api_url, req_args=req_args)
    216
    217     def _get_url(self, api_method):

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _sync_send(self, api_url, req_args)
    346             files=files,
    347             json_body=_json,
--> 348             additional_headers=headers,
    349         )
    350

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _urllib_api_call(self, token, url, query_params, json_body, body_params, files, additional_headers)
    448                 url = f"{url}&{q}" if "?" in url else f"{url}?{q}"
    449
--> 450             response = self._perform_urllib_http_request(url=url, args=request_args)
    451             if response.get("body", None):
    452                 response_body_data: dict = json.loads(response["body"])

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _perform_urllib_http_request(self, url, args)
    579         except Exception as err:
    580             self._logger.error(f"Failed to send a request to Slack API server: {err}")
--> 581             raise err
    582
    583     def _build_urllib_request_headers(

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _perform_urllib_http_request(self, url, args)
    560
    561                 resp: HTTPResponse = urlopen(
--> 562                     req, context=self.ssl, timeout=self.timeout
    563                 )
    564                 charset = resp.headers.get_content_charset()
~/anaconda3/lib/python3.6/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    221     else:
    222         opener = _opener
--> 223     return opener.open(url, data, timeout)
    224
    225 def install_opener(opener):

~/anaconda3/lib/python3.6/urllib/request.py in open(self, fullurl, data, timeout)
    524             req = meth(req)
    525
--> 526         response = self._open(req, data)
    527
    528         # post-process response

~/anaconda3/lib/python3.6/urllib/request.py in _open(self, req, data)
    542         protocol = req.type
    543         result = self._call_chain(self.handle_open, protocol, protocol +
--> 544                                   '_open', req)
    545         if result:
    546             return result

~/anaconda3/lib/python3.6/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
    502         for handler in handlers:
    503             func = getattr(handler, meth_name)
--> 504             result = func(*args)
    505             if result is not None:
    506                 return result

~/anaconda3/lib/python3.6/urllib/request.py in https_open(self, req)
   1359         def https_open(self, req):
   1360             return self.do_open(http.client.HTTPSConnection, req,
-> 1361                 context=self._context, check_hostname=self._check_hostname)
   1362
   1363         https_request = AbstractHTTPHandler.do_request_

~/anaconda3/lib/python3.6/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1318                           encode_chunked=req.has_header('Transfer-encoding'))
   1319             except OSError as err: # timeout error
-> 1320                 raise URLError(err)
   1321             r = h.getresponse()
   1322         except:

URLError: <urlopen error [Errno 32] Broken pipe>

[seratch]

Is there a limit on file size?

It's up to 1Gb in size but, when your operation to upload a file takes long, timeouts or network issues between Slack API and your end may occur much more often.

If you set run_async=True in the WebClient constructor, this API client internally uses aiohttp. In this mode, it runs asynchronously (you need to use async/await in your code). If the async way works more stably for your use case, switching to the async mode may be an option.

As your file size is huge, another way to handle would be to utilize yet another online storage platform (e.g., Google Drive, Box, etc.) and share the permalink of the file in Slack.

If you use files.remote.add API for sharing a file link in Slack, you can customize the appearance by adding a preview image in Slack (preview_image) and also can register text data (indexable_file_contents) for the file into Slack's search engine index.

[praveen2593]

I'll try the async mode, to see if that works. And about the remote sharing, is it possible to use aws s3 platform here (not sure if it's considered as an online storage platform)?

[seratch]

You can use S3 as long as the files are accessible to your end-users. You don't need to give a permission to Slack API servers.

[Arszilla]

@seratch is it possible to set the timeout to be "indefinite" i.e never timeout? If I use timeout=0 I get urllib.error.URLError: <urlopen error [Errno 115] Operation now in progress>, which can be resolved by setting timeout=0 to any other number besides 0, but I'd rather have an "indefinite" timeout where the upload would retry until it succeeds.

[seratch]

@Arszilla We're not planning to implement such. Please set an extremely large timeout and implement your own retry logic.

[Arszilla]

Understandable. Right now I have a 600 second timeout but was curious if an "indefinite" setting was to be implemented.

Cheers @seratch

[seratch]

@Arszilla I'm glad to know you've found your solution around this issue. Can we close this issue now as it seems we don't have anything further to discuss here?

[praveen2593]

@seratch Thanks. We can close it. I'm good on my end!! 👍