Aws-data-wrangler: Error when sending DataFrame to S3 using a boto3 Session

Created on 13 Mar 2020  路  16Comments  路  Source: awslabs/aws-data-wrangler

Hi.

I am trying to send a DataFrame to S3 by using a previously created boto3 session and get the following error:

Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 908, in _data_to_s3_dataset_writer_remote
    isolated_dataframe=True))
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 853, in _data_to_s3_dataset_writer
    isolated_dataframe=isolated_dataframe)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 962, in _data_to_s3_object_writer
    extra_args=extra_args)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 999, in _write_csv_dataframe
    Pandas._write_csv_to_s3_retrying(fs=fs, path=path, buffer=csv_buffer)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 241, in wrapped_f
    return self.call(f, *args, **kw)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 330, in call
    start_time=start_time)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 279, in iter
    return fut.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 333, in call
    result = fn(*args, **kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 1009, in _write_csv_to_s3_retrying
    f.write(buffer)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/fsspec/spec.py", line 1245, in __exit__
    self.close()
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/fsspec/spec.py", line 1213, in close
    self.flush(force=True)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/fsspec/spec.py", line 1085, in flush
    self._initiate_upload()
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/s3fs/core.py", line 1002, in _initiate_upload
    Bucket=self.bucket, Key=self.key, ACL=self.acl)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/s3fs/core.py", line 991, in _call_s3
    **kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/s3fs/core.py", line 184, in _call_s3
    return method(**additional_kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/client.py", line 276, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/client.py", line 573, in _make_api_call
    operation_model, request_dict, request_context)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/client.py", line 592, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/signers.py", line 160, in sign
    auth.add_auth(request)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 908, in _data_to_s3_dataset_writer_remote
    isolated_dataframe=True))
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 853, in _data_to_s3_dataset_writer
    isolated_dataframe=isolated_dataframe)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 962, in _data_to_s3_object_writer
    extra_args=extra_args)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 999, in _write_csv_dataframe
    Pandas._write_csv_to_s3_retrying(fs=fs, path=path, buffer=csv_buffer)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 241, in wrapped_f
    return self.call(f, *args, **kw)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 330, in call
    start_time=start_time)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 279, in iter
    return fut.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 333, in call
    result = fn(*args, **kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 1009, in _write_csv_to_s3_retrying
    f.write(buffer)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/fsspec/spec.py", line 1245, in __exit__
    self.close()
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/fsspec/spec.py", line 1213, in close
    self.flush(force=True)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/fsspec/spec.py", line 1085, in flush
    self._initiate_upload()
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/s3fs/core.py", line 1002, in _initiate_upload
    Bucket=self.bucket, Key=self.key, ACL=self.acl)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/s3fs/core.py", line 991, in _call_s3
    **kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/s3fs/core.py", line 184, in _call_s3
    return method(**additional_kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/client.py", line 276, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/client.py", line 573, in _make_api_call
    operation_model, request_dict, request_context)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/client.py", line 592, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/signers.py", line 160, in sign
    auth.add_auth(request)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
Traceback (most recent call last):
  File "awswrangler_test.py", line 14, in <module>
    wr_session.pandas.to_csv(df, "s3://blu-datalake/test/foobar")
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 592, in to_csv
    columns_comments=columns_comments)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 729, in to_s3
    extra_args=extra_args)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 805, in data_to_s3
    objects_paths += receive_pipes[i].recv()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

I believe it i trying to get the credentials from the default profile in ~/.aws/credentials.

Here is an example that replicate the error:

import pandas as pd
import awswrangler as wr
import boto3

boto3_session = boto3.Session(aws_access_key_id="****", aws_secret_access_key="****",)

wr_session = wr.Session(boto3_session=boto3_session)

df = pd.DataFrame(dict(a=[1, 2], b=[3, 4]))

wr_session.pandas.to_csv(df, "s3://foo/bar")

Is this the right way to use a boto3 Session?

Major Release bug
Source
picture gabraganca

All 16 comments

Hi @gabraganca, thank you for reaching out.

Probably something internal is trying to find the related region.

Please try to add the region_name argument to your boto3.Session.

e.g. 

boto3_session = boto3.Session(
    aws_access_key_id="...",
    aws_secret_access_key="...",
    region_name="us-east-1"
)
igorborgest picture igorborgest on 16 Mar 2020

Hi @igorborgest . Thanks for your reponse.

I tried adding the region_name but I got the same error.

But when I use the profile_name arguments, it works:

boto3_session = boto3.Session(
    # aws_access_key_id="...",
    # aws_secret_access_key="...",
    # region_name="us-east-1",
    profile_name="my-other-profile"
)

This is very strange... Unfortunately, I can't use the profile since I am not using the credentials file.

I'm using the following lib versions :

awswrangler==0.3.2
boto3==1.12.21
botocore==1.15.21

Any clues?

gabraganca picture gabraganca on 16 Mar 2020

Hi @gabraganca, now it really seems like a bug. 馃悰

I will start work to reproduce it here. But in this meanwhile I can think in three alternatives that could help you to overcome it:

1 - Pass your credentials directly to awswrangler.Session like:

import awswrangler as wr

wr_session = wr.Session(
    aws_access_key_id="...",
    aws_secret_access_key="...",
    region_name="us-east-1"
)

2 - Avoid the session serialization/deserialization through multiple processing passing procs_cpu_bound=1 in the to_csv method like:

wr_session.pandas.to_csv(df, "s3://foo/bar", procs_cpu_bound=1)

3 - 1 and 2 combined.

I will work on that right now, so please, let me know any updates from your side.

igorborgest picture igorborgest on 16 Mar 2020
👍1

Great, @igorborgest . Actually, I found this bug when running to_parquet, but the to_csv method returns the same error.

gabraganca picture gabraganca on 16 Mar 2020
👍1

I just tried the following code and got the same error:

import pandas as pd
import awswrangler as wr
import boto3

boto3_session = boto3.Session(
    aws_access_key_id="...",
    aws_secret_access_key="...",
    region_name="us-east-1",
)

wr_session = wr.Session(boto3_session=boto3_session)

df = pd.DataFrame(dict(a=[1, 2], b=[3, 4]))

wr_session.pandas.to_csv(df, "s3://foo/bar", procs_cpu_bound=1)

Traceback (most recent call last):
File "awswrangler_test.py", line 15, in
wr_session.pandas.to_csv(df, "s3://blu-datalake/test/foobar", procs_cpu_bound=1)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 592, in to_csv
columns_comments=columns_comments)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 729, in to_s3
extra_args=extra_args)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 818, in data_to_s3
isolated_dataframe=isolated_dataframe)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 853, in _data_to_s3_dataset_writer
isolated_dataframe=isolated_dataframe)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 962, in _data_to_s3_object_writer
extra_args=extra_args)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 999, in _write_csv_dataframe
Pandas._write_csv_to_s3_retrying(fs=fs, path=path, buffer=csv_buffer)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 241, in wrapped_f
return self.call(f, args, kw)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 330, in call
start_time=start_time)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 279, in iter
return fut.result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 333, in call
result = fn(args, *kwargs)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 1009, in _write_csv_to_s3_retrying
f.write(buffer)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/fsspec/spec.py", line 1245, in __exit__
self.close()
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/fsspec/spec.py", line 1213, in close
self.flush(force=True)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/fsspec/spec.py", line 1085, in flush
self._initiate_upload()
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/s3fs/core.py", line 1002, in _initiate_upload
Bucket=self.bucket, Key=self.key, ACL=self.acl)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/s3fs/core.py", line 991, in _call_s3
*kwargs)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/s3fs/core.py", line 184, in _call_s3
return method(additional_kwargs)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/client.py", line 613, in _make_api_call
operation_model, request_dict, request_context)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/client.py", line 632, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request
request = self.create_request(request_dict, operation_model)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request
operation_name=operation_model.name)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, kwargs)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(kwargs)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/signers.py", line 160, in sign
auth.add_auth(request)
File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/auth.py", line 357, in add_auth
raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials

gabraganca picture gabraganca on 16 Mar 2020
👍1

@gabraganca

It makes sense. Testing it here, and everything points to be a really a session serialization/deserialization issue.

So probably the better approach will be focus in the version 1.0.0 that is coming with a new strategy remaining sessions.

We will keep no more states inside AWS Wrangler, this mean that all the session will be explicit under the user control.

If you could take a look under our non-productive branch, you see that to_parquet and to_csv are being developed with explicit boto3_session arguments.

You can easily install and test the version 1.0.0 through pip install git+https://github.com/awslabs/[email protected]

And check the current documentation under: https://aws-data-wrangler.readthedocs.io/en/dev-1.0.0/api/

igorborgest picture igorborgest on 16 Mar 2020

@gabraganca could you also test the alternative [1], please?

igorborgest picture igorborgest on 16 Mar 2020

@igorborgest thanks for looking into it.

About alternative [1], I think I've already tried since it is similar of the example I provided. Or am I missing something?

I am looking forward to see version 1.0 on the master branch soon.

gabraganca picture gabraganca on 16 Mar 2020

@gabraganca Actually the alternative [1] is different from your code snippets. It basically means to pass your keys directly to wr.Session() w/o any boto3.Session() in the middle.

igorborgest picture igorborgest on 17 Mar 2020

@igorborgest Oh, I see. I didn't know that was possible. That makes everything easier.

I'll try that and return back to you.

gabraganca picture gabraganca on 17 Mar 2020
👍1

@igorborgest I tried the following, passing the credentials directly to awswrangler.Session:

import pandas as pd
import awswrangler as wr

wr_session = wr.Session(
    aws_access_key_id="...",
    aws_secret_access_key="...",
    region_name="us-east-1",
)

df = pd.DataFrame(dict(a=[1, 2], b=[3, 4]))

wr_session.pandas.to_csv(df, "s3://test/foobar", procs_cpu_bound=1)

and I've got a different error this time:

Traceback (most recent call last):
  File "awswrangler_test.py", line 20, in <module>
    wr_session.pandas.to_csv(df, "s3://blu-datalake/test/foobar", procs_cpu_bound=1)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 592, in to_csv
    columns_comments=columns_comments)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 729, in to_s3
    extra_args=extra_args)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 818, in data_to_s3
    isolated_dataframe=isolated_dataframe)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 853, in _data_to_s3_dataset_writer
    isolated_dataframe=isolated_dataframe)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 962, in _data_to_s3_object_writer
    extra_args=extra_args)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 999, in _write_csv_dataframe
    Pandas._write_csv_to_s3_retrying(fs=fs, path=path, buffer=csv_buffer)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 241, in wrapped_f
    return self.call(f, *args, **kw)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 330, in call
    start_time=start_time)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 279, in iter
    return fut.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/tenacity/__init__.py", line 333, in call
    result = fn(*args, **kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/awswrangler/pandas.py", line 1009, in _write_csv_to_s3_retrying
    f.write(buffer)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/fsspec/spec.py", line 1245, in __exit__
    self.close()
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/fsspec/spec.py", line 1213, in close
    self.flush(force=True)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/fsspec/spec.py", line 1085, in flush
    self._initiate_upload()
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/s3fs/core.py", line 1002, in _initiate_upload
    Bucket=self.bucket, Key=self.key, ACL=self.acl)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/s3fs/core.py", line 991, in _call_s3
    **kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/s3fs/core.py", line 184, in _call_s3
    return method(**additional_kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/client.py", line 613, in _make_api_call
    operation_model, request_dict, request_context)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/client.py", line 632, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/signers.py", line 160, in sign
    auth.add_auth(request)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/auth.py", line 371, in add_auth
    self._inject_signature_to_request(request, signature)
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/auth.py", line 374, in _inject_signature_to_request
    l = ['AWS4-HMAC-SHA256 Credential=%s' % self.scope(request)]
  File "/home/gbra/.virtualenvs/blu_airflow/lib/python3.6/site-packages/botocore/auth.py", line 324, in scope
    return '/'.join(scope)
TypeError: sequence item 0: expected str instance, tuple found
gabraganca picture gabraganca on 17 Mar 2020

Thanks @gabraganca, I will keep trying to figure out.

igorborgest picture igorborgest on 17 Mar 2020
👍1

@igorborgest Let me know if you need more testing.

Could you replicate the issue?

gabraganca picture gabraganca on 17 Mar 2020

@gabraganca I could reproduce the first two stack traces and then we decided to keep the focus on the version 1.0.0. We will have specific test cases to cover all this kind of situation related to boto3 session.

igorborgest picture igorborgest on 18 Mar 2020
👍1

Hi! We will release AWS Data Wrangler 1.0.0 on April 10 and we are collecting as much feedback as possible.

This issue should be resolved with this release. So, please test it if possible, and any feedback about the docs, the library or this specific functionality will be very welcome.

igorborgest picture igorborgest on 7 Apr 2020

Thank you for the contribution. Please, reopen the issue if it persists even after the version 1.0.0.

igorborgest picture igorborgest on 11 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jasonamyers picture jasonamyers  路  4Comments
bryanyang0528 picture bryanyang0528  路  4Comments
vfrank66 picture vfrank66  路  3Comments
bryanyang0528 picture bryanyang0528  路  5Comments
sapientderek picture sapientderek  路  3Comments