Aws-data-wrangler: wr.s3.read_csv exception

[BrainMonkey]

When calling wr.s3.read_csv("<csv s3 path>") I am getting exception

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.6/dist-packages/awswrangler/s3.py", line 1289, in read_csv **pandas_kwargs, File "/usr/local/lib/python3.6/dist-packages/awswrangler/s3.py", line 1510, in _read_text sort=False, File "/usr/local/lib/python3.6/dist-packages/pandas/core/reshape/concat.py", line 281, in concat sort=sort, File "/usr/local/lib/python3.6/dist-packages/pandas/core/reshape/concat.py", line 326, in __init__ objs = list(objs) File "/usr/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator yield fs.pop().result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.6/dist-packages/awswrangler/s3.py", line 1541, in _read_text_full fs: s3fs.S3FileSystem = _utils.get_fs(session=boto3_session, s3_additional_kwargs=s3_additional_kwargs) File "/usr/local/lib/python3.6/dist-packages/awswrangler/_utils.py", line 146, in get_fs skip_instance_cache=True, File "/usr/local/lib/python3.6/dist-packages/fsspec/spec.py", line 54, in __call__ obj = super().__call__(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/s3fs/core.py", line 171, in __init__ self.s3 = self.connect() File "/usr/local/lib/python3.6/dist-packages/s3fs/core.py", line 229, in connect self.s3 = self.session.client('s3', config=conf, use_ssl=ssl, AttributeError: 'Session' object has no attribute 'client'

I am able to run other s3 commands such as wr.s3.does_object_exist() without any issues.
To Reproduce
I am using

• python 3.6.10
• awswrangler version 1.0.2
• boto3 version 1.13.10

Calling wr.s3.read_csv("<s3_path>") . I also tried passing in a boto3 session as a parameter to the method read_csv() and the exception is still thrown.

[igorborgest]

Hi @BrainMonkey, thanks for reporting it!

We did several fixes since the version 1.0.2.

Do you mind to test our current development branch?

pip install git+https://github.com/awslabs/aws-data-wrangler.git@dev

I would like to make sure that it will be fixed on our next version 1.2.0. Thanks!

[BrainMonkey]

Hi @igorborgest Installing from pip install git+https://github.com/awslabs/aws-data-wrangler.git@dev gives me version 1.1.2 and the issue it still in that version.

[igorborgest]

@BrainMonkey thanks for the feedback.

Yeah, the development branch still with the 1.1.2 metadata, but for sure you got the updated version.

I couldn't able to reproduce this error here. Are you running on SageMaker or local? Which OS?

One last detail, what is the s3fs version in your environment?

[igorborgest]

@BrainMonkey are you doing something different than that?

[mrshu]

@igorborgest The problem seems to be with the s3fs package -- version 0.4.0 does not seem to work, and even running pip install git+https://github.com/awslabs/aws-data-wrangler.git@dev -U won't update that (it's currently set as s3fs~=0.4.0 in requirements.txt, so 0.4.0 is good enough).

I therefore suggest that the s3fs version in requirements.txt gets bumped to at least 0.4.2 when a new version of awswrangler gets released.

[igorborgest]

@mrshu thanks a lot, you are absolutely right, now I'm able to reproduce it.

I've just bumped s3fs version in the requirements.txt on the dev branch and it worked for me.

Could someone double check it, please?

bash pip install -U git+https://github.com/awslabs/aws-data-wrangler.git@dev

[BrainMonkey]

@igorborgest That worked !

[mrshu]

@igorborgest I can confirm that as well!

[hyperloglog]

I just discovered aws-data-wrangler itself within the past hour, ran into this same issue, found this thread, and successfully used your fix. :-) Thanks, @igorborgest !

[igorborgest]

Thanks a lot @BrainMonkey, @mrshu and @hyperloglog !

It's already available on version 1.2.0