Aws-data-wrangler: EOFError: when wr.pandas.to_parquet() in glue job

[LeoHsu0802]

I get anEOFError: when wr.pandas.to_parquet() in glue job

  File "/glue/lib/installation/awswrangler/pandas.py", line 729, in to_s3
    extra_args=extra_args)
  File "/glue/lib/installation/awswrangler/pandas.py", line 805, in data_to_s3
    objects_paths += receive_pipes[i].recv()
  File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/runscript.py", line 137, in <module>
    raise e_type(e_value).with_tracsback(new_stack)
AttributeError: 'EOFError' object has no attribute 'with_tracsback'

[igorborgest]

Hi @LeoHsu0802, thanks for report it.

Any chance of your data be bigger than the available memory?

Could you try to run the same workload with procs_cpu_bound=1? (It will slowdown the parallelism but will save a considerable piece of memory.)

[LeoHsu0802]

Hi @igorborgest thanks for reply
I try procs_cpu_bound=1 and it works !