Pandas: .astype() and Decimal

[shadchin]

Begin from 0.24.0 next code raise exception

import decimal
import pandas as pd
ser = pd.Series([1, 2])
ser.astype(decimal.Decimal)

Traceback

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/shadchin/t_d/venv/local/lib/python2.7/site-packages/pandas/core/generic.py", line 5691, in astype
    **kwargs)
  File "/home/shadchin/t_d/venv/local/lib/python2.7/site-packages/pandas/core/internals/managers.py", line 531, in astype
    return self.apply('astype', dtype=dtype, **kwargs)
  File "/home/shadchin/t_d/venv/local/lib/python2.7/site-packages/pandas/core/internals/managers.py", line 395, in apply
    applied = getattr(b, f)(**kwargs)
  File "/home/shadchin/t_d/venv/local/lib/python2.7/site-packages/pandas/core/internals/blocks.py", line 534, in astype
    **kwargs)
  File "/home/shadchin/t_d/venv/local/lib/python2.7/site-packages/pandas/core/internals/blocks.py", line 595, in _astype
    dtype = pandas_dtype(dtype)
  File "/home/shadchin/t_d/venv/local/lib/python2.7/site-packages/pandas/core/dtypes/common.py", line 2029, in pandas_dtype
    raise TypeError("dtype '{}' not understood".format(dtype))
TypeError: dtype '<class 'decimal.Decimal'>' not understood

Problem description

This begin from #21160

After was https://github.com/pandas-dev/pandas/blob/0409521665bd436a10aea7e06336066bf07ff057/pandas/core/dtypes/common.py#L2013
Now https://github.com/pandas-dev/pandas/blob/cb00deb94500205fcb27a33cc1d0df79a9727f8b/pandas/core/internals/blocks.py#L595

pandas_dtype() does not understand decimal.Decimal

Is it expected behavior or regress?

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.12.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.101-26
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.24.2
pytest: None
pip: 19.1.1
setuptools: 41.0.1
Cython: None
numpy: 1.16.4
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.8.0
pytz: 2019.1
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

[WillAyd]

There is no first class support in pandas for Decimal objects. You would need to instantiate the Decimal objects within your container and use an an object dtype

[WillAyd]

There is however a DecimalArray extension type in our test suite. @TomAugspurger is there a plan to move this into the core at some point?

[TomAugspurger]

Nope. Arrow has a decimal type, that would be the recommended way to store them.

On Jun 8, 2019, at 06:36, William Ayd [e-mail] wrote:

There is however a DecimalArray extension type in our test suite. @TomAugspurger is there a plan to move this into the core at some point?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[sid-kap]

@TomAugspurger Is storing decimals via the Arrow backend possible today with Pandas? Or do you mean that we will be able to do this once we hit Pandas 1.0 and Pandas is using Arrow as a backend?

[TomAugspurger]

No it's not possible today, at least not without writing your own extension type.

Pandas 1.0 won't be using Arrow. That will come after.

[Sajaki]

is this in your pipeline / realised now ?