Astropy: Initialising Table with structured dtype produces inconsistent/confusing results

[dhomeier]

Description

A discussion in the Facebook group brought up that a new, empty table can be constructed from an existing table by passing its dtype:

>>> oldtable.dtype
dtype([('wavelength', '<f8'), ('flux', '<f4'), ('uncertainty', '<f4')])
>>> Table(dtype=oldtable.dtype)
<Table length=0>
wavelength   flux  uncertainty
 float64   float32   float32  
---------- ------- -----------

and it is equally possible to initialise it in the more pedestrian manner

>>> newtable = Table(names=oldtable.colnames, dtype=[dt for n, dt in oldtable.dtype.descr])

but directly using the old dtype if names are already specified throws these exceptions:

 >>> newtable = Table(names=oldtable.colnames, dtype=oldtable.dtype)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/derek/lib/python3.8/site-packages/astropy/table/table.py", line 621, in __init__
    self._check_names_dtype(names, dtype, n_cols)
  File "/Users/derek/lib/python3.8/site-packages/astropy/table/table.py", line 873, in _check_names_dtype
    raise ValueError(f'{inp_str} must be a list or None')
ValueError: dtype must be a list or None

>>> newtable = Table(names=oldtable.colnames, dtype=oldtable.dtype.descr)
Traceback (most recent call last):
  File "/Users/derek/lib/python3.8/site-packages/astropy/table/table.py", line 1088, in _convert_data_to_col
    col = col_cls(name=name, data=data, dtype=dtype,
  File "/Users/derek/lib/python3.8/site-packages/astropy/table/column.py", line 879, in __new__
    self = super().__new__(
  File "/Users/derek/lib/python3.8/site-packages/astropy/table/column.py", line 254, in __new__
    if np.dtype(dtype).char == 'S':
TypeError: data type 'wavelength' not understood

The same happens when the Table is initialised with a structured array.

Expected behavior

Table should either raise a more meaningful error, or (perhaps preferably) accept the structured dtype, possibly with a warning that dtype.names will be replaced with the specified names.

Actual behavior

See above. While the recommended method in the docs newtable = oldtable[:0].copy() works (and has the advantage of additionally copying any unit properties), the discrepant behaviour between specifying dtype alone, and together with names is potentially confusing, and the exceptions not very clear.

https://github.com/astropy/astropy/blob/eeb9b7ac0e18b262fae64cd2fa46d0e1cff1e4d5/astropy/table/table.py#L592-L594

"unpacks" the dtype automatically when no other names information is available, and the same might be done with names set (but discarding dtype.names, since names should certainly take priority).

On a slightly related note, it would be convenient to have a method to directly get all units for the table columns in a form that could be passed as units=. Of course one can return to the newtable = oldtable[:0].copy() method, but if modifying names or dtypes is desired, such an option would come in handy.

System Details

Astropy version 4.2.dev112+g595484597.
Date: 2020-05-22T19:44:39
Platform: macOS-10.12.6-x86_64-i386-64bit
Executable: /sw/bin/python3.8
Full Python Version:
3.8.3 (default, May 14 2020, 22:17:35)
[Clang 9.0.0 (clang-900.0.39.2)]

Package versions:
Numpy: 1.19.0rc1
Scipy: 1.4.1
Matplotlib: 3.2.1
h5py: 2.9.0
Pandas: not available
Cython: 0.29.16

[pllim]

Thanks! What exactly was oldtable? Is it possible to provide a completely reproducible snippet?

[dhomeier]

import numpy as np
from astropy.table import Table
arr = np.ones(2, dtype=np.dtype([('A', 'i'), ('B', 'f4'), ('C', 'f8')]))
tab1 = Table(arr)
tab1

tab2 = Table(dtype=tab1.dtype)
tab2

tab3 = Table(names=['X', 'Y', 'Z'], dtype=[dt for n, dt in tab1.dtype.descr])
tab3

produces the following output:

<Table length=2>
  A      B       C   
int32 float32 float64
----- ------- -------
    1     1.0     1.0
    1     1.0     1.0

<Table length=0>
  A      B       C   
int32 float32 float64
----- ------- -------

<Table length=0>
  X      Y       Z   
int32 float32 float64
----- ------- -------

whereas

>>> tab3 = Table(names=['X', 'Y', 'Z'], dtype=tab1.dtype)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/sw/lib/python3.8/site-packages/astropy/table/table.py", line 621, in __init__
    self._check_names_dtype(names, dtype, n_cols)
  File "/sw/lib/python3.8/site-packages/astropy/table/table.py", line 873, in _check_names_dtype
    raise ValueError(f'{inp_str} must be a list or None')
ValueError: dtype must be a list or None

>>> tab3 = Table(names=['X', 'Y', 'Z'], dtype=tab1.dtype.descr)
Traceback (most recent call last):
  File "/sw/lib/python3.8/site-packages/astropy/table/table.py", line 1088, in _convert_data_to_col
    col = col_cls(name=name, data=data, dtype=dtype,
  File "/sw/lib/python3.8/site-packages/astropy/table/column.py", line 879, in __new__
    self = super().__new__(
  File "/sw/lib/python3.8/site-packages/astropy/table/column.py", line 254, in __new__
    if np.dtype(dtype).char == 'S':
TypeError: data type 'A' not understood

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/sw/lib/python3.8/site-packages/astropy/table/table.py", line 624, in __init__
    init_func(data, names, dtype, n_cols, copy)
  File "/sw/lib/python3.8/site-packages/astropy/table/table.py", line 955, in _init_from_list
    col = self._convert_data_to_col(col, copy, default_name, dtype, name)
  File "/sw/lib/python3.8/site-packages/astropy/table/table.py", line 1092, in _convert_data_to_col
    raise ValueError('unable to convert data to Column for Table')
ValueError: unable to convert data to Column for Table



md5-9116051d321cf97db138d26f38a78850



>>> tab3 = Table(arr, dtype=tab1.dtype.descr)
Traceback (most recent call last):
  File "/sw/lib/python3.8/site-packages/astropy/table/table.py", line 1088, in _convert_data_to_col
    col = col_cls(name=name, data=data, dtype=dtype,
  File "/sw/lib/python3.8/site-packages/astropy/table/column.py", line 879, in __new__
    self = super().__new__(
  File "/sw/lib/python3.8/site-packages/astropy/table/column.py", line 254, in __new__
    if np.dtype(dtype).char == 'S':
TypeError: data type 'A' not understood

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/sw/lib/python3.8/site-packages/astropy/table/table.py", line 624, in __init__
    init_func(data, names, dtype, n_cols, copy)
  File "/sw/lib/python3.8/site-packages/astropy/table/table.py", line 1108, in _init_from_ndarray
    self._init_from_list(cols, names, dtype, n_cols, copy)
  File "/sw/lib/python3.8/site-packages/astropy/table/table.py", line 955, in _init_from_list
    col = self._convert_data_to_col(col, copy, default_name, dtype, name)
  File "/sw/lib/python3.8/site-packages/astropy/table/table.py", line 1092, in _convert_data_to_col
    raise ValueError('unable to convert data to Column for Table')
ValueError: unable to convert data to Column for Table

[taldcroft]

See #10419.