Pandas: melt does not recognize numeric column names

[nils-braun]

Code Sample, a copy-pastable example if possible

import pandas as pd

df = pd.DataFrame(columns=[1, "string"])
pd.melt(df, id_vars=[1, "string"])

Problem description

The shown example fails with

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    pd.melt(df, id_vars=[1, "string"])
  File "/home/nils/projects/tsfresh/venv/lib/python3.6/site-packages/pandas/core/reshape/melt.py", line 52, in melt
    "".format(missing=list(missing))
KeyError: "The following 'id_vars' are not present in the DataFrame: ['1']"

and I guess the reason is that the call of

Index(np.ravel(id_vars))

in pd.melt somehow casts the numerical column name 1 to the string "1".

I am not sure if this is intended behavior or if the case of numerical column names is just not supported, but at least in older pandas versions (e.g. 0.23.4) this still worked.

Thanks for looking into this! I am also fine if this is closed with "won't fix" :-)

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.6.8.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-65-generic
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 0.25.3
numpy : 1.17.4
pytz : 2019.3
dateutil : 2.8.1
pip : 19.3.1
setuptools : 41.6.0
Cython : None
pytest : 5.2.4
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.4.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.3
IPython : 7.9.0
pandas_datareader: 0.8.1
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : 4.4.1
matplotlib : 3.1.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.3.2
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

[mroeschke]

This looks like a bug to me. Thanks for the report! Investigations and PR's welcome.

[nils-braun]

So concerning investigation: the root cause is that

>>> np.ravel(["string", 1])
array(['string', '1'], dtype='<U5')

will give an np.array just of strings.

Concerning PRs: the check for having the columns could also be implemented via just using the Index directly without the np.ravel. Do you (or someone) know why it was introduced?

I see it was added here https://github.com/pandas-dev/pandas/pull/23575/commits/fba641fdb53beb3ec1b952cf9329544af1c378e5 to cope with multiindex. Maybe there is another way to do this?

[jschendel]

Maybe use pandas.core.common.flatten instead?

diff --git a/pandas/core/reshape/melt.py b/pandas/core/reshape/melt.py
index 16c044548..2d3a40fdb 100644
--- a/pandas/core/reshape/melt.py
+++ b/pandas/core/reshape/melt.py
@@ -10,6 +10,7 @@ from pandas.core.dtypes.generic import ABCMultiIndex
 from pandas.core.dtypes.missing import notna

 from pandas.core.arrays import Categorical
+import pandas.core.common as com
 from pandas.core.frame import _shared_docs
 from pandas.core.indexes.base import Index
 from pandas.core.reshape.concat import concat
@@ -45,7 +46,7 @@ def melt(
         else:
             # Check that `id_vars` are in frame
             id_vars = list(id_vars)
-            missing = Index(np.ravel(id_vars)).difference(cols)
+            missing = Index(com.flatten(id_vars)).difference(cols)
             if not missing.empty:
                 raise KeyError(
                     "The following 'id_vars' are not present"

Which seems to make things work:

In [1]: import pandas as pd

In [2]: df = pd.DataFrame(columns=[1, "string"])

In [3]: pd.melt(df, id_vars=[1, "string"])
Out[3]: 
Empty DataFrame
Columns: [1, string, variable, value]
Index: []