Pandas: to_latex shifts column names if MultiIndex contains empty label

[toobaz]

Code Sample, a copy-pastable example if possible

In [2]: df = pd.DataFrame([[1, 2]], columns=pd.MultiIndex.from_tuples([('', 'first'), ('2', 'second')]))

In [3]: df
Out[3]: 
             2
  first second
0     1      2

In [4]: print(df.to_latex())
\begin{tabular}{lrr}
\toprule
{} &      2 \\
{} & first & second \\
\midrule
0 &     1 &      2 \\
\bottomrule
\end{tabular}

Problem description

Notice that the headers line has 2 elements rather than 3.

Expected Output

\begin{tabular}{lrr}
\toprule
{} &    {} &      2 \\
{} & first & second \\
\midrule
0 &     1 &      2 \\
\bottomrule
\end{tabular}

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.0-5-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: it_IT.UTF-8
LOCALE: it_IT.UTF-8

pandas: 0.23.0.dev0+422.g0bfb61b21
pytest: 3.2.3
pip: 9.0.1
setuptools: 36.7.0
Cython: 0.25.2
numpy: 1.14.1
scipy: 0.19.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.5.6
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.0dev
tables: 3.3.0
numexpr: 2.6.1
feather: 0.3.1
matplotlib: 2.0.0
openpyxl: 2.3.0
xlrd: 1.0.0
xlwt: 1.3.0
xlsxwriter: 0.9.6
lxml: 4.1.1
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.0.15
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.2.1

[gfyoung]

What can I say? Addressing missing data can be hard... 😄. Patch and PR are welcome!

[Michael-E-Rose]

It's not only when the label is empty, but also when it contains a blank or when it contains a protected blank.

I think the blank gets removed in https://github.com/pandas-dev/pandas/blob/master/pandas/io/formats/format.py#L782 or in https://github.com/pandas-dev/pandas/blob/master/pandas/io/formats/format.py#L785. Unfortunately I can't test it because I can't compile my pandas fork.

[gfyoung]

@Michael-E-Rose : I can test if you aren't able to compile (though curious to know what your issue is with compilation). What patch are you imagining to those lines of the code?

[YunhaoHe]

I got the same problem. I got around it by specifying a "placeholder name" and replace this placeholder with empty string later. But still, it would be great to address this problem directly...

[sgsaenger]

You will get the expected output if you call df.to_latex(multicolumn=False).

The cause is that the latex formatting depends on the output of DataFrameFormatter._to_str_columns, which uses empty labels to remove repeated labels, therefore LatexFormatter._format_multicolumn cannot distinguish between empty labels that should stay and empty labels that should be converted to multicolumn.

I agree that this particular behavior is not ideal, but also don't know about the intended semantics of empty labels. They will always clash with the use of multicolumns in latex, but already in df.to_string() you will get ambiguities:

pd.DataFrame([[1, 2]], columns=pd.MultiIndex.from_tuples([('2', 'first'),
                                                          ('2', 'second')])).to_string()

vs:

pd.DataFrame([[1, 2]], columns=pd.MultiIndex.from_tuples([('2', 'first'),
                                                          ('', 'second')])).to_string()

which both yield:

      2       
  first second
0     1      2

[toobaz]

Probably the same bug, but just in case:

In [2]: df = pd.DataFrame(1, index=range(3), columns=pd.MultiIndex.from_tuples([('a', 1), ('a', 2), ('b', '')]))                                                                                                                                              

In [3]: print(df.to_latex())                                                                                                                                                                                                                                  
\begin{tabular}{lrrr}
\toprule
{} & \multicolumn{2}{l}{a} &  b \\
{} &  1 & \multicolumn{2}{l}{2} \\
\midrule
0 &  1 &  1 &  1 \\
1 &  1 &  1 &  1 \\
2 &  1 &  1 &  1 \\
\bottomrule
\end{tabular}