Reticulate: pandas > data frame conversion 'corrupt data frame' error

[mdingemanse]

On Windows 7, using R 3.5.1, Python 3.7.1, RStudio 1.2.1139, and reticulate_1.10.0.9003, running the toy example from the reticulate website results in a 'corrupt data frame' error. Not sure whether this is to do with type conversion or some other issue. Similar issue reported (for Windows, too) on StackOverflow here.

First a demonstration that the Python end of things works well via Reticulate:

Python 3.7.1 (conda\envs\r-reticulate\python.exe)
Reticulate 1.10.0.9003 REPL -- A Python interpreter in R.
>>> import pandas
>>> flights = pandas.read_csv("flights.csv",encoding="utf-8")
>>> flights = flights[flights['dest'] == 'ORD']
>>> flights = flights[['carrier', 'dep_delay', 'arr_delay']]
>>> flights = flights.dropna()
>>> print(flights.head())
   carrier  dep_delay  arr_delay
6       UA      227.0      219.0
17      AA        0.0      -19.0
20      UA       -3.0        7.0
22      UA       -3.0       -2.0
51      AA       -4.0      -28.0
>>> 

But if we try to reach the flights object in R we get this:

>>> quit
> py$flights %>% head()
                            carrier                         dep_delay
1 <environment: 0x000000001e25ce08> <environment: 0x000000001e3be750>
2                              <NA>                              <NA>
3                              <NA>                              <NA>
4                              <NA>                              <NA>
5                              <NA>                              <NA>
6                              <NA>                              <NA>
                          arr_delay
1 <environment: 0x000000001e51e178>
2                              <NA>
3                              <NA>
4                              <NA>
5                              <NA>
6                              <NA>
Warning message:
In format.data.frame(x, digits = digits, na.encode = FALSE) :
  corrupt data frame: columns will be truncated or padded with NAs

[kevinushey]

Can you provide a reproducible example? (I don't know what version of flights.csv you're using)

[mdingemanse]

Sorry — was using this one from another repo.

[kevinushey]

Doesn't work for me:

>>> import pandas
>>> flights = pandas.read_csv("flights.csv",encoding="utf-8")
>>> flights = flights[flights['dest'] == 'ORD']
KeyError: 'dest'

Can you provide a self-contained reproducible example?

[mdingemanse]

Python:
```{python}
import pandas as pd
d = {'A' : pd.Series([1., 2., 3.]), 'B' : pd.Series(['x', 'y', 'z'])}
df = pd.DataFrame(d)
print(df)

R:
```{r}
> py$df
                                  A                                 B
1 <environment: 0x0000000017b9ded0> <environment: 0x0000000017cc04f8>
2                              <NA>                              <NA>
3                              <NA>                              <NA>
Warning message:
In format.data.frame(x, digits = digits, na.encode = FALSE) :
  corrupt data frame: columns will be truncated or padded with NAs
> 

[mdingemanse]

Did that help — was it reproducible?

[kevinushey]

Sorry, no -- I'm still not able to reproduce. I didn't test in a conda environment though; this was just with a system Python install.

[grayskripko]

I got similar errors when I had the conda version of numpy only.
I'm not sure it will help to reproduce the bug but it worths to try
https://github.com/magnusfurugard/featuretoolsR/issues/4

I have Windows OS too.

[fmannhardt]

I can reproduce this problem with the following code:

library(reticulate)
py_to_r(r_to_py(data.frame(x=c("A","B"))))

which results in:

1 <environment: 0x00000157d227ec68>
2                              <NA>
Warning message:
In format.data.frame(if (omit) x[seq_len(n0), , drop = FALSE] else x,  :
  corrupt data frame: columns will be truncated or padded with NAs

The issue (or at least a part of it) seems to be that the column is a factor, since the following code works fine:

library(reticulate)
py_to_r(r_to_py(data.frame(x=c("A","B"), stringsAsFactors = FALSE)))

results in:

1 A
2 B

Environment:

R version 3.5.2 (2018-12-20)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17134)

Matrix products: default

locale:
[1] LC_COLLATE=Norwegian Bokmål_Norway.1252  LC_CTYPE=Norwegian Bokmål_Norway.1252    LC_MONETARY=Norwegian Bokmål_Norway.1252
[4] LC_NUMERIC=C                             LC_TIME=Norwegian Bokmål_Norway.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] reticulate_1.10.0.9003

loaded via a namespace (and not attached):
[1] compiler_3.5.2  Matrix_1.2-15   tools_3.5.2     Rcpp_1.0.0      grid_3.5.2      jsonlite_1.6    packrat_0.5.0   lattice_0.20-38

Python:

python:         C:\Users\..\AppData\Local\Continuum\anaconda3\envs\r-reticulate\python.exe
libpython:      C:/Users/../AppData/Local/Continuum/anaconda3/envs/r-reticulate/python37.dll
pythonhome:     C:\Users\..\AppData\Local\CONTIN~1\ANACON~1\envs\R-RETI~1
version:        3.7.2 (default, Jan  2 2019, 17:07:39) [MSC v.1915 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:\Users\..\AppData\Local\CONTIN~1\ANACON~1\envs\R-RETI~1\lib\site-packages\numpy
numpy_version:  1.16.1
[1] "conda 4.6.1"

(r-reticulate) C:\Users\..>pip list
Package          Version
---------------- ----------
certifi          2018.11.29
ciso8601         2.1.1
colorama         0.4.1
cvxopt           1.2.2
cycler           0.10.0
decorator        4.3.2
graphviz         0.10.1
intel-openmp     2019.0
ipython          5.3.0
ipython-genutils 0.2.0
Jinja2           2.10
kiwisolver       1.0.1
lxml             4.3.0
MarkupSafe       1.1.0
matplotlib       2.2.2
mkl              2019.0
mkl-fft          1.0.10
mkl-random       1.0.2
networkx         2.2
numpy            1.16.1
pandas           0.23.4
pickleshare      0.7.5
pip              18.1
prompt-toolkit   1.0.15
PuLP             1.6.9
Pygments         2.3.1
pyparsing        2.3.1
python-dateutil  2.7.5
pytz             2018.9
pyvis            0.1.5.0
scikit-learn     0.20.2
scipy            1.2.0
setuptools       40.7.2
simplegeneric    0.8.1
six              1.12.0
traitlets        4.3.2
wcwidth          0.1.7
wheel            0.32.3
wincertstore     0.2

[kevinushey]

Thanks, I can reproduce with this. My best guess is that the class hierarchy has changed for categorical variables. We have a method for objects of class pandas.core.categorical.Categorical, but the class I see in this case is pandas.core.arrays.categorical.Categorical. I'll take a look next week.

[kevinushey]

I've committed a fix to master (it looks like it was fortunately a simple fix): https://github.com/rstudio/reticulate/commit/e2e408062b1146de8af519d836fef789df5f9693

Let me know if things work well in the development version of reticulate in your case. Use devtools::install_github("rstudio/reticulate") to install.

Thanks for the reproducible example!

[grayskripko]

it helped

[fmannhardt]

Thanks, it fixes the issue for me.

[sergiojsj76]

I still have the same error

In format.data.frame(x, digits = digits, na.encode = FALSE) :
  corrupt data frame: columns will be truncated or padded with NAs

after updating reticulate to 1.10.0.9004. I have Python 3.7.1 on Windows 10

[kevinushey]

@sergiojsj76 do you have a reproducible example? What code are you running?