Describe the bug
We seem to be reading in an integer column with nulls as a floating column. This is similar to pandas behavior but since we have support for nullable dtypes in all dtypes, should we even type-cast to float?
Steps/Code to reproduce bug
In[20]: df = cudf.DataFrame({'a':cudf.Series([1, 2, 3], dtype='uint64')})
In[21]: df.to_csv('temp.csv')
In[22]: cudf.read_csv('temp.csv')
Out[22]:
Unnamed: 0 a
0 0 1
1 1 2
2 2 3
In[23]: pd.read_csv('temp.csv')
Out[23]:
Unnamed: 0 a
0 0 1
1 1 2
2 2 3
In[24]: df = cudf.DataFrame({'a':cudf.Series([1, 2, 3, None], dtype='uint64')})
In[25]: df.to_csv('temp.csv')
In[26]: cudf.read_csv('temp.csv')
Out[26]:
Unnamed: 0 a
0 0 1.0
1 1 2.0
2 2 3.0
3 3 <NA>
In[27]: pd.read_csv('temp.csv')
Out[27]:
Unnamed: 0 a
0 0 1.0
1 1 2.0
2 2 3.0
3 3 NaN
Expected behavior
Expected behavior is to not type-cast to float and only cast to the inferred dtype.
Environment overview (please complete the following information)
Environment details
Please run and paste the output of the cudf/print_env.sh script here, to gather any other relevant environment details
Click here to see environment details
**git***
commit 768b67c37c2e3539f3d80d3c881ff8b1b320e56b (HEAD -> branch-0.16, upstream/branch-0.16)
Author: GALI PREM SAGAR <[email protected]>
Date: Thu Sep 24 09:18:00 2020 -0500
Implement Fuzz tests for cuIO (#6114)
* add log reporting for failed tests
* add random data generation apis for all dtypes
* make size as mandatory param
* add random dataframe generation api
* add Fuzz tests for parquet reader
* add support for datetime, timedelta and category dtypes datageneration
* add parquet writer tests
* add csv writer and reader tests
* add all dtypes support for random dataframe generation
* add logging for exception
* add all dtypes in CSV and parquet testing
* cleanup
* code cleanup
* typo
* add support for regression run from crash files
* add json reader writer tests
* fix issue related to picking random dtypes
* replace prints with logging
* exclude timedelta dtypes in CSV reader / writer tests
* use seed in generate_input apis
* add max_columns param
* change param file extension to json from xml
* add columns param
* fix issues with regression run
* add Readme for fuzz tests
* add call method to fuzzer
* add readme for fuzz tests
* change main block code
* add url
* fix issue with float dtype data generation
* add tips in readme
* Update CHANGELOG.md
* apply seeds
* change the format of the dtypes_meta for rand_dataframe
* re-organize tests into their respective files
* move common code
* use 3x faster np.radom apis instead of mimesis
* make optimizations in _generate_column and fix lambda scope issues
* remove redundant numpy computation
* moved common code into utils.py
* make max string length generation random
* add todo for list dtype support
* add copyright and use string buffer
* change print to an error
* handle keyboard interrupts and exit
* replace writing to parquet directly from pyarrow
* remove skipping of timedelta types in csv
* give limits to datetime and duration types
* parametrize max_string_length
* move common code into a base class implementation
* push down common logic into IOBase
* use common logic from IOBase
* remove unused variables
* change default max_rows to 100000
* add utility function to translate to pandas nullable dtypes
* rename IOBase to IOFuzz
* copyright
* change default max_rows_size to 100000
* change from saving to file to return buffer
* Update python/cudf/cudf/fuzz_tests/readme.md
Co-authored-by: Christopher Harris <[email protected]>
* add comparison of file contents
* handle deafult param for dirs
* remove redundant test code
* change default value for max rows:
Co-authored-by: Christopher Harris <[email protected]>
**git submodules***
***OS Information***
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Linux pgali-HP-Z8-G4-Workstation 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
***GPU Information***
Thu Sep 24 15:26:03 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.66 Driver Version: 450.66 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro RTX 8000 Off | 00000000:22:00.0 Off | Off |
| 33% 30C P8 4W / 260W | 874MiB / 48601MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 Quadro RTX 8000 Off | 00000000:2D:00.0 On | Off |
| 33% 35C P8 20W / 260W | 588MiB / 48592MiB | 18% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 960 G /usr/lib/xorg/Xorg 4MiB |
| 0 N/A N/A 1820 G /usr/lib/xorg/Xorg 4MiB |
| 0 N/A N/A 5056 C .../envs/cudf_dev/bin/python 695MiB |
| 0 N/A N/A 6256 C ...ffice/program/soffice.bin 165MiB |
| 1 N/A N/A 960 G /usr/lib/xorg/Xorg 39MiB |
| 1 N/A N/A 1820 G /usr/lib/xorg/Xorg 287MiB |
| 1 N/A N/A 2020 G /usr/bin/gnome-shell 200MiB |
| 1 N/A N/A 5616 G /usr/lib/firefox/firefox 3MiB |
| 1 N/A N/A 5788 G ...AAAAAAAAA= --shared-files 44MiB |
+-----------------------------------------------------------------------------+
***CPU***
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 12
On-line CPU(s) list: 0-11
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
Stepping: 4
CPU MHz: 1200.141
CPU max MHz: 3700.0000
CPU min MHz: 1200.0000
BogoMIPS: 6800.00
Virtualization: VT-x
L1d cache: 192 KiB
L1i cache: 192 KiB
L2 cache: 6 MiB
L3 cache: 19.3 MiB
NUMA node0 CPU(s): 0-11
Vulnerability Itlb multihit: KVM: Vulnerable
Vulnerability L1tf: Mitigation; PTE Inversion
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d
***CMake***
/home/pgali/anaconda3/envs/cudf_dev/bin/cmake
cmake version 3.18.2
CMake suite maintained and supported by Kitware (kitware.com/cmake).
***g++***
/usr/bin/g++
g++ (Ubuntu 9.3.0-10ubuntu2) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
***nvcc***
/usr/local/cuda/bin/nvcc
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0
***Python***
/home/pgali/anaconda3/envs/cudf_dev/bin/python
Python 3.7.8
***Environment Variables***
PATH : /home/pgali/anaconda3/envs/cudf_dev/bin:/home/pgali/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda/bin
LD_LIBRARY_PATH : :/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
NUMBAPRO_NVVM :
NUMBAPRO_LIBDEVICE :
CONDA_PREFIX : /home/pgali/anaconda3/envs/cudf_dev
PYTHON_PATH :
***conda packages***
/home/pgali/anaconda3/condabin/conda
# packages in environment at /home/pgali/anaconda3/envs/cudf_dev:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
abseil-cpp 20200225.2 he1b5a44_2 conda-forge
alabaster 0.7.12 py_0 conda-forge
appdirs 1.4.3 py_1 conda-forge
argon2-cffi 20.1.0 py37h8f50634_1 conda-forge
arrow-cpp 1.0.1 py37hf00d4d6_5_cuda conda-forge
arrow-cpp-proc 1.0.1 cuda conda-forge
async_generator 1.10 py_0 conda-forge
attrs 20.2.0 pyh9f0ad1d_0 conda-forge
aws-c-common 0.4.57 he1b5a44_0 conda-forge
aws-c-event-stream 0.1.6 h72b8ae1_3 conda-forge
aws-checksums 0.1.9 h346380f_0 conda-forge
aws-sdk-cpp 1.7.164 h69f4914_4 conda-forge
babel 2.8.0 py_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.1 py_0 conda-forge
black 19.10b0 py_4 conda-forge
bleach 3.2.1 pyh9f0ad1d_0 conda-forge
bokeh 2.2.1 py37hc8dfbb8_0 conda-forge
boost-cpp 1.74.0 h9359b55_0 conda-forge
brotli 1.0.9 he1b5a44_0 conda-forge
brotlipy 0.7.0 py37h8f50634_1000 conda-forge
bzip2 1.0.8 h516909a_3 conda-forge
c-ares 1.16.1 h516909a_3 conda-forge
ca-certificates 2020.6.20 hecda079_0 conda-forge
certifi 2020.6.20 py37hc8dfbb8_0 conda-forge
cffi 1.14.3 py37h2b28604_0 conda-forge
cfgv 3.2.0 py_0 conda-forge
chardet 3.0.4 py37hc8dfbb8_1007 conda-forge
clang 8.0.1 hc9558a2_2 conda-forge
clang-tools 8.0.1 hc9558a2_2 conda-forge
clangxx 8.0.1 2 conda-forge
click 7.1.2 pyh9f0ad1d_0 conda-forge
cloudpickle 1.6.0 py_0 conda-forge
cmake 3.18.2 h5c55442_0 conda-forge
cmake_setuptools 0.1.3 py_0 rapidsai
commonmark 0.9.1 py_0 conda-forge
cryptography 3.1 py37hb09aad4_0 conda-forge
csv-diff 0.6 pypi_0 pypi
cudatoolkit 11.0.221 h6bb024c_0 nvidia
cudf 0.16.0a0+1960.g848b4266fa pypi_0 pypi
cudnn 8.0.0 cuda11.0_0 nvidia
cupy 7.8.0 py37h0ce7dbb_0 rapidsai
cython 0.29.21 py37h3340039_0 conda-forge
cytoolz 0.10.1 py37h516909a_0 conda-forge
dask 2.27.0+5.g5fa77484 pypi_0 pypi
dask-cudf 0.16.0a0+1956.g0d50234f35.dirty pypi_0 pypi
decorator 4.4.2 py_0 conda-forge
defusedxml 0.6.0 py_0 conda-forge
dictdiffer 0.8.1 pypi_0 pypi
distlib 0.3.1 pyh9f0ad1d_0 conda-forge
distributed 2.27.0+1.g8aefbf36 pypi_0 pypi
dlpack 0.3 he1b5a44_1 conda-forge
docutils 0.16 py37hc8dfbb8_1 conda-forge
double-conversion 3.1.5 he1b5a44_2 conda-forge
editdistance 0.5.3 py37h3340039_1 conda-forge
entrypoints 0.3 py37hc8dfbb8_1001 conda-forge
expat 2.2.9 he1b5a44_2 conda-forge
fastavro 1.0.0.post1 py37h8f50634_0 conda-forge
fastrlock 0.5 py37h3340039_0 conda-forge
filelock 3.0.12 pyh9f0ad1d_0 conda-forge
flake8 3.8.3 py_1 conda-forge
flatbuffers 1.12.0 he1b5a44_0 conda-forge
freetype 2.10.2 he06d7ca_0 conda-forge
fsspec 0.8.2 py_0 conda-forge
future 0.18.2 py37hc8dfbb8_1 conda-forge
gflags 2.2.2 he1b5a44_1004 conda-forge
glog 0.4.0 h49b9bf7_3 conda-forge
gmp 6.2.0 he1b5a44_2 conda-forge
grpc-cpp 1.30.2 heedbac9_0 conda-forge
heapdict 1.0.1 py_0 conda-forge
hypothesis 5.28.0 py_0 conda-forge
icu 67.1 he1b5a44_0 conda-forge
identify 1.5.4 pyh9f0ad1d_0 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
imagesize 1.2.0 py_0 conda-forge
importlib-metadata 1.7.0 py37hc8dfbb8_0 conda-forge
importlib_metadata 1.7.0 0 conda-forge
iniconfig 1.0.1 pyh9f0ad1d_0 conda-forge
ipykernel 5.3.4 py37h43977f1_0 conda-forge
ipython 7.18.1 py37hc6149b9_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
isort 5.0.7 py37hc8dfbb8_0 conda-forge
jedi 0.17.2 py37hc8dfbb8_0 conda-forge
jinja2 2.11.2 pyh9f0ad1d_0 conda-forge
jpeg 9d h516909a_0 conda-forge
jsonschema 3.2.0 py37hc8dfbb8_1 conda-forge
jupyter_client 6.1.7 py_0 conda-forge
jupyter_core 4.6.3 py37hc8dfbb8_1 conda-forge
jupyterlab_pygments 0.1.1 pyh9f0ad1d_0 conda-forge
krb5 1.17.1 hfafb76e_3 conda-forge
lcms2 2.11 hbd6801e_0 conda-forge
ld_impl_linux-64 2.35 h769bd43_9 conda-forge
libblas 3.8.0 17_openblas conda-forge
libcblas 3.8.0 17_openblas conda-forge
libcurl 7.71.1 hcdd3856_6 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.10 hcdb4288_2 conda-forge
libffi 3.2.1 he1b5a44_1007 conda-forge
libgcc-ng 9.3.0 h24d8f2e_16 conda-forge
libgfortran-ng 7.5.0 hdf63c60_16 conda-forge
libgomp 9.3.0 h24d8f2e_16 conda-forge
liblapack 3.8.0 17_openblas conda-forge
libllvm10 10.0.1 he513fc3_3 conda-forge
libllvm8 8.0.1 hc9558a2_0 conda-forge
libnghttp2 1.41.0 h8cfc5f6_2 conda-forge
libopenblas 0.3.10 pthreads_hb3c22a3_4 conda-forge
libpng 1.6.37 hed695b0_2 conda-forge
libprotobuf 3.12.4 h8b12597_0 conda-forge
librmm 0.16.0a200922 cuda11.0_ge3ce8c5_384 rapidsai-nightly
libsodium 1.0.18 h516909a_0 conda-forge
libssh2 1.9.0 hab1572f_5 conda-forge
libstdcxx-ng 9.3.0 hdf63c60_16 conda-forge
libthrift 0.13.0 hbe8ec66_6 conda-forge
libtiff 4.1.0 hc7e4089_6 conda-forge
libutf8proc 2.5.0 h516909a_2 conda-forge
libuv 1.39.0 h516909a_0 conda-forge
libwebp-base 1.1.0 h516909a_3 conda-forge
llvmlite 0.34.0 py37h5202443_1 conda-forge
locket 0.2.0 py_2 conda-forge
lz4-c 1.9.2 he1b5a44_3 conda-forge
markdown 3.2.2 py_0 conda-forge
markupsafe 1.1.1 py37h8f50634_1 conda-forge
mccabe 0.6.1 py_1 conda-forge
mimesis 4.0.0 pyh9f0ad1d_0 conda-forge
mistune 0.8.4 py37h8f50634_1001 conda-forge
more-itertools 8.5.0 py_0 conda-forge
msgpack-python 1.0.0 py37h99015e2_1 conda-forge
nbclient 0.5.0 py_0 conda-forge
nbconvert 6.0.5 py37hc8dfbb8_0 conda-forge
nbformat 5.0.7 py_0 conda-forge
nbsphinx 0.7.1 pyh9f0ad1d_0 conda-forge
nccl 2.7.8.1 h4962215_0 nvidia
ncurses 6.2 he1b5a44_1 conda-forge
nest-asyncio 1.4.0 py_1 conda-forge
nodeenv 1.5.0 pyh9f0ad1d_0 conda-forge
notebook 6.1.4 py37hc8dfbb8_0 conda-forge
numba 0.51.2 py37h9fdb41a_0 conda-forge
numpy 1.19.1 py37h7ea13bd_2 conda-forge
numpydoc 1.1.0 pyh9f0ad1d_0 conda-forge
olefile 0.46 py_0 conda-forge
openssl 1.1.1h h516909a_0 conda-forge
packaging 20.4 pyh9f0ad1d_0 conda-forge
pandas 1.1.2 py37h3340039_0 conda-forge
pandoc 1.19.2 0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
parquet-cpp 1.5.1 2 conda-forge
parso 0.7.1 pyh9f0ad1d_0 conda-forge
partd 1.1.0 py_0 conda-forge
pathspec 0.8.0 pyh9f0ad1d_0 conda-forge
pexpect 4.8.0 py37hc8dfbb8_1 conda-forge
pickleshare 0.7.5 py37hc8dfbb8_1001 conda-forge
pillow 7.2.0 py37h718be6c_1 conda-forge
pip 20.2.3 py_0 conda-forge
pluggy 0.13.1 py37hc8dfbb8_2 conda-forge
pre-commit 2.7.1 py37hc8dfbb8_0 conda-forge
pre_commit 2.7.1 0 conda-forge
prometheus_client 0.8.0 pyh9f0ad1d_0 conda-forge
prompt-toolkit 3.0.7 py_0 conda-forge
psutil 5.7.2 py37h8f50634_0 conda-forge
ptyprocess 0.6.0 py_1001 conda-forge
py 1.9.0 pyh9f0ad1d_0 conda-forge
pyarrow 1.0.1 py37h72578d1_5_cuda conda-forge
pycodestyle 2.6.0 pyh9f0ad1d_0 conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pyflakes 2.2.0 pyh9f0ad1d_0 conda-forge
pygments 2.7.1 py_0 conda-forge
pyopenssl 19.1.0 py_1 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyrsistent 0.17.3 py37h8f50634_0 conda-forge
pysocks 1.7.1 py37hc8dfbb8_1 conda-forge
pytest 6.0.2 py37hc8dfbb8_0 conda-forge
python 3.7.8 h6f2ec95_1_cpython conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python_abi 3.7 1_cp37m conda-forge
pytz 2020.1 pyh9f0ad1d_0 conda-forge
pyyaml 5.3.1 py37h8f50634_0 conda-forge
pyzmq 19.0.2 py37hac76be4_0 conda-forge
rapidjson 1.1.0 he1b5a44_1002 conda-forge
re2 2020.08.01 he1b5a44_1 conda-forge
readline 8.0 he28a2e2_2 conda-forge
recommonmark 0.6.0 py_0 conda-forge
regex 2020.7.14 py37h8f50634_0 conda-forge
requests 2.24.0 pyh9f0ad1d_0 conda-forge
rhash 1.3.6 h14c3975_1001 conda-forge
rmm 0.16.0a200922 cuda_11.0_py37_ge3ce8c5_384 rapidsai-nightly
send2trash 1.5.0 py_0 conda-forge
setuptools 49.6.0 py37hc8dfbb8_1 conda-forge
six 1.15.0 pyh9f0ad1d_0 conda-forge
snappy 1.1.8 he1b5a44_3 conda-forge
snowballstemmer 2.0.0 py_0 conda-forge
sortedcontainers 2.2.2 pyh9f0ad1d_0 conda-forge
spdlog 1.7.0 hc9558a2_2 conda-forge
sphinx 3.2.1 py_0 conda-forge
sphinx-copybutton 0.3.0 pyh9f0ad1d_0 conda-forge
sphinx-markdown-tables 0.0.14 pyh9f0ad1d_1 conda-forge
sphinx_rtd_theme 0.5.0 pyh9f0ad1d_0 conda-forge
sphinxcontrib-applehelp 1.0.2 py_0 conda-forge
sphinxcontrib-devhelp 1.0.2 py_0 conda-forge
sphinxcontrib-htmlhelp 1.0.3 py_0 conda-forge
sphinxcontrib-jsmath 1.0.1 py_0 conda-forge
sphinxcontrib-qthelp 1.0.3 py_0 conda-forge
sphinxcontrib-serializinghtml 1.1.4 py_0 conda-forge
sphinxcontrib-websupport 1.2.4 pyh9f0ad1d_0 conda-forge
sqlite 3.33.0 h4cf870e_0 conda-forge
streamz 0.5.6 pypi_0 pypi
tblib 1.6.0 py_0 conda-forge
terminado 0.9.1 py37hc8dfbb8_0 conda-forge
testpath 0.4.4 py_0 conda-forge
thrift-compiler 0.13.0 hbe8ec66_6 conda-forge
thrift-cpp 0.13.0 6 conda-forge
tk 8.6.10 hed695b0_0 conda-forge
toml 0.10.1 pyh9f0ad1d_0 conda-forge
toolz 0.10.0 py_0 conda-forge
tornado 6.0.4 py37h8f50634_1 conda-forge
traitlets 5.0.4 py_0 conda-forge
typed-ast 1.4.1 py37h516909a_0 conda-forge
typing_extensions 3.7.4.2 py_0 conda-forge
urllib3 1.25.10 py_0 conda-forge
virtualenv 20.0.20 py37hc8dfbb8_1 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_1 conda-forge
webencodings 0.5.1 py_1 conda-forge
wheel 0.35.1 pyh9f0ad1d_0 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
zeromq 4.3.2 he1b5a44_3 conda-forge
zict 2.0.0 py_0 conda-forge
zipp 3.1.0 py_0 conda-forge
zlib 1.2.11 h516909a_1009 conda-forge
zstd 1.4.5 h6597ccf_2 conda-forge
Additional context
Surfaced while running fuzz tests: #6001
This is an issue in pandas as well. Our code is conformant with pandas behavior.
https://github.com/rapidsai/cudf/blob/60d25ea04bf018905836310a18133806f780cb22/cpp/src/io/csv/reader_impl.cu#L583-L587
In [18]: pdf = pd.DataFrame({'a':pd.Series([1, 2, 3, None], dtype='Int64')})
In [19]: pdf.dtypes
Out[19]:
a Int64
dtype: object
In [20]: pdf.to_csv('temp.csv')
In [21]: cat temp.csv
,a
0,1
1,2
2,3
3,
In [22]: pd.read_csv('temp.csv').dtypes
Out[22]:
Unnamed: 0 int64
a float64
dtype: object
I guess the question is with nullable dtypes support introduced in pandas aswell do we still want to mimic pandas behavior?
I guess the question is with nullable dtypes support introduced in pandas aswell do we still want to mimic pandas behavior?
Agreed. This sounds like a bug in Pandas at this point and we shouldn't replicate the behavior. @galipremsagar can you open an issue on Pandas for this?
Agreed. This sounds like a bug in Pandas at this point and we shouldn't replicate the behavior. @galipremsagar can you open an issue on Pandas for this?
Linking for reference: https://github.com/pandas-dev/pandas/issues/36712
From the Pandas issue: Pandas will not change this behavior until they start using nullable types by default.
We can still go ahead with the change.
Most helpful comment
From the Pandas issue: Pandas will not change this behavior until they start using nullable types by default.
We can still go ahead with the change.