Tvm: [CI][Docker] xgboost 1.0.1 causes segfault on test_autotvm_xgboost_model.py

Created on 27 Feb 2020 · 61Comments · Source: apache/tvm

With the release of XGBoost 1.0.x (i.e xgboost-1.0.1-py3-none-manylinux1_x86_64.whl), it seems that installing TVM from scratch (rebuilding Docker containers) makes tests/python/unittest/test_autotvm_xgboost_model.py to fail with a segfault.

Investigating it a bit further, if I manually revert it to xgboost-0.90 it works fine. Using xgboost-1.0.1, this is the message I see:

tests/python/unittest/test_autotvm_xgboost_model.py::test_fit Fatal Python error: Segmentation fault

Thread 0x00007f4f98de4700 (most recent call first):
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 379 in _recv
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 407 in _recv_bytes
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 250 in recv
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 463 in _handle_results
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f4f905e3700 (most recent call first):
  File "/usr/lib/python3.6/threading.py", line 295 in wait
  File "/usr/lib/python3.6/queue.py", line 164 in get
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 415 in _handle_tasks
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f4f8fde2700 (most recent call first):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 406 in _handle_workers
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007f4fb514c700 (most recent call first):
  File "/usr/local/lib/python3.6/dist-packages/xgboost/core.py", line 1248 in update
  File "/usr/local/lib/python3.6/dist-packages/xgboost/training.py", line 74 in _train_internal
  File "/usr/local/lib/python3.6/dist-packages/xgboost/training.py", line 209 in train
  File "/workspace/python/tvm/autotvm/tuner/xgboost_cost_model.py", line 272 in fit_log
  File "/workspace/tests/python/unittest/test_autotvm_xgboost_model.py", line 35 in test_fit
  File "/usr/local/lib/python3.6/dist-packages/_pytest/python.py", line 167 in pytest_pyfunc_call
  File "/usr/local/lib/python3.6/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.6/dist-packages/_pytest/python.py", line 1445 in runtest
  File "/usr/local/lib/python3.6/dist-packages/_pytest/runner.py", line 134 in pytest_runtest_call
  File "/usr/local/lib/python3.6/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.6/dist-packages/_pytest/runner.py", line 210 in <lambda>
  File "/usr/local/lib/python3.6/dist-packages/_pytest/runner.py", line 237 in from_call
  File "/usr/local/lib/python3.6/dist-packages/_pytest/runner.py", line 210 in call_runtest_hook
  File "/usr/local/lib/python3.6/dist-packages/_pytest/runner.py", line 185 in call_and_report
  File "/usr/local/lib/python3.6/dist-packages/_pytest/runner.py", line 99 in runtestprotocol
  File "/usr/local/lib/python3.6/dist-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol
  File "/usr/local/lib/python3.6/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.6/dist-packages/_pytest/main.py", line 271 in pytest_runtestloop
  File "/usr/local/lib/python3.6/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.6/dist-packages/_pytest/main.py", line 247 in _main
  File "/usr/local/lib/python3.6/dist-packages/_pytest/main.py", line 197 in wrap_session
  File "/usr/local/lib/python3.6/dist-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/usr/local/lib/python3.6/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.6/dist-packages/_pytest/config/__init__.py", line 93 in main
  File "/usr/local/lib/python3.6/dist-packages/pytest/__main__.py", line 7 in <module>
  File "/usr/lib/python3.6/runpy.py", line 85 in _run_code
  File "/usr/lib/python3.6/runpy.py", line 193 in _run_module_as_main
./tests/scripts/task_python_unittest.sh: line 27: 24582 Segmentation fault      (core dumped) TVM_FFI=ctypes python3 -m pytest -v tests/python/unittest

@tqchen, I didn't see any PR or discussion about it, but are you aware about any ongoing initiative to move TVM to XGBoost 1.0.x, or shall we pin xgboost to be 0.90, to prevent the error to happen? (note: I'm happy to send a patch to pin the version)

Source

leandron

Most helpful comment

With https://github.com/dmlc/xgboost/pull/5590, I can now run tests/python/unittest/test_autotvm_xgboost_model.py::test_fit without crashing.

hcho3 on 24 Apr 2020

🎉3

All 61 comments

@leandron can you create a minimum reproducible example? e.g. pickle the data that causes segfault in XGBoost. Then we can start to bring attention of the XGBoost dev community. In the meanwhile, we can pin xgboost to 0.9

tqchen on 27 Feb 2020

cc @merrymercy @hcho3 Who might be interested in this issue

tqchen on 27 Feb 2020

It would be nice if we can get a reproducible example. We are currently working on the patch release 1.0.2 and I want to get a patch to fix this issue.

hcho3 on 27 Feb 2020

@leandron @hcho3 please followup :)

tqchen on 11 Mar 2020

@leandron can you please comment about the current state?

tqchen on 30 Mar 2020

Hi, I only managed to investigate it further, today. XGBoost now is version 1.0.2, and I can still reproduce this issue.

To give some context, this is the function call that triggers the issue:
https://github.com/apache/incubator-tvm/blob/54975a3fd24fa45b815be39075f4614e53009444/python/tvm/autotvm/tuner/xgboost_cost_model.py#L262-L272

I tried just creating pickle files of some inputs (self.xgb_params and dtrain) and simplifying the function call, but this is not enough to reproduce the issue. The issue seems to be in the context custom_callback, below:

https://github.com/apache/incubator-tvm/blob/54975a3fd24fa45b815be39075f4614e53009444/python/tvm/autotvm/tuner/xgboost_cost_model.py#L419-L526

Now, something that could help me a bit to narrow down where the problem is if I run XGBoost in debug mode. @hcho3 what is the simplest way I can do that?

leandron on 3 Apr 2020

@leandron Thanks for pointing out which part of TVM test is failing. Not sure if running in debug mode would help, since XGBoost is crashing with segfault here. I will take a look some time this week.

hcho3 on 7 Apr 2020

❤1

I compiled TVM from source and tried running the test tests/python/unittest/test_autotvm_xgboost_model.py::test_fit and I cannot reproduce the issue. Do I need a specific Docker container to reproduce the problem?

hcho3 on 20 Apr 2020

I also tried building ci-cpu Docker image from scratch and running the unit test inside the container. The test runs without crashing.

hcho3 on 20 Apr 2020

We applied a workaround, pinning the xgboost version to be 0.90. Which XGBoost version you see in the image you created from scratch?

leandron on 20 Apr 2020

@leandron I checked out commit 8502691b5b7ca152da9eb626529070db53d479c8 so that XGBoost 1.0.x is used.

hcho3 on 20 Apr 2020

@leandron can you also provide a bit more details

e.g. does directly run tests/python/unittest/test_autotvm_xgboost_model.py fails or do we need to run the entire unittest. It would also be nice if you can send a CI binary hashtag(perhaps in docker hub) to confirm the problematic issue.

I tried to build a docker image with xgboost==1.0.2 and seems cannot repro the issue.

tqchen on 22 Apr 2020

I see this reliably with a virtualenv on bionic on AWS.

environment:
ami: 099720109477/ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200408
using tvm revision: 72f2aea2dd219bf55c15b3cf4cfc21491f1f60dd

command: TVM_FFI=ctypes python3 -m pytest -s -v tests/python/unittest -k 'test_autotvm_xgboost_model'

python version
```$ python --version
Python 3.7.5

installed python packages:

antlr4-python3-runtime==4.8
Cython==0.29.16
decorator==4.4.2
psutil==5.7.0
pylint==2.4.4

astroid [required: >=2.3.0,<2.4, installed: 2.3.3]
- lazy-object-proxy [required: ==1.4., installed: 1.4.3]
- wrapt [required: ==1.11., installed: 1.11.2]
isort [required: >=4.2.5,<5, installed: 4.3.21]
mccabe [required: >=0.6,<0.7, installed: 0.6.1]
pytest==5.4.1
attrs [required: >=17.4.0, installed: 19.3.0]
importlib-metadata [required: >=0.12, installed: 1.6.0]
- zipp [required: >=0.5, installed: 3.1.0]
more-itertools [required: >=4.0.0, installed: 8.2.0]
packaging [required: Any, installed: 20.3]
- pyparsing [required: >=2.0.2, installed: 2.4.7]
- six [required: Any, installed: 1.14.0]
pluggy [required: >=0.12,<1.0, installed: 0.13.1]
- importlib-metadata [required: >=0.12, installed: 1.6.0]
  - zipp [required: >=0.5, installed: 3.1.0]
py [required: >=1.5.0, installed: 1.8.1]
wcwidth [required: Any, installed: 0.1.9]
tornado==6.0.4
xgboost==1.0.2
numpy [required: Any, installed: 1.18.3]
scipy [required: Any, installed: 1.4.1]
- numpy [required: >=1.13.3, installed: 1.18.3]

backtrace:

~/ws/tvm$ gdb python
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...(no debugging symbols found)...done.
(gdb) run tests/python/unittest/test_autotvm_xgboost_model.py
Starting program: /home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/bin/python tests/python/unittest/test_autotvm_xgboost_model.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4104700 (LWP 11905)]
[New Thread 0x7ffff3903700 (LWP 11906)]
[New Thread 0x7fffef102700 (LWP 11907)]
[New Thread 0x7fffec901700 (LWP 11908)]
[New Thread 0x7fffea100700 (LWP 11909)]
[New Thread 0x7fffe78ff700 (LWP 11910)]
[New Thread 0x7fffe50fe700 (LWP 11911)]
[New Thread 0x7fffe28fd700 (LWP 11912)]
[New Thread 0x7fffe00fc700 (LWP 11913)]
[New Thread 0x7fffdd8fb700 (LWP 11914)]
[New Thread 0x7fffdb0fa700 (LWP 11915)]
[New Thread 0x7fffd88f9700 (LWP 11916)]
[New Thread 0x7fffd60f8700 (LWP 11917)]
[New Thread 0x7fffd38f7700 (LWP 11918)]
[New Thread 0x7fffd10f6700 (LWP 11919)]
[Thread 0x7fffe50fe700 (LWP 11911) exited]
[Thread 0x7fffd88f9700 (LWP 11916) exited]
[Thread 0x7fffd10f6700 (LWP 11919) exited]
[Thread 0x7fffd60f8700 (LWP 11917) exited]
[Thread 0x7fffdb0fa700 (LWP 11915) exited]
[Thread 0x7fffdd8fb700 (LWP 11914) exited]
[Thread 0x7fffe00fc700 (LWP 11913) exited]
[Thread 0x7fffe28fd700 (LWP 11912) exited]
[Thread 0x7fffe78ff700 (LWP 11910) exited]
[Thread 0x7fffea100700 (LWP 11909) exited]
[Thread 0x7fffec901700 (LWP 11908) exited]
[Thread 0x7fffef102700 (LWP 11907) exited]
[Thread 0x7ffff3903700 (LWP 11906) exited]
[Thread 0x7ffff4104700 (LWP 11905) exited]
[Thread 0x7fffd38f7700 (LWP 11918) exited]
[New Thread 0x7fffd10f6700 (LWP 11936)]
[New Thread 0x7fffd38f7700 (LWP 11937)]
[New Thread 0x7fffd60f8700 (LWP 11938)]
[Thread 0x7fffd10f6700 (LWP 11936) exited]
[Thread 0x7fffd60f8700 (LWP 11938) exited]
[Thread 0x7fffd38f7700 (LWP 11937) exited]
[New Thread 0x7fffd38f7700 (LWP 11955)]
[New Thread 0x7fffd60f8700 (LWP 11956)]
[New Thread 0x7fffd10f6700 (LWP 11957)]

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffb5ba9e37 in std::vector, std::allocator > > xgboost::XGBoostParameter::UpdateAllowUnknown, std::allocator > > >(std::vector, std::allocator > > const&, bool*) ()
from /home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
(gdb) bt

0 0x00007fffb5ba9e37 in std::vector, std::allocator > > xgboost::XGBoostParameter::UpdateAllowUnknown, std::allocator > > >(std::vector, std::allocator > > const&, bool*) ()