Taichi: Python 3.8 support

[yuanming-hu]

Is your feature request related to a problem? Please describe.
Some users want to use taichi with Python 3.8.

Describe the solution you'd like
Simply test building from source with Python 3.8 and execute ti test to see if the test fails. If it does fail, it's probably because Python 3.8 changes the internal AST organization and the Python frontend compiler (essentially a Python AST transformer, transformer.py) fails due to potential compatibility issues: https://github.com/taichi-dev/taichi/blob/f294c9436615ee5523ff3ab803ca73af550f2795/python/taichi/lang/transformer.py#L31

We welcome contributions on this!

[Psycho7]

No issue found with Python 3.8.1. Below is the test log.

https://gist.github.com/Psycho7/82b76e28e2e71d9c8ca0ed26c035b223

[yuanming-hu]

This is great news! I thought we need to modify the AST transformer. Trying to add Python 3.8 to the build bots...

[yuanming-hu]

Added Python 3.8 to Appveyor (our Win32 buildbot). Working on Linux releases.

Meanwhile, I tried to add Python 3.6 & 3.8 to Travis for our OS X releases yet had no success. If someone familiar with OS X/Travis can help with this, that's greatly appreciated. My failure attempts: 4093c67fec14a3fa223e95afbf182fea8a10d6d6 8650dce048726a77b5d7f718a0bbb2a01a51ede9 57115356b04cca7bb98d7dd3ab809296ee431774 8481bb08492c56839a197009c784c3f42efd527e A standalone issue for OS X builds: https://github.com/taichi-dev/taichi/issues/409

[masahi]

I'm using Python 3.7.4 that comes with anaconda, is this version supposed to work? I'm having trouble running tests. Here is my log:

$ ti test
[T 02/07/20 06:35:42.698] [logging.cpp:Logger@68] Taichi core started. Thread ID = 874
[Taichi version 0.4.1, cpu only, commit 7b7c1d56]

 *******************************************
 **     Taichi Programming Language       **
 *******************************************


Running python tests...

============================================================== test session starts ===============================================================
platform linux -- Python 3.7.4, pytest-5.2.1, py-1.8.0, pluggy-0.13.0
rootdir: /home/masa/projects/graphics/taichi
plugins: doctestplus-0.4.0, remotedata-0.3.2, openfiles-0.4.0, arraydiff-0.3
collected 192 items                                                                                                                              

../tests/python/test_abs.py Fatal Python error: Aborted

Current thread 0x00007f5eedaf8740 (most recent call first):
  File "/home/masa/projects/graphics/taichi/python/taichi/lang/impl.py", line 125 in materialize
  File "/home/masa/projects/graphics/taichi/python/taichi/lang/expr.py", line 186 in __setitem__
  File "/home/masa/projects/graphics/taichi/tests/python/test_abs.py", line 23 in test_abs
  File "/home/masa/projects/graphics/taichi/python/taichi/lang/__init__.py", line 137 in test
  File "/home/masa/anaconda3/lib/python3.7/site-packages/_pytest/python.py", line 170 in pytest_pyfunc_call
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/masa/anaconda3/lib/python3.7/site-packages/_pytest/python.py", line 1423 in runtest
  File "/home/masa/anaconda3/lib/python3.7/site-packages/_pytest/runner.py", line 125 in pytest_runtest_call
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/masa/anaconda3/lib/python3.7/site-packages/_pytest/runner.py", line 201 in <lambda>
  File "/home/masa/anaconda3/lib/python3.7/site-packages/_pytest/runner.py", line 229 in from_call
  File "/home/masa/anaconda3/lib/python3.7/site-packages/_pytest/runner.py", line 201 in call_runtest_hook
  File "/home/masa/anaconda3/lib/python3.7/site-packages/_pytest/runner.py", line 176 in call_and_report
  File "/home/masa/anaconda3/lib/python3.7/site-packages/_pytest/runner.py", line 95 in runtestprotocol
  File "/home/masa/anaconda3/lib/python3.7/site-packages/_pytest/runner.py", line 80 in pytest_runtest_protocol
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/masa/anaconda3/lib/python3.7/site-packages/_pytest/main.py", line 256 in pytest_runtestloop
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/masa/anaconda3/lib/python3.7/site-packages/_pytest/main.py", line 235 in _main
  File "/home/masa/anaconda3/lib/python3.7/site-packages/_pytest/main.py", line 191 in wrap_session
  File "/home/masa/anaconda3/lib/python3.7/site-packages/_pytest/main.py", line 228 in pytest_cmdline_main
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
  File "/home/masa/anaconda3/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/masa/anaconda3/lib/python3.7/site-packages/_pytest/config/__init__.py", line 90 in main
  File "/home/masa/projects/graphics/taichi/python/taichi/main.py", line 16 in test_python
  File "/home/masa/projects/graphics/taichi/python/taichi/main.py", line 94 in main
  File "/home/masa/projects/graphics/taichi/bin/ti", line 4 in <module>

[yuanming-hu]

@masahi Could you upgrade to v0.4.4 and execute

ti test_verbose

to see what's wrong exactly?

[masahi]

ok I've upgraded to the latest version. I was using the older one because difftaichi examples were broken when I tried last night. But it seems to be fixed now, will try again. You move so fast :)

When I ran ti test_verbose with the latest version, I got the same error. I don't know what happened, but on a second try I got it working! Next I'll try GPU test.

For reference, here is my log for the first try with 0.4.4 that failed.

[yuanming-hu]

I guess that might relate to clang. Could you let me know your clang version? Thanks!

[masahi]

I'm using the same version of clang and llvm that are specified in DockerFile. CXX=clang++-7 (binary download) and llvm-8.0.1 (built from source).

Here is my cmake log.

Taichi Version 0.4.4
       Commit c042fba76e186f4db0ada55115efda1cd4c76a7a
Using C++ compiler: /home/masa/projects/compiler/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-18.04/bin/clang++
Clang compiler detected. Using std=c++17.
Building for processor x86_64
Setting -march=nehalem for x86_64 processors
Using float32 (single) precision as real
Using /home/masa/anaconda3/bin/python as python executable.
Python 3.7.4
    version: 3.7
    include: /home/masa/anaconda3/include/python3.7m
    library: /home/masa/anaconda3/lib/libpython3.7m.so
    numpy include: /home/masa/anaconda3/lib/python3.7/site-packages/numpy/core/include
    pybind11 include: /home/masa/anaconda3/include;/home/masa/anaconda3/include
-- Found LLVM 8.0.1
-- Using LLVMConfig.cmake in: /home/masa/projects/compiler/llvm-8.0.1.src/build/lib/cmake/llvm
llvm include dirs /home/masa/projects/compiler/llvm-8.0.1.src/include;/home/masa/projects/compiler/llvm-8.0.1.src/build/include
PYTHON_LIBRARIES/home/masa/anaconda3/lib/libpython3.7m.so
C++ Flags:  -DTC_ISE_NONE -std=c++17 -fsized-deallocation -Wall  -march=nehalem -DTI_ARCH_x86_64 -DTC_PASS_EXCEPTION_TO_PYTHON -DTC_INCLUDED -g
Build type: RelWithDebInfo
-- Configuring done
-- Generating done
-- Build files have been written to: /home/masa/projects/graphics/taichi/build

[yuanming-hu]

Taichi will look for clang-7 or clang to compiler the runtime into bitcode:
https://github.com/taichi-dev/taichi/blob/c042fba76e186f4db0ada55115efda1cd4c76a7a/taichi/runtime/runtime.cpp

[T 02/07/20 06:58:10.630] [taichi_llvm_context.cpp:compile_runtime_bitcode@110] Compiling runtime module bitcode...
llvm-as: assembly parsed, but does not verify as correct!
Intrinsic has incorrect argument type!
void (i8*, i8*, i64, i1)* @llvm.memcpy.p0i8.p0i8.i64
Intrinsic has incorrect argument type!
void (i8*, i8*, i64, i1)* @llvm.memcpy.p0i8.p0i8.i64
Intrinsic has incorrect argument type!
void (i8*, i8*, i64, i1)* @llvm.memcpy.p0i8.p0i8.i64
Intrinsic has incorrect argument type!
void (i8*, i8*, i64, i1)* @llvm.memcpy.p0i8.p0i8.i64
Intrinsic has incorrect argument type!
void (i8*, i8*, i64, i1)* @llvm.memcpy.p0i8.p0i8.i64
Intrinsic has incorrect argument type!
void (i8*, i8*, i64, i1)* @llvm.memcpy.p0i8.p0i8.i64
Intrinsic has incorrect argument type!
void (i8*, i8, i64, i1)* @llvm.memset.p0i8.i64
[W 02/07/20 06:58:10.992] [taichi_llvm_context.cpp:module_from_bitcode_file@168] Bitcode loading error message:
Invalid bitcode signature
[E 02/07/20 06:58:10.992] [taichi_llvm_context.cpp:module_from_bitcode_file@170] Bitcode /home/masa/projects/graphics/taichi/taichi/runtime//runtime_x86_64.bc load failure.
Fatal Python error: Aborted

For some reason, LLVM is complaining Intrinsic has incorrect argument type! void (i8*, i8*, i64, i1)* @llvm.memcpy.p0i8.p0i8.i64

[yuanming-hu]

I've been using clang-7 with no problem. Do you have a clang command in your path with high version?

clang version 7.0.0-3~ubuntu0.18.04.1 (tags/RELEASE_700/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

[masahi]

Yes, clang-7 is on my PATH. The only thing I did between the first and second try was ldd libtaichi_core.so. Will try ti test on different machines to see if I get the same issue.

[masahi]

A bit off topic, but it seems if there are another LLVM libs present on the system, taichi crashes. Here is the log of running diffmpm.py example.

i= 93 loss= -0.49842146039009094
i= 94 loss= -0.49916335940361023
i= 95 loss= -0.49980902671813965
i= 96 loss= -0.5005872249603271
i= 97 loss= -0.5007992386817932
i= 98 loss= -0.5013236403465271
i= 99 loss= -0.5019235610961914
[E 02/07/20 07:28:42.007] Received signal 11 (Segmentation fault)
***********************************
* Taichi Compiler Stack Traceback *
***********************************
/home/masa/projects/graphics/taichi/build/taichi_core.so: taichi::signal_handler(int)
/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7f48c7673f20]
/usr/lib/x86_64-linux-gnu/libLLVM-9.so.1: llvm::StringMapImpl::LookupBucketFor(llvm::StringRef)
/usr/lib/x86_64-linux-gnu/libLLVM-9.so.1(+0x98ec5f) [0x7f481efc4c5f]
/usr/lib/x86_64-linux-gnu/libLLVM-9.so.1(+0x98ee0c) [0x7f481efc4e0c]
/usr/lib/x86_64-linux-gnu/libLLVM-9.so.1(+0x984fc5) [0x7f481efbafc5]
/usr/lib/x86_64-linux-gnu/libLLVM-9.so.1(+0x990e9d) [0x7f481efc6e9d]
/usr/lib/x86_64-linux-gnu/libLLVM-9.so.1: llvm::ManagedStaticBase::RegisterManagedStatic(void* (*)(), void (*)(void*)) const
/usr/lib/x86_64-linux-gnu/libLLVM-9.so.1: llvm::cl::OptionCategory::registerCategory()
/usr/lib/x86_64-linux-gnu/libLLVM-9.so.1(+0x8a1fa2) [0x7f481eed7fa2]
/lib64/ld-linux-x86-64.so.2(+0x10733) [0x7f48c7c55733]
/lib64/ld-linux-x86-64.so.2(+0x151ff) [0x7f48c7c5a1ff]
/lib/x86_64-linux-gnu/libc.so.6: _dl_catch_exception
/lib64/ld-linux-x86-64.so.2(+0x147ca) [0x7f48c7c597ca]
/lib/x86_64-linux-gnu/libdl.so.2(+0xf96) [0x7f48c7431f96]
/lib/x86_64-linux-gnu/libc.so.6: _dl_catch_exception
/lib/x86_64-linux-gnu/libc.so.6: _dl_catch_error
/lib/x86_64-linux-gnu/libdl.so.2(+0x1735) [0x7f48c7432735]
/lib/x86_64-linux-gnu/libdl.so.2: dlopen
/usr/lib/x86_64-linux-gnu/libGLX_mesa.so.0(+0x55f51) [0x7f4839a8ff51]
/usr/lib/x86_64-linux-gnu/libGLX_mesa.so.0(+0x43120) [0x7f4839a7d120]
/usr/lib/x86_64-linux-gnu/libGLX_mesa.so.0(+0x4b856) [0x7f4839a85856]
/usr/lib/x86_64-linux-gnu/libGLX_mesa.so.0(+0x37f26) [0x7f4839a71f26]
/usr/lib/x86_64-linux-gnu/libGLX_mesa.so.0(+0x337d2) [0x7f4839a6d7d2]
/usr/lib/x86_64-linux-gnu/libGLX_mesa.so.0(+0x34f12) [0x7f4839a6ef12]
/usr/lib/x86_64-linux-gnu/libGLX.so.0: glXChooseFBConfig
/home/masa/anaconda3/plugins/xcbglintegrations/libqxcb-glx-integration.so(+0xc30f) [0x7f486055030f]
/home/masa/anaconda3/plugins/xcbglintegrations/libqxcb-glx-integration.so(+0xc505) [0x7f4860550505]
/home/masa/anaconda3/plugins/xcbglintegrations/libqxcb-glx-integration.so(+0x98d1) [0x7f486054d8d1]
/home/masa/anaconda3/plugins/platforms/../../lib/libQt5XcbQpa.so.5: QXcbWindow::create()
/home/masa/anaconda3/plugins/platforms/../../lib/libQt5XcbQpa.so.5: QXcbIntegration::createPlatformWindow(QWindow*) const
/home/masa/anaconda3/lib/python3.7/site-packages/PyQt5/../../../libQt5Gui.so.5: QWindowPrivate::create(bool, unsigned long long)
/home/masa/anaconda3/lib/python3.7/site-packages/PyQt5/../../../libQt5Widgets.so.5: QWidgetPrivate::create_sys(unsigned long long, bool, bool)
/home/masa/anaconda3/lib/python3.7/site-packages/PyQt5/../../../libQt5Widgets.so.5: QWidget::create(unsigned long long, bool, bool)
/home/masa/anaconda3/lib/python3.7/site-packages/PyQt5/../../../libQt5Widgets.so.5: QWidget::setVisible(bool)
/home/masa/anaconda3/lib/python3.7/site-packages/PyQt5/QtWidgets.so(+0x26f063) [0x7f48a6fd6063]
/home/masa/anaconda3/lib/python3.7/site-packages/PyQt5/QtWidgets.so(+0x345d12) [0x7f48a70acd12]
python(_PyMethodDef_RawFastCallKeywords+0x1f0) [0x55bbc14e16e0]
python(_PyCFunction_FastCallKeywords+0x21) [0x55bbc14e1861]
python(_PyEval_EvalFrameDefault+0x4ecd) [0x55bbc154d2bd]
python(_PyFunction_FastCallKeywords+0xfb) [0x55bbc14e0ccb]
python(_PyEval_EvalFrameDefault+0x6a3) [0x55bbc1548a93]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55bbc1491539]
python(_PyFunction_FastCallKeywords+0x325) [0x55bbc14e0ef5]
python(_PyEval_EvalFrameDefault+0x6a3) [0x55bbc1548a93]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55bbc1491539]
python(_PyFunction_FastCallDict+0x1d5) [0x55bbc1492635]
python(_PyEval_EvalFrameDefault+0x1e42) [0x55bbc154a232]
python(_PyEval_EvalCodeWithName+0xac9) [0x55bbc1491d09]
python(_PyFunction_FastCallDict+0x1d5) [0x55bbc1492635]
python(_PyObject_Call_Prepend+0x63) [0x55bbc14b0e53]
python(PyObject_Call+0x6e) [0x55bbc14a3dbe]
python(_PyEval_EvalFrameDefault+0x1e42) [0x55bbc154a232]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55bbc1491539]
python(_PyFunction_FastCallKeywords+0x387) [0x55bbc14e0f57]
python(_PyEval_EvalFrameDefault+0x4b39) [0x55bbc154cf29]
python(_PyFunction_FastCallKeywords+0xfb) [0x55bbc14e0ccb]
python(_PyEval_EvalFrameDefault+0x416) [0x55bbc1548806]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55bbc1491539]
python(PyEval_EvalCodeEx+0x44) [0x55bbc1492424]
python(PyEval_EvalCode+0x1c) [0x55bbc149244c]
python(+0x22ab74) [0x55bbc15a7b74]
python(PyRun_FileExFlags+0xa1) [0x55bbc15b1eb1]
python(PyRun_SimpleFileExFlags+0x1c3) [0x55bbc15b20a3]
python(+0x236195) [0x55bbc15b3195]
python(_Py_UnixMain+0x3c) [0x55bbc15b32bc]
/lib/x86_64-linux-gnu/libc.so.6: __libc_start_main
python(+0x1db062) [0x55bbc1558062]

[yuanming-hu]

Oh interesting. Are you building with LLVM-9, or libLLVM-9.so somehow gets preloaded to your process?

[masahi]

I built taichi with 8.0.1 (see my cmake log above), but yes, libs under /usr/lib/x86_64-linux-gnu/ seem to be preloaded on ubuntu.

[masahi]

And libtaichi_core.so is statically linked against LLVM.

$ ldd libtaichi_core.so 
    linux-vdso.so.1 (0x00007fffdcdc0000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb83eccc000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fb83e943000)
    libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007fb83e60b000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fb83e3ee000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fb83e1e6000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb83dfe2000)
    libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007fb83ddb8000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb83da1a000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb83d629000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fb841c9b000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb83d411000)
    libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007fb83d1e9000)
    libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007fb83cfe5000)
    libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007fb83cddf000)
    libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007fb83cbca000)

[yuanming-hu]

LOL. I see. I don't think this can be fixed within Taichi, since someone can just preload a faulty glibc and hack memset to crash Taichi in various ways.

[masahi]

Yes, this is user's responsibility to get it right. I just have so many projects dependent on LLVM ...

[archibate]

README.md:

Python 3.6/3.7 needed for all platforms. Python 3.8 supported only on OS X and Windows

Python 3.8 is build passing on my Linux environment. I think there is no reason for us not releasing it.