Taichi: building procedure is not prescribed

[davidleon]

can't build on win/linux. i modify the source a little bit to work on llvm9. though leaving the source mostly untouched, the import fails at different place. i got a version that windows failed to load bitcode. And after i pull the source from repo, the linux vesion fails at import taichi_core. i explicitly change the sys.path to contain the taichi_core.pyd file. I believe is something wrong during the taichi_core import. and the windows version now fails at:

(base) D:\dev\taichi\python>python build.py
[Release mode]
[T 02/04/20 19:26:01.480] [logging.cpp:taichi::Logger::Logger@68] Taichi core started. Thread ID = 16448
[Taichi version 0.3.25, cuda 10.0, commit c5ce590f]
Traceback (most recent call last):
  File "build.py", line 6, in <module>
    import taichi as ti
  File "D:\dev\taichi\python\taichi\__init__.py", line 15, in <module>
    from taichi.lang import *
  File "D:\dev\taichi\python\taichi\lang\__init__.py", line 26, in <module>
    cuda = core.cuda
AttributeError: module 'taichi_core' has no attribute 'cuda'

any ideas?

[yuanming-hu]

Hi @davidleon,

Thanks for reporting. Two things might be the cause:

• Could you update your Taichi code base? The latest version is v0.4.3
• On Windows if you are doing development please set the environment variables
  
  
  
  • TAICHI_REPO_DIR=/home/XXX/taichi # Path to your taichi repository
  
  
  • PYTHONPATH=$TAICHI_REPO_DIR/python/
  
  

Two other common issues and solutions on Windows: https://github.com/taichi-dev/taichi/issues/370 and https://github.com/taichi-dev/taichi/issues/428

[davidleon]

oh works now.

[davidleon]

[T 02/05/20 01:29:14.072] [taichi_llvm_context.cpp:taichi::Tlang::compile_runtime_bitcode@110] Compiling runtime module bitcode...
Assertion failed: Index < Length && "Invalid index!", file C:\Users\david\Downloads\llvm-9.0.0.src\include\llvm/ADT/ArrayRef.h, line 240
however many difftaichi examples run into this error, some works fine

[davidleon]

I think it's much better to standardize the building process and the testing process. This speeds up contribution from community

[yuanming-hu]

I feel like the Linux/OS X building process is pretty much standard: https://taichi.readthedocs.io/en/latest/dev_install.html

Windows users usually just follow https://github.com/taichi-dev/taichi/blob/master/appveyor.yml#L28

[davidleon]

'_ti test_' this testing comand is not listed in windows section. I think this should be in a different section. And describe how testing should return results, and how to write test cases.

[davidleon]

developers might not want to completely replicate the building environment. e.g. in my case i want to use the newest llvm. and some linux distro have cuda installed in different dir arch. iirc, in source code the cuda lib is hardcoded to /usr/lib/cuda/XX. in my distro the cuda is installed in /opt/cuda/. i will check the windows _ti test_ later to see whatever test cases fail. currently linux version c++ part compiles, python build.py refuses to build
(base) [davidl@davidl-pc python]$ python build.py
[Release mode]
Traceback (most recent call last):
File "build.py", line 6, in
import taichi as ti
File "/run/media/davidl/Data/dev/taichi/python/taichi/__init__.py", line 1, in
from taichi.main import main
File "/run/media/davidl/Data/dev/taichi/python/taichi/main.py", line 6, in
from taichi.tools.video import make_video, interpolate_frames, mp4_to_gif, scale_video, crop_video, accelerate_video
File "/run/media/davidl/Data/dev/taichi/python/taichi/tools/video.py", line 3, in
import taichi.core as core
File "/run/media/davidl/Data/dev/taichi/python/taichi/core/__init__.py", line 1, in
from .util import tc_core, build, format, load_module, start_memory_monitoring, \
File "/run/media/davidl/Data/dev/taichi/python/taichi/core/util.py", line 161, in
os.symlink(link_src, link_dst)
FileExistsError: [Errno 17] File exists: '/run/media/davidl/Data/dev/taichi/python/taichi/core/../lib/taichi_core.so' -> '/run/media/davidl/Data/dev/taichi/python/taichi/core/../lib/libtaichi_core.so'

[davidleon]

umm removing the file and copy the prebuilt file libtaichi_core.so to the /dev/taichi/python/taichi/lib solves the linux building problem. and setting TAICHI_REPO_DIR is the key to solve import taichi_core error.

[davidleon]

running difftaichi in linux shows this errors in smoke series.

[T 02/05/20 09:20:56.814] [taichi_llvm_context.cpp:compile_runtime_bitcode@110] Compiling runtime module bitcode...                                         
/run/media/davidl/Data/dev/taichi//taichi/runtime/runtime.cpp:89:6: error: functions that differ only in their return type cannot be overloaded
void vprintf(Ptr format, Ptr arg);
~~~~ ^
/usr/include/stdio.h:347:12: note: previous declaration is here
extern int vprintf (const char *__restrict __format, __gnuc_va_list __arg);
       ~~~ ^
/run/media/davidl/Data/dev/taichi//taichi/runtime/runtime.cpp:970:3: error: no matching function for call to 'vprintf'
  vprintf((Ptr)format, (Ptr)&t);
  ^~~~~~~
/usr/include/stdio.h:347:12: note: candidate function not viable: no known conversion from 'Ptr' (aka 'unsigned char *') to 'const char *__restrict' for 1st argument
extern int vprintf (const char *__restrict __format, __gnuc_va_list __arg);
           ^
2 errors generated.
[E 02/05/20 09:20:57.246] [taichi_llvm_context.cpp:compile_runtime_bitcode@122] Runtime compilation failed.
[E 02/05/20 09:20:57.246] Received signal 6 (Aborted)

windows version only gpu version shows the assertion error.

Assertion failed: Index < Length && "Invalid index!", file C:\Users\david\Downloads\llvm-9.0.0.src\include\llvm/ADT/ArrayRef.h, line 240

Ti test results(linux, home build version and pip install prebuilt version ) show this error:

(base) [davidl@davidl-pc ~]$ ti test
[Release mode]
[T 02/05/20 09:43:52.376] [logging.cpp:Logger@68] Taichi core started. Thread ID = 8178                                                                     
[Taichi version 0.4.3, cuda 10.1, commit 6b8ac35e]

 *******************************************
 **     Taichi Programming Language       **
 *******************************************


Running python tests...

============================ test session starts =============================
platform linux -- Python 3.7.4, pytest-5.3.4, py-1.8.1, pluggy-0.13.1
rootdir: /home/davidl
collected 192 items                                                          

miniconda3/lib/python3.7/site-packages/taichi/tests/test_abs.py Fatal Python error: Aborted

Current thread 0x00007fb465341740 (most recent call first):
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/taichi/lang/impl.py", line 117 in materialize
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/taichi/lang/expr.py", line 186 in __setitem__
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/taichi/tests/test_abs.py", line 23 in test_abs
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/taichi/lang/__init__.py", line 149 in test
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/python.py", line 167 in pytest_pyfunc_call
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/python.py", line 1445 in runtest
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/runner.py", line 134 in pytest_runtest_call
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/runner.py", line 210 in <lambda>
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/runner.py", line 237 in from_call
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/runner.py", line 210 in call_runtest_hook
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/runner.py", line 185 in call_and_report
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/runner.py", line 99 in runtestprotocol
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/main.py", line 271 in pytest_runtestloop
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/main.py", line 247 in _main
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/main.py", line 197 in wrap_session
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/config/__init__.py", line 93 in main
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/taichi/main.py", line 14 in test_python
  File "/home/davidl/miniconda3/lib/python3.7/site-packages/taichi/main.py", line 94 in main
  File "/home/davidl/miniconda3/bin/ti", line 8 in <module>

[yuanming-hu]

'ti test' this testing comand is not listed in windows section. I think this should be in a different section. And describe how testing should return results, and how to write test cases.

Updated the doc. On Windows, the ti command should be replaced by python -m taichi. If you know how to make ti work on Windows, that would be great!

build.py refuses to build
build.py is for building the pypi packages. You probably don't need that...

The vprintf issue is probably due to a name conflict with stdio.h..

Could you open up a PR with your LLVM 9.0.0 fixes? That will help a lot of people and we plan to upgrade LLVM very soon. Thanks!

[yuanming-hu]

On linux it's probably the same LLVM error. In the latest commit you can try

ti test_verbose

so that error messages will get printed.

[davidleon]

1. some difftaichi code fails AttributeError: module 'taichi' has no attribute 'global_var'. seems like it's a update change.
2. some difftaichi now fails cuda out of memory.

I need to understand more of the code to really initiate a PR.

ti test fails both on windows and linux at the very beginning.
../../taichi/tests/python/test_abs.py Fatal Python error: Aborted

Current thread 0x00007fe33e4e3740 (most recent call first):
File "/home/davidl/miniconda3/lib/python3.7/site-packages/taichi/lang/impl.py", line 128 in materialize
File "/home/davidl/miniconda3/lib/python3.7/site-packages/taichi/lang/expr.py", line 186 in __setitem__
File "/run/media/davidl/Data/dev/taichi/tests/python/test_abs.py", line 23 in test_abs
File "/home/davidl/miniconda3/lib/python3.7/site-packages/taichi/lang/__init__.py", line 149 in test
File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/python.py", line 167 in pytest_pyfunc_call
File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 87 in
File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/home/davidl/miniconda3/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/python.py", line 1445 in runtest
File "/home/davidl/miniconda3/lib/python3.7/site-packages/_pytest/runner.py", line 134 in pytest_runtest_call

umm, from the ti test_verbose result, it's the vprintf name conflict.
The most simple fix should be rename the vprintf in the runtime. I will check that tmr.

[yuanming-hu]

@davidleon Thanks for reporting!

• global_var is fixed in DiffTaichi.
• vprintf might be fixed with the same trick on Windows: https://github.com/taichi-dev/taichi/blob/1ca769de3f6b64182ed0911c04767e9838f24b0f/taichi/runtime/runtime.cpp#L6

[davidleon]

i don't quite understand why bother a template Printf to call an underlying vprintf.
on Linux, I just simply #define Printf printf to pass the compilation phase. Though ti test now fails at:

---

• Taichi Compiler Stack Traceback *

---

/run/media/davidl/Data/dev/taichi/build/taichi_core.so: taichi::signal_handler(int)
/usr/lib/libpthread.so.0(+0x14930) [0x7f3a24b9e930]
/usr/lib/libpthread.so.0: raise
/usr/lib/libpthread.so.0(+0x14930) [0x7f3a24b9e930]
/run/media/davidl/Data/dev/taichi/build/taichi_core.so: llvm::TypeFinder::incorporateType(llvm::Type)
/run/media/davidl/Data/dev/taichi/build/taichi_core.so: llvm::TypeFinder::run(llvm::Module const&, bool)
/run/media/davidl/Data/dev/taichi/build/taichi_core.so: llvm::IRMover::IRMover(llvm::Module&)
/run/media/davidl/Data/dev/taichi/build/taichi_core.so: llvm::Linker::linkModules(llvm::Module&, std::unique_ptr >, unsigned int, std::function const&)>)
/run/media/davidl/Data/dev/taichi/build/taichi_core.so: taichi::Tlang::TaichiLLVMContext::link_module_with_libdevice(std::unique_ptr >&)
/run/media/davidl/Data/dev/taichi/build/taichi_core.so: taichi::Tlang::TaichiLLVMContext::clone_runtime_module()
/run/media/davidl/Data/dev/taichi/build/taichi_core.so: taichi::Tlang::TaichiLLVMContext::get_init_module()
/run/media/davidl/Data/dev/taichi/build/taichi_core.so: taichi::Tlang::StructCompilerLLVM::StructCompilerLLVM(taichi::Tlang::Program, taichi::Tlang::Arch)
/run/media/davidl/Data/dev/taichi/build/taichi_core.so: taichi::Tlang::StructCompiler::make(bool, taichi::Tlang::Program*, taichi::Tlang::Arch)
/run/media/davidl/Data/dev/taichi/build/taichi_core.so: taichi::Tlang::Program::materialize_layout()
/run/media/davidl/Data/dev/taichi/build/taichi_core.so: taichi::Tlang::layout(std::function /run/media/davidl/Data/dev/taichi/build/taichi_core.so(+0x7af8cf) [0x7f3a0d2a98cf]
/run/media/davidl/Data/dev/taichi/build/taichi_core.so(+0x630b90) [0x7f3a0d12ab90]

[yuanming-hu]

On CUDA printf has a different calling convention, that's why we need the template.

[xumingkuan]

'ti test' this testing comand is not listed in windows section. I think this should be in a different section. And describe how testing should return results, and how to write test cases.

Updated the doc. On Windows, the ti command should be replaced by python -m taichi. If you know how to make ti work on Windows, that would be great!

On my computer, %TAICHI_REPO_DIR%\bin\RelWithDebInfo\ti.exe works. I can run ti test after adding %TAICHI_REPO_DIR%\bin\RelWithDebInfo into PATH.

[yuanming-hu]

On my computer, %TAICHI_REPO_DIR%\bin\RelWithDebInfo\ti.exe works. I can run ti test after adding %TAICHI_REPO_DIR%\bin\RelWithDebInfo into PATH.

Awesome! Didn't expect that to work - we haven't used Windows and ti.exe for ~2 years.
%TAICHI_REPO_DIR%\bin\RelWithDebInfo\ is slightly cumbersome. Would it be possible to generate ti.exe under %TAICHI_REPO_DIR%\bin? (without the RelWithDebInfo folder) You may need to change the CMakeFile for this purpose. Also please feel free to open a PR to update the doc for ti.exe.

[xumingkuan]

Actually when I was using CLion + clang + cmake 3.15.3, ti.exe was generated under %TAICHI_REPO_DIR%\bin. However, that didn't work.
I don't know why when I switched to Visual Studio 2019 + cmake-gui 3.17.0 (without changing CMakeFile?), ti.exe comes into the RelWithDebInfo folder.

[yuanming-hu]

Yeah, CMake may generate binaries at different folders depending on the target IDE...

[github-actions[bot]]

Warning: The issue has been out-of-update for 50 days, marking stale.