Taichi: [Infra] Setup Codecov for coverage report

Created on 9 Jun 2020 · 8Comments · Source: taichi-dev/taichi

Concisely describe the proposed feature
Currently we have too much features untested and behavior can easily get wrong. For example, we just released a bug in imwrite and ti.Vector, the latter cause cornell_box.py failed to run..
It would be great if we could set up the test coverage report so that:

We don't accidentally merge a PR without strong enough test come with it.
We can see where and which branch is untested and needs further refinement.
We can see legacy codes that are no longer used so that we can remove them safely.
We can finally systematically defeat regression!

Describe the solution you'd like (if any)
Codecov is a online coverage report tool, can be easily integrated with CIs.
I've just set up codecov for my toy repo taichi_glsl, it's quite easy indeed :)
But I guess it will become much harder when we trying to set up for Taichi, which is far more complicated than taichi_glsl.
See https://github.com/codecov, they have official mature solutions for Python projects. But only offer a not very mature solution for C++11, not sure how they deal with multi-language projects like Taichi.. Maybe we should try to setup for Python-side first?

Additional comments
Check out https://codecov.io/gh/taichi-dev/taichi_glsl/tree/master/taichi_glsl for Codecov UI preview.
Check out https://github.com/taichi-dev/taichi_glsl/blob/master/taichi_glsl/__main__.py for python -m coverage usage.

(AD: Install me using pip install taichi_glsl! Feed back welcomed!)

discussion feature request

Source

archibate

👍2

All 8 comments

Yeah, I agree we should first cover the Python end. As a scripting language Python will go wrong more easily than C++.

Btw, taichi_glsl is so cool! Great job!

yuanming-hu on 9 Jun 2020

I can't run tests now... Log:

C:\Users\xmk\Desktop\taichi>ti test
[Taichi] mode=development
[Taichi] <dev mode>, supported archs: [cpu only], commit b52db42f, python 3
.7.4

*******************************************
**      Taichi Programming Language      **
*******************************************

Running Python tests...

Starting 8 testing thread(s)...
D:\Anaconda3\lib\site-packages\pytest_doctestplus\plugin.py:10: Deprecation
Warning: the imp module is deprecated in favour of importlib; see the modul
e's documentation for alternative uses
  import imp
D:\Anaconda3\lib\site-packages\pyreadline\py3k_compat.py:8: DeprecationWarn
ing: Using or importing the ABCs from 'collections' instead of from 'collec
tions.abc' is deprecated, and in 3.8 it will stop working
  return isinstance(x, collections.Callable)
ERROR: usage: ti [options] [file_or_dir] [file_or_dir] [...]
ti: error: unrecognized arguments: -n 8
  inifile: None
  rootdir: C:\Users\xmk\Desktop\taichi

>>> Running time: 0.30s

C:\Users\xmk\Desktop\taichi>ti benchmark
[Taichi] mode=development
[Taichi] <dev mode>, supported archs: [cpu only], commit b52db42f, python 3
.7.4

*******************************************
**      Taichi Programming Language      **
*******************************************

Running Python tests...

Traceback (most recent call last):
  File "ti", line 4, in <module>
    exit(taichi.main())
  File "C:\Users\xmk\Desktop\taichi\python\taichi\main.py", line 883, in ma
in
    return cli()
  File "C:\Users\xmk\Desktop\taichi\python\taichi\main.py", line 27, in wra
pper
    result = func(*args, **kwargs)
  File "C:\Users\xmk\Desktop\taichi\python\taichi\main.py", line 95, in __c
all__
    return getattr(self, args.command)(sys.argv[2:])
  File "C:\Users\xmk\Desktop\taichi\python\taichi\main.py", line 743, in be
nchmark
    TaichiMain._test_python(args)
  File "C:\Users\xmk\Desktop\taichi\python\taichi\main.py", line 660, in _t
est_python
    if args.coverage:
AttributeError: 'Namespace' object has no attribute 'coverage'

C:\Users\xmk\Desktop\taichi>ti test -C
[Taichi] mode=development
[Taichi] <dev mode>, supported archs: [cpu only], commit b52db42f, python 3
.7.4

*******************************************
**      Taichi Programming Language      **
*******************************************

Running Python tests...

Starting 8 testing thread(s)...
D:\Anaconda3\lib\site-packages\pytest_doctestplus\plugin.py:10: Deprecation
Warning: the imp module is deprecated in favour of importlib; see the modul
e's documentation for alternative uses
  import imp
D:\Anaconda3\lib\site-packages\pyreadline\py3k_compat.py:8: DeprecationWarn
ing: Using or importing the ABCs from 'collections' instead of from 'collec
tions.abc' is deprecated, and in 3.8 it will stop working
  return isinstance(x, collections.Callable)
ERROR: usage: ti [options] [file_or_dir] [file_or_dir] [...]
ti: error: unrecognized arguments: --cov-branch --cov=python/taichi -n 8
  inifile: None
  rootdir: C:\Users\xmk\Desktop\taichi

>>> Running time: 0.33s

C:\Users\xmk\Desktop\taichi>ti benchmark -C
[Taichi] mode=development
[Taichi] <dev mode>, supported archs: [cpu only], commit b52db42f, python 3
.7.4

*******************************************
**      Taichi Programming Language      **
*******************************************
usage: ti benchmark [-h] [-T] [-v] [-r RERUN] [-t THREADS] [files [files ..
.]]
ti benchmark: error: unrecognized arguments: -C

C:\Users\xmk\Desktop\taichi>

xumingkuan on 9 Jun 2020

I can't run tests now... Log:

python3 -m pip install --user pytest pytest-rerunfailures pytest-xdist pytest-cov?

yuanming-hu on 10 Jun 2020

👀1 👍1

I can't run tests now... Log:

python3 -m pip install --user pytest pytest-rerunfailures pytest-xdist pytest-cov?

ti benchmark keeps giving the same error message.

ti test -t 1 works. ti test (ti test -t 8) and ti test -t 2 don't work:

C:\Users\xmk>ti test
[Taichi] mode=development
[Taichi] <dev mode>, supported archs: [cpu only], commit b52db42f, python 3.7.4

*******************************************
**      Taichi Programming Language      **
*******************************************

Running Python tests...

Starting 8 testing thread(s)...
D:\Anaconda3\lib\site-packages\win32\lib\pywintypes.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp, sys, os
D:\Anaconda3\lib\site-packages\pyreadline\py3k_compat.py:8: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  return isinstance(x, collections.Callable)
================================================= test session starts =================================================
platform win32 -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: C:\Users\xmk\Desktop\taichi
plugins: cov-2.9.0, forked-1.1.3, rerunfailures-9.0, xdist-1.32.0, arraydiff-0.3, doctestplus-0.3.0, openfiles-0.3.2, remotedata-0.3.2
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 Iusage:
    baseline            |-> Archive current benchmark result as baseline
    benchmark           |-> Run Python tests in benchmark mode
    build               |-> Build C++ files
    changelog           |-> Display changelog of current version
    debug               |-> Debug a single script
    doc                 |-> Build documentation
    example             |-> Run an example by name (or name.py)
    format              |-> Reformat modified source files
    format_all          |-> Reformat all source files
    gif                 |-> Convert mp4 file to gif in the same directory
    regression          |-> Display benchmark regression test result
    release             |-> Make source code release
    run                 |-> Run a specific task
    test                |-> Run the tests
    update              |-> Update the Taichi codebase
    video               |-> Make a video using *.png files in the current directory
    video_crop          |-> Crop video in the same directory
    video_scale         |-> Scale video resolution in the same directory
    video_speed         |-> Speed up video in the same directory
ti.exe: error: the following arguments are required: command
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "D:\Anaconda3\lib\site-packages\_pytest\main.py", line 211, in wrap_session
INTERNALERROR>     config.hook.pytest_sessionstart(session=session)
INTERNALERROR>   File "D:\Anaconda3\lib\site-packages\pluggy\hooks.py", line 289, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "D:\Anaconda3\lib\site-packages\pluggy\manager.py", line 87, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "D:\Anaconda3\lib\site-packages\pluggy\manager.py", line 81, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "D:\Anaconda3\lib\site-packages\pluggy\callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "D:\Anaconda3\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "D:\Anaconda3\lib\site-packages\pluggy\callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "C:\Users\xmk\AppData\Roaming\Python\Python37\site-packages\xdist\dsession.py", line 78, in pytest_sessionstart
INTERNALERROR>     nodes = self.nodemanager.setup_nodes(putevent=self.queue.put)
INTERNALERROR>   File "C:\Users\xmk\AppData\Roaming\Python\Python37\site-packages\xdist\workermanage.py", line 68, in setup_nodes
INTERNALERROR>     nodes.append(self.setup_node(spec, putevent))
INTERNALERROR>   File "C:\Users\xmk\AppData\Roaming\Python\Python37\site-packages\xdist\workermanage.py", line 72, in setup_node
INTERNALERROR>     gw = self.group.makegateway(spec)
INTERNALERROR>   File "C:\Users\xmk\AppData\Roaming\Python\Python37\site-packages\execnet\multi.py", line 134, in makegateway
INTERNALERROR>     gw = gateway_bootstrap.bootstrap(io, spec)
INTERNALERROR>   File "C:\Users\xmk\AppData\Roaming\Python\Python37\site-packages\execnet\gateway_bootstrap.py", line 100, in bootstrap
INTERNALERROR>     bootstrap_import(io, spec)
INTERNALERROR>   File "C:\Users\xmk\AppData\Roaming\Python\Python37\site-packages\execnet\gateway_bootstrap.py", line 35, in bootstrap_import
INTERNALERROR>     assert s == "1".encode("ascii"), repr(s)
INTERNALERROR> AssertionError: b'['
>>> Running time: 1.30s
D:\Anaconda3\lib\subprocess.py:858: ResourceWarning: subprocess 19948 is still running
  ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
sys:1: ResourceWarning: unclosed file <_io.FileIO name=15 mode='wb' closefd=True>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
sys:1: ResourceWarning: unclosed file <_io.FileIO name=16 mode='rb' closefd=True>
ResourceWarning: Enable tracemalloc to get the object allocation traceback

C:\Users\xmk>

xumingkuan on 10 Jun 2020

If I print pytest_args before this line and run ti test,
https://github.com/taichi-dev/taichi/blob/f7596204c3c291fed1afb94701d2a49bd81e94dc/python/taichi/main.py#L679
I get ['C:\\Users\\xmk\\Desktop\\taichi\\tests\\python', '-n', '8'].

Interestingly, different from ti test, pytest -n 8 works:

(base) C:\Users\xmk\Desktop\taichi\tests\python>pytest C:\\Users\\xmk\\Desktop\\taichi\\tests\\python -n 8
================================================= test session starts =================================================
platform win32 -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: C:\Users\xmk\Desktop\taichi
plugins: cov-2.9.0, forked-1.1.3, rerunfailures-9.0, xdist-1.32.0, arraydiff-0.3, doctestplus-0.3.0, openfiles-0.3.2, remotedata-0.3.2
gw0 [421] / gw1 [421] / gw2 [421] / gw3 [421] / gw4 [421] / gw5 [421] / gw6 [421] / gw7 [421]
................................................................................................................ [ 26%]
................................................................................................................ [ 53%]
................................................................................................................ [ 79%]
.....................................................................................                            [100%]
================================================== warnings summary ===================================================
C:\Users\xmk\Desktop\taichi\python\taichi\lang\kernel.py:337
C:\Users\xmk\Desktop\taichi\python\taichi\lang\kernel.py:337
C:\Users\xmk\Desktop\taichi\python\taichi\lang\kernel.py:337
C:\Users\xmk\Desktop\taichi\python\taichi\lang\kernel.py:337
  C:\Users\xmk\Desktop\taichi\python\taichi\lang\kernel.py:337: DeprecationWarning: ti.Matrix(rows=[...]) or ti.Matrix(cols=[...]) is deprecated, use ti.Matrix.rows([...]) or ti.Matrix.cols([...]) instead.
    compiled()

-- Docs: https://docs.pytest.org/en/latest/warnings.html
====================================== 421 passed, 4 warnings in 102.24 seconds =======================================

xumingkuan on 10 Jun 2020

And python -m taichi test works.

xumingkuan on 10 Jun 2020

👀1

python bin/ti test and python python/taichi/main.py test all work. Why only ti.exe fails?

xumingkuan on 10 Jun 2020

AppVeyor didn't reproduce. May related to your local env? Could you add print(args, dir(args)) to obtain more info?

ti benchmark

That's expected, since -C is only for ti test.

archibate on 10 Jun 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

[CUDA] [Perf] The performance of cornell_box.py on CUDA is still poor than OpenGL

archibate · 3Comments

module 'taichi' has no attribute 'gpu'

GeoffreyPlitt · 4Comments

No taichi package on linux

kazimuth · 4Comments

runtime [verify.cpp:basic_verify@39] stmt 5249 cannot have operand 4663

zdxpan · 3Comments

[Bug]"ti.sin()" doesn't work with a local variable.

Xayahp · 3Comments