Codeception: Fail on low coverage

[Kolyunya]

What are you trying to achieve?

Restrict minimal code-coverage via CI specifying low_limit in config. I expect to get exit code distinct from 0 when coverage is lover than low_limit.

What do you get instead?

codecept run unit --coverage doesn't fail (process return code is 0) when actual coverage is lower than low_limit.

Details

• Codeception version: 2.2.9
• PHP Version: 7.1.3
• Operating System: GNU/Linux
• Installation type: Composer
• Suite configuration:

actor: Tester
paths:
    tests: tests
    log: tests/_output
    data: tests/_data
    support: tests/_support
    envs: tests/_envs
settings:
    bootstrap: _bootstrap.php
    colors: true
    memory_limit: 1024M
extensions:
    enabled:
        - Codeception\Extension\RunFailed
coverage:
    enabled: true
    low_limit: 80
    high_limit: 90
    include:
        - src/*

[edno]

low_limit and high_limit are only for coverage report purpose (see Code Coverage). They do not affect the result of the execution.

[Kolyunya]

Yeah, I see. But it's a common scenario - to fail CI tests when coverage is lower than n%. Maybe it's worth implementing?

[edno]

From my point of view, Codeception is a test execution framework, and the behaviour you've described is usually the role of a code analysis tools.
In my projects using Travis, I am using Codeception for running tests, and Coverall for analysing the coverage. Coverall tags any PR as failed if the coverage is too low.

[Kolyunya]

Codeception already has a coverage support so it's a code analysis tool already. I can't see any reasons not implementing this.

P.S. Coveralls is a no go since it's a non-free software.

[dave-newson]

I was just looking for this feature, as I'd really like my CI pipeline to kick my ass if I (or my colleagues) let coverage drop past a given threshold.

Would it be possible as a simple new parameter, like minimum or fail_limit ?
If the value isn't set, then there will be no failure, so adding this parameter to the config is backwards compatible.

[Nebulosar]

+1 on this issue.
Why would it possible to set a low and high limit. It only marks hihg and low in the reports, so it already kind of does some analytics.

In my opinion, you should be able to say: low_limit_fail: true or something like that.
It would really be a big advantage and since codeception already sets the mark, it's a simple addon

[Nebulosar]

Ok, since this issue has been created two years ago, it is save to say that this has become a bit of a stale issue.
I've made an extension for codeception. I still need to create a proper repo for it, but for now I've made a gist. It can be found here.

@Kolyunya maybe you can use this to :smile:

[Kolyunya]

@Nebulosar thanks for your work. Looking forward for a repository.

[Nebulosar]

@Kolyunya I just finished my first draft of a repo. I'm kinda new to php packages so I hope it works like intented :)

@dave-newson maybe this will keep your colleagues in line 💃

Repo: https://github.com/Nebulosar/CodeceptionCoverageChecker