Cidram: Question: Tests: 0/1

[eurobank]

Hello.

Today i see in the updated menu

"Tests: 0/1"

What does it mean ?

[DanielRuf]

Hi @eurobank,

Please provide some more info (the CIDRAM version and your settings).

[eurobank]

I use

CIDRAM version used: 1.13.0 as for my settings, not sure what you mean but i didn't touched anything recently.

[Maikuolan]

When "Tests" appears next to a component (CIDRAM itself, in this case), it refers to any publicly available test results for any testing done against the latest available commit for the component in question.

Clicking on the [+] should expand the information, including any links to said publicly available test results.

Ideally, whenever tests are available, we want them to pass. In this particular case, the only publicly available test results are from continuous-integration/travis-ci/push (i.e., Travis CI), and unfortunately, they indicate failure.

(Travis CI is a "continuous integration" service that CIDRAM, as well as a significant portion of PHP ecosystem, uses to run tests against PHP packages, to help detect problems, bugs, etc).

If we follow the link, we can see more detailed information provided. It shows that Travis CI wasn't able to run the tests properly for PHP 5.4 and PHP 5.5, and that the actual failures occured when testing against PHP 7.0 and PHP 7.1. The tests for PHP 5.6, PHP 7.2 and PHP 7.3 passed successfully. TBH, I'm not too sure what's going on there. The fact that the PHP 5.6 tests passed successfully, I don't think the PHP 7.0 and PHP 7.1 failures are due to any accidentally introduced compatibility breaks (because if it was due to any accidentally introduced compatibility breaks, PHP 5.6 should fail as well).

The reasons that Travis CI gives for why the PHP 5.4 and PHP 5.5 weren't able to run, suggests that those PHP versions aren't supported by Travis CI.. but we've been testing against those versions since we began using Travis CI, they've clearly been supported in the past, Travis CI hasn't announced dropping support for them anywhere AFAICT, and the repository where Travis CI gets its images from includes everything from PHP 5.2 onward, so.. my current thinking, is that there's probably just been some minor, temporary hiccup with Travis CI's systems, maybe too much server load, or maybe someone from their team was experimenting with some changes that accidentally introduced a bug, or something like that. I'm thinking that we should wait until the next time that tests are ran again, before complaining to Travis CI about dropping support for PHP versions or anything like that (if it's just a temporary thing, like I suspect, then the problem will likely disappear the next time that tests are ran again). Same thing could apply for the failed tests, maybe. Anyway.. at the moment, I've got no idea exactly why the failures occurred, beyond speculation.

But, to answer the question, as to why this information is included on the CIDRAM front-end updates page: It provides a mechanism to help users determine whether the latest available version of a tested component is trustworthy (trustworthy, in terms of how likely it is to introduce potential problems, bugs, breakage, etc). Obviously, we don't intend to commit anything that isn't trustworthy, and we always endeavour that our a commits are bug-free, but nobody is perfect, and test information (when available) can help identify when we slip up somewhere (and is therefore why we ideally want tests to always pass). Of course, this isn't perfect: Sometimes, a bug or a problem could be introduced that testing isn't able to catch, so, all tests passing, is not any guarantee that something is entirely trustworthy or bug-free (though, when tests are more comprehensive, and written to a higher standard, they should in theory be able to catch a little more). A passed test, just means that none of the potential problems that the tests specifically looks for were detected. It's also possible that tests can fail due to things other than bugs or code problems; things like server errors, timeouts during testing, etc.

So, in short: A pass/fail, doesn't provide a definitive conclusion, as to whether the tested component is trustworthy. But, what should happen, if someone sees a failed test: Before trusting the component which has the failed test, ask more questions. There could be number of different reasons why a test fails, but there must always be some kind of reason, whatever it is.

FYI, IMO, the latest commit introduces only very small changes, nothing too complex, nothing particularly difficult to write, and should be trustworthy (tested locally on my machine prior to committing, and I didn't encounter any problems). ;-)

But regardless, failed tests are still a problem, and seeing a failed test, should always be a reason to ask questions, before trusting whatever the failed test belongs to. So, I'll keep a close eye on this, the next time that I commit something.

[eurobank]

Thanks for the details, so nothing to worry about.

[DanielRuf]

@Maikuolan the 7.x failures are due to phpunit which is not locked (we can use something like 4 || 5 || 6 in composer.json) and the 5.x failed due to issues with the PHP version archives from Travis CI.

[DanielRuf]

https://travis-ci.org/CIDRAM/CIDRAM/jobs/545668339

[DanielRuf]

https://travis-ci.org/CIDRAM/CIDRAM/builds/547198355

Xenial does not support PHP 5 which is now the default distribution for builds, see https://travis-ci.community/t/php-5-4-and-5-5-archives-missing/3723/2 and https://changelog.travis-ci.com/xenial-as-the-default-build-environment-99476

[DanielRuf]

CI status for the v1 branch is green now.

PS: TLDR @Maikuolan, see https://storage.googleapis.com/travis-ci-language-archives/php/binaries/ubuntu/16.04/x86_64/php-5.4.tar.bz2 vs https://storage.googleapis.com/travis-ci-language-archives/php/binaries/ubuntu/14.04/x86_64/php-5.4.tar.bz2

(Ubuntu 14 = Trusty, Ubuntu 16 = Xenial)

[Maikuolan]

@Maikuolan the 7.x failures are due to phpunit which is not locked (we can use something like 4 || 5 || 6 in composer.json) and the 5.x failed due to issues with the PHP version archives from Travis CI.

Sounds good. :-)