Phpunit: [Proposal] Configuring phpunit via env vars

[rishavsharan]

Hi guys

I work at MS Azure Devops which is a CI/CD system with a major focus on testing in pipelines. To give our users a rich reporting experience without forcing them to make any platform specific config, we want to be able to make changes to the reporting configs on the fly, in our pool machines.

This is something pytest allows via "PYTEST_ADDOPTS".
Essentially, something like PHPUNIT_OPTS = --log-junit which would supercede the saved config.

Currently, phpunit doesn't allows us to to do this. Adding this option will enable all the CI/CD platforms, and not just Azure DevOps, in providing a better test experience to phpunit users so that they will not have to do any platform specific configuration in their project code just to leverage the test reporting capabilities of the platform.
This will allow us to do very platform specific things like run custom config that we need to generate test metadata and run custom reporters.

Let me know what do you guys think. We would love to contribute to the community and submit PRs for this.

[keradus]

:-1: for magical, hidden env var.

you can always do ./phpunit $MY_FLAGS

[sebastianbergmann]

Do I understand this proposal correctly that when the environment variable PHPUNIT_OPTS is set that PHPUnit should not parse argv for commandline options but the contents of said environment variable?

[theseer]

I fail to see how having an env var with a platform specific value is the opposite of a platform specific build option?

Setting it in some UI rather than having it version controlled within the repository feels wrong as well.

[theseer]

Do I understand this proposal correctly that when the environment variable PHPUNIT_OPTS is set that PHPUnit should not parse argv for commandline options but the contents of said environment variable?

I understand it, that the idea would be that it amends the CLI and config options, potentially being applied last and thus overwriting values.

While I can imagine a use case where the CI/CD environment magically abstracts some settings away from the user to make things like coverage handling work, all the side effects I can already come up with clearly out way any benefits.

Imagine the phpunit.xml having a setting for junit logs, then you specifiy something else as CLI parameter and have an env variable setting for it as well. Which source should win? While it makes a lot of sense to me that a CLI passed option overwrites the config xml, how does the env var play along here?

If the env var doesn't overwrite the CLI passed option the whole idea falls apart because a user can break it rendering the abstraction useless. If it does, there is no obvious way to understand why a seemingly valid cli option gets ignored.
I'd consider that rather unexpected.

The only clean way would be what @keradus already said: ./phpunit --my-opt $others. So again, no env var handling within PHPUnit needed.

Am I missing anything?

[rishavsharan]

EDIT:
I missed your last comment@theseer

You are right in your interpretation of the problem.
The env vars overwriting the cli and the file based configs seem to be the most practical option for our usecase. Something like this would make testing in CI/CD platforms much easier and at the same time,
I think it wouldnt be much of a problem for user testing on their own systems. Most users wouldn't manually use these env vars at all and in the rare scenario that they do end up using them, it would likely be in the context of a shared pipeline like Azure DevOps, jenkins, Gitlab etc.

I think one issue that lies with using shell variables is that we need to actively know what is the testing framework being invoked and that intelligence is hard to bring in without parsing through the entire project. env vars are a passive way of configuring the system and the config will be automatically be picked up if the test runner is phpunit.

---

@theseer We do not want to add platform specific values in the environment variable but rather normal phpunit config items like --verbose and --log-junit, the same ones which can be added via command line.
We would be happy with append only approach as well, but then we may have an issue where the user has defined a config in their project as --testdox-html <file> while we have appended --log-junit <file>.
Is this a valid command? phpunit --testdox-html td.html --log-junit ju.xml <unit test>

As a CI/CD provider, a challenge that we are facing is that we want to provide our users with rich testing reporting and insights. We, along with most other platforms prefer a structured and universally accepted test result format like junit-esque XML.
These configs are something we dont expect the user to set in a UI. Its more from the perspective of our pool machines where we want all tests to provide an expected set of metadata that we can show as test reports.

[theseer]

@theseer We do not want to add platform specific values in the environment variable but rather normal phpunit config items like --verbose and --log-junit, the same ones which can be added via command line.

What I meant with "platform specific" is that the values that these options get set to are platform specific. And I fail(ed) to see how that would be any different from passing them via CLI.

We would be happy with append only approach as well, but then we may have an issue where the user has defined a config in their project as --testdox-html <file> while we have appended --log-junit <file>.
Is this a valid command? phpunit --testdox-html td.html --log-junit ju.xml <unit test>

Sure, why wouldn't it? As far as I can tell, you are only going to run into issues if the user already has a setting for --log-junit. Having a magic env var though will not fix that? It will rather make the build fail, assuming the user didn't specify the cli/config value just because it's possible but because some other tool in the build requires it.

We, along with most other platforms prefer a structured and universally accepted test result format like junit-esque XML.

I totally see that. Given there is no real standard junit format, PHPUnit even features its own xml format with (PHPUnit specific) additional details.

These configs are something we dont expect the user to set in a UI. Its more from the perspective of our pool machines where we want all tests to provide an expected set of metadata that we can show as test reports.

I believe I understand your line of thought. The solution described though feels like a rather fragile construct with many drawbacks.

[epdenouden]

@rishavsharan If you insist on using an environment variable, please do it the way @keradus has suggested or use that environment to instruct your orchestration tool like Ansible.

Here at Mollie we run it like this:

tests: unittests integrationtests

unittests: vendor/autoload.php
    $(PHP_BIN) vendor/bin/paratest $(PARATEST_CONFIG) --no-test-tokens $(PHPUNIT_CONFIG) tests/unittests

integrationtests: migrations-development vendor/autoload.php
    $(PHP_BIN) vendor/bin/phpunit $(PHPUNIT_CONFIG) tests/integration

Works like a charm.