Suitecrm: Automated Testing improvements

[connorshea]

I didn't want to open a bunch of issues at once, so I figured I could just write out my suggestions as one larger issue.

I've been working on migrating the test suite to GitLab CI and I've noticed some problems while doing it. I've also had significant trouble setting up the test environment locally. I intend to fix some of these myself, but I wanted to discuss them to see if I'm missing anything important that these changes would break.

• [x] In ./vendor/bin/robo driver:run-chrome, the ChromeDriver version that's installed should always be the latest version to make sure it remains compatible with newer versions of Chrome. (#7345)
• [x] Remove most I->wait(x) from the acceptance test suite. Use waitForElementVisible instead. This makes the test suite take longer than necessary. (#7442)
• [x] Remove all specified timeouts from waitForElementVisible and similar. If a test fails, this unnecessarily adds 30-120 seconds to the test run. In special cases we can add a longer timeout, but for the most part we should always use the default 10 seconds timeout. (#7414)
• [x] Remove all the extra test environments (Browser Stack, probably also Selenium?) (#7347)
• [x] Remove the functional test suite (it's unused) (#7346)
• [x] Remove the demo test suite (no one uses it, I think?) (#7352)
• [ ] Add support for a config.test.php file, which would be used to define configuration values for the test environment specifically (e.g. so you can run the tests on a separate database from the development site). Alternatively, maybe just config_override.test.php? (#7621)
• [ ] Make the code:coverage Robo command work outside of CI. (#7622)
• [x] Add a config.codeception.yml.dist file and gitignore the file config.codeception.yml in the tests/_env directory. These can be used to set up a test environment locally without needing to mess with any files that'd be committed to the repo. (#7433)
• [ ] Adopt .env files for use with testing, either phpdotenv or Symfony Dotenv. This is mostly useful for running multiple CRMs side-by-side on the same computer, and also because it makes it easier to figure out what the local environment variables are. (Issue: #7620)
• [x] Add Robo tasks for running the test suite instead of needing to use the full command paths. (#7374)
• [x] Update Automated Testing documentation for these changes, and to provide proper setup instructions, etc. (https://github.com/salesagility/SuiteDocs/pull/287, https://github.com/salesagility/SuiteDocs/pull/288, https://github.com/salesagility/SuiteDocs/pull/289, https://github.com/salesagility/SuiteDocs/pull/290, https://github.com/salesagility/SuiteDocs/pull/300, https://github.com/salesagility/SuiteDocs/pull/302)
• [x] Fix flaky tests (#7417, #7565)
• [x] Acceptance tests need to clean up after themselves. (Issue: #7419, PR: #7536)
• [ ] Tests should be run in random order (Issue: #7022)
• [x] Misc. cleanup of test files (#7449, #7477, #7491, #7535, #7607)

[connorshea]

I've been working on improving the automated test suite in various ways and I was hoping I could get some information about whether people run the test suite and how they do it. I'd do this on the forums, but I expect I'd get more responses from people who'd actually run – or want to run – the test suite via GitHub.

My goal is to use this data to inform the improvements I'm making and to improve the documentation to make sure it focuses on the most common ways people run the test suite (or would want to run it).

If you could post a comment based on the following template, I'd appreciate it greatly. This is open to anyone who has contributed to the project! Please answer even if you don't think you're 'important' enough to :)

### What OS do you develop on primarily?
<!-- Windows, macOS, or Linux -->

### Do you run the test suite locally when making changes to the CRM?
<!-- Yes or No -->

### If you run the test suite locally, how do you do it?
<!-- Run it with just a local copy of PHP, with Docker, with a VM, or something else -->
<!-- Also, please mention whether you use Selenium Hub or a Selenium Docker container for the acceptance tests. -->

### If you're not running the test suite locally, why? And would you want to, if you could/if it were made easier to set up?
<!-- Explanation here -->

### Any other feedback here
<!-- Thanks! -->

[pgorod]

Ok, let me try answering that...

What OS do you develop on primarily?

Windows and Linux (Ubuntu)

Do you run the test suite locally when making changes to the CRM?

No...

If you run the test suite locally, how do you do it?

...

If you're not running the test suite locally, why? And would you want to, if you could/if it were made easier to set up?

I don't do extensive developments, just occasional bug fixing. It's also easy to run the tests from Github, so I save myself the hassle of installing the tests locally. But I do think they would be nice to have locally if I developed more often.

Any other feedback here

I really value the role that automated tests have for our project, and I would love to see this evolve positively, and getting rid of the downsides of Travis. So a big thank you to you and to all others contributing to improve our automated tests system!

[isleshocky77]

What OS do you develop on primarily?

Linux

Do you run the test suite locally when making changes to the CRM?

Have attempted to, got some going, generally only try while doing a bug or feature branch

If you run the test suite locally, how do you do it?

Docker

If you're not running the test suite locally, why? And would you want to, if you could/if it were made easier to set up?

Need to make sure they all run through the current setup.

Any other feedback here

docker-compose.yml file which just works out of the box would be delightful.

Something like

cd /tmp
wget https://github.com/suitecrm/docker-tests/docker-compose.yml
docker-compose run tests

[Abuelodelanada]

What OS do you develop on primarily?

GNU+Linux (Ubuntu)

Do you run the test suite locally when making changes to the CRM?

Yes.
Have attempted to, got some going... The documentation was not so clear when I tried.

If you run the test suite locally, how do you do it?

I only ran locally, unit tests

If you're not running the test suite locally, why? And would you want to, if you could/if it were made easier to set up?

When we tried, the docs were not so clear... but we have no good reason to not run tests on every commit.... We are trying to setup our internal gitlab to run CI.

Any other feedback here

[pribeiro42]

What OS do you develop on primarily?

Linux (Debian)

Do you run the test suite locally when making changes to the CRM?

No

If you run the test suite locally, how do you do it?

If you're not running the test suite locally, why? And would you want to, if you could/if it were made easier to set up?

I usually don't do extensive development, although, if the test suite were easier to set up, i would run it...

Any other feedback here

[gunnicom]

What OS do you develop on primarily?

Linux

Do you run the test suite locally when making changes to the CRM?

No

If you run the test suite locally, how do you do it?

N/A

If you're not running the test suite locally, why? And would you want to, if you could/if it were made easier to set up?

Most changes i do are not that big, so i trust in me :D

Any other feedback here

[lazka]

What OS do you develop on primarily?

Debian/Ubuntu

Do you run the test suite locally when making changes to the CRM?

Yes.

If you run the test suite locally, how do you do it?

Locally, only the phpunit tests.

If you're not running the test suite locally, why? And would you want to, if you could/if it were made easier to set up?

I would run the other tests if it would be easy to set up and would allow me to run a relevant subset of them.

[connorshea]

I guess I should answer as well.

What OS do you develop on primarily?

macOS

Do you run the test suite locally when making changes to the CRM?

I've tried, but I haven't really been able to get it to run consistently.

If you run the test suite locally, how do you do it?

I run it with a local copy of PHP. Docker for Mac has been too slow when I've attempted to use it (this almost definitely isn't as much of an issue with Linux).

If you're not running the test suite locally, why? And would you want to, if you could/if it were made easier to set up?

I pretty much covered everything I want from the test suite in this issue's description, but essentially:

• Well-documented setup instructions
• Being able to repeatedly run the test suite locally without needing to perform manual cleanup or having the test suite behave differently from the previous run (except for test order, that can differ).
• Consistent commands
• Easy configuration for my local environment, especially with a .env file.
• Tests should be run with a separate database from the one used for development, so the tests don't impact the data I have for development. I can kind of do this already by manually swapping the config files, but this should be automated.

Any other feedback here

IMO, if you don't ever run the test suite outside of CI, you're significantly limiting the usefulness of having a test suite. I understand that people don't want to run the test suite locally since it can take 20+ minutes, but I mean even just running one test file so you can rapidly validate that the test works. This is especially important when a test breaks, because waiting 20+ minutes for CI to do it before you know whether your fix worked is a really long feedback cycle. (This is another benefit of randomization, since it allows you to verify that tests don't depend on each other and can be run independently).

[connorshea]

@lazka @pgorod @isleshocky77 @Abuelodelanada @pribeiro42 @gunnicom thanks!

So based on the replies:

• Most contributors don't run the tests locally, or if they do they only run the unit tests.
• No one uses Selenium for running the acceptance tests (mostly because no one runs the acceptance tests)
• Contributors pretty much all use Linux (except me, on macOS :P), and pgorod sometimes with Windows. My coworkers use Windows so I'll probably end up supporting it regardless, but I think that's interesting.

@Dillon-Brown @gymad @cameronblaikie could you answer the survey Qs just so I can make sure I'm getting as many responses from SalesAgility devs as possible? :)

[isleshocky77]

@connorshea Yeah, one big want / need from my perspective is to make sure the tests are isolated and atomic and documented to show how to run just a single test pertaining to what you're trying to show is broken and or working on.

[Dillon-Brown]

What OS do you develop on primarily?

Ubuntu 18.04.

Do you run the test suite locally when making changes to the CRM?

Yes, both the unit tests and acceptance tests through chrome-driver.

If you run the test suite locally, how do you do it?

No local copy of PHP, I run everything through docker containers.

If you're not running the test suite locally, why? And would you want to, if you could/if it were made easier to set up?

I don't usually run through the entire test suite before putting up PRs but I typically run through any I have created for the change or any pre-existing ones that I think might be likely to fail. Some things that would make the tests easier:

• Better documentation.
• Stateless tests that clean up after themselves.

Any other feedback here

Would very much like to see as many people running the tests locally (SalesAgility devs too). With Travis starting to take 20+ minutes it's just not feasible to solely rely upon pushing changes up and waiting around for Travis to pass/fail.

[connorshea]

Here are the slowest acceptance tests as of hotfix-7.10.x (only includes the the ones above 10 seconds):

| Test | Time (seconds) |
| ------ | ------|
| jjwg_MapsCest: Create a Map: | 30.14029598236084 |
| HistoryCest: See the due date field on Account History subpanel: | 22.763580083847046 |
| ActivitiesCest: See the due date field on Account Activities subpanel: | 21.289443969726562 |
| BasicModuleCest: Create a basic module for testing: | 21.02902102470398 |
| IssueModuleCest: Create a issue module for testing: | 19.52377200126648 |
| FileModuleCest: Create a file module for testing: | 19.29529094696045 |
| SaleModuleCest: Create a sale module for testing: | 19.074450969696045 |
| PersonModuleCest: Create a person module for testing: | 18.96099591255188 |
| UsersCest: View the collapsed subpanel hints on Accounts: | 18.88200807571411 |
| CompanyModuleCest: Create a company module for testing: | 18.772393941879272 |
| ModuleBuilderFieldsCest: Create a module for testing fields: | 18.65506100654602 |
| OpportunitiesCest: Create an opportunity: | 17.190773963928223 |
| ModuleBuilderFieldsCest: Deploy Test Module: | 16.77246403694153 |
| ContractsCest: Create a Contract: | 14.65644097328186 |
| CasesCest: Create a Case: | 14.458054065704346 |
| CampaignsCest: Create Non-Email Campaign: | 12.749990940093994 |
| AccountsCest: Create an Account: | 12.334502220153809 |
| EventsCest: Create an Event: | 11.36881709098816 |
| ModuleBuilderFieldsCest: Add relate field: | 11.041476964950562 |

Based on the data in the JSON report from here: https://travis-ci.org/salesagility/SuiteCRM/jobs/550989935

[connorshea]

@lazka @pgorod @isleshocky77 @Abuelodelanada @pribeiro42 @gunnicom @Dillon-Brown @cameronblaikie @Mac-Rae @samus-aran thanks for the help! ❤️

I'm really happy with the progress we've made on the automated test suite :) It should be much easier to set up locally and I encourage everyone to do so whenever you make changes. It should also be far less flaky, much simpler to work with, and a bit faster.

You can follow the steps in the automated testing documentation to run the test suite locally now: https://docs.suitecrm.com/developer/automatedtesting/

If you run into any problems with that documentation, feel free to comment here or in the SuiteCRM Gitter with an explanation of the problem, and I'll try to improve the docs. 👍

Keep in mind that right now some of the acceptance tests can only be run once since they don't clean up after themselves, though I have a PR open to fix this that should hopefully be merged sometime this week. You should also note that these improvements mostly apply to hotfix-7.10.x, other branches should receive them in the next few weeks when hotfix-7.10.x is merged into hotfix and then develop.

I've split the remaining TODOs into separate issues:

• Add support for a config_override.test.php file (issue: #7621)
• Make the code:coverage Robo command work outside of CI. (Issue: #7622)
• Introduce dotenv files (Issue: #7620)
• Acceptance tests need to clean up after themselves. (Issue: #7419, PR: #7536)
• Tests should be run in random order (Issue: #7022, PR: #7609)

If anyone has any other suggestions for improving the test suite, I'd be happy to hear them.