Suitecrm: Add unit and functional tests

[gagarine]

This is a big project, this absolutely need functional and unit testing. Adding only 20 functional tests will really avoid stupid mistake and speed up pull request reviewing.

Functional testing I can think of:

• a full install path, check it work without any error
• Checking all the main page display some data without errors.
• adding leads
• adding contact
• moving leads to contact
• ...

[JoshuaQ]

Totally agree, I made a proposal, but nobody answered me. https://github.com/salesagility/SuiteCRM/issues/641
I think that the priority should be the robustness and stability.

Are you agree?

[Mausino]

:+1: great suggestion

[gagarine]

http://casperjs.org/ is pretty good for function testing. Or a PHP crawler (goute) and PHPunit....

[ghost]

Hi there,SalesAgility are well into the process of writing unit tests for SuiteCRM. I believe that these will be used in a CI approach to allow for automated testing on commits. @JimMackin can you comment about a time scale for this?

[JimMackin]

Can't give a concrete timescale but plans are underfoot (within the next few months I'd imagine). We're also looking into TravisCI to provide automated running of the Unit Tests.

[chris001]

:+1: for Travis-CI for Suite
Links to get this party started....
https://docs.travis-ci.com/user/for-beginners
https://docs.travis-ci.com/user/getting-started/
https://docs.travis-ci.com/user/customizing-the-build/
https://docs.travis-ci.com/user/languages/php

https://github.com/mbonaci/mbo-storm/wiki/Integrate-Travis-CI-with-your-GitHub-repo

[gagarine]

@JimMackin can you please share what are your plans so we can have this discussion in an open way? I'm mainly interested by how do you think test should be organized. Should they be at the root of the project for the core and a test folder in every modules?

And witch testing tools? phpunit? behat?

I really don't care about CI, because it's the step 2 and it's an fairly easy step. First you need tests...

EDIT: I will love to write a simple functional test than load every pages a fail if we have a single PHP warning so we can tackle #942

[ricardovfreixo]

There will be a lot of changes in the next couple of weeks. In every way. Bot technical (code changes, unit tests, API, new features, improving UI, etc) and regarding the community. We're working to bring all the technical talk here to Github and leave more business and user help to the forum. Also plans are being made to make the app more community driven with cleaning push requests/bug reports backlog and creating a more interactive process for everyone contribute. This is going to be a smashing year!!!

[gagarine]

Ok! Thanks. I totaly understand that if it's a broader shift you want to temporise things.

[ricardovfreixo]

We do realize we were so focused on new devs that kinda neglected you guys. We'll make sure we get back on track with everyone. After all, this is everyone's software. We're making all the changes necessary to make sure that every collaboration is evaluated and promoted in timely manner.

[gunnicom]

Hope you are right, but lost trust in you guys.

[adamjakab]

+1000 for unit and functional testing - I think this is no more an option for a project of this size but a must.
Another +1000 for travis CI and I'd suggest to look into scrutinizer CI as well (optional but gives a lot of insight to code quality problems)

In the past few weeks I opened 2 PRs and a few issues for problems which could have been avoided to make it to final release merely by doing automated tests on the project. Moreover, some of these problems(like the properly working dynamic dropdown in past versions where files are now completely missing in official 7.5.1 package downloadable from suitecrm site) are just 'stupid' mistakes due to the impossibility to be able to track changes in source tree by humans.
This is exactly why CI has been created so that no PRs can make it to be pulled into project tree unless they pass all tests.

This is an enormous task for this project because in my experience this means to write nearly double of the code that is present at the moment in the codebase. Yet, it is to be started ASAP.

I'd also seriously consider adding composer to the project. If we want to do automated testing we need to have automated command line tool installation. I have a test repository where I have already done this with Symfony's 'symfony/console' and I added some of the most frequently used admin tool commands to it from (index.php?module=Administration&action=Upgrade) - it works grate! I'd be happy to help out with these.

[jobvector]

Hi,

Sugarcrm CE had "a lot of" tests available, why not make them available for suitecrm as a starting point: https://github.com/sugarcrm/sugarcrm_dev/tree/master/tests
unittestingsugarcrm.pdf

[JimMackin]

Hi,

We had looked at using these as a starting point however the tests referred to non CE functionality and relied on a very old PHPUnit version.

However we would welcome people porting any relevant CE tests to SuiteCRM.

Thanks,
Jim

[samus-aran]

This is really a duplicate of this suggestion, but since this has more activity I will keep this open and close https://github.com/salesagility/SuiteCRM/issues/641

[chris001]

@adamjakab If you could make a PR with your command line symfony/console admin tools, against https://github.com/jakubpas/suitecrm/tree/develop that'd be great. Some background, this is the branch from @jakubpas, PR #3237 here, to add support for installing the application by composer, and start moving 3rd party libraries over to composer, so composer is already there ready for your PR to add console. #3237 really ought to be merged into here soon, I'd expect. Have heard no word back on reason for the delay.

[adamjakab]

Hi @chris001 , of course, I'd be more than happy to do so - just give me a few days to get my things together and I will. Whilst you're at it pls do the comunity a favor ;) - get rid of Sugar_Logger (I have the cold chills every time I see a "$GLOBALS['log']->" statement. Implement something like monolog/monolog - so simple yet so powerful. This project is entering a state where picking up momentum and it it really important to be able to see what is happening behind the scene. A good logger is essential for that.

[samus-aran]

Add unit and functional tests

[samus-aran]

We have now set up a new home for suggestions at Trello. All github issues that were labeled 'suggestion' have been moved and will be closed. Certain ones will be progressed within the new Suggestion Box and may be re-opened.

Announcement of moving Suggestions:
https://suitecrm.com/forum/suggestion-box/13691-moving-suggestions-from-github-to

New SuiteCRM Suggestion Box
https://trello.com/b/Ht7LbMqw/suitecrm-suggestion-box

(P.S) I have also moved this into the discussion lane so it can be progressed.

[chris001]

Hey there @JimMackin

1. With respect to the SugarCRM CE 6.5 tests referring to non-CE functionality, which non-CE functionality are you referring to? Teams? These Teams tests should just be marked as Skip, so they won't fail a build.
2. You mentioned the SugarCRM CE 6.5 tests rely on a very old PHPUnit version. That's true, they were written for PHPUnit 3.5 in 2009. These tests, however, are compatible with PHPUnit 3.7 required in SuiteCRM composer, and will also run in PHPUnit 4.8 (used by SugarCRM 7.8), and these tests are also compatible with PHPUnit 5 which includes a BC layer to support tests extending the class PHPUnit_Framework_TestCase.
   Reference: https://github.com/sebastianbergmann/phpunit/issues/1595#issuecomment-71826978

Conclusion, the tests from SugarCRM CE 6.5 should be merged immediately. The parts of some of those tests which fail because they refer to modules only included in the paid versions of SugarCRM, which are non-existent in SuiteCRM CE, can be easily found, when they run and fail, and marked as Skip.

[chris001]

Update: the SugarCRM 6.5 unit tests (1202 of them) were finally merged to the branch https://github.com/salesagility/SuiteCRM/tree/feature/codeception with PR #3600
There's also folders for functional and acceptance tests (now empty).

For these tests to actually do the job and run on this github repo automatically and test the code, as they are intended to do, @samus-aran must do one more step, merge the tests from this feature/codeception branch PR #3600 to the master, hotfix, and develop branches. The .travis.yml from feature/codeception contains new code needed for Travis to actually run this codeception test runner, on all of the code, every time a new PR is created against master, hotfix, develop, or feature.

[chris001]

@adamjakab For monolog/monolog and symfony/console - these two packages you've already added to your fork, no? Would it be possible to make a PR against either https://github.com/jakubpas/suitecrm/tree/develop (which jakub already added composer to, and is expected to merge here, barring any unmentioned objection) or directly here https://github.com/salesagility/suitecrm/tree/develop ?

[chris001]

Paging @murdal Are you able to merge #3600 into master, hotfix, and develop so the tests will run as they should.