Suitecrm: Consider moving away from Travis CI

[connorshea]

With the recent news of the Travis acquisition last month followed by the significant layoffs last week I think it's worth discussing options for moving to a different CI provider if the need arises in the near future.

My two personal preferences would be either GitLab CI or CircleCI:

GitLab CI

GitLab CI: GitLab CI provides 50k build minutes/month (Note that the page says that the GitHub integration will be moved from the GitLab.com Free plan to the Silver (paid) plan, but my understanding is that open source projects have all Gold features available to them by default, we can get that clarified if we're concerned about it).

Job concurrency is dependent on the availability of the GitLab.com Shared Runners, which can vary somewhat (there's a public Grafana instance with this kind of info available in realtime if that's interesting to anyone).

You can also use your own runners you host yourself, if that's of any interest.

• GitLab CI Documentation
• The .gitlab-ci.yml config file documentation

CircleCI

CircleCI: CircleCI offers 4 Linux containers with no build limits for open source projects. I used CircleCI a few years ago – in 2015/16ish – and enjoyed it at the time, though I don't have as much recent experience with it.

• CircleCI Documentation

---

Either way, we'd need to get SuiteCRM running in a Docker container to run tests on it (GitLab CI supports non-Docker environments but I'm not familiar with them and I'm not sure if you can do that kind of thing with the Shared Runners provided by GitLab.com).

Both GitLab CI and CircleCI allow you to provide your own containers for running your test suite on. GitLab has a Container Registry you can upload custom images to, I think CircleCI requires you to upload the container to Docker Hub. Both can use container images from Docker Hub.

If you want to see GitLab CI in use on an actual GitHub project, I recently switched to it for a personal project of mine. See the .gitlab-ci.yml, Dockerfile.ci (which is the container used by GitLab CI), and any recent pull requests.

Obligatory disclaimer: I'm a former GitLab employee, so I'm obviously biased here. I'm open to any other suggestions people have :)

[connorshea]

Here's an example of what the .gitlab-ci.yml might look like:

image: "php:7.2.15-apache-stretch"

services:
  # Probably specify a version for mysql.
  - mysql
  # Add elasticsearch here as well.

variables:
  - INSTANCE_URL: http://localhost
  - DATABASE_DRIVER: MYSQL
  - DATABASE_NAME: automated_tests
  - DATABASE_HOST: localhost
  - DATABASE_USER: automated_tests
  - DATABASE_PASSWORD: automated_tests
  - INSTANCE_ADMIN_USER: admin
  - INSTANCE_ADMIN_PASSWORD: admin1
  - COMPOSER_MEMORY_LIMIT: -1

before_script:
  - php -v
  - curl -sSL https://dl.google.com/linux/linux_signing_key.pub | apt-key add - && echo "deb https://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list
  - apt-get update -qqy && apt-get install -qqyy google-chrome-stable mysql-client
  # Install composer here (if we're using stretch you can just use apt-get and it should have a pretty recent version)
  - composer install

php-lint:
  script:
    # Put the actual lint command here:
    - php lint

# Use anchors to reuse common configuration options.
# https://docs.gitlab.com/ee/ci/yaml/README.html#anchors
.php:
  script:
    - ./vendor/bin/codecept run install --env travis-ci-hub -f -vvv -d
    # Run API functional tests
    - ./vendor/bin/codecept run tests/api/V8/ -f -vvv -d
    # Run basic acceptance tests
    - ./vendor/bin/codecept run acceptance --env travis-ci-hub -f -vvv -d
  # Save 'var/log/apache2/error.log' and 'suitecrm.log' as artifacts
  # https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html
  artifacts:
    paths:
      - var/log/apache2/error.log
      - suitecrm.log
    expire_in: 2 weeks

php-5.6:
  image: "php:5.6.40-apache-stretch"
  # Cache composer packages for reuse on the next run of the test suite.
  cache:
    # Set a key for the cache, e.g. based on the PHP version.
    key: 'php-5.6'
    paths:
      - vendor/
  <<: *php

# I don't see a 7.0.x version of the official php containers on Docker Hub.
# php-7.0:
#   image: "php:7.0.30-apache-stretch"
#   <<: *php

php-7.1:
  image: "php:7.1.26-apache-stretch"
  # Cache composer packages for reuse on the next run of the test suite.
  cache:
    # Set a key for the cache, e.g. based on the PHP version.
    key: 'php-7.1'
    paths:
      - vendor/
  <<: *php

php-7.2:
  # Cache composer packages for reuse on the next run of the test suite.
  cache:
    # Set a key for the cache, e.g. based on the PHP version.
    key: 'php-7.2'
    paths:
      - vendor/
  <<: *php

This _definitely_ won't work, but it's a decent base to start from. I think you'll need to enable some PHP extensions and add a bunch of the commands from the .travis.yml that I didn't include here.

---

For now I'd say ignore custom Dockerfiles and just focus on getting the CI YML working (Docker container building + uploading is slow). But for if/when you want to do it, you'll need a Dockerfile for each of the versions of PHP you want to support. Something like this:

# Dockerfile for running the application in a CI environment.
# Base image from https://hub.docker.com/_/php/
FROM php:7.2.15-apache-stretch

RUN curl -sSL https://dl.google.com/linux/linux_signing_key.pub | apt-key add - && echo "deb https://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list
RUN apt-get update -qqy && apt-get install -qqyy google-chrome-stable mysql-client

[connorshea]

This issue will probably be a show-stopper for GitLab CI :/ https://gitlab.com/gitlab-org/gitlab-ee/issues/5667

Essentially, when forking a project that uses GitLab CI, opening a PR won't trigger the GitLab CI job(s) because it relies on syncing the whole git repository from GitHub to GitLab, and there's no logic in the GitLab codebase for handling the case where a repository on GitHub is forked and a PR is opened from the fork into the origin repository.

It also has some security implications, e.g. if you have an API token in GitLab CI's environment variables a PR could abuse this fact and, e.g. have the CI job print the token in plain text (or otherwise do something that isn't as easy to detect and automatically block, e.g. printing it with spaces in between each character).

TL;DR if you fork a project and then open a PR using that fork, CI won't be run.

[lazka]

I just want to add azure-pipelines: 10 parallel jobs, max 6h per job, no limits.

[connorshea]

This might be useful for resolving the issue of PRs from forks on GitHub: https://github.com/brndnmtthws/labhub

[samus-aran]

Hey Peeps. As stated in the chatroom we are looking at this ASAP and will be trying out a few things in the next few sprints.. So We'll keep this thread going and provide updates but GitLab we saw and that was the biggest issue we had about the PRs.

[connorshea]

I've been working on getting the CI ported to GitLab CI today and made some decent progress. I don't know if it'll actually be used in SuiteCRM proper because of the aforementioned problems with forks in GitLab CI (though the bot I linked previously seems like a decent workaround?), but we intend to start running CI in GitLab on our fork, so I figured I may as well try it out.

The branch is here: https://github.com/connorshea/SuiteCRM/tree/gitlab-ci

Some notes:

• Travis is locked to an older version of Chrome, so reproducing the exact same version of Chrome as Travis uses isn't really possible. That means we have to upgrade to Chrome 74 and ChromeDriver 74 (or switch to whatever version of Chromium is provided by Debian).
• Currently, I'm installing Chrome from Google's official Debian repo, but I can't specify the version because it's always just the latest stable. This could be a problem as new builds of Chrome could potentially break the build.
• If we use a custom Docker container we can cut the CI time down quite a bit, since currently the first few minutes of the CI job are mostly spent installing Debian packages. This would also allow us to freeze the Chrome version.
• Most of the before_script in the current travis config is from the travis docs: https://docs.travis-ci.com/user/languages/php/#apache--php
• So far I'm only working with PHP 7.2, once that works it shouldn't be too hard to make it work for 7.0 and 7.1, I'm not so sure about 5.6, though.

At the moment I'm stuck on how to get SuiteCRM to actually run in CI. I don't have FastCGI installed currently (is that strictly necessary?) just because it seems quite complicated to get it set up. Since the other test suites require that SuiteCRM be installed before they can be run, I haven't been able to verify whether the test suite actually works.

I'm fairly certain ChromeDriver works, I've got a MySQL server set up in parallel to the test runner, and I've managed to get Composer to install the project dependencies.

[connorshea]

I'm still stuck on how to getting the Apache configuration set up, unfortunately. I've kind of hit a wall and the server returns a 403 Forbidden error.

I went so far as to set up a custom Dockerfile and everything locally but I haven't been able to get it to respond with anything other than a 403 error in my browser. It also doesn't seem to log any errors, bizarrely.

If anyone has a better understanding of Apache I'd appreciate help here :) I _think_ I've got all the permissions right but I don't know if there's something I'm overlooking.

EDIT: I got it to work! Woo!

~Though now it's got errors about theme files not being there.~

~Maybe something to do with not having run the SuiteP Build Robo task? CI log is here You can see the artifacts on the right sidebar if you hit "Browse", including screenshots.~

The theme errors seem to be a red heering, the actual problem is that either that the database credentials are incorrect or the database connection isn't configured right. I'm assuming the latter. I know the CI log says there are problems with the MySQL service booting, but it doesn't look like it's actually a problem to me.

EDIT 2: Got it working! The install step and database, at least.

[connorshea]

Current progress:

• MySQL works
• ChromeDriver works
• Install test suite finally passes!
• The PHPUnit test suite works but does not pass yet, the only failing tests seem to be from having not configured ElasticSearch yet.
  
  
  
  • Need to setup ElasticSearch.
  
  
• The API test suite does not work yet, need to set that up. I'm not sure how to insert the data into MySQL given that the MySQL server is in a separate container. Any ideas here are appreciated.
• The acceptance test suite works but does not pass yet.
  
  
  
  • Need to figure out why the tests/acceptance/Core/BasicModuleCest.php:testScenarioViewBasicTestModule acceptance test is failing (CI log here)
  
  

[lazka]

Just a thought, if you'd move your test scripts into a bash script we could use the same docker image and test script on travis-ci and gitlab.

[connorshea]

That assumes there’s no difference between the two :)

[lazka]

With docker there shouldn't be any difference :)

[connorshea]

I don’t think Travis supports custom Docker containers, does it?

[lazka]

I don’t think Travis supports custom Docker containers, does it?

Not in the sense that the travis-script gets executed inside of the container, but you can use docker like you would locally on your machine (docker build/pull/run etc): https://docs.travis-ci.com/user/docker/

1) pull image
2) start container
3) run script in container
...

[connorshea]

@lazka for now I don't expect that that'd be worth the effort. It'd mostly just add overhead for not much gain.

[connorshea]

An update: I figured out how to inject the data into the MySQL service and I've got the API test suite running. Unfortunately, it fails because the /Api/access_token route 404s.

FWIW, my branch is based off master, so #7173 isn't in my branch. I'm not sure how much that'd matter, since I'm not using FastCGI anyway (the reason I'm not using it is because it's less well-supported by the official PHP containers, and I'd rather rely on continued support from them than run our own PHP base containers). I'll probably try 'backporting' it to my branch to see if that fixes the problem.

CI log: https://gitlab.com/connorshea/SuiteCRM/-/jobs/219111143

EDIT: As I expected, the backport didn't fix the problem. Neither did using the .htaccess in this issue. (it just caused a 500 error instead).

[connorshea]

Pivoting to getting the PHPUnit suite to pass, it almost passes.

Tests: 1600, Assertions: 5006, Errors: 14, Skipped: 15, Incomplete: 192.

The 14 errors are mostly to do with the StateSaverChecker, ElasticSearch, and InboundEmail.

[adriangibanelbtactic]

An update: I figured out how to inject the data into the MySQL service and I've got the API test suite running. Unfortunately, it fails because the /Api/access_token route 404s.

FWIW, my branch is based off master, so #7173 isn't in my branch. I'm not sure how much that'd matter, since I'm not using FastCGI anyway (the reason I'm not using it is because it's less well-supported by the official PHP containers, and I'd rather rely on continued support from them than run our own PHP base containers). I'll probably try 'backporting' it to my branch to see if that fixes the problem.

CI log: https://gitlab.com/connorshea/SuiteCRM/-/jobs/219111143

EDIT: As I expected, the backport didn't fix the problem. Neither did using the .htaccess in this issue. (it just caused a 500 error instead).

So in your last test did you apply both https://github.com/salesagility/SuiteCRM/pull/7173 PR and .htaccess from https://github.com/salesagility/SuiteCRM/issues/7306 issue?

In that case do you have the associated GitlabCI log?
Assuming the GitlabCI log includes the associated apache's error_log I am curious on knowing what caused that 500 error.

Thank you.

[lazka]

The 14 errors are mostly to do with the StateSaverChecker, ElasticSearch, and InboundEmail.

InboundEmail is likely because of #7107

[connorshea]

I've opened a PR, it's very much experimental and unfinished, but if anyone wants to offer feedback or help out, I've included instructions to set up GitLab CI for your own fork :)

[connorshea]

@adriangibanelbtactic for some reason, CI never generates a /var/log/apache2/error.log file. I'm fairly certain that's where the file is supposed to be, though, so it's odd to me that it doesn't show up. The test with the .htaccess file was never run on GitLab CI, just in a local Docker container, so I don't have the logs anyway, unfortunately.

[connorshea]

FWIW, I added the coq bot to a personal project of mine to make the GitLab CI status checks work on PRs, and it's actually a lot easier than I thought it'd be. Only takes maybe 15 minutes to set up. So that shouldn't really be a concern.

[computamike]

Hi - I'm not sure if this helps - but I started making a docker container for SuiteCRM (well - a set of containers... An NGINX container, a PHP-FPM container, and a mariaDB container) - does that help in terms of testing / CI?