Cphalcon: Internals - Set up standardized environment for testing and development

[niden]

There is a need for a standardized environment to be used for testing and/or development

Requirements

• Easy to setup
• Offer all the necessary services for the framework
• Ability to run the testing suite locally

Implementation

• The use of docker and nanobox.io satisfies the above needs
• Installation for various platforms
• Containers offer the necessary services for the application
• Can compile Phalcon and run suite

Status

• [x] Set up environment with the boxfile.yml
• [x] Add all the necessary services and related extensions (Redis, Postgresql etc.)
• [x] Rebuild the test suite to cover as many methods as possible (for now)
• [x] Create Integration test suite
• [x] Create Cli test suite
• [x] Create Unit test suite

Working Branch

https://github.com/niden/cphalcon/tree/T13578-setup-phalcon-environment

Coding standard

• We are using the Cest format of Codeception.
• Each test file represents a method of a component.
• Tests are written for each method accordingly.
• Keep each test file as small as possible to improve readability
• Keep each test with as few assertions as possible and as few steps as possible. It makes it easier to read
• Docblocks on each test are required
• The name of each test is the namespace and method and then the purpose. For instance

Phalcon\Acl\Adapter\Memory :: allow() has a test of aclAdapterMemoryAllow.

• Each test must have a call to $I->wantToTest() describing what is being tested. This helps with reading the output and identifying where an error has happened. The message contains the namespace/component and the method (see also docblock). For instance

$I->wantToTest('Acl\Adapter\Memory - allow() - try to allow twice');

[dschissler]

It also needs be very easy to use it in a forked repo for development without relying on phalcon/cphalcon. It also should be able to use any repo or copy of Zephir as well. Before I ended up just using an ongoing PR to run the tests and so I was mocked for not having mergeable code since people didn't realize that I was routing around a difficult development process that I didn't care to learn.

[niden]

@dschissler Agreed. The environment should allow you to use any Zephir version from the relevant repo and compile Phalcon as needed.

[dschissler]

Consider adapting and improving the Webird provision system. I adapt this to all of my projects and sometimes I feed the refinements back into Webird. I was disgusted by needing to install plugins into a provision system in order to install plugins for my plugins and so I did it all in bash.

https://github.com/perchlayer/webird/blob/master/setup/provision-system.sh

Its also very flexible in that I can just comment out a "list" line with a pound symbol to have it not install that package and to not install it or to leave documentation about something. There are lists files for pecl and system packages and on some systems I use a npm list for global install. A init script can read a list file by a bash function that is exported into the init script.

This is a package list file for example:

# languages
language-pack-en
language-pack-de
language-pack-ru
language-pack-uk
# translation
# utilities
vim
git
curl
gcc
make
pkg-config
re2c
# environments
nodejs
nginx-extras
mariadb-server
mariadb-client
php-pear
php-common
php-cli
php-fpm
php-curl
php-mbstring
php-zip
php-mysql
#php7.2-phalcon
# dev libraries
php-dev
composer
libpcre3-dev
libczmq4
libczmq-dev

This is the pecl list file for example:

# You must add -alpha or -beta if the package is not yet stable
# Manually downloading, patch and installing mailparse
# in pecl init script until mbstring issue is resolved
#mailparse
apcu
ev
zmq-beta
msgpack

Here is the exports file that gets sourced and exported into every init script. It allows for me to change it up to build phalcon from release tarball or tree. I plan to someday improve the variables here but its easy and gets the job done.

#!/usr/bin/env bash

if [[ -z "$DATABASE_PASSWORD" ]]; then
  export DATABASE_PASSWORD=open
fi


export PHP_VERSION=7.2
export NODE_VERSION=8.11.1

# export PHALCON_COMPILE=

export PHALCON_COMPILE=tarball
export PHALCON_VERSION=3.3.2

# export PHALCON_COMPILE=tree
# export PHALCON_VERSION=3.4.x

export ZEPHIR_VERSION=0.10.9
export ZEPHIR_PARSER_VERSION=1.1.2

export PHP_ETC=/etc/php/$PHP_VERSION
export NGINX_ETC=/etc/nginx
export SSL_DIR=$NGINX_ETC/ssl

export DEBIAN_FRONTEND=noninteractive

The init scripts just go in order from 00 to 99 and an init file can be ignored by simply prefixing the name with a pound symbol (commented out file).

00-prepare.sh
10-package.sh
#20-node.sh
21-npm.sh
30-php.sh
31-pecl.sh
32-phalcon.sh
80-mysql.sh
80-nginx.sh
99-finalize.sh

Here is the Phalcon and Zephir init script. Currently I'm installing Phalcon from system packages so "PHALCON_COMPILE" isn't set and this script just exits with a success error code. This script could be easily improved to merge too variables together and to explode it for easier setup and also to allow a forked repo to be used as well.

#!/usr/bin/env bash
# Environment variables OS_DIR and TEMP_DIR are available

# If Phalcon compile is not set then don't compile.
[[ -z "$PHALCON_COMPILE" ]] && exit 0

# You can skip compiling Phalcon by manually exporting this.
[[ ! -z "${SKIP_PHALCON+x}" ]] && exit 0

# Install Zephir
echo "Installing Zephir"
ZEPHIR_INSTALL_DIR=/opt/zephir
rm -Rf "$ZEPHIR_INSTALL_DIR"
cd "$TEMP_DIR"
wget "https://github.com/phalcon/zephir/archive/${ZEPHIR_VERSION}.tar.gz"
[[ $? -ne 0 ]] && exit $?
tar -xf "${ZEPHIR_VERSION}.tar.gz"
[[ $? -ne 0 ]] && exit $?
mv "$TEMP_DIR/zephir-${ZEPHIR_VERSION}" "$ZEPHIR_INSTALL_DIR"
cd "$ZEPHIR_INSTALL_DIR"
./install -c > /dev/null

# Install Zephir parser
echo "Installing zephir_parser extension "
cd "$TEMP_DIR"
wget "https://github.com/phalcon/php-zephir-parser/archive/v${ZEPHIR_PARSER_VERSION}.tar.gz"
[[ $? -ne 0 ]] && exit $?
tar -xf "v${ZEPHIR_PARSER_VERSION}.tar.gz"
[[ $? -ne 0 ]] && exit $?
cd "$TEMP_DIR/php-zephir-parser-${ZEPHIR_PARSER_VERSION}"
./install
php-extension-enable zephir_parser 50
echo "Zephir parser extension installed."

# Install Phalcon
echo "Installing phalcon extension"
cd "$TEMP_DIR"
case "$PHALCON_COMPILE" in
"tree")
  git clone --depth=1 -b "$PHALCON_VERSION" git://github.com/phalcon/cphalcon.git "$TEMP_DIR/cphalcon" > /dev/null
  [[ $? -ne 0 ]] && exit $?
  cd "$TEMP_DIR/cphalcon"
  ;;
"tarball")
  wget "https://github.com/phalcon/cphalcon/archive/v${PHALCON_VERSION}.tar.gz"
  [[ $? -ne 0 ]] && exit $?
  tar -xf "v${PHALCON_VERSION}.tar.gz"
  [[ $? -ne 0 ]] && exit $?
  cd "$TEMP_DIR/cphalcon-${PHALCON_VERSION}/"
  ;;
*)
  echo "Invalid Phalcon compile source.  Aborting."
  exit 1
  ;;
esac
zephir build
[[ $? -ne 0 ]] && exit $?
echo "Phalcon extension installed."

php-extension-enable phalcon 50

exit 0

In short I really like my scripts because I can read them after not working with them much for 6-12 months. The system always has bash available and so I can easily just open up the 5-15 init files to quickly scan them to see what is going on. I've had very good luck running these scripts multiple times on a development system and idea is to make the process idempotent (although I never can trust that in production but it works great in development). I spent countless hours years ago running this entire system over and over again until it finished and also could repair or upgrade.

[dschissler]

Additionally multiple OSs can be supported at once. For example; the latest Fedora, Ubuntu LTS and Debian could be offered.

[dschissler]

I looked at nanobox.io and it might be great but unless its The Thing then needing to pick that up and learn it in a time crunch just seems miserable. I think that the question needs to be asked if you are more interested in getting new blood into the project with just a few small commits here and there or if you want to have the best and most tricked out environment for the extended team. If you want to get new people into the project then nanobox.io is perhaps too much as it also assumes fluent docker knowledge as well. I can see someone working with Phalcon and noticing a single bug and then wanting to fix it and knowing how to but then not having the time to learn a bunch of new other stuff in addition to learn a new programming language. This is why I think that my provision script wins out in this use case because you can just git clone and then run ./provision ubuntu1804 and then reboot and then there is a full working Phalcon dev environment. I personally loathe having to learn a DSL just to enter a project before I can do what I was there to do. Perhaps there can be a place for both sorts of provision system.

[niden]

@dschissler Valid points.

The issue is not to get more people to use nanobox. We don't own stock there or anything like that. It has been however used for over a year now and it has worked as advertised. There is no need to know docker or anything like that. All you do is run a command.

The goal of this issue is to create quickly an environment so that people can develop in Zephir for Phalcon and be able to run the test suite without having to install a bunch of prerequisites.

The reason we are leaning towards nanobox is that there are no complex scripts or anything like that to provision the environment.

Your approach takes into account the environment (ubuntu, fedora etc.) which nanobox does not. That could very well be something we might want to check a bit later.

For now, the goal is to create this environment and to do so a lot of the tests have to be rewritten and organized so this is the focus for now.

[danhunsaker]

@dschissler I'm a former Nanobox employee, so I expect my input will be ignored or at least glossed over, but the fact it uses Docker under the hood doesn't at all mean it requires knowledge thereof. I'm not sure which aspects of Nanobox you investigated, but the version of the product being used, here, is a single config file that sets up the entire dev environment, no knowledge of the build system itself required, nor the stack it runs within. Since Docker is readily available on any Linux OS, and can be installed on macOS and Windows, and Nanobox itself will happily install VirtualBox and configure its _own_ Docker VM (the same way macOS and Windows are supported by Docker's so-called "Native" versions) if you like, it bypasses most or all of the OS-specific issues folks face. Maybe you saw some early documentation on the Dockerfile approach which still hasn't been released?

At any rate. This approach requires users to install Nanobox, then run the nanobox run command, to ensure a full working dev environment which _won't interfere or otherwise directly interact with other environments on the host_. Your approach, while solid and tested, seems to install dependencies directly _onto_ the host, so it misses those advantages.

That said, alternative approaches could be used. Some have been considered internally, and at the moment Nanobox has won out. It might be helpful to increase the transparency around this decision, here?

I'm not technically part of the Phalcon team (I'm assigned to Zephir at the moment), but I'm more than willing to provide what insight I have, as desired.

[dschissler]

@niden

Your approach takes into account the environment (ubuntu, fedora etc.) which nanobox does not. That could very well be something we might want to check a bit later.

I've encountered major bugs in even an Ubuntu LTS for a PECL extension that I had to figure out and then usually the patch existed in just one place but for some reason was not yet applied. How do you deal with weird stuff like that on the major supported distros?

@danhunsaker

Your approach, while solid and tested, seems to install dependencies directly onto the host, so it misses those advantages.

Its meant to be run in a VM guest and I also used Docker to repeatedly run it in the early phase.

I haven't run this in forever and I can see that it runs xenial instead of bionic, but it can easily be applied to any system without worrying about the pre-setup. The script to get it all going is just 54 lines with comments and reasonable spacing.

https://github.com/perchlayer/webird/blob/master/Dockerfile

---

I know that my shit already works now for setting up Phalcon and Zephir (or at least my private version does if the Webird is currently not perfect) and its really a separate matter just to run the tests. So just make the tests easier to get going and with more than just one way in mind and then other good stuff can happen if there is interest.

[danhunsaker]

For my part, if you're willing to maintain those scripts, we can easily list the link to them in the README, for those who want an alternative approach.

[dschissler]

For my part I can do that before after years of neglect I'm not going to jump into something with manic enthusiasm until something has been accomplished.

[dschissler]

I think that the Windows Subsystem for Linux should be the official promoted way to run/test Phalcon on Windows. I'll be getting a Surface Go on Cyber Monday (week) and I plan to heavily experiment with it. There the nanobox is probably a lot easier. I plan to heavily investigate how runtime services integrate into Windows. I'll probably need to modify my distro scripts to support it like ubuntu-bionic and ubuntu-bionic-wsl. I'll do that for my personal uses and then see how that goes.

[niden]

This has been addressed

[dschissler]

I plan to support the following:

export PHP_VERSION=7.2

export ZEPHIR_PARSER_VERSION=1.1.4

# export ZEPHIR_VERSION=tarball:0.11.8
export ZEPHIR_VERSION=phar:0.11.8

# export PHALCON_VERSION=repository:stable
# export PHALCON_VERSION=repository:mainline
# export PHALCON_VERSION=repository:nightly
# export PHALCON_VERSION=tarball:3.4.2
# export PHALCON_VERSION=tarball:4.0.0-alpha1
# export PHALCON_VERSION=tarball:https://github.com/phalcon/cphalcon/archive/v4.0.0-alpha1.tar.gz
# export PHALCON_VERSION=git:4.0.x
# export PHALCON_VERSION=shell:"git clone https://github.com/dschissler/cphalcon"

For the time being it will just be a matter of uncommenting a line. In the case of the tarball I'll get it with curl so the file:// protocol could be used on a local file.

The PHP_VERSION was already necessary for the /etc/php` directory and now it seems it has additional use for the Phalcon repository package.