Codeception: Mission, Vision and Values

[Naktibalda]

We need to have them in order to know the direction of project and what to expect in the future.
Please help me to fill it out.

Mission

???

Vision

Toy project or enterprise level quality assurance tool?

Values

• New features
• Quality ?
• Backwards compatibility ?
• ???

[javigomez]

proposal:

a few ideas:

Mision: to provide a flexible platform for digital testing.

Vision: People increasing the quality of software products around the world;

Values:

• Openness
• Adaptability
• Community
• Collaboration
• Reusability

[DavertMik]

I really like the @javigomez points )
Not sure I can add something more to it.

Mission

Flexible platform for automated testing. We provide common and reusable set of helpers to solve 90% for acceptance tests. We aim to make tests easy to write, read, and maintain. Codeception reduces costs on creating and maintaining automatic test suite by sharing best practices and solutions.

Vision

Quality assurance simplified to be open for everyone: developers, managers, testers.
Provide acceptance/functional/integration testing levels for popular PHP frameworks and all web-based applications.

Values

• encourage best practices in testing. There is still lot of work to do but at least we separate acceptance/functional/unit tests, provide reusable methods, use BDD the right way and so on.
• include best ideas to make Codeception fit for most PHP projects and web based apps.
• dynamical evolution - we should keep movement with time. We are not resisting changes.
• maximal backwards compatibility - provide BC overlays when things changes, so most of users not affected by it
• stability - provide an internal test setup to ensure all the features work as they expect to. Its really hard to maintain stability while supporting all PHP versions, PHP frameworks, and follow all Selenium changes. But we are getting better with it
• community - right now we are not as good as we could be, there are still few codeception experts that help newbies to get it, and we should work more with it. StackOverflow, PHPTestClub, Laracasts forum, Gitter, etc.

---

These are my first thoughts. Probably I can extend it. But we can start the discussion )

[Naktibalda]

I put quality and backwards compatibility values with questionmarks, because these are the areas where Codeception is seriously lacking. You seem to be interested only in adding new features and you stop supporting them after a month.

When I started using Codeception (2.0.13 version of many projects is expected to be very stable and bug free) I had to fix 3 bugs in different modules to complete my first functional test. Afterwards I fixed many trivial bugs reported 1-2 years ago. I find it hard to sell Codeception to my own team, because I have to say that it is good, but it is very buggy( but it is improving :).

Also breaking backwards compatibility every year is too often.
2.1 migration affected every user and 2.2 doesn't look any better.

[DavertMik]

@Naktibalda I understand your doubts. Much easier is to maintain a library, something like legendary left-pad which just works and doesn't change for ages. In case of large product which is constantly challenged by changing dependencies, platforms, and frameworks it is really hard to do the job. Will be it a good way to just drop support for Laravel/Symfony/Doctrine/..... and concentrate only on WebDriver? Or drop PHP 5.4, 5.5. and support only PHP 5.6? But ensure those things will be working 100% for everyone?

_As an example of software company facing similar problems I'd name Ubuntu. They have their operating system and you probably know how it becomes terribly broken after all updates. Maybe you know what mess happened once they released first versions of Unity. I'm not fan of breaking everything but Ubuntu is still major Linux player and the only real alternative to Windows and OSX just because they didn't stop with the standard things provided by Gnome and KDE desktop environments. It gets better from year to year but still faces old issues, and adds new one. Probably this is because Ubuntu is not a software monster-corporation like Google-Apple-Microsoft, they can't sell vendor laptops with preinstalled OS, and they just don't need that resources to encourage all others vendors to support them._

Should we draw a top line for adding new features? And stick to old solutions even if we see they are becoming broken by design? You probably know that fully BC-compatible software becomes a legacy codemess (yes, you can look into PHPUnit's internals).

New features are important as they can bring us new users. They are not just nice shiny things which I want to develop, but a reflection to real needs. And we can't avoid those requirements and answer to pretty everything: not possible in Codeception. Every feature has its edge cases, and they can't be done ideally from start before testing in userland. Like the issue with "killed friends". I didn't have a chance to test it with Selenium Grid but it worked with regular Selenium. And it works for others too.

This is always a conflict in stability and improvements. And I see that we can just keep good balance in it.

Also breaking backwards compatibility every year is too often

No, it is not. There was no LTS release yet. And we don't have enough resources to maintain it yet.
Also it depends on kind of changes. Yes, we are breaking the things but they are about to be as minimal as they could. This is what I meant. Users upgrading from Codeception 1.x don't need to change their Guy classes to Actor, the same way as upgrading to 2.1 does not require anything far more than rebuilding Actor classes. But sure there are always edge cases in such upgrade which can't be predicted.

---

What can we do now?

Issues will always happen because

• dependent libraries, frameworks, selenium/firefox/chrome do change
• edge cases for features can't be tested from start

What can we do with that is to provide constant improvements for those features, extending regression test suite, and what is most important to encourage more users to participate in development. This way we can ensure that all faced issues will not become hidden for others but to be fixed.

Talking about philosophy: our mission is to have testing framework that helps build a test suite for all PHP projects without reinventing the wheel. Yes, we need much stronger community to do this. And if you ask me, I see this is our the biggest problem. Probably this is related to BC, but there are so many other reasons why it is not as powerful as it could be.

[raistlin]

Most important points to me are:

• Strictly follow semantical version, so don't do any breaking change without doing a major release, new features increments minor version, and bugfixes/ minor enhacements increments third number.
• Stability: bug free
• Documentation, for users there are enough documentation, but when I try to improve or extend Codeception I think documentation could do better.
• Efficiency, I need Codeception runs as fast as possible and use as less resources as possible.

Currently, we have a not so bad efficiency and both decent documentation and stability.
I don't think we need to have a LTS, there is no need for it, just strictly following semver we can add new features and do BC without breaking other projects dependant on Codeception. If someone want to use our last added feature it would be able to update to a new major version with the knowledge that something can go wrong. If other project is not interested on new features because it could not afford the migration, may be they could simply cherry-pick desired bug-fix patch into their preferred branch and submit a pull-request.

Apart from SemVer, my proposals would be:

• split Codeception-core and Codeception-_put-your-preferred-framework-here_ into separate projects, fill codeception-core with hooks that codeception-frameworks could easily subscribe and interact with it. I feel quite upset when Codeception installs packages or code related with Laravel, when I'm working on Symfony, for example.
• If we need to use some features from new PHP version, stop supporting old PHP version. People can still use Codeception with PHP5.3 if they need, just not the shiny new version, so no-one will become upset. And project will continue its evolution.

[DavertMik]

Yes, I good points. I just feel we are not ready to follow semver and have 3.0 released yet.

Strictly follow semantical version, so don't do any breaking change without doing a major release, new features increments minor version, and bugfixes/ minor enhacements increments third number.

How about starting to follow this practice from 3.0?
Also we should agree on release cycle. Will this be year or 6 months per major version.

split Codeception-core and Codeception-put-your-preferred-framework-here into separate projects, fill codeception-core with hooks that codeception-frameworks could easily subscribe and interact with it. I feel quite upset when Codeception installs packages or code related with Laravel, when I'm working on Symfony, for example.

I think we can do this in 3.0
Currently each framework is just 2 files (Connector and Module) so it's not about code or packages that are installed. It's more about splitting core into pieces.

Probably this way we could have flexile releases. If we fix bug in Laravel module we could release it once it is done and not to wait for next minor version of Codeception to be released with it. Same can go to other modules, especially WebDriver and others.

This way we can get stability + changes in one.

Documentation, for users there are enough documentation, but when I try to improve or extend Codeception I think documentation could do better.

Yes, I agree, documentation MUST be extended. Can you point to things which is missing? I have some my ideas what to improve but I need your thoughts. I was just about to finish updating chapters for 2.0

---

I think we all can agree on point that 3.0 release should aim for

• decoupling modules
• stabilizing things

Probably this will require some architectural updates going in master, so migration can go smoothly, and move features out of core

[DavertMik]

Related https://github.com/Codeception/Codeception/issues/3014

[raistlin]

Time fixed release cycle is an artificial method to force a version bump.
For some big projects like Ubuntu or even Symfony, it can make sense. Users and developers will know exactly when new versions will be released, and can plan it's projects and work with this knowledge in mind, and will know exactly the project's EOL. They also have hundreds or thousands or contributors, so they will generate enough new content every release cycle.
Codeception is not that type of project, it's a tool with less than 10 regular contributors. Nobody needs to know when new Codeception version will be released to plan it's new project around. Codeception is used to make easier testing, so we only need to be very reliable and fast.

If you want to force a version end of life, we can simply choose to support current stable and last version of old stable branches.
For example if latest released branch is 3.2.1, we can say that we are supporting 2.4.8 (last version of old stable) and 3.2.1 (last version of current cycle branch). With a working SemVer, any user using 2.x.x or 3.x.x should be able to update to 2.4.8 or 3.2.1 without any problems, so there is no need to support other branches.

I simply don't see any benefit on fixed time release cycle on Codeception.