Prestashop: Provide an automated build tool for github ZIP archives

[matks]

For some of the modules we store on github, we attach with releases a production-ready ZIP archive (example: https://github.com/PrestaShop/ps_facetedsearch/, https://github.com/PrestaShop/autoupgrade). However the process to create this ZIP archive is specific to each module and is written nowhere.

This build process involves for example:

• removing git and CI related files/folder : .git, .github, .travis.yml, .gitignore ...
• when module uses Composer, run composer install --no-dev --optimize-autoloader
• removing tests, usually located into /tests folder
• when module uses a frontend build tool (for example webpack), build the frontend assets and remove the source files

We should:

• write down a global process for building a ZIP for all modules
• maybe automate it using PHP or bash scripts (maybe put this an executable in https://github.com/PrestaShop/php-dev-tools ?)
• find a way to handle module-specific build steps

One solution, for example, would be to have

• one php binary able to build a tool following the standard process
• and each module can provide some extra steps, maybe in a .ps_module_build.php

[khouloudbelguith]

Ping @PrestaShop/prestashop-core-developers

[Quetzacoalt91]

We (the module team) are also interested in this step. Our first try is using Travis (https://github.com/PrestaShopCorp/ps_checkout/blob/5e1a134c98783d947958c9428b99fdb6d86dc081/.travis.yml#L37-L69) in which we run the js scripts compilation, run a composer install then clean all the unneeded files.

However all the builds try to upload the artifact and setting it is fastidious. Switching to a github action may be interesting.

[matks]

Project "ZIP Archive Module Builder"

Expected outcome

We want to create a php binary able to build a ready-for-production ZIP archive of a module.

So we want this to happen:

$ php ps-zip-module-builder 4.2.1 modules/ps_faceted
Building ZIP archive ...
ps_faceted-4.2.1.zip has been built !

Technical specifications

I think ps-zip-module-builder should apply a standard prestashop modules building process that can be customized by the module to add extra steps.

Here are things we want ps-zip-module-builder to perform for all builds

• remove development-related files and directories such as .github/, .gitignore, .travis.yml, phpunit.xml, .php_cd.dist, .docker, test/
• run composer --no-dev --optimize-autoloader to download php dependencies
• run npm install & npm run build to download JS dependencies and build assets
• perform a series of checks to find whether some mandatory files and practices for modules are valid
  
  
  
  • check version number
  
  
  • check index.php files in every directory
  
  
  • check mandatory module files such as config.xml
  
  

Idea

I think we can lay out ps-zip-module-builder building strategy like this:

Steps

1. customisable step for modules
2. mandatory checks
3. customisable step for modules
4. run composer install if eligible
5. run npm install if eligible
6. customisable step for modules
7. remove files and directories using a mapping
8. customisable step for modules
9. build the ZIP archive
10. customisable step for modules

Modules would be able to customize this process by providing a .ps_build.php file with a pre-defined format.

In this .ps_build.php files, they would be able to:

• alter the mapping of files and directories to remove (add or remove items in the mapping)
• provide additional steps to be run either in position 0, position 2, position 5, position 7 or position 9 (like prestashop hooks)
• disable standard steps from being run

[matks]

I started a POC on https://github.com/matks/ps-modules-builder

[matks]

@Quetzacoalt91 POC is now quite advanced. I think the best place for it would be in https://github.com/PrestaShop/php-dev-tools repository. Wdyt ? Shall I make a PR ?

[Quetzacoalt91]

I kept your builder in my todo list so we can discuss about it, indeed. Can you give me some time until the end of next week so I can provide a feedback?

[matks]

I kept your builder in my todo list so we can discuss about it, indeed. Can you give me some time until the end of next week so I can provide a feedback?

Take all the time you need 😊

[Progi1984]

@matks : As you can see in PS_Checkout : https://github.com/PrestaShopCorp/ps_checkout/blob/master/.github/workflows/build-release.yml, Build Release can do the job ;)

[matks]

@Progi1984 not exactly 🤔 https://github.com/PrestaShopCorp/ps_checkout/blob/master/.github/workflows/build-release.yml a set of scripts / command lines that, when run in GitHub Action environment, build a ZIP release. This cannot be used by a PHP developer to simply build a ready-for-prod ZIP archive on its localhost and it depends on the GA environment.

For example yesterday I wanted to build a ZIP of http://github.com/prestashop/ps_qualityassurance and I had to check the README to see what I had to compile before building the ZIP. I wish I could have run something like ps-zip-builder build ps_qualityassurance 😉 . And the ZIP attached to GitHub releases are only stable versions, it does not provide you ZIP archives from git branches.

[PierreRambaud]

@Progi1984 not exactly https://github.com/PrestaShopCorp/ps_checkout/blob/master/.github/workflows/build-release.yml a set of scripts / command lines that, when run in GitHub Action environment, build a ZIP release. This cannot be used by a PHP developer to simply build a ready-for-prod ZIP archive on its localhost and it depends on the GA environment.

For example yesterday I wanted to build a ZIP of http://github.com/prestashop/ps_qualityassurance and I had to check the README to see what I had to compile before building the ZIP. I wish I could have run something like ps-zip-builder build ps_qualityassurance . And the ZIP attached to GitHub releases are only stable versions, it does not provide you ZIP archives from git branches.

But it's specific to each module.
Sometimes you have package.json, not placed in the same directory, a composer.json or not, tests directories, commands inside package.json & composer.json but not the same or with the same name, etc.
Adding one more tool and we all know that we already have too many tools, will add work for us. We will have to manage it, make sure dependencies are updated and it's harder in PHP or JS because of the ecosystem which is changing every day.
I'm not sure it's a good idea :/

[Quetzacoalt91]

And the ZIP attached to GitHub releases are only stable versions, it does not provide you ZIP archives from git branches.

Actually it does.
If you look at the workflow for a given commit, you will find in the artifacts the module ZIP. In the case of a GitHub release, this ZIP is copied in the release attachments.

[matks]

@Progi1984 not exactly https://github.com/PrestaShopCorp/ps_checkout/blob/master/.github/workflows/build-release.yml a set of scripts / command lines that, when run in GitHub Action environment, build a ZIP release. This cannot be used by a PHP developer to simply build a ready-for-prod ZIP archive on its localhost and it depends on the GA environment.
For example yesterday I wanted to build a ZIP of http://github.com/prestashop/ps_qualityassurance and I had to check the README to see what I had to compile before building the ZIP. I wish I could have run something like ps-zip-builder build ps_qualityassurance . And the ZIP attached to GitHub releases are only stable versions, it does not provide you ZIP archives from git branches.

But it's specific to each module.
Sometimes you have package.json, not placed in the same directory, a composer.json or not, tests directories, commands inside package.json & composer.json but not the same or with the same name, etc.

But it's already being (wrongly) done. 😄

1. Consider all our native modules. How many of them are specific? I dare say that 75% of them will be fine with a simple composer install.

2. Today, everytime we push to branch master the Addons Cron checkouts the last commit, does composer install and zip and pushes this to Addons API. This is what merchants download. We can quite simply do better 😄 .

See the things I wrote in the ticket description, I already mentioned what you said 😉

This build process involves for example:
- removing git and CI related files/folder : `.git`, `.github`, `.travis.yml`, `.gitignore` ...
- when module uses [Composer](https://getcomposer.org/), run `composer install --no-dev --optimize-autoloader`
- removing tests, usually located into `/tests` folder
- when module uses a frontend build tool (for example webpack), build the frontend assets and remove the source files

We should:
- write down a global process for building a ZIP for all modules
- maybe automate it using PHP or bash scripts (maybe put this an executable in https://github.com/PrestaShop/php-dev-tools ?)
- find a way to handle module-specific build steps

Adding one more tool and we all know that we already have too many tools, will add work for us. We will have to manage it, make sure dependencies are updated and it's harder in PHP or JS because of the ecosystem which is changing every day.

Like I said, _we are already doing badly this "build and ZIP" process_. But it's Addons API that does the job and because of that we do dirty things like committing the bundled assets.

So today there is an opaque limited system that take care of "build and ZIP" that we have no easy control on 😄 my suggestion is to do better. Yes it means "We will have to manage it" and today because we dont manage it we have to commit bundled assets when using webpack 😉

EDIT: also if we start adding workflows like https://github.com/PrestaShopCorp/ps_checkout/blob/master/.github/workflows/build-release.yml on all of our modules it becomes also a maintenance burden as any change we need to do globally, we need to do it on every module. The idea is to build something we can aim to reuse everywhere, following DRY principle.

[matks]

And the ZIP attached to GitHub releases are only stable versions, it does not provide you ZIP archives from git branches.

Actually it does.
If you look at the workflow for a given commit, you will find in the artifacts the module ZIP. In the case of a GitHub release, this ZIP is copied in the release attachments.

Indeed 😉 thanks I did not watch closely enough!

[kpodemski]

It would be great to have an official tool for this, right now in my daily work I'm using Grunt which is great for this kind of tasks, this is an example config: https://gist.github.com/kpodemski/eb424775dc3b3bc796677cd654e20535

it's not perfect but it's simple and does the job, also it's fully compatible with github actions :)

[PierreRambaud]

I let you decide, but don't forget we have already too many things to do, we are already overworked :/

[Progi1984]

@matks Build release Workflow is currently implemented (or setup or installed) at each pass of Presthubot. (https://github.com/PrestaShop/presthubot/blob/master/src/App/Service/PrestaShop/ModuleChecker.php#L56)

[matks]

I let you decide, but don't forget we have already too many things to do, we are already overworked :/

We dont have to do it now, the question is rather whether we should do it (I think YES) 😄 .

We have other topics like this: for example _we know we need to build a new Webservice_ because of current one age and limitation. We have this planned for ... 3 years ? and I'm not sure we'll do it before 2022 or 2023. But we keep the topic in the backlog :)

[PierreRambaud]

I let you decide, but don't forget we have already too many things to do, we are already overworked :/

We dont have to do it now, the question is rather whether we should do it (I think YES) .

We have other topics like this: for example _we know we need to build a new Webservice_ because of current one age and limitation. We have this planned for ... 3 years ? and I'm not sure we'll do it before 2022 or 2023. But we keep the topic in the backlog :)

We can go further with this and create 10k issues only with features requests but as @SimonGrn and others already said, we should also focus on what we can do and what has priority.
About the API, it's a mandatory requirement for the next major, maybe before, a lot of people are waiting for it whereas this tool is not. It's like the StarterTheme, even if it's needed for the community, we were not able to maintain it :(

[matks]

Closing because @PierreRambaud does not like me 😢