Prestashop: Help developers to deploy a shop from the sources

[matks]

Is your feature request related to a problem? Please describe.
Today, when a developer wants to improve the prestashop project he clones this git repository and opens a PR.

But if he wants to deploy a new shop for a customer (or himself), he rather downloads the zip archive from prestashop.com as it is "production-ready".

Then, if he wants to modify the shop, he does it from the zip. Usually he creates a new git repository, commits the whole shop in a 1st commit then works by iterations.

However, doing so, he does not have some necessary files for local development. For example the _dev folder in themes/classic that can be used to work on a FO theme using webpack. Yesterday I talked with a developer for @MDWeb-Lille who told me he would copy the _dev folder from the source and paste it into the zip in order to have both the development files AND the production-ready code.

We should avoid this. People should be able to build their own production-ready shop from the source.

Describe the solution you'd like
A script / tool / CLI command / process that configures a shop to be production-ready.

This topic can also over the question "how do you deploy a prestashop from the source" (for example using https://deployer.org/). For developers who work often on their shop, having to download the zip is not a suitable solution IMO.

Ideal solution
1) a CLI tool to setup the shop to be production-ready: build assets, clear cache, switch settings from DEV to PROD
2) standard deployer recipes to deploy a prestashop on a webserver, I'm thinking about: composer install, npm install && build, removing dev folders such as tests/, clearing cache, if possible cache warmup ...
3) documentation that covers it all 😄

[khouloudbelguith]

@PrestaShop/prestashop-core-developers, what do you think?

[PierreRambaud]

You're already able to run the script
php tools/build/CreateRelease.php --destination-dir=/tmp/ps-release

[matks]

@PierreRambaud This will build a usage Zip Archive, this is not the goal.

When you develop a standard Symfony application, when deploying to production, you do not copy your all app into a Zip archive production-ready. You modify the Symfony app settings to be prod-ready.

[PierreRambaud]

In automated environement you can:

• Create an archive
• scp to production (really faster than 10 000 files)
• Extract content
• remove parameters files (for database) or create config_prod.yml file
• copy to the final path

And eveyrthing's ok. This is how deployments work :thinking:

[matks]

This tool is called CreateRelease because it creates a new release 😛and that is its purpose. Not "DeployApp" or "CreateArchive" 😉

Also your process works to deploy a new shop. What about an existing one, with assets (product images, modules, translation catalogs) and local files such as logs ?

[matks]

If you look at deployer standard deploy recipe, it's

task('deploy', [
    'deploy:prepare',
    'deploy:lock',
    'deploy:release',
    'deploy:update_code',
    'deploy:shared',
    'deploy:writable',
    'deploy:vendors',
    'deploy:clear_paths',
    'deploy:symlink',
    'deploy:unlock',
    'cleanup',
    'success'
]);

It's the standard deployment strategy and we should try to make prestashop fit it IMO

[jf-viguier]

Thanks @matks. it's exactly what we need : a deployment tool to update already running production or staging shop. The first install is a concern but not as much as an upgrade.
Upgrading means modules upgrades, themes upgrades, ps core upgrade.

[PierreRambaud]

@matks Ohhhh my bad.
Let's create a little script to do that :)

[PierreRambaud]

@matks as @Quetzacoalt91 said, in fact we're already able to create an archive with this tool without updating the version :)

[matks]

@PierreRambaud Still, the archive contains useless files for a shop deployment, such as the install directory* 😐I dont think we should hijack the release/archive building process for deployment.

*Well actually we might need to perform an installation - at least some steps of it for full deployment...

---

I've been thinking about it, and I think that in order to be able to deploy a shop deployer-like (or capistrano-like if you prefer) we need:
1) An accurate mapping of needed code files for production (=> must be what's left after the install process)

These files are expected to be always the same, no matter the deployed environment (localhost, preprod or prod).

Exotic usecase: what do we do about modules ?

2) An accurate mapping of all the folders where PrestaShop puts/generates files, including

• uploaded files (example: images) => asset
• i18n required files (translation catalogs, CLDR data) => asset
• templating cache files => cache
• Symfony cache files => cache
• any other cache directory 😅=> cache
• logs directory => logs
• generated code files, if we have some => generated code
• generated content (example: sitemap) => generated content
• exported files directories (PDF ? backups ?) => asset

All these files are expected to be specific to a shop and environment. 2 prod shops would have a totally different set of these files. If you deploy the same shop codebase on preprod and prod, you expect none of these files to be identical between prod and preprod.

3) An accurate mapping of all settings for prestashop: be it files, ENV variables or database content (example: database credentials, shop URL, shop name...)

Settings are expected to be specific to a shop and environment too.

---

Requirements for th

• every code file from 1) is versioned into git
• assets must be kept for 1 shop between 2 deploys (so if you deploy shop A 10 times, assets must remain between the 10 deployments)
• logs must be preserved too through deployments
• cache files must be wiped out between 2 deploys and if possible warmed up before the deployed version being made available to public (because the cache depends on the code, and the code can change between 2 deploys)
• generated code must be wiped between 2 deploys and re-generated (because the generated code depends on the code, and the code can change between 2 deploys)
• generated content must be wiped between 2 deploys and re-generated (because the generated content depends on the code, and the code can change between 2 deploys)
• settings must be secured and copied into the new shop being deployed

[PierreRambaud]

the archive contains useless files for a shop deployment, such as the install directory* neutral_faceI dont think we should hijack the release/archive building process for deployment

You're able to remove files you don't want before create archive, it's base one your current working directory.

An accurate mapping of needed code files for production (=> must be what's left after the install process)

Always depends on what you want to deploy. If you have a glusterfs or anything else to manage static files, you only want to deploy modules / Core for example.

templating cache files => cache
Symfony cache files => cache
any other cache directory sweat_smile=> cache

Bad idea to deploy cache. It must be generate on the targetting server or while building the docker container (for example).

All lines in your list still depends on what you want to deploy, and what you target. Server, container, cluster, with lb or not.
This is why the current generated archive is enough.

logs must be preserved too through deployments

Same as install dir, you can remove dev logs before creating archive. More, if you're working in dev environement, you must not run the script too release / create archive because, it will 1) break your dev env because of prod env needed, 2) create archive with dependencies?

settings must be secured and copied into the new shop being deployed
It's not depending on the deployment part, but we need to allow environement variable to specify database access and anything in the parameters.yml files.

I'm not in favor to create a specific script just for "your" case. I can find many others things where what you want will not work :/

[matks]

Interesting links:

• https://github.com/deployphp/deployer/blob/master/recipe/prestashop.php
• https://github.com/iaintshine/ansible-prestashop/

[Bonobomagno]

This is really interesting. Actually there is not a nice way to use stateless container for quick horizontal scaling. You actually need to put the all the webroot under nfs/distribuited fs, but this dont seem really good.
If we could automatize the updates/upgrades process, using distribuited fs only for static files, would be really nice.

[muyncky]

@matks thanks for opening this subject! This is in reaction to your response on Gitter. I'm struggling to deploy my PrestaShop 1.7.6.x. Well, not a big struggle, everything goes smoothly really, but the translations are not working. It seems that the cache files are not updated/placed correctly. I still need to manually(!) upload 2 files from var/cache/dev/translations/. A cache folder! They have some sort of key in the name, one is called catalogue.nl-NL.L8dqxxF.php. Maybe there is a mis-match somehow, I cannot find.
Once this is fixed, I will have a lovely recipe for PrestaShop.

Recipe https://github.com/deployphp/deployer/blob/master/recipe/prestashop.php is clearly a PrestaShop 1.6.x recipe, and is useless for 1.7.x.

I want to publish my recipe soon. I have to tweak some stuff and remove garbage. For now I wanna share the following settings:

set('shared_files', [
    'app/config/parameters.php',
    '.htaccess',
    '.user.ini',
]);

set('shared_dirs', [
      'docs',
      'download',
      'img',
      'js',
      'localization',
      'mails/themes',
      'modules/ps_imageslider/images',
      'pdf',
      'tools',
      'translations',
      'upload',
      'var',
      'app/logs',
      'app/Resources/translations/default',
      'app/Resources/translations/nl-NL',
      'modules/blockreassurance/img',
      'modules/blockreassurance/views/img',
      'themes/classic/translations/nl-NL',
      'themes/{your_theme}/translations/nl-NL',
    ]
);


// Writable dirs by web server 
set('writable_dirs', [
      'app',
      'config',
      'config/xml',
      'download',
      'img',
      'mails/themes',
      'pdf',
      'translations',
      'upload',
      'var',
      'app/Resources/translations/default',
      'app/Resources/translations/nl-NL',
      'modules/blockreassurance/img',
      'modules/blockreassurance/views/img',
      'themes/classic/translations/nl-NL',
      'themes/{your_theme}/translations/nl-NL',
  ]
);

I can help with giving instructions on connecting with GitHub and keyless login. I honestly don't have time to write a whole tutorial. But I anyone struggles with Deployer and PrestaShop I'm happy to help.

[PierreRambaud]

We also get inspiration from https://github.com/staegi/puppet-prestashop and https://supermarket.chef.io/cookbooks/prestashop

[muyncky]

We also get inspiration from https://github.com/staegi/puppet-prestashop and https://supermarket.chef.io/cookbooks/

These are installation tools right?

_Deployer_ is a deployment tool (as the name suggests :D), my previous post is only concerned with deployment from local dev environment to staging and production environment. Learn about Deployer-php. Deployer is completely written in PHP, see the GitHub repo deployerphp/deployer.

@matks Maybe we need two topics to cover the things that we are talking about here?

[PierreRambaud]

@muyncky Puppet & chef help to deploy app / tools / software / system configuration :)

[muyncky]

@muyncky Puppet & chef help to deploy app / tools / software / system configuration :)

Can explain your process of versioning and deploying the changes in PrestaShop?

[PierreRambaud]

@muyncky I currently don't have one for PrestaShop, it still depend on how and what you want to deploy.
This was just a suggestion if we want to extend the deployment process to well-known tools, such as Chef, Puppet, Ansible, which are most of the time used by system administrator.

[muyncky]

I wanna give an update on this topic about _Deployer_ and PrestaShop.

1. Translations

One issue with Deployer is that translation cache files are not linked properly, see this issue https://github.com/PrestaShop/PrestaShop/issues/17011

2. Clearing cache

Works, since about 1.7.6.2.
php bin/console cache:warmup --env=prod warms up the cache, but it does not warm up a lot, you still need to visit pages to get it really filled up. Maybe simulate page visites to get al the smarty cache in there too?

3. Enable theme

Since 1.7 we have php bin/console prestashop:theme:enable {{theme}}, this one is broken (I have not opened in issue). It promises a very useful tool for continues deployment.

Enabling the theme does at this moment:

• Set the default hooks for the theme configured in theme.yml, so you remove a module from een hook in, let's say, displayNav2
• Enabling/disabling module, in list to_enable and to_disable
• Change default image size (that happens in theme development, you don't want to go and configure these things in the BO every time you deploy a new version), I did not test this myself
• Change layouts for pages, like category: layout-left-column, I did not test this myself

The issue with prestashop:theme:enable is that it only does this for 1 shop, probably the default shop, or it tries to detect the Context, but there should be no context in CLI, IMO. Setting the shop in the CLI with the option --id_shop did not work for me. Is there already an issue for this?

4. Upgrade/downgrade

In the far future I would like to see role backs. We can upgrade a module, but not downgrade. I have this in mind: in the theme.yml we can define a module list, like this:

modules:
  ps_emailsubscription: 3.4.5
  ps_banner: 1.4.5

This is sort of a dependency list if you will, that allows you to upgrade/downgrade.
The modules would have a /downgrade folder, that works similar to /upgrade, but it has scripts defined for rolling back the changes. PrestaShop would have to look at the current version and decide whether it should down grade or upgrade.

So those are some ideas and issues. But the translation issue https://github.com/PrestaShop/PrestaShop/issues/17011, is by far the most important. Once that is solved, we have a decent deployment tool, and we can start doing continues deployment with a very accessible tool that is written in php; Deployer!

[mcdado]

How do people generally handle module upgrade and similar taks? Because upgrading a module that requires upgrade functions needs — to my knowledge — to be done through the backoffice (at least it was in 1.6, I'm not very familiar with 1.7 yet) which means that backoffice actions can change the cache. In our case we resorted to deploying mostly just the theme folder and little else, everything else is handled by hand, which is not ideal. Documentation in this area is scarse, as @matks suggests.

[muyncky]

@mcdado I can recommend, to deploy ./modules, ./override and ./themes. Module upgrade will be a manual process, however, you can upgrade modules with the command line. Check php bin/console list.

[mcdado]

The reason I don't deploy (or manage through git) the override directory is that it is handled by modules at run time. If I have to make an override I'd rather make a specific module that i can upgrade and keep track of that way. Otherwise let's say that I install a new third party module that installs an override and upgrades it on updates. The deployed would get out of sync with scm.

I mean yes, if you manage modules through git then it would work... but it's a bit messy, no? On deploy there are overrides that would not be already installed otherwise.

[awitkutarahil]

I use Deployer to deploy my shop too. My recipe looks like what has been provided above but it has some flaws.

It would be very useful to have these:

An accurate mapping of needed code files for production
An accurate mapping of all the folders where PrestaShop puts/generates files, including
An accurate mapping of all settings for prestashop: be it files, ENV variables or database content (example: database credentials, shop URL, shop name...)

Even better would be to have an official how-to from PrestaShop.

[muyncky]

@mcdado in my opinion the process _should_ be like this:

1. Place the module locally/in git
2. Define the module in theme.yml
3. Deploy & activate theme (and thus enabling the module, which means installing it if not installed, however this function is still buggy in version 1.7.6.x)
4. Module will be installed on deployment and create overrides

To extends this idea
Of course we still have to deal with configuring the module, be maybe a sort of export/import function from PrestaShop could be created, that exports the module configs in a .yml. The contents could look like this:

In themes/{theme name}/config/modules/ps_wirepayments.prod.yml

BANK_WIRE_CURRENCIES: 2,1
BANK_WIRE_PAYMENT_INVITE: 1
BANK_WIRE_DETAILS: NLddd023908
BANK_WIRE_OWNER: PrestaShop
BANK_WIRE_ADDRESS: Bankstreet 100
BANK_WIRE_RESERVATION_DAYS: NULL
BANK_WIRE_CUSTOM_TEXT: NULL

On installation these configurations will be saved. Maybe a configuration value for forcing overriding the configs could be used.

Just an idea! Lot's of work for the PrestaShop development team to agree on a design :-) ... @ethernoendless @matks

[mcdado]

Hmm interesting idea! It would be nice to allow configuration trough deployment, but I think it should be an installation-time thing and that's it. Because in many cases it's unreasonable to change all settings through yml files and deployment.

I'll look into defining modules in theme.yml, thank you for the tip!

[Quetzacoalt91]

Hi there,

You may use the command php bin/console prestashop:module configure <module name> [<YML file path>] to configure your modules from yml files, it seems it may cover your needs.

See https://devdocs.prestashop.com/1.7/modules/testing/resources/#configuration for more details.

[muyncky]

@Quetzacoalt91 that is great!

[PierreRambaud]

PrestaShop already provides tools to create releases and allow admins to play with themes/modules.
We can't create a custom puppet, chef, ansible, or Deployer scripts for all use cases. People having PrestaShop in production are the best to improve the documentation or purpose custom repositories with what they need.

I'm in favor of closing this issue.

[Progi1984]

I approve. This issue is not an issue. It's a way. We must provide tools for helping developers to deploy a shop from the sources. Actually, we do that. If developers have needs, they must create specific tickets.

[matks]

We can't create a custom puppet, chef, ansible, or Deployer scripts for all use cases. People having PrestaShop in production are the best to improve the documentation or purpose custom repositories with what they need.

Well we do provide docker images 😅 it's a bit the same idea

[PierreRambaud]

We can't create a custom puppet, chef, ansible, or Deployer scripts for all use cases. People having PrestaShop in production are the best to improve the documentation or purpose custom repositories with what they need.

Well we do provide docker images it's a bit the same idea

It's not the same, you can't compare a docker image with an infrastructure automation tool or a configuration management tool.

[matks]

@PierreRambaud what I mean is that "providing docker images" is something that is not necessarily needed from PrestaShop that helps people run it in production. And an infrastructure automation tool is also not necessarily needed from PrestaShop and it helps people run it in production.

But anyway we never succeeded to find the time to work on this topic 😢 so I agree to close it

[PierreRambaud]

@matks Wrong, we use all our docker images for our CI :sweat_smile:

[muyncky]

It would be great if the PrestaShop Team tries some of the technologies for deployment and write a blog about it. Or even better, if some of the deployment tools are proven valuable, it could be included in the docs. The docs can be used as a suggestion, as this is done other external tools:

• testing tools
• migration tools
• benchmark tools

[Progi1984]

@muyncky We are opened to contributions so don't hesitate to contribute to our documentation.

[PierreRambaud]

It would be great if the PrestaShop Team tries some of the technologies for deployment and write a blog about it. Or even better, if some of the deployment tools are proven valuable, it could be included in the docs. The docs can be used as a suggestion, as this is done other external tools:

• testing tools
• migration tools
• benchmark tools

I already worked with puppet, ansible and chef, all of them have advantages and disadvantages, it depends if you're familiar with json (DSL Puppet), Ruby (Chef), or Yaml (for ansible), and also if you want to add tests on it (rspec tests, or serverspecs tests).

We already prepare a benchmark article (afaik).

[muyncky]

@muyncky We are opened to contributions so don't hesitate to contribute to our documentation.

@Progi1984 I have posted an issue #17011. Once this is solved, as promised before, I'm going to provide an elaborate documentation on how to deploy with Deployer.

I think Deployer could be a good tool, because it is the only tool written in php. It makes the learning curve less steep. Deploying Laravel with Deployer is flawless.

@PierreRambaud For me Puppet and Ansible seem rather abstract, search result for PrestaShop gives me really obscure Github repo's. I'm not a hardcore DevOps kinda guy.

[PierreRambaud]

@muyncky We are opened to contributions so don't hesitate to contribute to our documentation.

@Progi1984 I have posted an issue #17011. Once this is solved, as promised before, I'm going to provide an elaborate documentation on how to deploy with Deployer.

I think Deployer could be a good tool, because it is the only tool written in php. It makes the learning curve less steep. Deploying Laravel with Deployer is flawless.

@PierreRambaud For me Puppet and Ansible seem rather abstract, search result for PrestaShop gives me really obscure Github repo's. I'm not a hardcore DevOps kinda guy.

Haha, I understand about the obscure world the automation is :sweat_smile:
But I'm not a huge fan of using PHP for deploying, but I understand backend developers like the same technology as they learned back in days.

[PierreRambaud]

FYI you can contribute to the DeployerPhp tool if you need specific usecase https://github.com/deployphp/deployer/blob/master/recipe/prestashop.php

[muyncky]

@PierreRambaud I will contribute, when I have something that works.