Prestashop: Expose a single public/ or web/ folder in order to allow keeping dev-only unprotected files

[elboletaire]

Describe the bug

When downloading a release from github or the official prestashop website, the downloaded zip file contains files like composer.lock and package-lock.json, but the main .json files are removed, making it completely unusable for most of the use cases.

Right now, if you upgrade or install from zip (this also affects docker, as it uses zip files for the installation), you can't run commands like npm run build, because it cannot file a build script, as the file does not exist.

Expected behavior

To have these files, so we can properly do stuff like compiling the assets, as it's described in the official documentation: https://devdocs.prestashop.com/1.7/development/compile-assets/

Steps to Reproduce

Steps to reproduce the behavior:

1. Install prestashop or upgrade a current version using a zip file either from the releases page here at github, or from the official prestashop downloads page.
2. In the installation root folder, try to run composer update or composer require somepackagename.
3. You won't be able, as there's no composer.json file.

You can repeat this process, but changing the root folder by one of the admin's themes paths, and using npm run build or any other expected script instead. You'll see the same problem there, as package.json files are not included.

Screenshots

Additional information

• PrestaShop version: > 1.6
• PHP version: Does not affect.

[prestashop-issue-bot[bot]]

Thanks for opening this issue! We will help you to keep its state consistent

[khouloudbelguith]

Hi @elboletaire,

Thanks for your report.
Ping @PrestaShop/prestashop-product-team what do you think of this improvement?

Thanks!

[matks]

Hi, removing these files is intentional because the ZIP is built to be production-ready. In production, you never run npm or composer.

If you are willing to use composer or npm on prestashop to modify some of its elements, I think it will a lot more efficient to close the git repository. You will benefit for a lot of dev-only resources such as tests (you can run them to monitor whether your changes are breaking something), linters, scripts to (recently we added phpstan) ... and you can even build a ZIP similar to the official ZIP using the embedded Release Creator tool https://github.com/PrestaShop/PrestaShop/blob/develop/tools/build/README.md

[elboletaire]

Yeah, but here's the truth: I have to work on an already production project which was installed using the normal procedures. Now I need to change code stuff, and I cannot due to all these files being removed.

I'm pretty sure there's plenty of people out there with the same problem: they're asked to take a project which was installed following production procedures, and then they stuck due to the lack of these files.

I cannot move now to the git version, due to how the installation is working these days (and due to many related issues, like the composer installation done in the install process).

Any ideas or workarounds for moving from a non-git project with a lot of modules and stuff installed to a git project (speaking about prestashop, ofc...)?

Edit: note that I do understand why they are removed; but I'm asking for a proper workflow with prestashop, which currently is broken both in dev and production (from my point of view).

[elboletaire]

On the other hand, what's the reason keeping the .lock files without their respective json files? Neither manager works properly without their json files AFAIK.

[matks]

On the other hand, what's the reason keeping the .lock files without their respective json files? Neither manager works properly without their json files AFAIK.

We initially removed it as well but we found out that Symfony Requirements Checker uses the composer.lock file as part of its process, when triggered by the Installer, to look for server configuration issues (see https://github.com/PrestaShop/PrestaShop/pull/14739).

So we keep it for now but in the long term it will be gone indeed.

[elboletaire]

What about the workflow? When I say the dev workflow is broken, I'm not speaking about prestashop's dev workflow. Obviously the way you work works for you.

But if I, as a shop developer, want to start creating a prestashop shop and making commits to my own repo to keep the (soon in production) project up to date, there are things that will broke the shop: modules and vendors are ignored, but some of them are installed during the installation process (I know there's a fix on the way for this, but takes too long), so I can't ignore them in my development project, otherwise the project won't work due to missing modules and vendors.

Anyway, there's the other (very common) case: I have to start working on a shop which was directly installed on the server. What I did was to cherry pick the .json files from the project, but I had to ensure those files were the proper ones (browsing files of the exact same tag my shop was installed) to ensure no different versions were installed. Even though I had downloaded what seemed to be the proper version, as soon as I ran composer install it said composer.json file was updated, forcing me to run composer update and updating packages.

So... could you reconsider keeping these files? I don't see why keeping those json files in the production tarballs was banned in the first place, I find more cons than pros in removing them.

Edit: In fact, I don't see any pros in removing these files.

[matks]

What about the workflow? When I say the dev workflow is broken, I'm not speaking about prestashop's dev workflow. Obviously the way you work works for you.

But if I, as a shop developer, want to start creating a prestashop shop and making commits to my own repo to keep the (soon in production) project up to date, there are things that will broke the shop: modules and vendors are ignored, but some of them are installed during the installation process (I know there's a fix on the way for this, but takes too long), so I can't ignore them in my development project, otherwise the project won't work due to missing modules and vendors.

If we follow your mind, you are describing 3 environments:

• prestashop open source project development, which aims to provide an environment where code changes are built then submitted as PRs
• shop development, which aims to build a shop on top of prestashop, and code changes aim to be pushed into production eventually
• shop production

The .gitignore in this repository is targeted for the project development, which is why it does not fit your needs. You are however free to modify it to fit your needs.

AFAIK there are multiple "trends" about modules and vendor folder, whether to version them or not. Some people add them to main git shop repo, some keep them apart, some do not store them into git. I think it highly depend on your own workflow.

Anyway, there's the other (very common) case: I have to start working on a shop which was directly installed on the server. What I did was to cherry pick the .json files from the project, but I had to ensure those files were the proper ones (browsing files of the exact same tag my shop was installed) to ensure no different versions were installed. Even though I had downloaded what seemed to be the proper version, as soon as I ran composer install it said composer.json file was updated, forcing me to run composer update and updating packages.

So here we are in your "shop development" environment. You actually did the right thing: fetching .json files from sources to add them into your project. How did you pick them ? Did you target the git tag ? I dont understand why it said it was outdated.

So... could you reconsider keeping these files? I don't see why keeping those json files in the production tarballs was banned in the first place, I find more cons than pros in removing them.

Edit: In fact, I don't see any pros in removing these files.

Security, simply. Keeping these files accessible from outside is an Information disclosure.

A long term solution ?

So I understand that removing these files from the ZIP is not helping you setup your "shop development" environment. Right now the main workaround is to copy the .json files into your shop and start building, and it's not a nice workaround (but at least it exists !).

One ideal solution would be to provide 2 ZIP:

• one for the shop development, with not only .json files but also dedicated .gitignore, helper tools, maybe the tests folders
• one for production

Right now we dont have this "shop development ZIP". The idea is nice but it'll take some time to obtain something 100% compliant, however we are already working on this track.

It could be a ZIP or something else 🤔 though. The idea is to have a "prestashop-ready-to-run" environment with developer tools, is that right ?

Keep in mind that even if tomorrow we are able to say "Hello ! we have released PS 1.7.7.4, you can find the production ZIP here and the developer ZIP here !" you will still struggle when working on the 2nd usecase, where you start working on a shop which was directly installed on the server.

[elboletaire]

AFAIK there are multiple "trends" about modules and vendor folder, whether to version them or not. Some people add them to main git shop repo, some keep them apart, some do not store them into git. I think it highly depend on your own workflow.

From my point of view, vendors should NEVER be in the repo. That's why you have a composer.json AND a composer.lock file. The lock file has the exact versions of the vendors to be installed. Why fill hard drives with stuff when it's not needed?

Edit: even composer's official documentation says it's a bad practice: https://getcomposer.org/doc/faqs/should-i-commit-the-dependencies-in-my-vendor-directory.md

Security, simply. Keeping these files accessible from outside is an Information disclosure.

Well, this is another stuff to be debated. Only be the public folder should be accessible by the browser. And now you're using symfony, which embraces this way to work (publish only public path, the others are invisible to the browser, but obviously not for the php process manager). But, as said, this is something we could widely discuss in another thread, not my main issue here.

Right now the main workaround is to copy the .json files into your shop and start building, and it's not a nice workaround (but at least it exists !).

Yes, it's a PITA :smile:

Right now we dont have this "shop development ZIP". The idea is nice but it'll take some time to obtain something 100% compliant, however we are already working on this track.

I do not think there should be two zip files. We would still have the same problem I described in the beginning: people tend to use the production ready zip file. As soon as a developer has to work on an already deployed project, the "development zipfile" is useless, as the work will be the same as today: cherry pick the .json files, because probably the client have changed core files (note that I do NEVER change core files, I've opened multiple issues were I cleary show I work using modules, but I'm not the only prestashop developer in the world, and the latest two prestashop projects I've been working on, had core files heavily modified), and a complete replacement with the development file won't be possible.

I truly believe that the production zip file should be the same used by everyone. As we're already using symfony, how problematic would be to try to change the way the shop is published, and try to move to the proper symfony way? Meaning, only publish the public path to the server.

https://symfony.com/doc/current/setup/web_server_configuration.html

Doing that, we could maintain our .json and .lock files, and also, we could remove those ugly index.php files stored everywhere on every prestashop shop and module.

As a curiosity, prestashop and wordpress, are the only (well known) CMS that do not follow this way to work, having all their files published in the webserver, instead of just the public path, which would be the more secure way to work.

[matks]

AFAIK there are multiple "trends" about modules and vendor folder, whether to version them or not. Some people add them to main git shop repo, some keep them apart, some do not store them into git. I think it highly depend on your own workflow.

From my point of view, vendors should NEVER be in the repo. That's why you have a composer.json AND a composer.lock file. The lock file has the exact versions of the vendors to be installed. Why fill hard drives with stuff when it's not needed?

Edit: even composer's official documentation says it's a bad practice: https://getcomposer.org/doc/faqs/should-i-commit-the-dependencies-in-my-vendor-directory.md

Some people use git as a rsync alternative, I am not saying it's a good practice, just that it can be convenient for some people 😄 .

Security, simply. Keeping these files accessible from outside is an Information disclosure.

Well, this is another stuff to be debated. Only be the public folder should be accessible by the browser. And now you're using symfony, which embraces this way to work (publish only public path, the others are invisible to the browser, but obviously not for the php process manager). But, as said, this is something we could widely discuss in another thread, not my main issue here.

Oh, there is nothing to be debated: the use of public folder is the good way to go. But changing folders locations and usage are a huge BC break so such a change will not come before next PrestaShop major version. Until then we have to find a solution that suits current PrestaShop folder structure.

how problematic would be to try to change the way the shop is published, and try to move to the proper symfony way? Meaning, only publish the public path to the server.

This is actually planned 😄 when the full BO has been migrated to Symfony we will be able to go this way. Until then we have to remain backward compatible.

To conclude

The long term goal is to make PrestaShop a big Symfony application, benefiting from all of Symfony features, standards, tools and usages (which are known to be of high quality). When we reach this point, a lot of architecture issues in PrestaShop will be solved because Symfony has been providing solutions for them since a long time. Rather than re-building the wheel we better be dwarfs standing on shoulders of giants.

However this is a long work, and we cannot expect this day to come before 2021 or 2022. So we try to provide solutions that can be implemented now to help current users 😉 this is why I mention an imaginary 2nd ZIP.

[elboletaire]

But then, this is planned, right? And I guess, in the end, these files won't be removed.

If it's really planned and we'll be able to properly work both as core prestashop developers and as shop developers, I guess I can wait.

If you can confirm this will be actually the way to work, this issue can be closed and/or tagged accordingly.

Edit: Just to be clear, I prefer to close the issue and not add more work load to the current ongoing process. Let's keep things simple, this is the way prestashop has been working for years, people can wait one or two more years.

In case others stumble with this, at least they'll know both the current (ugly) way to work, and also that a new way to work is on the way (the symfony one).

[matks]

Well if you can wait 1 or 2 years it's good news 😄 a lot of people refuse to wait so long. But indeed it's planned and I dont see why we could change our mind on a plan that, in theory, brings only benefits, no issues.

Nobody can tell the future so I will not promise it will 100% happen but this is what we are aiming for

[elboletaire]

Then I leave it up to you what to do with this issue. Maybe the best solution would be to properly tag it and assign it to a milestone, at least to not loose track of it.

[matks]

We have a dedicated Kanban for that: https://github.com/PrestaShop/PrestaShop/projects/6 I put this issue there

[matks]

Renaming the title to follow latest items in the discussion