Pim-community-dev: Avoid reading all the dependencies of pim-commutiny-dev

[nietzscheson]

Hi, guys.

I know that the title is very confused.

I understand how composer works (maybe not very well) but it is quite tedious to update the vendors of the project. Composer reads all dependencies in pim-community-dev. I understand that this happens when there are private dependencies and they are not public in the packagist. Is there a way to optimize?

Thanks...

[damien-carcel]

Hello @nietzscheson.

Indeed, composer needs to scan all the dependencies, and on the first run it is always quite long, as the PIM has quite a lot of dependencies. However, there is several things to do to speed up the process.

Composer cache

First, I see on your screen shot that you are using our Docker images with Docker Compose. But your composer cache and configuration are not set properly, as you have a warning about the cache directory that is not writable.

This is very likely because you kept the default configuration (with composer home folder being in ~/.composer/, and the cache inside this home folder). And in fact, this folder does not exist on your computer, so Docker created it for you, but with root access only (it is Docker normal behavior), so composer cannot write cache in it as the container run with a user named docker.

You need to locate the configuration and the cache directories on your machine as mentionned in our doc. They are probably in ~/.config/composer and ~/.cache/composer.

If it is the case, then your fpm service in the compose file should look like that (I removed some stuff, keeping only the relevant environment variables and volumes):

services:
  fpm:
    environment:
      COMPOSER_CACHE_DIR: '/home/docker/.cache/composer'
      COMPOSER_HOME: '/home/docker/.config/composer'
    volumes:
      - ~/.cache/composer:/home/docker/.cache/composer
      - ~/.config/composer:/home/docker/.config/composer

If it does not exist at all, then you can keep the default compose file, but you need to be sure your container can write in the composer folder by running sudo chown -R 1000:1000 ~/.composer.

By doing that, you'll have access to a composer cache in your container, which will considerably accelerate future installations.

composer.lock

If you are working on a project, using the akeneo/pim-community-standard package, after running the install for the first time, you should commit your composer.lock and run composer install instead of composer udpate.

This will provide a very fast install, and you ensure that all your project deployments use the same vendors.

Kind regards,
Damien

[oliverde8]

I have trouble understanding why Akeneo isn't on packagist? That would make the update faster as it won't need to switch on each branch individually, because even with caches doing an update is very long.

There is another issue about it here #7689

[nietzscheson]

@damien-carcel thanks for the info. I'm agreed with @oliverde8, and consider that is a "problem". My question is how Akeneo behaves in CI/CD processes (Travis-CI - Jenkins) When everything, each new build is built again...

[oliverde8]

@nietzscheson You shouldn't be doing composer update in a CI process but a composer install so the issue doesn't present itself.

Devs also should not run composer update, you need to run that only when you actually need to update something which is not something you would do that often. Most of the time it's just a composer install

But still, when actually doing an update the packages not being on packagist means (even with cache) that composer will checkout in each branch individually to see the dependencies vs retrieving all the data at once from packagist.

[damien-carcel]

Hello @oliverde8 and @nietzscheson.

The reason Akeneo is not on Packagist is because akeneo/pim-community-dev is a full Symfony application we use for development and run the CI on. So we don't want to add it on Packagist because we don't want people to think it is the main Akeneo entry point. This entry point is akeneo/pim-community-standard, that is dedicated to projects (with or without integration). But I agree having Akeneo on Packagist would make update faster.

Maybe one day we'll extract the content of src from akeneo/pim-community-dev to split it in a read-only repository, as Symfony does with their bundles and components. Then this read-only repo would be on Packagist. But it is not planned for now.

Concerning composer update or composer install, it depends of what you are doing. If you put in place a CI for a client project, indeed you must use install to ensure you always install the same thing everywhere.

But in our development process of the product, we use akeneo/pim-community-dev, and need to always use composer update to ensure we tests the latest versions of our dependencies (except a few that we explicitly fix in composer.json and update only when we decide too, like Doctrine or Symfony).

Regards,
Damien.

[nietzscheson]

Hello ... @damien-carcel and @oliverde8 again with this.

I think it would be important to give priority to moving from Akeneo to Packgist. The Sylius project has done well. They have the Sylius base and a standard version for production projects. It's much faster...

• Sylius on Packgist
• Sylius-Standard

Today, I am having problems with CI/CD for Akeneo in Gitlab. Only building the whole project (composer and node) it takes up to 9 minutes or more (when caching could last 19+ min)... when in other projects, the construction would take 2 ~ 5 (maximum).

Sylius build

Do you plan to move to version 4 of Symfony?

[damien-carcel]

Hi @nietzscheson .

akeneo/pim-community-dev is now on packagist, since a couple of days. We finally did it to speed up the install of the dependencies.

Regarding Symfony 4, Akeneo is always based on LTS versions of Symfony. It means we will not move on Symfony 4 before its 4.4 release, and only on a major version of Akeneo. This means it will not be before Akeneo 4, and more probably Akeneo 5.

Regards,
Damien

[nietzscheson]

Thanks @damien-carcel for the info...

I just tried akeneo-community-dev from Packagist and the times in CI (Gitlab) improved:

Now: Duration: 7 minutes 9 seconds.
Latest: 08:01 - 09:12 - 11:17 - 12:48

[hayuns]

Hi,

1. I'm using the pim-community-standart and i don't have the cache folder. I downloaded the project from the link. I have at the root folder: - compose.json, compose.lock and docker-compose.yml.dist. I know I need to copy it as docker-compose.yml but firstly I want to follow step by step. What I need to do to set my compse and configuration properly?
2. My errors in my prevent installations were database is unknown and then I succeed to resolve it I had an error like "don't find the test file".
   Can you please help me?