Drupal-project: Do not ignore composer generated files

[rsanzante]

Composer it's a great tool for dependency management, but is it a good tool for deployment? There're good reasons to think is not. This article explains why is not a good idea to use Composer to deploy applications and why it's better to commit vendor directory.

I'll highlight some parts:

Why are you doing all this [Running Composer in production to recreate the vendor dir]? To "reproduce your initial files layout exactly how they were". Then why don’t you just commit the files and push them, then?

So, to sum it up, if you use Composer to build your code in production, you get:

• (-) Un-needed and time consuming deployment complexity increase, with small but real risks of failure on each and every build for external cause

• (-) No auditing of changes that are not your own custom code

• (+) Easier handling of .git « false » submodules for a few dev dependencies

On the other hand, if you commit the "vendor" directory, you get:

• (+) Easier and straightforward deployment

• (+) All code that lands on production gets audited/versioned

• (-) Small amount of work involved in dealing with possible .git « false » submodules

This article has been endorsed by Merlin of Chaos, Damien McKenna, Dave Reid‏ , sun and Wim Leers among others. Please see this tweet: https://twitter.com/merlinofchaos/status/844271616225787904

This project states in the FAQ:

Should I commit the contrib modules I download?

Composer recommends no. They provide argumentation against but also workrounds if a project decides to do it anyway.

According to the linked article I think is time to change this and say: yes. The reasons given by the Composer team are discussed in the article, but the main point is Composer recommendations are for _distributing_ software, not for _deployin_ a web app. This project, drupal-composer, is aimed to be the base for a web app that lives, not to build a project to be distributed.

Also, .gitignore file should changed to not ignore Composer generated directories.

Ah, some say that Composer may clone complete repos in the vendor dir and it's not a good idea to commit them. I agree, but I guess it's easier to find a solution for this (may be ignoring .git directories?). I would like to see this, but I don't know a PHP package that does this. If you know anyone please tell in the comments.

[worthwhileindustries]

"Un-needed and time consuming deployment complexity increase, with small but real risks of failure on each and every build for external cause"

If you are worried about things failing or bandwidth then you can create your own internal repo to store and to pull your dependencies from instead of relying on external sources.

"No auditing of changes that are not your own custom code"

They have repos of their own you can diff.

[worthwhileindustries]

Also one more thing to note is that decoupling it as it is now allows that vendor/module directory to change as it goes up the build chain. I might not want Devel module or some other dev dependency pulled in when it goes to upper environments so, you don't want to commit everything to the repo. You lose flexibility. This all depends on your resources though. If you don't have the resources to manage configurations and deployment then modify the .gitignore on a per project basis.

[frob]

I consider all of these issues valid. However, this project is more about the composer way and ignoring all the the dependencies is the composer way. My suggestion is to fork this repo when creating your project and modifying as you need.

This project is a starting point and you shouldn't expect non-breaking upstream changes. I change the .gitignore to not ignore all the composer generated files so that we do not have to maintain two repos: one working and one for deployment artifacts.

One big problem with this is that composer's installed.json file is created non-deterministically. But there is a PR that fixes that https://github.com/composer/composer/pull/6696

[worthwhileindustries]

@frob Totally agreed and more to the point. However, I'm not sure what exactly you are downing my comments for as if what I stated is a bad idea.

[worthwhileindustries]

Also, albeit a different article to read for npm but, the industry has figured out that this is generally a bad idea and warrants smell. There's also a term for it called "vendoring" http://blog.bithound.io/why-we-stopped-vendoring-our-npm-dependencies/

[frob]

@worthwhileindustries I thumbs downed your comment because it is dismissive and does nothing to actually answer the question or even deal with the concerns brought up by the original post.

I have heard this a thousand times from composer people and frankly, sometimes vendoring isn't a bad way to go, especially with legacy code. It could, in fact, be the only reasonable way to go. Brushing off these concerns helps no one.

You will notice I didn't thumb down your second comment which was far more constructive.

[rsanzante]

worthwhileindustries:

If you are worried about things failing or bandwidth then you can create your own internal repo to store and to pull your dependencies from instead of relying on external sources.

Well, or just a use a successful approach: use git. It's far simpler, you don't need to setup an infraestructure because you are already using git.

They have repos of their own you can diff.

Yes, but it's obviously easier if you have those changes with just a command (git diff) instead of having to go to every repository and diff them.

I might not want Devel module or some other dev dependency pulled in when it goes to upper environments so, you don't want to commit everything to the repo. You lose flexibility.

Then do not commit that dependency. Or even you can commit them to another branch. Flexibility is not affected.

This all depends on your resources though. If you don't have the resources to manage configurations and deployment then modify the .gitignore on a per project basis.

I DO have resources to manage configurations and deployments, but this is not related. The real problem is deploying code in a certain release, that's a code snapshot. It's same you have just a little project or thousands of them.

frob:

However, this project is more about the composer way and ignoring all the the dependencies is the composer way.

That's what the article says at the end. Composer recommendations are for distributing software, not for deploying a web app.

About the article you link: In my team we have been committing composer generated files without running into those issues. Anyway you don't have to review changes in vendor dir. Committing generated code allows you to check them if you want, but you don't need to. Following the Composer recommendation only the person that commits the changes in the json file reviews the changes (well, it should). Applying patches to vendor code could be a good point. I prefer to have a file with patches applied (same you have to do with composer, but less 'formal'). The slow composer process of regenerating files and the dependency of third party infraestructures scare me more than maintaining that file. The uninstall issue seems a lazy-worker issue. Delete it from composer file and update composer.

[weitzman]

On one of my projects, tracked vendor code for a while and then decided to stop. The main issue is the Github PR UI. We rely on that for our code review process. Many PRs had thousands (or tens of thousands) of files in them, and only a few changes that matterred. It became hard to find the files that matter and focus on their changes in the presence of so much noise. The Guthub UI refuses to show you more than 3000 changed files IIRC. Our quality was sufferring.

So, there are good philosophical reasons to keep /vendor code in git but as a practical matter it worked poorly on my project.

[rsanzante]

weitzman: it seems that the problem was the GitHub UI. I agree, but drupal-project is used as base for your own projects that can be hosted in any other git provider or even do not use a git web frontend at all. So I think is not a good reason to maintain vendor dir ignored.

[jcnventura]

@rsanzante Anyone running code reviews will be horrified if vendor dir is un-ignored. I already suffer enough with composer.lock, and just wish there was a way to hide that by default on git diff and github PRs.

This project should cover the needs of the majority. If you want to comment out the 1 line in .gitignore that ignores /vendor/ go ahead and commit that in your projects. The project exists for a few years already, and I think you're the first one to suggest adding /vendor to the repo.

BTW, if you follow that, you should probably delete most of .gitignore, as you should also commit all of drupal core and all the contrib modules and themes. Your .gitignore should only ignore the settings files and sites/*/files.

[rodrigoaguilera]

It seems very non-intuitive to keep the history of other repos in your own repo bringing unnecessary growth to the size of the repo.

I feel composer is not a tool for deployment (as is stated in the first post) nor should be git or any other VCS. Deployment tools nowadays are sophisticated enough this to handle these cases and don't involve having or running composer in production.

And from personal experience I can say I started vendoring and now I can't even think about going back there.

[jcnventura]

I totally get the composer is not for deployment. But then, I wonder if git is, as @rodrigoaguilera also stated. You don't need the entire repo history in the deployed server. The only reason to use git in deployment is for configuration management. To put it another way, I hear that somewhere in Drupal's older commits are passwords by core maintainers. It's a very common mistake to do. If for some reason that leaked into your repo at some point in time, should it be left there for hackers to use?

The best solution would be a deploy command that can do a binary diff between files in the previous deployment and the new one, that would efficiently deploy the new configuration. The only files kept in the deployed system would be the generated project snapshot, and maybe a tamper-proof file of the last deployment state (extra points for a system able to detect local file changes).

[frob]

Honestly, I don't find this type of discussion constructive at all. Composer has a recommended workflow which doesn't involve committing the dependencies and this project is a starting place for bringing together a Drupal composer workflow. That said, I start here and then remove those lines from the .gitignore on every Drupal 8 project. I commit the dependencies and I don't look back.

Git isn't for deployment, but is a damn good tool for that. I use git for deployments. Tags as snapshots of working code made ideal deployments.

[jcnventura]

Yes, I've been reading about it, actually. There's a lot of people using a deploy-only git repo. You develop on your development repo, then you do a 'git archive' into the deployment repo. Best of both worlds, without any intermediate development history (aka leaked passwords in settings.local.php).

[alcohol]

Composer is not a deployment tool. Neither is Git. Committing your vendor directory is just silly. Read up on CI/CD and build/deploy/release artefacts.

[jcnventura]

I think we're converging on the consensus that ignoring the composer-generated files is the right solution within the scope of this project. As such, I'll close this issue.