Deployer: Deployer unable to find composer installed recipes

[jamieburchell]

| Q | A
| ----------------- | ---
| Issue Type | Question
| Deployer Version | 6.0.3
| Local Machine OS | WSL
| Remote Machine OS | CentOS 7

Description

Deployer fails to find and require recipes installed via the method included in the third party recipe instructions

Steps to reproduce

Install recipes:

composer require deployer/recipes --dev

Add a recipe as per docs:

require 'recipe/cachetool.php';

Run deployment:

dep deploy production
OR
php deployer.phar deploy production

Result:

PHP Warning:  require(recipe/cachetool.php): failed to open stream: No such file or directory in /mnt/d/web/deploy.php on line 3
PHP Fatal error:  require(): Failed opening required 'recipe/cachetool.php' (include_path='phar:///mnt/d/web/deployer.phar/bin/../:.:/usr/share/php') in /mnt/d/web/blog.ib3.uk/deploy.php on line 3

Content of deploy.php

<?php
namespace Deployer;

require 'recipe/composer.php';
require 'recipe/cachetool.php';

// Project name
set('application', 'Example');

// Project repository
set('repository', '[email protected]:foo');

// Hosts
host('xxx')
    ->stage('production')
    ->user('deploy')
    ->set('deploy_path', '/var/www')
    ->set('cachetool', '127.0.0.1:9000');

// Shared files/dirs between deploys
add('shared_dirs', [
    'web/app/upgrade',
    'web/app/uploads',
    'web/app/wfcache',
    'web/app/wflogs'
]);

// Copy config
task('enable_prod_config', function() {
    run('cp "{{release_path}}/.env.prod" "{{release_path}}/.env"');
})->onStage('production');
after('deploy:update_code', 'enable_prod_config');

// Clear opcache
after('deploy:symlink', 'cachetool:clear:opcache');

// [Optional] Allocate tty for git clone. Default value is false.
set('git_tty', true);

// [Optional] if deploy fails automatically unlock.
after('deploy:failed', 'deploy:unlock');

Worked around the issue thus:

ini_set('include_path', ini_get('include_path') . PATH_SEPARATOR . 'vendor/deployer/recipes');

Should that be part of the documtation or is something broken?

[antonmedv]

I think it's a bug

[antonmedv]

Can you find and fix it?

[jamieburchell]

I can’t see how Deployer was designed to find the composer installed recipes. Presumably, it needs to know where the vendor directory is (read composer.json/assume default?) and to add the path to those recipes to the include path? Or the expectation on the user is to simply provide the full path to the third party recipes. Perhaps, if Deployer is run and installed using the local composer method, it would find the recipes?

[antonmedv]

It's using include path.

[jordanbrauer]

I am having this problem as well.

Perhaps recipes should be redesigned in how they are implemented? My initial thought would be to do something similar to how Symfony handles extension – simple abstract classes and interfaces.

Deployer already uses Symfony Console, so why not utilize the class autoloading of composer? It will make extension much easier I think.

[antonmedv]

@jordanbrauer what do u suggest?

[jamieburchell]

At a minimum update the documentation to show that the include path needs to include the deployer recipes directory. As it stands all the examples and documents suggest it "just works" without, when it doesn't.

[antonmedv]

Hmm, maybe better to fix include path of all repos? :) I'll do it.

[jordanbrauer]

@antonmedv I would handle recipes the same way Symfony handles their single components and autoloading.

• Have it so that a user only needs to require deployerphp/deployer for the core functionality (common recipe)
• Any additional recipes should be required as singletons (again, akin to Symfony single components), _e.g._, deployerphp/rsync
  
  
  
  • This allows users to select only the recipes they really need for their project
  
  
• To use a recipe, simply require/use the file/namespace in your deployer file (again, see usage of Symfony standalone components if you need an idea of what I mean)

Ideally, this would mean ditching the use of require "recipe/rsync.php"; in favour of use Deployer\Recipe\Rsync;. Since deployer seems to work off of global functions, using use will work fine for importing.

[jamieburchell]

Would this still work if like me, you've installed the deployer.phar in to the project root and not via composer? That is the first installation method shown on the install page.

[jordanbrauer]

Good question – I am sure that there would be a way to make it work yes. As an example, when you add entries to the "scripts" key of a composer.json file, Composer will update it's set of commands, adding yours to it (try it by running composer list, and you will see your commands).

My point is, is that if a PHP CLI application is installed globally via a .phar rather than as a local composer binary, the application can still be extended for a local instance only.

[antonmedv]

I think we should keep it simple and do not reinvent the wheel. Use what already in php and composer.

[jordanbrauer]

I am not sure I understand @antonmedv? None of what I mentioned requires reinventing the wheel and uses default composer autoloading with native PHP imports (use). 😮

[antonmedv]

It's impossible to autoload tasks. In php and composer we can autoload only classes (not even functions, but you can use function ).

I think something like this will be good:

require 'vendor/autoload.php'; // Or move it to deployer and automatically detect
require 'common.php'
require 'slack.php'

All recipes have to provide autoload.php file with set_include_path to composer.json.

[jordanbrauer]

Ahh, you are right! My bad.

I think that user should be responsible for handling the inclusion of their autoload file. This way they are able to bootstrap however they please, even if they are not using composer.

[kszymukowicz]

From my experience doing require 'vendor/autoload.php'; from deploy.php level can make a conflicts with deployer packages.

The reason is that composer uses spl_autoload_register(..., true, true); The important is the second 'true' param in spl_autoload_register function. Its "prepend" and means the classes searching for this spl_autoload_register will be done before other registered ealier (by composer autoload from deployer in that case). That means that if deployer will want to create symfony console object then it will use class from root project vendors and not from itself vendors (inside phar). That of course can lead to problems if there are different version of symfony console for deployer phar and for root project.

The solution I know is to make light loader and register it with spl_autoload_register but without the third "prepend" param set to true. Then if there are the same libraries used in deployer and root project - deployer will use itself library first.

The example of such loader which uses information generated by composer itself is in this little library: https://github.com/sourcebroker/deployer-loader

I'd be glad to hear for better solution.
So far its working for me well.

[antonmedv]

I think it will be cool to have possibility to include recipes like this without worried about autoload:

require 'common.php'
require 'slack.php'

Deployer already has autoload.php detection https://github.com/deployphp/deployer/blob/master/bin/dep#L52-L56
So it should work if proper include paths will be configured.
In case when deployer installed globally and locally, local version should be used (dep automatically will switch to local version). In this case recipes installed by composer will be used and this is correct.

[antonmedv]

So now, then using phar one think should be done is requiring projects autoload.php

[antonmedv]

As another solution: fork all used symfony components and rename namespace.

[kszymukowicz]

@antonmedv
How the "deployer already has autoload.php detection" should work then ?
I can not create object from my vendor class if I have no require 'vendor/autoload.php' (or my autoload) in deploy.php. So for my understanding the autoload from root project is not loaded.

[antonmedv]

So for my understanding the autoload from root project is not loaded.
Here is a problem: when using phar it will load only phar autoload, when composer installed – project root.

Yes, it's true.

One more solution: when building phar archive create a bootstrap file (with all needed classes) and require it as first line in bin/dep file. After load project root's autoload.php file.
This way symfony classes will be already preloaded and no need of deployer's autoload.
I like this approach better when others. But it may have a bug: if using something in deploy.php what is relaying on older symfony components when bootstraped with deployer.

[kszymukowicz]

@antonmedv
IMO there will be never problem with adding require 'vendor/autoload.php' to deploy.php when you install deployer with composer require deployer/deployer as at this level developer already needs to solve the potential conflicts.

Problem will only occur ONLY when we use phar based deployer. For that case there is simple solution however. Deployer needs to detect if there is vendors/autoload.php from root project and if yes they it should require that file BEFORE require autoload from phar. This way the deployer classes will have precedence over that one from root vendors.

[antonmedv]

Deployer needs to detect if there is vendors/autoload.php from root project and if yes they it should require that file BEFORE require autoload from phar.

Yes, this is a solution too. I think I'll try to implement this.

[antonmedv]

After lots of thinking, here is what I came up with:

Installing with composer

• Deployer will try to locate autoload.php file (local or global) and require it.
• Scan vendor dir for composer.json files and fetch stuff like this:
  js "extra": { "deployer": { "include_path": "recipe/" } }
• Next add all found include_path to set_include_path — this allows to simple recipe requirement:
  php require 'laravel.php'; require 'slack.php';

Installing with archive

• Download phar to local deployer.phar or bin/dep.
• Commit phar file to repository (to save version number).
• Save scan on vendor step as on composer installation.
• No autoload.php file required, you need to manually add it to your deploy.php file if needed.
• All required files by deployer included in single bootstrap file inside phar.
• If called global installed dep it will check existing next files in project:
  
  
  
  • deployer.phar
  
  
  • bin/dep
  
  
  • vendor/bin/dep
  
  
• If on of this files found, call it instead of global on (this allows global dep (for example of version v7) to call project installed deployer).

[ihipop]

I build a docker image for CI/DEPLOYER there with deployer/recipes installed globally and deployer is install with phar

composer global require deployer/recipes --dev 

then in my deployer.php of project

$composerHome = getenv("COMPOSER_HOME") ?: (getenv("HOME") . '/.composer/');
if (is_dir($composerHome)) {
    include $composerHome . '/vendor/autoload.php';
} else {
    $xdgComposerHome = getenv('XDG_CONFIG_HOME') ?: (getenv("HOME") . '/.config');
    $xdgComposerHome = $xdgComposerHome . '/composer';
    if (is_dir($xdgComposerHome)) {
        include $xdgComposerHome . '/vendor/autoload.php';
    }
}

It Just works well

@antonmedv

[barryvdh]

Can't you just include the recipes in the phar? Or build a phar with and without plugins? I'm not sure how many people are building custom recipes anyways?
I'm also having trouble including the Slack recipe when using the phars.. (Because of Symfony conflicts etc)

[antonmedv]

I would like to keep them separately. And definitely going to this it for phars and composers require 'recipe.php'.

But may be popular stuff like slack should go to main repo.

[staabm]

Can't you just include the recipes in the phar? Or build a phar with and without plugins? I'm not sure how many people are building custom recipes anyways?
I'm also having trouble including the Slack recipe when using the phars.. (Because of Symfony conflicts etc)

I got the feeling that you need to composer require the deployer in your project (and also the custom recipes/plugins/classes etc) in case you need something more customized then the phar.

IMO using the phar only works for rather simple/standard use-cases... most use-cases need more flexibility and dont work with the phar.

[barryvdh]

yes but that's pretty difficult with conflicting dependencies, eg. like symfony console and Magento2.

I haven't looked at most recipes, but I guess they have similar dependencies (like the HTTP utilities), so it wouldn't impact the phar too much to include Slack alone, or alle of them, right?

[antonmedv]

Yes, but this is wrong path. See my note upper.

[hooch]

Example remedy:

require 'vendor/deployer/recipes/recipe/rsync.php';

[jkobus]

Updating phar on our build server broke our deployment process due to:

dep6 deploy test -vvv
PHP Warning:  require_once(recipe/symfony4.php): failed to open stream: No such file or directory in /home/xxxxx/jenkins-slave/workspace/xxxxx.com_develop-

These recipes were present in that phar previously?

[antonmedv]

What version of dep?

[jkobus]

Well it seems that this problem is somewhere on my side, although I have no idea why this happens.
I downloaded deployer from this url - its 6.0.5:

https://deployer.org/releases/v6.0.5/deployer.phar (in bin as dep6)

I also have two other versions in bin:

6.3.0 (as dep)
4.3.1 (as dep4)

I only updated the 6.3.0 as two others are version-locked for compatibility.

The one used in deployment that went broken is 6.0.5 (dep6).

Im wondering if its possible that in bin I had an old phar with this receipt and after update to 6.3 it was removed. My include path during install:

PHP Fatal error:  require_once(): Failed opening required 'recipe/symfony4.php' (include_path='phar:///usr/local/bin/dep6/bin/../:.:/usr/share/php') in /home/xxx/jenkins-slave/workspace/xxx.com_develop-BE4ZFIKIUWRUVBXYXPR6E5MC2IFLQN4GNM2FRXXL35N7JLIBZUEA/deploy.php on line 9

[antonmedv]

include_path looks normal. Try running this script locally.

[davefarthing]

Just thought I'd share my experience with php 7.0.32:
I had the same problem require(): Failed opening required 'recipe/ .......

I finally traced it to the presence of the ioncube_loader extension. Disabled ioncube and everything works normally.

[ihipop]

include_path looks normal. Try running this script locally.

why not just include the autoloader？ https://github.com/deployphp/deployer/issues/1371#issuecomment-359814638

[alinalexandru]

I am having same problem caused by ioncube.
Everything works perfectly without ioncube

[alinalexandru]

This is how I "solved" the issue

require 'phar:///usr/local/bin/dep/recipe/common.php';

Even though the get_include_path() was
phar:///usr/local/bin/dep/bin/../:.:/usr/local/lib/php

[antonmedv]

Well. This one is over. Almost 3 years since opening. Now recipes gonna live in deployer repo: https://github.com/deployphp/deployer/tree/master/contrib

[jamieburchell]

What's the status of this please? I just noticed deployer/recipes is abandoned but including recipe/cachetool.php without them doesn't work for me and the composer and CodeIgniter recipes I use in older projects are missing it seems?

Was the recipe repo prematurely archived and abandoned before v7 of Deployer was released?