Liipimaginebundle: Help: resolver AwsS3Resolver / loader: (Gaufrette + AwsS3) => thumb creation takes ages

[faneca]

Hello,

I'm trying to get your bundle to play nice with my setup. Both source images and cache files are to be stored on AWS S3. I'm using Gaufrette as a loader and AwsS3Resolver as the cache resolver.

Things seem to work, but the time it takes to generate the thumbnails is simply unbearable: 24 seconds for two thumbnail versions of a 600kb .jpg, several minutes for a 3.3 Mb .png, etc., with the server CPU rocketing to 100% usage.
Among other things, I routinely hit the PHP's "maximum execution time" when invoking through the web (though app/console liip:imagine:cache:resolve <file> works, just so slowly) and, after that condition happens, things start to get weird: I've seen things like fopen(gaufrette://...): failed to open stream: infinite recursion prevented in the logs on second invocation, just to mention the most esoteric.

Also, falling back to a local cache doesn't appear to improve the situation and, given the CPU usage, it doesn't seem S3 related. So I'm wondering if anyone has hit a similar problem before, if I did something incorrectly, if I'm wrong in my assumption that S3 is not at fault... ¿Could anyone be kind enough to help me, please ;-)?

Here's the setup:

• OS: Ubuntu 14.04.5 LTS
• Nginx: 1.10.1-3+trusty2
• PHP: 5.6.27-1+deb.sury.org~trusty+1
• Symfony: 2.8.4-DEV
• Libraries:
  
  
  
  • Libgd:
    
    $ dpkg --search $(locate gd.so) php5.6-gd: /usr/lib/php/20131226/gd.so libgd3:amd64: /usr/lib/x86_64-linux-gnu/libgd.so.3 libgd3:amd64: /usr/lib/x86_64-linux-gnu/libgd.so.3.0.3
  
  

• Packages:

  • knp-gaufrette-bundle: 0.3.0
  • liip/imagine-bundle: 1.6
  • aws-sdk-php: 3.19

• Environment: dev (didn't try in production), tried both with and without web_profiler: intercept_redirects: false

• Relevant config:

  • config.yml

    (...)
    
    liip_imagine:
        loaders:
            loader_gaufrette_images:
                stream:
                    # This refers to knp_gaufrette filesystems configuration
                    wrapper: gaufrette://filesystem_images/
        filter_sets:
            test_medium_thumb:
                data_loader: loader_gaufrette_images
                quality: 75
                filters:
                    thumbnail: { size: [400, 400], mode: outbound }
            test_small_thumb:
                data_loader: loader_gaufrette_images
                quality: 75
                filters:
                    thumbnail: { size: [100, 100], mode: outbound }
    

  • config_dev.yml

    (...)
    
    parameters:
        liip_imagine.cache_prefix: %amazon.s3.directories.dev.images%
    
    (...)
    
    knp_gaufrette:
        stream_wrapper: ~
        adapters:
            adapter_images:
                aws_s3:
                    service_id: app.amazon.s3
                    bucket_name: %amazon.s3.bucket%
                    options:
                        directory: %amazon.s3.directories.dev.images%
                        create: true
        filesystems:
            filesystem_images:
                adapter: adapter_images
    
    liip_imagine:
    #    cache: ~ # also tried this as mentioned above
        cache: image_thumbnails_cache
    

  • services.yml:

    services:
    
    (...)
    
        app.amazon.s3.credentials:
            class: Aws\Credentials\Credentials
            arguments: ["%amazon.s3.key_id%", "%amazon.s3.secret%"]
    
        app.amazon.s3:
            class: Aws\S3\S3Client
            arguments:
                -
                    version: %amazon.s3.version%
                    region: %amazon.s3.region%
                    credentials: "@app.amazon.s3.credentials"
    
        app.imagine.cache.resolver.amazon_s3:
            class: Liip\ImagineBundle\Imagine\Cache\Resolver\AwsS3Resolver
            arguments:
                - "@app.amazon.s3"
                - "%amazon.s3.bucket%"
                - "public-read" # Aws\S3\Enum\CannedAcl::PUBLIC_READ (default)
                - { Scheme: https }
                - { CacheControl: "max-age=86400" }
            calls:
                - [ setCachePrefix, ["%liip_imagine.cache_prefix%"]]
            tags:
                - { name: 'liip_imagine.cache.resolver', resolver: 'image_thumbnails_cache' }
    

  • routing.yml
(...) _liip_imagine: resource: "@LiipImagineBundle/Resources/config/routing.xml" (...)
  • AppKernel.php
(...) new Knp\Bundle\GaufretteBundle\KnpGaufretteBundle(), new Liip\ImagineBundle\LiipImagineBundle(), (...)

[alexwilson]

Hey, thanks for reporting - This does indeed sound quite strange!

Would you be able to post some more information about your connection to S3, i.e. from where you're connecting, and to which region / etc. In the meantime I will try and replicate (actually using a setup similar to this right now!).

Cheers,
Alex

[faneca]

Hey, thanks for your time!

Well, the server is an AWS EC2 t2.micro in eu-west-1a (Ireland). The bucket is also located in eu-west-1.

Monitoring the server through htop while running app/console liip:imagine:cache:resolve <file>, the only resource that seems to be used to the limit is the CPU itself (being a micro instance, I was afraid that maybe memory was being used over the limit forcing too much swap hits, but that doesn't seem the case).

If you need any other bit of info, let me know ;-)

Cheers

[faneca]

Also, do you have any recommendations to isolate the issue? Any concrete setup you want me to try with Gaufrette and LiipImagineBundle (for example, making every file access local to the server)?

[faneca]

Ok, never mind. I ditched gaufrette in favor of flysystem and everything is working really, really smoothly (well, except for a serializing exception with vichUploaderBundle, but that has nothing to do with this issue nor this repository ;-).
Thanks for your time anyway, @antoligy!

[alexwilson]

Hey, sorry I haven't had a chance to look at this yet. I'm still going to take a look at this as we've got a similar use-case, and will reopen if I can replicate.

Pleased to hear that you've got this working now with Flysystem!

[davidromani]

@faneca can you copy (or explain) how did you configure flysystem as a stream adapter, please?

[faneca]

That was 8 months ago, so the memories aren't exactly clear but, for what I can recall, and if I'm reading my config correctly, I'm not using stream adapters at all.

Also, I'm not quite sure the concept exists in flysytem. Maybe you can take a look at additional bundles, like https://github.com/twistor/flysystem-stream-wrapper, for instance, but (DISCLAIMER) I never used it myself, so I'm in no position to recommend that bundle (or any other one, for that matter).

Wish I could be of more help, sorry.

[davidromani]

Thanks for your answer @faneca

Well, indeed my problem is that I'm storing file uploads in a AWS S3 instance with VichUploaderBundle+Flysystem adapter.

I can obtain S3 stored URLs with this config well:

# Flysystem
oneup_flysystem:
    adapters:
        aws_s3_adapter:
            awss3v3:
                client: app.aws_s3.client
                bucket: "%amazon_s3_bucket_name%"
                prefix: ~
    filesystems:
        aws_filesystem:
            adapter: aws_s3_adapter
            mount: aws_filesystem

# Vich Uploader
vich_uploader:
    db_driver: orm
    storage:   flysystem
    mappings:
        company_image:
            namer:              vich_uploader.namer_uniqid
            upload_destination: aws_filesystem
            uri_prefix:         "https://s3.%amazon_s3_region%.amazonaws.com/%amazon_s3_bucket_name%"

# Liip Imagine
liip_imagine:
    loaders:
        aws_loader:
            flysystem:
                filesystem_service: oneup_flysystem.aws_filesystem_filesystem
        stream_aws_loader:
            stream:
                wrapper: flysystem://aws_filesystem
    data_loader: aws_loader

But Liip data loader it seems that can't undestand where is the image stored because it is building this cache resolve path:
http://.dev/app_dev.php/media/cache/resolve/60x60/https://s3.eu-central-1.amazonaws.com//

Any hint?

[faneca]

Well, that being the case, I'll drop here the relevant bits of my current
config, hoping it'd be of use to you.

I've used a few tricks, the "documentation" I added tries to explain them.

Some notes:

1. Had to remove some private bits, so even if I did my best to
   give you a clean and working example, take into account it can contain errors.

2. I'm using the same bucket in AWSs3 for prod and dev, just adding a prefix
   to the dev case. But I've also included (commented out) the needed alterations
   to configure flysystem to access local files instead (useful to e.g. put dev
   into a Vagrant VM or a Docker container).

3. As a last comment, I'm using my own classes for naming directories and/or objects:

    app.vich_uploader.directory_namer.object_id:
        class: AppBundle\Services\ObjectIdDirectoryNamer

    app.vich_uploader.directory_namer.organization_request:
        class: AppBundle\Services\OrganizationRequestDirectoryNamer

Ok, let's get to business!

Excerpt from parameters.yml.dist:

parameters:
   (...)
    # AWS S3 credentials
    amazon.s3.version: "2006-03-01"
    amazon.s3.key_id: IAM_user_access_key_id
    amazon.s3.secret: IAM_user_access_key_secret
    amazon.s3.region: AWS_region
    amazon.s3.bucket: S3_bucket_name
    amazon.s3.directories.prod.prefix: ~
    amazon.s3.directories.dev.prefix: "dev/" # trailing slash needed

Then, in config_prod.yml:

(...)
parameters:
    files_prefix: "%amazon.s3.directories.prod.prefix%"
(...)

and, equivalently, in config_dev.yml:

(...)
parameters:
    files_prefix: "%amazon.s3.directories.dev.prefix%"
(...)

As for config.yml:

(...)

parameters:
    (...)
    files_prefix.images: "%files_prefix%pictures"
    files_prefix.attachments: "%files_prefix%attachments"
    files_prefix.documents: "%files_prefix%documents"

(...)

#######################################################
# FlySystem + VichUploader + LiipImagine orchestration
#
# Parameters:
#
# - Environment dependent values (defined in "config_*.yml"):
#
#   * "images_prefix" is a "parameter" used to configure (if it's in use) the
#     "liip_imagine.cache.resolver" service (tagged "image_thumbnails_cache")
#     and, of course, everything else here (this value is NOT CURRENTLY USED).
#     Effectively, this means that we have equal configuration for both dev &
#     prod environments, except for a different prefix in the filesystems used.
#

oneup_flysystem:
    adapters:
        # if you want to use local access instead of aws s3, uncomment the
        # following local adapters, comment the next awss3v3 ones, and
        # also change "root_url" in liip_imagine's cache resolver below

        # flyroot_adapter:
        #     local:
        #         directory: "%kernel.root_dir%/../web"

        # profile_user_adapter:
        #     local:
        #         directory: "%kernel.root_dir%/../web/%files_prefix.images%/profile/user"

        # profile_organization_adapter:
        #     local:
        #         directory: "%kernel.root_dir%/../web/%files_prefix.images%/profile/organization"

        # attachments_organization_adapter:
        #     local:
        #         directory: "%kernel.root_dir%/../web/%files_prefix.attachments%/organization"

        flyroot_adapter:
            awss3v3:
                client: app.amazon.s3.client
                bucket: "%amazon.s3.bucket%"
                prefix: ""

        profile_user_adapter:
            awss3v3:
                client: app.amazon.s3.client
                bucket: "%amazon.s3.bucket%"
                prefix: "%files_prefix.images%/profile/user"

        profile_organization_adapter:
            awss3v3:
                client: app.amazon.s3.client
                bucket: "%amazon.s3.bucket%"
                prefix: "%files_prefix.images%/profile/organization"

        attachments_organization_adapter:
            awss3v3:
                client: app.amazon.s3.client
                bucket: "%amazon.s3.bucket%"
                prefix: "%files_prefix.attachments%/organization"

    filesystems:
        flyroot_fs:
            adapter: flyroot_adapter
            mount: flyrootfs
            alias: flyrootfs

        profile_user_images_fs:
            adapter: profile_user_adapter
            mount: userimgfs
            alias: userimgfs

        profile_organization_images_fs:
            adapter: profile_organization_adapter
            mount: organizationimgfs
            alias: organizationimgfs

        attachments_organization_fs:
            adapter: attachments_organization_adapter
            mount: organizationattfs
            alias: organizationattfs

vich_uploader:
    db_driver: fanecadb # just kidding, use mongodb or the like, if needed (or comment it out ;-)
    storage: flysystem
    mappings:
        profile_user_images:
            namer: vich_uploader.namer_uniqid
            directory_namer: app.vich_uploader.directory_namer.object_id
            uri_prefix: "/%files_prefix.images%/profile/user"
            upload_destination: userimgfs

        profile_organization_images:
            namer: vich_uploader.namer_uniqid
            directory_namer: app.vich_uploader.directory_namer.object_id
            uri_prefix: "/%files_prefix.images%/profile/organization"
            upload_destination: organizationimgfs

        attachments_organization:
            namer: vich_uploader.namer_uniqid
            directory_namer: app.vich_uploader.directory_namer.organization_request
            uri_prefix: "/%files_prefix.attachments%/organization"
            upload_destination: organizationattfs

liip_imagine:
    resolvers:
        flysystem_cache:
            flysystem:
                filesystem_service: oneup_flysystem.flyroot_fs_filesystem
                cache_prefix: "%files_prefix.images%/cache"

                # use the following value for amazon, or the next one for local access
                # (if you change this, switch also flysystem adapters above)
                root_url: "https://s3-%amazon.s3.region%.amazonaws.com/%amazon.s3.bucket%"
                # root_url: "/"


    # this cache resolver uses whatever we've configured in flysystem ^^
    cache: flysystem_cache

    # this one points instead to the liip_imagine cache resolver SERVICE we've
    # configured, which uses the aws_s3 client service directly, instead of
    # going through flysystem (and, as a side note, it's configured differently
    # in 'services.yml' for the different environments: prod, dev...)
    #
    # [NOTE: right now we have that resolver commented out in 'services.yml',
    #  as we are currently not using it]
    #
    # cache: image_thumbnails_cache


    loaders:
        # Here we have to use plain, unprefixed access to the flyroot filesystem
        # instead of the specialized, prefixed ones or else we'll have
        # to face problems with liipImagineBundle's interaction with
        # vichUploaderBundle:
        #
        # - setting vich_uploader's "uri_prefix"es to "/%files_prefix.images%/profile/{organization,user}"
        #
        #   would work for
        #   <img src="{{ vich_uploader_asset(user, 'photoFile') }}">
        #
        #   but would fail for
        #   <img src="{{ vich_uploader_asset(user, 'photoFile') | imagine_filter('profile_user_medium') }}">
        #
        #
        # - setting vich_uploader's "uri_prefix"es to "/" would do the opposite:
        #
        #   make imagine_filter work (and also make resolution routes
        #   prettier!) but break vich_uploader_asset() when used alone
        #
        #
        # So the alternatives are either:
        #
        # 1.- To keep the uri_prefix: "/" while remembering to add the URI
        #   prefix manually whenever vich_uploader_asset() is used standalone
        #   in any template
        #
        #  or
        #
        # 2.- To use "/%files_prefix.images%/profile/{organization,user}", then make
        #   liipImagineBundle use unprefixed access (with longer, uglier routes)
        #   and simply forget about doing anything special on top of the usual
        #   template and controller mechanisms
        #
        # We opted for 2, for mental sanity
        #
        flyroot:
            flysystem:
                filesystem_service: oneup_flysystem.flyroot_fs_filesystem

        # profile_user_images:
        #     flysystem:
        #         filesystem_service: oneup_flysystem.profile_user_images_fs_filesystem

        # profile_organization_images:
        #     flysystem:
        #         filesystem_service: oneup_flysystem.profile_organization_images_fs_filesystem


    # here we set *globally* the root (i.e. "unprefixed") flysystem fs, instead
    # of using the specific ones (profile_user_images, profile_organization_images)
    # in each filter (explanation a few lines above)
    data_loader: flyroot

    filter_sets:
        profile_user_medium:
            # data_loader: profile_user_images
            quality: 75
            filters:
                thumbnail: { size: [88, 88], mode: outbound }

        profile_user_mini:
            # data_loader: profile_user_images
            quality: 75
            filters:
                thumbnail: { size: [25, 25], mode: outbound }

        profile_organization_big:
            # data_loader: profile_organization_images
            quality: 75
            filters:
                thumbnail: { size: [950, 558], mode: outbound }
                upscale: { min: [950, 558] }
                relative_resize: { widen: 950 }

        profile_organization_medium:
            # data_loader: profile_organization_images
            quality: 75
            filters:
                thumbnail: { size: [260, 153], mode: outbound }
                upscale: { min: [260, 153] }
                relative_resize: { widen: 260 }

        profile_organization_small:
            # data_loader: profile_organization_images
            quality: 75
            filters:
                thumbnail: { size: [153, 90], mode: outbound }
                upscale: { min: [153, 90] }
                relative_resize: { widen: 153 }

        profile_organization_mini:
            # data_loader: profile_organization_images
            quality: 75
            filters:
                thumbnail: { size: [25, 25], mode: outbound }

And finally, excerpt from services.yml:

services:
    (...)
    app.amazon.s3.credentials:
        class: Aws\Credentials\Credentials
        arguments: ["%amazon.s3.key_id%", "%amazon.s3.secret%"]

    app.amazon.s3.client:
        class: Aws\S3\S3Client
        arguments:
            -
                version: "%amazon.s3.version%"
                region: "%amazon.s3.region%"
                credentials: "@app.amazon.s3.credentials"

    # NOT USING THIS CACHE CURRENTLY
    # app.imagine.cache.resolver.amazon_s3:
    #     class: Liip\ImagineBundle\Imagine\Cache\Resolver\AwsS3Resolver
    #     arguments:
    #         - "@app.amazon.s3.client"
    #         - "%amazon.s3.bucket%"
    #         - "public-read" # Aws\S3\Enum\CannedAcl::PUBLIC_READ (default)
    #         - { Scheme: https }
    #         - { CacheControl: "max-age=86400" }
    #     calls:
    #         - [ setCachePrefix, ["%images_prefix%/cache"]]
    #     tags:
    #         - { name: 'liip_imagine.cache.resolver', resolver: 'image_thumbnails_cache' }

    app.vich_uploader.directory_namer.object_id:
        class: AppBundle\Services\ObjectIdDirectoryNamer

    app.vich_uploader.directory_namer.organization_request:
        class: AppBundle\Services\OrganizationRequestDirectoryNamer

[davidromani]

@faneca many thanks for your detailed explanation, your time and your effort. Finally I could resolved it with this simple strategy (enough for my use case):

# config.yml

# Flysystem
oneup_flysystem:
    adapters:
        aws_s3_adapter:
            awss3v3:
                client: app.aws_s3.client
                bucket: "%amazon_s3_bucket_name%"
                prefix: ~
    filesystems:
        aws_filesystem:
            adapter: aws_s3_adapter
            mount:   aws_filesystem

# Vich Uploader
vich_uploader:
    db_driver: orm
    storage:   flysystem
    mappings:
        company_image:
            namer:              vich_uploader.namer_uniqid
            upload_destination: aws_filesystem
            uri_prefix:         ""

# Liip Imagine
liip_imagine:
    loaders:
        aws_loader:
            flysystem:
                filesystem_service: oneup_flysystem.aws_filesystem_filesystem
    data_loader: aws_loader
    resolvers:
        default:
            web_path:
                web_root:     "%kernel.root_dir%/../web"
                cache_prefix: "media/cache"
    cache: ~
    filter_sets:
        cache: ~
        400xY:
            quality: 90
            filters:
                thumbnail: { size: [ 400, null ], mode: outbound }

and S3 connector service:

# services.yml

app.aws_s3.client:
    class: Aws\S3\S3Client
    arguments:
        -
            version: "2006-03-01"
            region:  "%amazon_s3_region%"
            credentials:
                key:    "%amazon_s3_acces_key%"
                secret: "%amazon_s3_secret_key%"

The tricky part was understand how to tie VichUploader and LiipImage through the S3 Flysystem adapter, but finally I've done it quite simple and easy because I want to store Liip thumbnails in local file system.

Pay special attention at Vich "upload_destination" and "uri_prefix" configuration combinated with Liip "cache_prefix".