Liipimaginebundle: add a way to force early creation of the cache file

[lsmith77]

I have to dig in an see if and how it would make sense exactly. But when reading from a remote data source it can make sense to generate the cache entry in the request that renders the image tag.

[havvg]

What exactly do you mean?
When you request the cached version of the image the first time, it's created and its content is served directly.

Do you want to manually create the filtered images when saving/updating the original one? So you never get into creating the cached version on a request mentioned above?

[lsmith77]

Right now the common case is:

1) we read the resource to determine if it exists and to have access the relevant metadata and generate an html image tag
2) the browser reads the image tag and requests the image, in which case we read the resource and generate the image and if necessary cache it

I am talking about making it possible to force the creation of the cached image in step 1)

[trsteel88]

+1

[trsteel88]

This would also stop multiple database queries.

In step 1 the server may run queries on request. Eg the image path is requested from the database.

In step 2 this query may have to run again to verify the path.

[lsmith77]

right that is the idea .. when dealing with remote resources it might be more efficient to generate the cache file early without having to generate all possible cache files when the resources is created/updated

[havvg]

Ok, let me try to sum this up and let's verify we are talking about the same.

You got something (e.g. a model retrieved from a datasource) in your template. This model got an accessor on a related image path (e.g. a profile picture of a user). This code is reflecting the first step:

<img src="{{ model.image|imagine_filter('thumbnail', true) }} " />

This will eventually call ResolverInterface::getBrowserPath. For example the resolvers for Amazon S3 will in the case we want to address fallback to CacheManager::generateUrl which will result in the second step. The url is a local one, which will be routed to ImagineController::filterAction creating and caching the image in question.
Further requests will now result in a s3.amazonaws.com url for the cached image instead of CacheManager::generateUrl.

Basically the solution would be a new method combining ResolverInterface::resolve, ResolverInterface::store and ResolverInterface::getBrowserPath. It's very similar to the ImagineController::filterAction but without the dependency of a Request.

[lsmith77]

Yes, I guess resolvers like Amazon S3 are one scenario. However often the model with the image itself (though as a stream) maybe already be fetched by the controller. So the gist of it there are cases where in the current setup the same resources needs to be fetched both in the request generating the html and the subsequent request for the actual image binary. for this case i proper adding another optional parameter to force cache generation right when the html is generated.

[psytraxx]

I am not sure if this helps but i found a way around this by creating a new twig function - I was not able to pass on the PHPCR Image document as an argument to a filter

I then implemented a service which has a lot of dependencies to liip imagine bundle basically saving me the work to generate thumbs using the existing filtermanager

generating the thumbs at the time of the url generation should also allow us to invalidate cached thumbs more effective (document image creation time > thumbnail time)

unfortunately the image Document has no updatedAt property so i cant effective invalidate the cache

maybe its a good idea to have updatedAt added to Image in the future?

i am pretty sure the LIIP wizzards have better insight into their bundle and this can be MUCH done better but maybe some of my thoughts and code might be useful :)

/**
* Sets up all of the functions this extension makes available.
*
* @return array
*/
public function getFunctions()
{
return array(
'thumbnail' => new Twig_Function_Method($this, 'thumbnail')
);
}

 * generate a thumb straight from the CR
 *
 * @param \Doctrine\ODM\PHPCR\Document\Image $imageCR
 * @param $filter
 * @return mixed|string
 */
public function thumbnail(Image $imageCR, $filter )
{
    /** @var $thumbnailHelper ImageThumbnailHelper */
    $thumbnailHelper = $this->container->get('joiz.thumbnailhelper');
    return $thumbnailHelper->getUrlForPhpCR($imageCR,$filter);
}

joiz.thumbnailhelper:
      class: Joiz\HardcoreBundle\Helper\ImageThumbnailHelper
      arguments:
          - %liip_imagine.web_root%
          - @liip_imagine.filter.manager
          - @liip_imagine

/**
*
* get a thumbnail url for the given sourcefileß
*
* @param DoctrineODMPHPCRDocumentImage $imageCR
* @param $filter
* @param Request $request
* @return null|string
*/
public function getUrlForPhpCR(Image $imageCR, $filter, Request $request = null)
{

    try {

        //determine extensiom from mimetype
        switch($imageCR->getMimeType())
        {
            case 'image/gif':
                $extension = '.gif';
                break;
            case 'image/jpeg':
                $extension = '.jpg';
                break;
            case 'image/png':
                $extension = '.png';
                break;
            default:
                $extension = '';
        }

        //
        $mediabase = "/media/cache";
        $folder = $mediabase."/twigthumb_".$filter;

        $imagePath = $imageCR->getId(); //this is the phpcr name
        $fullPath = $this->webroot . $folder . $imagePath.$extension;

        $fileExits = file_exists($fullPath);


        if ($fileExits) {

            //regenerate file if its older than an hour
            if ($fileExits && time()-filemtime($fullPath) >= 3600) {
                unlink($fullPath);
                $fileExits = false;
            }

        }

        //we have no file - lets generate a thumb!
        if (!$fileExits) {

            $pathinfo = pathinfo($fullPath);

            if (!file_exists($pathinfo['dirname'])) {
                mkdir($pathinfo['dirname'],0777,true);
            }

            $stream = $imageCR->getContent();

            rewind($stream);

            /** @var $image ImageInterface */
            $image =  $this->imagine->load(stream_get_contents($stream));

            $result = $this->filtermanager->applyFilter($image,$filter);

            //TODO: use quality from config
            $result->save($fullPath,array('quality' => 85));

        }


        //this is the path returned for the browser
        $webPath = $folder.$imagePath.$extension;


        if ($request) {
            return $request->getUriForPath($webPath);
        } else {
            return $webPath;
        }

    } catch (\Exception $ex) {
        return null;
    }
}

[KeKs0r]

I am not sure but I think I have another use case. I want to serve my model via rest API and therefore have no access to the twig helpers.
I would like to create all filtered version on amazon s3 after the original is uploaded (e.g. through a worker). Of course I would need to recreate some of the helper logic within js, but I would like to skip the cache process and directly access the final version on S3. Otherwise I would need to recreate the whole caching logic in js.

@havvg : you already pointed in the correct direction. I was trying to reverse engeneer the controller, but am slightly confused. Because in order to just eagerly store the thumbnail, I would not need the cacheresolver. It would probably make sense for other use cases, though. You said that this is the flow that needs to happen:

Basically the solution would be a new method combining ResolverInterface::resolve, ResolverInterface::store and ResolverInterface::getBrowserPath. It's very similar to the ImagineController::filterAction but without the dependency of a Request.

But how does the Cache and DataManger play a role in this?

[KeKs0r]

I have now a resquejob that basically is manually creating the Image. Performance is not too good. And I also did not really all the parts, how they play together. I am also unsure how to test this. I started writing a test but I would like to assert that the targetFile does not exist before the test is run and then assert that its there after the test, but I don't know how to do this.

But here goes the Code of my Job:

        $resolver = $container->get('liip_imagine.cache.resolver.amazon_s3'); 
        $dataManager = $container->get('liip_imagine.data.manager');
        $filterManager = $container->get('liip_imagine.filter.manager');

        $req = new Request();

        $targetPath = $resolver->resolve($req, $path, $filter);  // 1s
        $image = $dataManager->find($filter, $path); // 7s
        $response = $filterManager->get($req, $filter, $image, $path); // 0.003s
        $response = $resolver->store($response, $targetPath, $filter);  // 1s

[pakaufmann]

If anybody is still interested in this, the fix is actually quite easy to implement. The following code creates the image as soon as the filter is called. It's basically the same code as in the filterAction of the ImagineController.

    new \Twig_SimpleFilter('imagine_create', function ($path, $filter)
        {
            try
            {
                if(!$this->cacheManager->isStored($path, $filter))
                {
                    try
                    {
                        $binary = $this->dataManager->find($filter, $path);
                    }
                    catch(NotLoadableException $e)
                    {
                        if($defaultImageUrl = $this->dataManager->getDefaultImageUrl($filter))
                        {
                            return $defaultImageUrl;
                        }

                        throw new NotFoundHttpException('Source image could not be found', $e);
                    }

                    $this->cacheManager->store(
                        $this->filterManager->applyFilter($binary, $filter),
                        $path,
                        $filter
                    );
                }

                return new \Twig_Markup(
                    $this->cacheManager->resolve($path, $filter),
                    'utf8'
                );
            }
            catch(RuntimeException $e)
            {
                throw new \RuntimeException(sprintf('Unable to create image for path "%s" and filter "%s". Message was "%s"', $path, $filter, $e->getMessage()), 0, $e);
            }
        })

Use it like this:

    'path_to_image'|imagine_create('filter')

[lsmith77]

can you submit a PR for this?

[pakaufmann]

I can, but I'm not sure if it should actually be a separate command when being in the bundle. Maybe a flag on the current imagine_filter would be better. Something like "forceEarlyCreation" or similar.

[makasim]

Maybe a flag on the current imagine_filter would be better. Something like "forceEarlyCreation" or similar.

:+1: on flag option.

[mikemix]

No flags, please. Just another filter, like imagine_filter_create. Wow, this is still open lol

[michellesanver]

@pscheit Is this something you are looking at? Given the eyes emoji. If not, I'd like to label this one as a "good first issue" and write some clear instructions what needs to be done. Let me know :)

[pscheit]

I was referring to: "Wow, this is still open lol".,. ;)

I'm not using imagine anymore