Twig: Impossible to update/change a template when running as daemon (ex: as symfony messenger worker)

[sarim]

This issue (and other unexpected behaviors) stems from how twig generates actual php classes for template. Once a class is loaded into php runtime, its there forever.

So my use case was a worker will re-render templates whenever it receives a command via messenger to do so. The templates gets updated independently. Unfortunately it was impossible to do in the current implementation of twig. (Don't suggest to restart worker every time a template gets updated, the uptime of the service would go down the drain).

the cache interfaces, auto_reload options etc... provides kinda false information in this context, as in the Environment::loadTemplate function, class_exists is called before all those logic. There is no api/method provided to change the generated class name, as its effectively always templateClassPrefix + hash ( template path ).

I ended up creating this monstrosity. I call this each time worker receives a render job.

    /**                                                                                           
     * Twig concatenates template name and Environment's `optionsHash` and makes a hash,          
     * then creates a PHP Class named by that hash. So to force twig to reload template content,  
     * we have to force twig to generate a new Class.                                             
     *                                                                                            
     * Problem with this solution is it leaks memory with each call,                              
     * as all the previous class signatures remain in php runtime.                                
     */                                                                                           
    private function updateTwig()                                                                 
    {                                                                                             
        $reflection = new \ReflectionClass($this->twig);                                          
        $property = $reflection->getProperty('optionsHash');                                      
        $property->setAccessible(true);                                                           
        $property->setValue($this->twig, \microtime());                                           
    }

Setting aside the memory leak problem for now.

Creating new instances of "Twig\Environment" is also kinda pointless as all of them will use the same generated php class.

Here's some potential idea for a solution.

1. Provide a way to change the TemplateClass name. Maybe expose templateClassPrefix via getter and setter. User then can set those to new values, effectively resulting in making new class.

2. Rearrange the logics in Environment::loadTemplate. Do the checking of cache and auto_reload before checking class_exists. If cache reports a non fresh template (modified in filesystem), make a new class, maybe adding a increment suffix to the name.

Also there should be documentations about this in both twig and symfony messenger, as twig's behavior is changed in daemon vs http. We had to do a deep dive into twig source to figure out the bug, some warning in docs would save time of future people suffering from same issues.

[stof]

Also there should be documentations about this in both twig and symfony messenger, as twig's behavior is changed in daemon vs http.

how is it changed ? The behavior is not changed. What changes is the lifetime of the PHP process, not the behavior of Twig.

[sarim]

@stof
My bad. I striked out that section. Ignore that for now.
Do we want to discuss the actual problem? AKA provide a solution to update template in daemon.

[stof]

Given that modifying templates without restarting the process is not the common use case for Twig (most projects have templates in the source code, which require deploying a new version for a change to happen), I don't think this algorithm will change as that would affect performance (and will probably become tricky for some parts of the implementation like the compilation of {% embed %}).
However, what could probably work for your use case (I haven't tried it to confirm it) would be to use a custom loader for your reloadable templates (instead of relying on the core FilesystemLoader), and implement the appropriate logic in getCacheKey so that the cache key changes when the content needs to be reloaded.

[sstok]

Not to mention over flooding the memory with to many classes, as only objects can be GC.

Instead of letting the daemon handle the rendering, you can delegate this to a separate process that only runs a short while.

[sarim]

Given that modifying templates without restarting the process is not the common use case for Twig

Thats fair. I'm aware of that. As a longtime user of symfony I'm familiar with that. That years of experience and familiarity with symfony developing traditional apps is why we decided on symfony for this project (and some other logistical reasons), which kinda falls under event driven microservices architecture.

What I wanted to discuss here kinda goes in both technical and philosophical domain. If there is not solid support for "non common case" ( event driven / microservice etc.. ), people won't choose php/symfony/twig for those use cases. And those uses cases remains non common case .....
Direction of symfony in recent years kinda points that it wants to cover more use cases (introduction of messenger component for example). As the "default" template engine for symfony, I wish twig would be worker friendly too.

What I'm trying to say is the world is moving forward, and we in php world must too.

---

implement the appropriate logic in getCacheKey so that the cache key changes when the content needs to be reloaded

Yes that would work, but still It wouldn't be a solution, it'd be a workaround.

Not to mention over flooding the memory with to many classes, as only objects can be GC.

Yes, I mentioned that in my original post. We're still kinda in prototype/MVP stage, so i'm ignoring the memory leak for now. (Managing it outside).

Instead of letting the daemon handle the rendering, you can delegate this to a separate process that only runs a short while.

For the final product it'd be more easier to move to an engine that is built for this use case. A templating engine that doesn't dynamically makes classes and that cleans up easily. That'd be more reasonable (and less complex architecturally) than spawning new processes for each render call.

[stof]

Twig is architectured for the use case where templates don't change all the time (even when using a long running app server, that assumption is still valid when templates are part of the source code), just like PHP classes don't change over time without reloading the process.
If your use case is precisely to change template sources dynamically without reloading the PHP process, you should indeed probably use a different templating engine which does not compile templates to PHP classes but to something which is temporary in memory instead. But Twig won't be rebuilt this way.

[fabpot]

Closing as we won't support the use case described here (as explained by @stof).