Caddy: Caddy sometimes serving stale PHP page content

[adamwathan]

(Sorry for deviating from the issue format, but this one is sort of weird and hard to intentionally reproduce)

We're using Caddy 0.9.3 on the master branch of Valet and have noticed that we very frequently need to refresh the page multiple times before our changes take effect.

For example, we might change some HTML in a template, refresh the page, and the old HTML is still there until refreshing again.

For me, it seems to happen when I haven't edited something in a few minutes, and I usually only need to refresh one more time to get the right content.

For @taylorotwell, it's happening more often and sometimes he has to refresh 3-5 times for the new content to show up.

This happens even when hitting the server via cURL, and reverting to Caddy 0.9.1 makes the issue completely disappear.

The only thing I can think is that it has something to do with the changes that were made for persistent connections.

I _really_ wish I had a way to reproduce it on demand, but it seems to happen very randomly, but frequently enough to be frustrating.

If there's anything I can do or try to help narrow down where the issue could be, please let me know!

Here's my Caddyfile:

import /Users/adamwathan/.valet/Caddy/*

:80 {
    root /

     # Slight hack, see: https://github.com/mholt/caddy/issues/1020
    internal /dev/null

    fastcgi / /var/run/fpm-valet.socket php {
        index /Users/adamwathan/.composer/vendor/laravel/valet/server.php
    }

    rewrite {
        to /Users/adamwathan/.composer/vendor/laravel/valet/server.php?{query}
    }

    log /Users/adamwathan/.valet/Log/access.log {
        rotate {
            size 10
            age 3
            keep 1
        }
    }

    errors {
        log /Users/adamwathan/.valet/Log/error.log {
            size 10
            age 3
            keep 1
         }
    }
}

[taylorotwell]

Yeah, I typically have to hit the URL 4-5 times every time I make a code change to my PHP before I start seeing updated content.

[mholt]

So by default, persistent connections are disabled, I think (as of 0.9.3) -- is that correct, @abiosoft? Can you look into this when you have a moment?

[abiosoft]

Yeah, persistent connections are disabled by default. However, I'll try to reproduce this.

@adamwathan I can see a caddy binary in bin directory. I assume I can keep replacing the binary while I'm tweaking caddy's source.

[adamwathan]

Hey @abiosoft yup totally, just need to run valet restart afterwards.

Thanks for looking into this for us!

[abiosoft]

Can't reproduce this yet 😕

Any publicly available laravel/php source I can reproduce this behaviour with?

[taylorotwell]

I think it's a little hard to give you a nice, re-creatable repo like that. You kinda have to use PHP with Caddy all day in a development environment. It's basically just pretty broken. If I were you, I would literally roll back to whatever your PHP handling was before the persistent connection stuff went in. Some of my projects just plain don't even work with Caddy anymore. Just white screens - had to move those back over to Nginx. All of them are impossible to use during development because you have to refresh the page 5-6 times to see every code change you make.

[tobya]

I'm using multiple sites on Windows with fastcgi with no problems since 0.9.3

Admittedly they are low traffic but I am not having any of these issues.

I also run sites with lumen (which is a sub project of Laravel)

[mholt]

@abiosoft Any progress in reproducing this? Granted, it's difficult since it appears to have some aspect of randomness to it... (do we know why?)

[Dexus]

@adamwathan please check your php settings (opcache or similar cache) its mostly the problem with fpm.

I think its has nothing to do with caddy.

Best regards

[adamwathan]

@Dexus Had no issues before these changes in Caddy and it has affected a lot of Valet users. We've all since switched to nginx using the same FPM configuration and the problem has gone away.

[jordic]

Could be something related to x-cache or APC? On the phone side... The behaviours seems like ...

[jordic]

S/phone/php

[taylorotwell]

As Adam mentioned, you can switch between Nginx and Caddy without touching your FPM config and Caddy will exhibit the broken behavior while Nginx will work fine. So it seems disconnected from any FPM settings. We never had any problem with Caddy until that one change to FPM handling went into it.

[mholt]

Speculation isn't helpful here at this point, @jordic, @Dexus -- what we need are well-documented reproductions of this behavior that can expose the bug in the fastcgi client code. I wish I had time to devote to this issue, but I'm busy with other (mostly Caddy-related) things right now.

Ideally, someone with PHP installed could take this setup described above, see the same thing that Adam and Taylor are describing, then twiddle the Caddy code a bit to see what causes any change in that behavior and report back.

I'll treat someone to lunch if they do this and are ever in the area to visit. :smile:

[withinboredom]

@adamwathan, @mholt:

I've created a basic dockerized reference implementation of caddy + php-fpm + docker. I was unable to reproduce the issue with basic php... @adamwathan if you want to try this with more advanced code, that would be amazing...

What is the output of phpinfo() when using caddy vs using nginx?

[mholt]

Wonder if this is related to https://forum.caddyserver.com/t/php-fpm-via-fastcgi-throws-502-bad-gateway-after-10-requests/1419?u=matt -- I would really appreciate the community's help with this while I am tied up getting other things prepared for Caddy.

[davidak]

I have added some more information to that thread.

Like @Dexus and @jordic said, it seams related to Zend OPcache.

After i disabled it, i have no problem with DokuWiki or phpinfo();.

Piwik still has similar problems. It loads the data with XHR-Requests so they are random and don't fit their heading.

When i reload the page several times i will also get random JS files or images as HTTP-Response.

[davidak]

When i load the phpinfo(); page several times and then try to open the Piwik site (served by same FPM Pool), i also get a phpinfo site as response. Feels like some cache, but opcache is disabled now.

Same when i open Piwik often and then open phpinfo IN INCOGNITO MODE, i get a response only visitors logged in to Piwik should see.

So this is a security issue!

[mholt]

Oops. @Echsecutor would you be able to help us out and look into this ASAP? I don't know if this is related to the changes you contributed, but you are at least very familiar with the fastcgi middleware.

@davidak What if you build Caddy before or without the changes in https://github.com/mholt/caddy/pull/1126 - does it still happen?

[davidak]

@mholt that change was merged on 24.9.16, Caddy 0.9.2 was released on 21.9, so i actually don't have that changes.

[mholt]

@davidak You'll probably have to do a git checkout to an older commit and use go build or go install.

[adamwathan]

We used to see that issue of random assets being served with Caddy 0.9.2 but it went away in 0.9.3.

https://github.com/laravel/valet/issues/165

[davidak]

@adamwathan good to know, thank you!

[withinboredom]

This smells like some kind of off-by-1 error...

[abiosoft]

Apologies if this question is a bit redundant.

What is the issue right now. Stale contents or random assets or both ?
I have some time on my hands now to help with this.

Thanks.

[davidak]

@abiosoft my issues are described in detail in the linked forum post and https://github.com/mholt/caddy/issues/1204#issuecomment-278198501.

I can test if it is fixed in 0.9.3 after https://github.com/NixOS/nixpkgs/pull/22544 is merged, what will happen soon i think. :)

[tobya]

Similar to @adamwathan I had issues with this in 0.9.2 (with PHP fast-cgi) but have had no issues with 0.9.3 - 0.9.5

[mholt]

@davidak Did you see if this is still a problem in the latest version?

[davidak]

@mholt i can now confirm that it is fixed in the latest version.

[wendigo]

Thx @davidak! Closing this issue :)

[mholt]

@adamwathan What would I have to do to convince you to try this again with Caddy 0.9.5? I want to make sure there are no bugs in Caddy that would stand in the way of Valet.

[mholt]

The random content issue was different, and the original issue may not still be fixed.

[mholt]

@Echsecutor I'm probably going to revert https://github.com/mholt/caddy/commit/8cb4e9085273e9678d73b4ebce51fbcb18852bf9 and https://github.com/mholt/caddy/commit/09a1f02971b99b8c7e4e583398608c09a9747976 since they are, as far as I can tell, the only changes that might have this effect between 0.9.1 and 0.9.2. Please speak up in the next few days if you are able to address this; I would really hate to have to take out this feature you worked on. :confused: But I won't have the time to debug it myself before the release on April 20...

[tobya]

@mholt Is this still an issue?

From the thread I can't see @adamwathan confirming it is still an issue on 0.9.5. I think the exact commits you plan to revert were the ones that fixes issues with fastcgi in 0.9.2.

I don't understand why this issue will require reverting these commits/

[mholt]

@tobya I don't see where @adamwathan said it was resolved, and I've heard reports of similar strange behavior with fastcgi since as well, but we haven't been able to reproduce it. I believe the commits I plan to revert are only for _adding_ persistent connections, a feature that I suspect may contain the faulty code.

Can somebody please publicly confirm where we are with this issue then? @adamwathan or @taylorotwell, using the latest Caddy from master branch, before I revert? A new release is going out in 8 days either way; if you want the commits reverted before then, I need to know.

[adamwathan]

We started seeing issues when persistent fastcgi connection support was added. It was super rough to use for local development after that, both Taylor and I had to switch to nginx and haven't tried Caddy again since so can't speak to whether or not it still happens.

[mholt]

@adamwathan Okay, very well. I will close this issue then and be done with it based on the reports of others that it has been fixed; if you still experience the problem, please let us know!