Caddy: Caddy process crashes after working for a while

[nordstern]

1. What version of Caddy are you using (caddy -version)?

v2.0.0 h1:pQSaIJGFluFvu8KDGDODV8u4/QRED/OPyIR+MWYYse8=

2. What are you trying to do?

Run caddy.

3. What is your entire Caddyfile?

    :80 {
    @http {
    protocol http
    }
    redir @http https://{host}{uri}
    bind 0.0.0.0 #changed IPs for security purposes
    bind 0.0.0.1
    bind 0.0.0.2
    }

    :443 {
    root * /var/www/app/public/
    bind 0.0.0.0 #changed IPs for security purposes
    bind 0.0.0.1
    bind 0.0.0.2
    file_server
    encode zstd gzip

    # Security headers
    header {
    Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
    X-Xss-Protection "1; mode=block"
    X-Content-Type-Options "nosniff"
    Content-Security-Policy "upgrade-insecure-requests"
    Referrer-Policy "strict-origin-when-cross-origin"
    Cache-Control "public, max-age=15, must-revalidate"
    Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
    }

    # Logs
    log {
    format single_field common_log
    output file /var/log/caddy/access.log {
    roll_size 100mb
    roll_keep 10
    roll_keep_for 168h
    }
    }

    log {
    format logfmt
    level error
    output file /var/log/caddy/error.log {
    roll_size 100mb
    roll_keep 10
    roll_keep_for 168h
    }
    }

    # SSL Settings
    tls {
    on_demand
    }

    # PHP-FPM Configuration
    php_fastcgi unix//run/php-fpm/www.sock

    # If the requested file does not exist, try index files
    try_files {path} {path}/index.php index.php
    }

4. How did you run Caddy (give the full command and describe the execution environment)?

    /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

lsb_release -a:

    LSB Version:    :core-4.1-amd64:core-4.1-noarch
    Distributor ID: CentOS
    Description:    CentOS Linux release 8.1.1911 (Core)
    Release:    8.1.1911
    Codename:   Core

Logs:

    May 17 20:34:01 caddy[20071]: runtime.systemstack_switch()
    May 17 20:34:01 caddy[20071]: #011/usr/lib/golang/src/runtime/asm_amd64.s:330 fp=0xc000062788 sp=0xc000062780 pc=0x562d7cb1c1c0
    May 17 20:34:01 caddy[20071]: runtime.main()
    May 17 20:34:01 caddy[20071]: #011/usr/lib/golang/src/runtime/proc.go:133 +0x74 fp=0xc0000627e0 sp=0xc000062788 pc=0x562d7caed5d4
    May 17 20:34:01 caddy[20071]: runtime.goexit()
    May 17 20:34:01 caddy[20071]: #011/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000627e8 sp=0xc0000627e0 pc=0x562d7cb1e301
    May 17 20:34:01 caddy[20071]: rax 0x0
    May 17 20:34:01 caddy[20071]: rbx 0x6
    May 17 20:34:01 caddy[20071]: rcx 0x7f927e06e8df
    May 17 20:34:01 caddy[20071]: rdx 0x0
    May 17 20:34:01 caddy[20071]: rdi 0x2
    May 17 20:34:01 caddy[20071]: rsi 0x7fff279eabc0
    May 17 20:34:01 caddy[20071]: rbp 0x562d7dcc289b
    May 17 20:34:01 caddy[20071]: rsp 0x7fff279eabc0
    May 17 20:34:01 caddy[20071]: r8 0x0
    May 17 20:34:01 caddy[20071]: r9 0x7fff279eabc0
    May 17 20:34:01 caddy[20071]: r10 0x8
    May 17 20:34:01 caddy[20071]: r11 0x246
    May 17 20:34:01 caddy[20071]: r12 0x562d8092e2a0
    May 17 20:34:01 caddy[20071]: r13 0x0
    May 17 20:34:01 caddy[20071]: r14 0x562d7dc829f0
    May 17 20:34:01 caddy[20071]: r15 0x0
    May 17 20:34:01 caddy[20071]: rip 0x7f927e06e8df
    May 17 20:34:01 caddy[20071]: rflags 0x246
    May 17 20:34:01 caddy[20071]: cs 0x33
    May 17 20:34:01 caddy[20071]: fs 0x0
    May 17 20:34:01 caddy[20071]: gs 0x0
    May 17 20:34:01 systemd[1]: caddy.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

Log file
logs.txt

[mholt]

@nordstern Thanks for the info, but that log is truncated and we can't see the top of the stack. Please attach a link to the full logs (and clean up your post a bit with formatting if you can, please -- it will make it easier to read; better yet, for such a long log, attach a file). Avoid altering the logs.

Also, please post your full, unredacted Caddyfile so we can troubleshoot.

[nordstern]

@mholt Sorry for the mess. I cleared it a bit according your comment.

[francislavoie]

Not related to the crash, but a couple notes:

    # If the requested file does not exist, try index files
    try_files {path} {path}/index.php index.php

That's unnecessary in v2, php_fastcgi has its own try_files built-in that does the same thing. See the expanded form here: https://caddyserver.com/docs/caddyfile/directives/php_fastcgi#expanded-form

Also, you can use the caddy fmt command to format your Caddyfile. As posted in the thread, it's somewhat difficult to read because of the lack of indentation.

Also, in GitHub and many other places that support markdown, you can instead use ``` on the lines before and after your config/logs to use code formatting, instead of prefixing 4 spaces on every line.

[mholt]

@nordstern Thanks! That's better, although the attached log file is still truncated at the top. We'll need the full logs. It should be a bunch of normal output, until some line that says "panic" (usually but not always).

[francislavoie]

@matt I see May 17 20:33:14 rsyslogd[3093]: imjournal: 1274 messages lost due to rate-limiting so I think it got truncated by the system :cry:

[mholt]

Ah, how unfortunate.

If they can be recovered, or if it happens again, that would be great. Until then, closing for lack of information.

[mholt]

It was pointed out to me that the logs might actually be complete at least for one invocation of the caddy process, around line 2153...

[mholt]

        May 17 20:34:01 systemd[1]: Started Caddy.
        May 17 20:34:01 caddy[20071]: runtime/cgo: pthread_create failed: Resource temporarily unavailable
        May 17 20:34:01 caddy[20071]: SIGABRT: abort
        May 17 20:34:01 caddy[20071]: PC=0x7f927e06e8df m=0 sigcode=18446744073709551610
        May 17 20:34:01 caddy[20071]: goroutine 0 [idle]:
        May 17 20:34:01 caddy[20071]: runtime: unknown pc 0x7f927e06e8df

So, I dunno what's going on here -- Caddy appears to be invoking cgo, which I know we don't in our own programming, but maybe the runtime is using it for something? I am not sure but I figure system calls _are_ needed to create system threads.

This is either a serious misconfiguration of your Linux system/service, OR a bug in Go, is my best guess.

[francislavoie]

How did you download Caddy? From GitHub or via the Fedora COPR? (shouldn't make much of a difference, but it's good to know)

[nordstern]

@francislavoie I installed it through Fedora COPR.

[mholt]

@nordstern Any changes to the systemd file? What is your full systemd service file?

[nordstern]

@mholt here is my service file, I just add Restart and RestartSec parameters.
```[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
Restart=on-abnormal
RestartSec=5s

[Install]
WantedBy=multi-user.target

[Mohammed90]

This issue has the same error golang/go#24484.

[mholt]

Okay, thanks.

@nordstern I asked some people smarter than me in the Gopher slack and it's quite possible that the system aborted the process because:

EAGAIN A system-imposed limit on the number of threads was encountered. There are a number of limits that may trigger this error: the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of processes and threads for a real user ID, was reached; the kernel's system- wide limit on the number of processes and threads, /proc/sys/kernel/threads-max, was reached (see proc(5)); or the maximum number of PIDs, /proc/sys/kernel/pid_max, was reached (see proc(5)).

The output of ulimit -a (as run by the caddy service, not just in a terminal) might be helpful. And/or df -i (the IUse% column).

I would also check to make sure that your PHP pages do eventually exit...

So, after that quick revival, I'm so far convinced this isn't a bug in Caddy as much as in some external factor -- so I'll close this, but feel free to continue discussion and update us on what you discover. We can reopen it if it is determined to be a bug in Caddy.