I'd love caddy to reload the config file when receiving a signal,
I would use caddy paired with consul and hashicorp/consul-template to generate the config file and when something changes in consul the config file is regenerated, i would like to be able to tell caddy that it's config file changed and to reload it when this happens.
Use case:
_Then:_
Other service discovery tools have similar processes to configure services automatically, i think reloading the config file from a signal is a nice portable way to do it.
This may link to #104 & #8
Glad you brought this up. Planning has already begun for this feature, including an automatic failsafe: Caddy will roll back to the last working configuration if the new one fails to start for some reason.
I like the signal idea - but which signal would be used that works cross-platform? SIGINT and SIGKILL are the only two that are guaranteed to be present on any OS, and those terminate the process. My other thought was to open a high port on the loopback interface so that a new Caddyfile could be streamed into that any time from the local machine.
We're also working on a RESTful API that can make both incremental and major changes to the server's configuration on-the-fly. You can see its progress on the api branch of this repo.
I don't know how they handle it with other web servers, would need to poke in code, or they communicate through an api (listening on another port with an external tool) or they send a signal.
Apache: https://httpd.apache.org/docs/2.2/programs/apachectl.html (graceful option)
Nginx: http://nginx.org/en/docs/beginners_guide.html
SIGHUP:
The SIGHUP signal is sent to a process when its controlling terminal is closed ...
... Many daemons will reload their configuration files and reopen their logfiles instead of exiting when receiving this signal....
SIGHUP is defined in : http://golang.org/pkg/syscall/#SIGHUP
and you can use it from the os/signal
package
I can check this a little, but the rest api to control Caddy is also a good plan.
In apache from ./include/mpm_common.h
/* Signal used to gracefully restart */
#define AP_SIG_GRACEFUL SIGUSR1
From mod_so.c
* This module is used to load Apache modules at runtime. This means that the
* server functionality can be extended without recompiling and even without
* taking the server down at all. Only a HUP or AP_SIG_GRACEFUL signal
* needs to be sent to the server to reload the dynamically loaded modules.
*
So I guess those 2 signals (HUP and USR1) are used. I've seen occurences where in windows they use something else
But i also like the idea of the rest api to control the server
Would an API be compatible with the etcd/consul workflow? Or possibly a port where a file could be piped in? (I'm not a consul/etcd user so I have no idea.)
By the way, thanks for doing the research -- this is very helpful.
An API would indeed be compatible, consul, etcd have a lot of programs (consul-template , kelseyhightower/confd) permit to run a command after a config file has been generated, so would be easy to run something like curl
to reload or send the config.
API would also permit new types of interractions, so i kinda like this way.
But signal is always more standard way to reload config. So having both would really be nice. If this is something you think easy to do I would even lend a hand
I'm making really good progress on the API - almost ready to "mass-produce" the endpoints for all the middleware.
As for reloading config, what do you mean by "more standard way"? Is there a way to get the signal to work on Windows?
i was meaning : SIGHUP
is standard on unix systems to reload config.
For windows systems, I really don't know since normally there is a windows service involved.I don't know how reload is handled. I've just started digging go programs on windows.
Okay. Well, I'm not opposed to a quick-and-easy way to reload config, but it does have to be cross-platform. That's why I'm leaning more toward opening a port on loopback and letting a new Caddyfile be piped in through there. But at that point, it's a network request, which is like what the API does anyway, so... it just depends if we want to be required to use HTTP to reload config. The port could just listen for raw bytes, but I'm still not convinced this is necessary since the API will be able to do this.
I'm seeing that even apache doesn't support _graceful_ reloading on windows without restarting the server completely :confused:
So, that is up to you if you want to do the same (unix only graceful reloading) or using the api, which i find a good idea (the api).
so if we rely on a socket rather than signal trapping we should probably have a caddyctl binary or something that speaks api to the socket. then we could accomplish hot reloads from the cli via caddyctl -s reload
or something
That's basically what the API will do. If we speak HTTP, then any HTTP client (like curl) can interact with Caddy, no special binaries needed (without precluding others from writing Caddy-specific API clients.)
personally i feel like we should ship a cli for the api with caddy, that does basic things like reload. The are must haves to use caddy as a server process in any sort of production environment.
@j-mcnally Sure, anyone is welcome to write clients for the Caddy API when it is ready.
Question for you. I'm starting the API endpoint that will do a graceful reload of the config without killing the whole process. The big question is _Where does it get the config from?_ Sure, it's easy if Caddy's input happens to be from a Caddyfile - just reload the Caddyfile and use it as input. But what about input that gets piped into Caddy? I guess in that case we can just return an error code and say that the original input is no longer available. _-or-_ we could just restart all the servers using the same configuration as was piped in, but that's inconsistent with loading from the file.
Then we need to remember that the API can make other incremental (or major) changes to the server configuration, altering it from what the Caddyfile had when the server first started. Do we reload the server with its current, mutated configuration or use the Caddyfile from disk?
Of course, if a reload does fail, it will roll back to last working configuration.
I can make this happen, but I need some help figuring out these details. Maybe the request could specify what kind of reload to do? How would that look?
After giving it some more thought, I think I've cleared some things up in my head. Here is what will probably happen, feel free to comment if I missed something:
If you use the API to invoke a reload, it only succeeds if Caddy was started with a Caddyfile. When you reload, it just reloads the same Caddyfile, and if it fails, it rolls back. Any changes made to Caddy by the API since it was started would be discarded, since if you invoke a reload, that must mean that you've updated the Caddyfile and want to use that now.
:+1:
I have a similar use-case except that I want to integrate it with Chef outside of docker.
But the whole idea is that a change in machine metadata will trigger chef to recreate Caddyfile and will issue a signaling for reload configuration.
Caddy has the potential to became the "go to" reverse proxy alternative for the DevOps era.
(Thanks for the hard work)
This is almost done.
Okay, this just landed in the le-graceful
branch. /cc @pastjean and @j-mcnally
You can trigger graceful reload with SIGUSR1
. This change will soon be merged into the letsencrypt
branch, which will then be merged into the master branch when nearing release. But if this is something you're very interested to try, go ahead by cloning it down.
If the new configuration is invalid and the child fails to start successfully, the parent process will not terminate and will keep listening. (This is not bulletproof yet: if a startup functions returns an error, for example, the parent may still terminate... same with trying to bind to new ports... I need to work that out later. But if the Caddyfile is invalid, it should failover properly.)
Keep in mind that startup functions _will_ execute on restarts, but shutdown functions will _not_ execute on SIGUSR1.
nice ! good job :+1:
Is this in the v0.8 beta?
@coolaj86 Yep
@mholt
If I understood the thread - currently, caddy has an API, at which I can fire Caddyfiles with an updated configuration. If I got something wrong, the new config will not take.
Is that right?
Is there an established way to use Caddy with Docker across multiple hosts?
Many thanks
-Jake
@faddat
Caddy does not have an API, but it is being worked on.
You cannot pass a new Caddyfile to caddy yet but you can update the existing Caddyfile and send USR1
signal to get caddy to restart with new config.
Is there an established way to use Caddy with Docker across multiple hosts?
I'm assuming you mean as a reverse proxy/load balancer. None yet but there's a work in progress mainly for proxy here with demonstration video here.
If I got something wrong, the new config will not take.
If you got something wrong _that prevents Caddy from using the new Caddyfile_, the new config will not take. So you could misspell a value and it would still work, but Caddy would not reload if the Caddyfile was malformed or something.
@abiosoft @mholt
Yes, as an RP/LB, you got it. I'll check out the repo and demo. I think I'll take a second and tell about my pans, so that if caddy isn't the right tool, you can let me know. Basically, I want to use caddy to act as the RP for many docker containers across many hosts because I love what it's got going on with let's encrypt. Is that reason enough? I mean-- it is certainly not a feature I have seen elsewhere. Thanks guys.
@mholt okay, that is what I figured.
Thanks guys! BTW..... CADDY IS GREAT!
Glad you like it :) And see #564 for a relevant PR for what you're wanting. I think.
there should be a CLI flag to tell caddy just dry check the config file syntax then print out errors and warnings
@myguidingstar You could just start another instance of Caddy with that Caddyfile and it will tell you if there are problems.
I thought that will be a bit noisy with unavailable ports (used by running instance) and HTTPS may take some time or even impossible in local machine if public ip is not available. But maybe that's just fine.
Well, if the Caddyfile has problems, it won't even get to that point. If the Caddyfile is good, it will just fail to bind the needed ports and abort anyway.
@mholt would you mind putting this comment in a more public location, perhaps in the documentation itself? Perhaps point me to the ideal place and I can submit a pull request.
@martindale It's on the CLI page already. :wink:
For future reference:
pkill -USR1 caddy
Reloads the configuration file, then gracefully restarts the server.
Why not use SIGHUP signal, I know nginx use SIGHUP for hot reload signal.
@codeskyblue Because Caddy doesn't "hang up" when doing a reload. :wink:
@mholt perfect answer
For people who don't know how to send SIGNAL.
sudo pkill -USR1 caddy
For future reference:
pkill -USR1 caddy
Reloads the configuration file, then gracefully restarts the server.
@hickford What about caddyv2? This signal doesn鈥檛 work there.
You run caddy reload
instead, which under the hood makes a request to the config API.
Please don't comment on years-old issues. Please instead ask your questions on the forums https://caddy.community
Most helpful comment
For future reference:
https://caddyserver.com/docs/cli#signal