Caddy: SRV proxy feature

[rstupek]

see relevant discussion here: https://caddy.community/t/dns-srv-record-support-in-proxy-directive/3592/7

1. What version of Caddy are you using (caddy -version)?

latest version

2. What are you trying to do?

use the srv record feature of the proxy directive to handle multiple backends and multiple domains with one configuration entry

[mholt]

Pinging @Gufran - can you help here?

[rstupek]

I did have some code investigations I've done which I can share. I could not figure out how to get the dialer to work though

[Gufran]

Let me address the questions raised in discourse thread.

I thought a request would come in for example.com, that would cause caddy to request the srv record for _http._tcp.example.com and find a hostname and port to use to proxy the request to. Is that what it was meant to do?

This is not what Caddy is supposed to do. First off, this behaviour ties your internal service registry with external traffic handlers and even with clients. And second, there is no way for caddy to know that resolving with _http._tcp prefix will actually work. For an example, the service may be deployed as web or frontend or app or whatever else and in those cases the name _http will be invalid.

I think its immaterial that there are non-standard implementations since the code uses net LookupSRV to perform the lookup and thus would depend on records being formatted per rfc 2782?

The code uses the function net.LookupSRV but Caddy does not care about RFC-2782 formatted service locator, the DNS lookup is performed for an FQDN with service name and protocol embedded in it. The implementation in Caddy does not understand the service name and protocol separate from service registry domain. For example, it can resolve the domain web.tcp.registry.internal but can't work with service=web proto=tcp domain=registry.internal. This is intentional because there is nothing special about latter, it eventually gets converted into the former FQDN before the lookup is performed.

and the SRV option becomes a much more useful feature than it is presently since it can provide a list of failover hosts and include a priority weighting which isn’t presently available with the current configuration options.

Failover, target priority, and target weight are handled by DNS resolver. https://golang.org/src/net/lookup.go?s=11084:11167#L335

The returned records are sorted by priority and randomized by weight within a priority.

Correct me if I’m wrong but I don’t believe the code uses more than one SRV entry?

I am not sure if by SRV entry you mean the locator entry in configuration block or the entries in lookup result, let me explain both.

1. The configuration block does not allow more than one locator because SRV lookup is supposed to abstract the upstreams away from configuration. If you want to put all upstream hostnames in configuration then you are better off with using the normal reverse proxy behaviour.
   With SRV resolver you are expected to manage available services on DNS level in the registry itself.
2. The result is randomised by resolver within priority so Caddy always picks up the first entry. You can adjust weightage and priority in service registry to shape traffic.
   SRV records are volatile by nature. that makes it a bit difficult to implement correct kind of load balancing algorithms, and random balancing often works out just fine.

Now to answer your original question:

You need to make sure that the configuration contains a valid domain.

:443 {
    proxy / srv+https://_http._tcp.example.com {
        transparent
    }
}

then you can add or remove upstreams in service registry and change weightage and priority without ever changing the configuration.

[francislavoie]

@Gufran could you propose some documentation changes to clarify this stuff for the future?

[mholt]

This is a great explanation @Gufran, thanks. @francislavoie : To be fair, I believe at least some of that is "expected" to be known by those who are using SRV records. I don't use SRV, so I'm not sure, but it seems like a lot of that is out of the scope of the Caddy documentation. Maybe the few Go-specific or Caddy-implementation-specific quirks mentioned in your explanation could/should be mentioned in the docs? If so, I'm willing to make the changes, just tell me what the wording should be. But let's not get out of scope and explain how DNS and SRV works in general. :+1:

[rstupek]

@Gufran :

This is not what Caddy is supposed to do. First off, this behaviour ties your internal service registry with external traffic handlers and even with clients

I'm not sure what you're trying to say there and how using SRV records means that at all.

I am not sure if by SRV entry you mean the locator entry in configuration block or the entries in lookup result, let me explain both.

I certainly meant #2 as we were discussing the results of the lookup and how the code only ever uses the first entry

1. The result is randomised by resolver within priority so Caddy always picks up the first entry. You can adjust weightage and priority in service registry to shape traffic.
   SRV records are volatile by nature. that makes it a bit difficult to implement correct kind of load balancing algorithms, and random balancing often works out just fine.

I don't believe you can adjust weightage and priorty in the service registry to shape traffic because the code only ever use the top most entry in the dns result.

I don't believe SRV records are being used in caddy how they're expected to be by anyone who sees it supports SRV records. If I list multiple SRV entries in DNS it implies I am expecting it to handle failover and load balancing. Yes the golang resolver sorts the records and randomizes them within a given priority but it does still return all of the records. What is presently in the code only handles load balancing if you set all of the priorities to be the same. It definitely doesn't handle failover as far as I can see.

Back to my original suggestion which was to enhance what caddy could do when using SRV records, for someone who does follow the SRV rfc, and properly formats their dns configuration and doesn't want to list every domain in caddy and yet can have one configuration setting which will handle any hostname requested of it by looking up in DNS and retrieving the SRV record and getting the backend service for that hostname to proxy to. I'm not sure why you wouldn't want it to be able to do that when it doesn't seem to me to be that much of a change based on my investigation not being a caddy or go expert I got 90% of the way to actually making it work.

[Gufran]

I'm not sure what you're trying to say there and how using SRV records means that at all.

Could you please ask a specific question? I'm not sure how to address this.

I certainly meant #2 as we were discussing the results of the lookup and how the code only ever uses the first entry

I see your point here. This is a deficiency in Caddy if you wish to have an elaborated setup with target weightage and priorities. I believe this can be discussed in a separate thread and someone can implement a load balancing algorithm for it.

If I list multiple SRV entries in DNS it implies I am expecting it to handle failover and load balancing.
...
What is presently in the code only handles load balancing if you set all of the priorities to be the same. It definitely doesn't handle failover as far as I can see.

I think you are confusing health checks with discovery. If you have a bunch of targets registered in DNS then the client is supposed to work with all of them. If some of the targets are unhealthy then that is a problem at some other level. ideally the service or the registry should keep the list of available targets up to date depending on their health. Consul catalog is a good example of this.
I agree that load balancing can be improved by a lot.

Back to my original suggestion which was to enhance what caddy could do when using SRV records, for someone who does follow the SRV rfc, and properly formats their dns configuration and doesn't want to list every domain in caddy and yet can have one configuration setting which will handle any hostname requested of it by looking up in DNS and retrieving the SRV record and getting the backend service for that hostname to proxy to.

It looks like you want some sort of variable substitution behaviour in configuration block where you can dynamically choose service name to look up in registry for every request. This behaviour has nothing to do with SRV or Caddy.

I'm not sure why you wouldn't want it to be able to do that when it doesn't seem to me to be that much of a change based on my investigation not being a caddy or go expert I got 90% of the way to actually making it work.

Could you please propose a pull request with those changes so that everyone can get a better idea of the behaviour you are trying to describe?

---

@mholt @francislavoie I agree with @mholt that this is out of scope of Caddy documentation.

[mholt]

Great, sounds good to me. If further discussion is needed, feel free, but at this point it seems to be just a discussion rather than something specific and actionable in Caddy, so I will close the issue. Thanks!