Consul: Question: Services with multiple ports

[bmonkman]

Just a question about best practices here..

We have internal services which expose multiple ports (similar to, say, Zookeeper or Consul itself) but the service definition within Consul only allows for a single port per service.
It is possible to configure multiple services, each with their own id, which seems to be the default for registrator, though that adds complications in things like health checks, maintenance mode, etc.
Another option would be to have a unique service id and then just store the ports somewhere in the consul KV store, keyed by the service id, though that would be a bunch of extra calls and it would be nice to have it all encapsulated in the service definition.

We are running in to this problem because on one host we have 1 consul container and various other containers, two of which may contain the same service, in which case they would need to use different ports, and we would need that to be easily discoverable.

Any thoughts?

Thanks!

[highlyunavailable]

I've run into this as well and unfortunately because the srv record exposes the port, I think it's a bit of a painful proposition from the view of DNS. I'd really love to have a multi-port option though.

[armon]

This is on our roadmap for a V2 API where we can redesign some of the assumptions of V1. As @highlyunavailable said, the DNS perspective makes it very difficult. That said, we have some ideas as to how to tackle it, but we are probably a few Consul versions away from this.

[chrismiller]

Have you considered using tags on the service itself to hold your port details? We use tags as a basic KV store to hold various bits of metadata about the service and this has worked well for us so far. In your case you could tag your service with "main_port:1234" and "admin_port:1235". The only slight downside is having to manage splitting the KV components yourself. Also currently it is not possible to filter by more than one tag when querying for services, though I understand this is coming in a future release.

[josdotso]

Can we revisit this discussion? Services with multiple ports are common (e.g. MySQL Galera) and until Consul treats multi-port services as first class citizens, I am hesitant to recommend it for enterprise use.

One "killer app" would be to register multi-port services in Consul and consume the resulting SRV and A records using the new Nginx Plus feature [1] that dynamic re-configures load-balancing by looking up SRV records. I'm still working to verify this, but it would seem Nginx Plus may be superior for this use case than HAProxy-based solutions -- because Nginx Plus includes dynamic reconfiguration and this new feature of SRV lookups.

This is particularly relevant in "scheduled container" scenarios where worker-hosts and container exposition ports are randomized for workload scheduling purposes.

[1] https://www.nginx.com/blog/nginx-plus-r9-released/#dns-srv

Related links:
https://www.nginx.com/blog/service-discovery-with-nginx-plus-and-consul/

[teenageorge]

Hello there, I've got a question related to the service ports.

I have to select an open random port at the time of starting my java micro service (I do it in my start.sh of the service) and assign it to the service. I will register this service to Consul using the HTTP API by curl and the Port field will have the same random port I selected. Because of this, I cannot map my domain and port in a proxy configuration.
My argument is, I gave the port number to Consul while registering the service, and Consul should be able to decide the port instead of using the default HTTP port 80.
Otherwise, is there anyway Consul will do this mapping: http://my-service.service.consul:7020 --> http://my-service.service.consul ?
Note: I won't be using Docker or Nginx.
This feature will greatly favour us for using consul for service discovery.

Thanks in advance!
Question posted in Stack Overflow Random ports for microservices - consul

[lord2800]

@teenageorge no, you would have to have a proxy that can reroute the port traffic from one to the other. Consul doesn't do that.

[teenageorge]

@lord2800 thank you for your reply. If I may ask, in your opinion, what would be the ideal tool for mapping HTTP requests from port 80 to any assigned random port? I believe I should be able to do this from start.sh of my micro service. Thanks again.

[lord2800]

Nginx is good. HAProxy is also good. @josdotso's comment above yours suggests that nginx might be the best option if you have nginx plus (as it can automatically configure itself from SRV records).

[teenageorge]

@lord2800 thank you! Let me have a look at Nginx.

[slackpad]

I'm going to close this out as service tags provide a working mechanism for this that is widely used.

[ethicalmohit]

@slackpad Is there any way to do the same with amazon ecs and dynamic host ports? Also, let me know if we can set service_tags with the registrator or any other method/ref.

[suchisubhra]

@slackpad "I'm going to close this out as service tags provide a working mechanism for this that is widely used." can you please elaborate a little bit more? can I add multiple service names? How does it work

[sibblegp]

@slackpad Also wondering how to use service tags.

[hashibot[bot]]

Hey there,

This issue has been automatically locked because it is closed and there hasn't been any activity for at least _30_ days.

If you are still experiencing problems, or still have questions, feel free to open a new one :+1:.