Calico: Advertise Kubernetes Service external IPs

[caseydavenport]

Expected Behavior

Calico already can advertise ClusterIPs. It would be awesome if we could also advertise external IPs. Raising this so we can gather some requirements and use-cases, since this is something we hear about a lot.

Current Behavior

We only advertise ClusterIPs.

Possible Solution

Some things to think about:

• The code should be straightforward and mimic what we do in confd for cluster IPs
• This allows anyone who can create a service to advertise an arbitrary IP address into the network. We probably want to put this behind a feature flag at least, or do some admission control.

[bensallen]

Use case:

Announcing ExternalIPs will allow Services via Kube-proxy and IPVS to be a native L3/L4 load balancer with virtual IPs while maintaining the logical split between CluserIPs being for clients within the cluster and ExternalIPs being for external clients.

Today with only ClusterIP announcement, the CALICO_ADVERTISE_CLUSTER_IPS subnet needs to be routable to external clients, to be used in this fashion.

There is a concern on restricting users from arbitrarily announcing IPs. One idea, ExternalIP address announcement should be restricted to be within one or more ipPools. IPPools will likely need a new attribute, eg. external: true. This setting would also need to have the added effect of the pool not being considered for pod IPs. Like other ipPools, assignment to one or more namespaces should be supported, thus restricting the use and announcement of that IP block to the configured namespace.

[tompntn]

I would be very interested in working to make this a reality. As far as I can tell, the high-level requirements are:

• Automatically advertise all ExternalIP's attached to a service.
• Whitelist these advertisements, to prevent users from advertising arbitrary addresses.
• Implement this whitelisting in a way that is easily and simplify configurable, and works well with Calico's existing data model.

@caseydavenport @neiljerram @bensallen Any other thoughts on the requirements/use cases?

[tompntn]

@bensallen Regarding the IP Pool external: true idea, I wonder if it might be cleaner to simply whitelist all advertisements that fall within _any_ IP Pool? Then to define an "external" IP Pool the user could just create a disabled IP Pool.

[bensallen]

@tompntn your high-level requirements look good.

There also needs to be a way to associate the specific BGP peers and AS numbers to use for announcing the external IPs. It seems possible that different routers may be used for external announcement versus pod networks, or there may also be different peers depending on the external IP pool.

Per your second point, I'd think allowing an advertisement of an IP within any pool that is either global or scoped within the namespace will work.

Perhaps instead of external: true, if a IP pool is associated with a BGP Peer configuration then announce it, otherwise complain in some fashion and do nothing.

[fasaxc]

There also needs to be a way to associate the specific BGP peers and AS numbers to use for announcing the external IPs.

That sounds like over-complicating things to me. It can be added later if someone comes along with a concrete need. For those who want to do such things, they can always do them at the router side by configuring their import policy.

[tompntn]

That sounds like over-complicating things to me. It can be added later if someone comes along with a concrete need. For those who want to do such things, they can always do them at the router side by configuring their import policy.

Agreed, I think for now we should focus on the core feature of advertising whitelisted IP's.

[lmm]

After some discussion on possible approaches with @caseydavenport, we could do this by adding to BGPConfiguration. It might look like:

apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: true
  asNumber: 63400
  serviceExternalIPs:
  - cidr: 104.244.42.129/32
  - cidr: 172.217.3.0/24

Some advantages over using IPPools or environment variables (on the calico/node daemonset):

• Allows updating external IPs advertisement without a rolling restart of the calico/node daemonset
• Allows immediate feedback and input validation
• Moves us towards a more narrowly-defined Calico configuration model (over env variables).

IPPools are probably not the best place since ippools are for endpoint IPs (so we're either overloading the concept of ippools or widening it into a generic pool of IPs).

As part of this feature, we'll probably want to migrate ClusterIP advertisement over to BGPConfiguration as well.

Edit: Based on feedback from @caseydavenport I changed the field from externalServiceIPs to serviceExternalIPs which is now a list.

[bensallen]

@lmm Extending BGPConfiguration seems like a pretty elegant approach.

Is it possible to extend BGPConfiguration to limit advertisements to specific namespaces? Perhaps via the naming scheme currently used for node specific configuration, eg. namespace.<name>. Although it might be more extensible to select namespaces in the same fashion as IPPools via annotation or perhaps an attribute similar to nodeSelector.

In addition to support ClusterIP and ExternalIP advertisement, LoadBalancerIP advertisement would be useful for integration of a service like MetalLB.

[lmm]

@bensallen if an operator wanted to limit ingress to a service to a specific namespace, they could use network policy.

In addition to support ClusterIP and ExternalIP advertisement, LoadBalancerIP advertisement would be useful for integration of a service like MetalLB.

I agree! 😄It would certainly be nice to improve our integration with MetalLB for our bare-metal users.

[bensallen]

@lmm, not seeing an obvious way to go about limiting ingress to specific service's IP(s) via NetworkPolicy. It seems like you'd have to use something like:

ingress:
  - to:
    - ipBlock:
        cidr: 104.244.42.129/32
    - ipBlock:
        cidr: 172.217.3.0/24

But ingress only supports from. Any more pointers on how you'd go about this?

Another issue, say a user allocates an IP that they're not supposed to, if we get network policies to work, then no ingress traffic gets routed to their pods. However will the legitimate user still be able to use their IPs if they're allocated already? In other words, does relying just on network policies allow a DoS of IP address resources?

[lmm]

@bensallen sorry for the delay. What I had in mind was that you could employ network policy to limit access to the set of pods backing a service to a specific namespace: https://docs.projectcalico.org/master/security/namespace-policy

[bensallen]

@lmm, no worries. Ideally I'd like to go the next step and allow users in a multi-tenant environment provision their own services, but I can see how network policies could be used as an interim solution, where services are only allowed to be provisioned via admins.

[lmm]

@bensallen I see. Tigera Secure has a bit more flexibility in this regard with the ability to slot policies into tiers: https://docs.tigera.io/v2.5/reference/resources/tier

Admins can set Tiers with baseline policies and give users access only to certain tiers/policies:
https://docs.tigera.io/v2.5/reference/cnx/rbac-tiered-policies#policy-and-tier-rbac