Cluster-api-provider-aws: Validate networking rules and configuration

[randomvariable]

/kind feature

Describe the solution you'd like
[A clear and concise description of what you want to happen.]
As a pre-req for cross-AZ support, we need to validate the security group ruleset to include IP in IP for cross-subnet traffic.

Additionally, Calico is deployed here with the default MTU of 1440 bytes, but we should bump this to 8981 bytes which is the max MTU for IP in IP inside standard jumbo ethernet frames, which is supported by AWS.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api-provider-aws version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[vincepri]

Cross-AZ shouldn't be a problem with default configuration (our or AWS), the intra-VPC traffic is always allowed within the private subnets. Calico and MTU is definitely something to consider to improve performance.

/milestone Next
/good-first-issue
/help-wanted

[k8s-ci-robot]

@vincepri:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

Cross-AZ shouldn't be a problem with default configuration (our or AWS), the intra-VPC traffic is always allowed within the private subnets. Calico and MTU is definitely something to consider to improve performance.

/milestone Next
/good-first-issue
/help-wanted

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@vincepri: You must be a member of the kubernetes-sigs/cluster-api-provider-aws-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Cluster API Provider AWS Maintainers and have them propose you as an additional delegate for this responsibility.

In response to this:

Cross-AZ shouldn't be a problem with default configuration (our or AWS), the intra-VPC traffic is always allowed within the private subnets. Calico and MTU is definitely something to consider to improve performance.

/milestone Next
/good-first-issue
/help-wanted

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[detiber]

Additionally, Calico is deployed here with the default MTU of 1440 bytes, but we should bump this to 8981 bytes which is the max MTU for IP in IP inside standard jumbo ethernet frames, which is supported by AWS.

We'd have to be careful with this, since not all instances types have support for jumbo frames.

[detiber]

/milestone Next

[detiber]

Cross-AZ shouldn't be a problem with default configuration (our or AWS), the intra-VPC traffic is always allowed within the private subnets.

While Security Groups shouldn't be an issue, we will have to make sure that the Load Balancer that we deploy supports multi-AZ, the routes are configured correctly, and Internet Gateways/NAT gateways are sanely handled as well.

[randomvariable]

It's the truly ancient instance types that won't support jumbo frames. I think it'd be safe from a supportability perspective to only support C3 or newer.

[randomvariable]

Just got some field reports on the MTU setting:

we're using m5.xlarge instance types. These instances have interfaces with MTUs set at 9001, instead of > the standard 1500...we were able to deploy an cluster properly but were having some issues with certain functionality, particularly the metrics api. The error we were seeing that keyed us into there being an issue was:

$ kubectl api-resources
couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request

The resolution to this was to set the MTU for Calico to the proper amount for our instance type.

Potentially will need to provide guidance for the different instance types, or standardise on jumbo frames in the AMIs

[sfzylad]

/lifecycle active