Amazon-eks-ami: kubelet occasionally fails on new nodes

[altaurog]

What happened: I created an ec2 autoscaling group for EKS cluster. Sometimes (more than 20% of the time) the instance does not join the cluster. kubelet produces an error message as follows:

Feb 11 14:07:04 ip-172-31-19-62.eu-central-1.compute.internal kubelet[21578]: F0211 14:07:04.216688   21578 server.go:245] unable to load client CA file /etc/kubernetes/pki/ca.crt: error reading /etc/kubernetes/pki/ca.crt: data does not contain any valid RSA or ECDSA certificates

What you expected to happen: nodes should register themselves with kubernetes master. If they are not able to do so, they should be replaced by ASG.

How to reproduce it (as minimally and precisely as possible):
Here’s a cloudformation template which is more or less what I use to launch my cluster and autoscaling group. Launching test stacks every couple of days, I saw this error twice in about a week.

Description: >
  Deploy an EKS cluster using an AutoScaling Group, an EKS service for
  the web server, and an elastic load balancer.

Parameters:
  TemplateS3Base:
    Type: String
    Default: https://s3.amazonaws.com/siemens-eks-test
    Description: |
      Base S3 URL for templates.  This is just a convenient way
      to make relative template references for sub-stacks.

  VPC:
    Type: AWS::EC2::VPC::Id

  Subnets:
    Type: List<AWS::EC2::Subnet::Id>

  SecurityGroup:
    Type: AWS::EC2::SecurityGroup::Id
    Description: Security group for k8s control plane

  InstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup::Id
    Description: Security group for ec2 instances (k8s nodes)

  ServiceRoleARN:
    Type: String
    Default: arn:aws:iam::account-id:role/eksServiceRole

  InstanceProfile:
    Type: String
    Default: eks-node-instance-profile

  ClusterName:
    Type: String
    Default: eks-mwe

  AMI:
    Type: AWS::EC2::Image::Id
    Description: AMI id for the node instances.
    Default: ami-010caa98bae9a09e2

  InstanceType:
    Description: EC2 instance type for the node instances
    Type: String
    Default: t3.medium

  KeyName:
    Type: AWS::EC2::KeyPair::KeyName

Resources:

  Cluster:
    Type: AWS::EKS::Cluster
    Properties:
      Name: !Ref ClusterName
      RoleArn: !Ref ServiceRoleARN
      ResourcesVpcConfig:
        SubnetIds: !Ref Subnets
        SecurityGroupIds:
          - !Ref SecurityGroup

  NodeGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      DesiredCapacity: 1
      LaunchConfigurationName: !Ref NodeLaunchConfig
      MinSize: 1
      MaxSize: 1
      VPCZoneIdentifier:
        !Ref Subnets
      Tags:
      - Key: Name
        Value: !Sub "${ClusterName}-Node"
        PropagateAtLaunch: 'true'
      - Key: !Sub 'kubernetes.io/cluster/${ClusterName}'
        Value: 'owned'
        PropagateAtLaunch: 'true'
    UpdatePolicy:
      AutoScalingRollingUpdate:
        MaxBatchSize: '1'
        MinInstancesInService: 1
        PauseTime: 'PT5M'

  NodeLaunchConfig:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
      AssociatePublicIpAddress: 'true'
      IamInstanceProfile: !Ref InstanceProfile
      ImageId: !Ref AMI
      InstanceType: !Ref InstanceType
      KeyName: !Ref KeyName
      SecurityGroups:
      - !Ref InstanceSecurityGroup
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash
          yum install -y aws-cfn-bootstrap
          /opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource NodeLaunchConfig
          /opt/aws/bin/cfn-signal -e $? --region ${AWS::Region} --stack ${AWS::StackName} --resource NodeGroup

Anything else we need to know?:
Using the following script as ExecStopPost in kubelet.service unit seems to do the trick, but granting the instance permissions to update only its own health status seems not to be as easy as it should be. (Actually I could probably accomplish the same thing by shutting the instance down. That wouldn’t require any additional permissions.)

#!/bin/bash
#
# this script is run by systemd every time kubelet service exits
# withdraw from autoscaling group if kubelet is failing
# by updating asg health check directly

if [[ $(journalctl -u kubelet | fgrep 'kubelet.service failed'  | wc -l) -lt 5 ]]
then
  exit
fi

INSTANCE_ID="$(curl http://169.254.169.254/latest/meta-data/instance-id)"
aws autoscaling set-instance-health --instance-id "${INSTANCE_ID}" --health-status Unhealthy >> /var/log/kubefail

This doesn’t solve the original problem, but does address a more general issue of keeping health status in sync with node’s participation in cluster.

Environment:

• AWS Region: eu-central-1
• Instance Type(s): t3.medium
• EKS Platform version: eks.1:
• Kubernetes version: 1.11
• Kernel: Linux ip-172-31-21-204.eu-central-1.compute.internal 4.14.88-88.76.amzn2.x86_64 #1 SMP Mon Jan 7 18:43:26 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Release information (run cat /tmp/release on a node): maybe that’s supposed to be /etc/eks/release? there doesn’t seem to be any other file named release on the node.
  

$ cat /etc/eks/release 
BASE_AMI_ID="ami-0a2f0b3efddb5a19a"
BUILD_TIME="Wed Jan  9 00:06:06 UTC 2019"
BUILD_KERNEL="4.14.88-88.76.amzn2.x86_64"
AMI_NAME="amazon-eks-node-1.11-v20190108"
ARCH="x86_64"

[micahhausler]

Thanks for this report, we'll take a look

[bphi]

Also having this issue

Edit: After banging my head against the wall for a few hours, I discovered adding DependsOn: Cluster to AutoScalingGroup and LaunchConfiguration fixed it.

[micahhausler]

Great, thanks!

[scottwilson312]

@bphi You are my savior. I've been tearing my hair out over this.

To anyone else coming here - my template was working fine in January. I hopped back on the project today and the same template would result in nodes with the above error. Not sure what changed.