Amazon-ssm-agent: Health ping failed with error - AccessDeniedException

[dwilches]

Hello,

I started seeing this error message in several of my Ubuntu VMs today after snap restarted the amazon-ssm-agent:

Failed to load instance info from vault. RegistrationKey does not exist.
Error occurred fetching the seelog config file path:  open /etc/amazon/ssm/seelog.xml: no such file or directory
Error adding the directory to watcher: no such file or directory
Health ping failed with error - AccessDeniedException: User: arn:aws:sts::<redacted>:assumed-role/<redacted> is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:<redacted>

My currently installed version is:

$ snap list
Name              Version    Rev   Tracking  Publisher   Notes
amazon-ssm-agent  2.3.274.0  930   stable/…  aws✓        classic

Do you know what can be wrong? We have not changed any permissions/roles recently.

I also checked at these issues that mention similar problems, but not quite the same: 109 and 114

[AmyZhangAmz]

Thank you for posting here. The reason you received “AccessDeniedException” is that the IAM role you attached to this instance doesn't contain correct policy. In order to use AWS System Manager service, your instance need to have permission for System Manager API operation.
Please verify your instance is configured correctly and find more detailed information from this link: https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-access.html

[forzagreen]

You will need to attach the following policy to your instance profile: arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

View policy details

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:DescribeAssociation",
                "ssm:GetDeployablePatchSnapshotForInstance",
                "ssm:GetDocument",
                "ssm:DescribeDocument",
                "ssm:GetManifest",
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:ListAssociations",
                "ssm:ListInstanceAssociations",
                "ssm:PutInventory",
                "ssm:PutComplianceItems",
                "ssm:PutConfigurePackageResult",
                "ssm:UpdateAssociationStatus",
                "ssm:UpdateInstanceAssociationStatus",
                "ssm:UpdateInstanceInformation"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssmmessages:CreateControlChannel",
                "ssmmessages:CreateDataChannel",
                "ssmmessages:OpenControlChannel",
                "ssmmessages:OpenDataChannel"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2messages:AcknowledgeMessage",
                "ec2messages:DeleteMessage",
                "ec2messages:FailMessage",
                "ec2messages:GetEndpoint",
                "ec2messages:GetMessages",
                "ec2messages:SendReply"
            ],
            "Resource": "*"
        }
    ]
}

Reference: https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-quick-setup.html#quick-setup-instance-profile

[shokunin-kishitsu]

I've just received this error in the logs when trying to start an SSM Agent manually. The instance definitely has the arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore policy.

[raxod502]

In my testing, I needed to reboot the instance after attaching the policy. Simply restarting the SSM agent did not have any effect, for some reason.