Amazon-ssm-agent: SSM document worker timed out

[pbobov]

SSM command execution occasionally fails on some of target instances with the following error:

document process failed unexpectedly: document worker timed out , check [ssm-document-worker]/[ssm-session-worker] log for crash reason

The document shell script execution did not even start.

Here is the SSM agent log from the EC2 instance where the command execution failed.

19-11-13 19:05:39 INFO [StartupProcessor] Write to serial port: Amazon SSM Agent v2.3.672.0 is running
...
2019-11-13 23:32:53 INFO [StartupProcessor] Write to serial port: OsProductName: Ubuntu
2019-11-13 23:32:53 INFO [StartupProcessor] Write to serial port: OsVersion: 18.04
2019-11-13 23:32:53 INFO [MessagingDeliveryService] [EngineProcessor] [OutOfProcExecuter] [975c3a2e-df8e-4212-a140-45a935d5a23c] discovered old channel object, trying to find detac
hed process...
2019-11-13 23:32:53 INFO [MessagingDeliveryService] [EngineProcessor] [OutOfProcExecuter] [975c3a2e-df8e-4212-a140-45a935d5a23c] process: 29882 not found, treat as exited
2019-11-13 23:32:53 INFO [MessagingDeliveryService] [EngineProcessor] [OutOfProcExecuter] [975c3a2e-df8e-4212-a140-45a935d5a23c] inter process communication started
2019-11-13 23:32:53 INFO [MessageGatewayService] Opening websocket connection to: wss://ssmmessages.us-east-2.amazonaws.com/v1/control-channel/i-03c4b3ebb738e8c1d?role=subscribe&st
ream=input
2019-11-13 23:32:53 INFO [MessageGatewayService] Successfully opened websocket connection to: wss://ssmmessages.us-east-2.amazonaws.com/v1/control-channel/i-03c4b3ebb738e8c1d?role=
subscribe&stream=input
2019-11-13 23:32:53 INFO [MessageGatewayService] Starting receiving message from control channel
2019-11-13 23:32:53 INFO [MessageGatewayService] [EngineProcessor] Initial processing
2019-11-13 23:32:53 INFO [MessageGatewayService] [EngineProcessor] Found in-progress document - 975c3a2e-df8e-4212-a140-45a935d5a23c
2019-11-13 23:32:56 ERROR [MessagingDeliveryService] [EngineProcessor] [OutOfProcExecuter] [975c3a2e-df8e-4212-a140-45a935d5a23c] received timedout signal!
2019-11-13 23:32:56 INFO [MessagingDeliveryService] [EngineProcessor] [OutOfProcExecuter] [975c3a2e-df8e-4212-a140-45a935d5a23c] channel /var/lib/amazon/ssm/i-3c4b3ebb738e8c1d/cha
nnels/975c3a2e-df8e-4212-a140-45a935d5a23c requested close
2019-11-13 23:32:56 INFO [MessagingDeliveryService] [EngineProcessor] [OutOfProcExecuter] [975c3a2e-df8e-4212-a140-45a935d5a23c] channel /var/lib/amazon/ssm/i-03c4b3ebb738e8c1d/channels/975c3a2e-df8e-4212-a140-45a935d5a23c closed
2019-11-13 23:32:56 INFO [MessagingDeliveryService] [EngineProcessor] [OutOfProcExecuter] [975c3a2e-df8e-4212-a140-45a935d5a23c] ipc channel closed, stop messaging worker
2019-11-13 23:32:56 ERROR [MessagingDeliveryService] [EngineProcessor] [OutOfProcExecuter] [975c3a2e-df8e-4212-a140-45a935d5a23c] messaging worker encountered error: document worker timed out
2019-11-13 23:32:56 INFO [MessagingDeliveryService] [EngineProcessor] [OutOfProcExecuter] [975c3a2e-df8e-4212-a140-45a935d5a23c] document failed half way, sending fail message...
2019-11-13 23:32:56 INFO [MessagingDeliveryService] [EngineProcessor] sending document: 975c3a2e-df8e-4212-a140-45a935d5a23c complete response
2019-11-13 23:32:56 INFO [MessagingDeliveryService] command: aws.ssm.975c3a2e-df8e-4212-a140-45a935d5a23c.i-03c4b3ebb738e8c1d complete
2019-11-13 23:32:56 INFO [MessagingDeliveryService] Sending reply {
  "additionalInfo": {
    "agent": {
      "lang": "en-US",
      "name": "amazon-ssm-agent",
      "os": "",
      "osver": "1",
      "ver": ""
    },
    "dateTime": "2019-11-13T23:32:56.553Z",
    "runId": "",
    "runtimeStatusCounts": {
      "Failed": 1
    }
  },
  "documentStatus": "Failed",
  "documentTraceOutput": "",
  "runtimeStatus": {
    "aws:runShellScript": {
      "status": "Failed",
      "code": 1,
      "name": "",
      "output": "document process failed unexpectedly: document worker timed out , check [ssm-document-worker]/[ssm-session-worker] log for crash reason",
      "startDateTime": "0001-01-01T00:00:00.000Z",
      "endDateTime": "0001-01-01T00:00:00.000Z",
      "outputS3BucketName": "",
      "outputS3KeyPrefix": "",
      "stepName": "",
      "standardOutput": "",
      "standardError": ""
    }
  }
}
2019-11-13 23:32:56 INFO [MessagingDeliveryService] [EngineProcessor] [OutOfProcExecuter] [975c3a2e-df8e-4212-a140-45a935d5a23c] Executer closed

[amzfp]

Hi, thank you for reaching out. Could you kindly provide us with the detailed document worker log for the further investigation?

[AsrarAhmad]

I created an AWS Support ticket and they couldn't identify the problem.
Later moving to SSM Agent Version 2.3.722.0 and then restarting SSM agent helped resolve the issue.

Will be happy to know the root cause here.

[pbobov]

The failure happened because the EC2 instances were restarted by CloudFormation stack update shortly after the SSM command execution started.

[AsrarAhmad]

I would disagree with this root cause, because I faced this same issue in EMR based EC2 which was always up

[dineshrathee12]

_I came across similar issue and there could be some possible reasons :_

• Memory Issue (OOM Killer) - Make sure you check Console Screenshot - https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/screenshot-service.html
• SSM Agent outdated - https://github.com/aws/amazon-ssm-agent , https://github.com/aws/amazon-ssm-agent/releases , Updated ( Agent Version : 3.0.529.0 to 3.0.655.0 ) using Run Command ---> Document name (AWS-UpdateSSMAgent)

_In my use-case, SSM Agent was crashing because of OOM killer and I had a process which was using a lot of memory. Simply stopped the process associated which was taking more memory and also upgraded the agent and I was able to start the session again and issue has been resolved._

References

[+] How do I install AWS Systems Manager Agent (SSM Agent) on an Amazon EC2 Linux instance at launch? : https://aws.amazon.com/premiumsupport/knowledge-center/install-ssm-agent-ec2-linux/
[+] Automating updates to SSM Agent : https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent-automatic-updates.html
[+] To update SSM Agent using Run Command : https://docs.aws.amazon.com/systems-manager/latest/userguide/rc-console.html#rc-console-agentexample
[+] Troubleshoot Linux Out-of-Memory: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstances.html#MemoryOOM