Amazon-ssm-agent: "No such file or directory" error on Ubuntu 16.04

[rarkins]

I'm using the latest Ubuntu 16.04 AMI with ssm-agent preinstalled. However it suffers from this problem:

$ sudo systemctl status amazon-ssm-agent
● amazon-ssm-agent.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

Here is /var/log/amazon/ssm/amazon-ssm-agent.log with timestamps and instance ID removed for brevity:

INFO Init the cloudwatchlogs publisher
INFO Create new startup processor
INFO [LongRunningPluginsManager] registered plugins: {}
INFO [MessagingDeliveryService] Starting document processing engine...
INFO [MessagingDeliveryService] [EngineProcessor] Starting
INFO [MessagingDeliveryService] [EngineProcessor] Initial processing
INFO [MessagingDeliveryService] Starting message polling
INFO [MessagingDeliveryService] Starting send replies to MDS
INFO Starting association polling
INFO [MessagingDeliveryService] [Association] [EngineProcessor] Starting
INFO [MessagingDeliveryService] [Association] Launching response handler
INFO [MessagingDeliveryService] [Association] [EngineProcessor] Initial processing
INFO [MessagingDeliveryService] [Association] Initializing association scheduling service
INFO [MessagingDeliveryService] [Association] Association scheduling service initialized
INFO [OfflineService] Starting document processing engine...
INFO [OfflineService] [EngineProcessor] Starting
INFO [OfflineService] [EngineProcessor] Initial processing
INFO [OfflineService] Starting message polling
INFO [OfflineService] Starting send replies to MDS
INFO [LongRunningPluginsManager] starting long running plugin manager
INFO [LongRunningPluginsManager] there aren't any long running plugin to execute
INFO [HealthCheck] HealthCheck reporting agent health.
INFO [LongRunningPluginsManager] There are no long running plugins currently getting executed - skipping their healthcheck
INFO [StartupProcessor] Executing startup processor tasks
INFO [StartupProcessor] Write to serial port: Amazon SSM Agent v2.2.619.0 is running
INFO [StartupProcessor] Write to serial port: OsProductName: Ubuntu
INFO [StartupProcessor] Write to serial port: OsVersion: 16.04

INFO [MessagingDeliveryService] [Association] No associations on boot. Requerying for associations after 30 seconds.
INFO [MessagingDeliveryService] [Association] Schedule manager refreshed with 0 associations, 0 new assocations associated
INFO [HealthCheck] HealthCheck reporting agent health.
INFO [HealthCheck] HealthCheck reporting agent health.
INFO [MessagingDeliveryService] [Association] Schedule manager refreshed with 0 associations, 0 new assocations associated
INFO [HealthCheck] HealthCheck reporting agent health.

And here's the error log:

ERROR [Start @ config_watcher.go.72] Error adding the directory to watcher: no such file or directory

Manual enabling fails:

$ sudo systemctl enable amazon-ssm-agent
Failed to execute operation: No such file or directory

There is no /etc/amazon directory and hence no /etc/amazon/ssm/seelog.xml.

[nehalaws]

Can you please try: sudo systemctl status snap.amazon-ssm-agent.amazon-ssm-agent the service name on ubuntu is different.

[minfrin]

Have the same problem:

minfrin@sql01:~$ sudo cat /var/log/amazon/ssm/amazon-ssm-agent.log 
2018-10-08 16:16:09 ERROR Error adding the directory to watcher: no such file or directory

ssm-agent is running, but is not visible to AWS.

It is not clear whether the above error message is relevant, the message complains that a directory does not exist, but does not state what the directory is.

This is the output of sudo systemctl status snap.amazon-ssm-agent.amazon-ssm-agent:

● snap.amazon-ssm-agent.amazon-ssm-agent.service - Service for snap application amazon-ssm-agent.amazon-ssm-agent
   Loaded: loaded (/etc/systemd/system/snap.amazon-ssm-agent.amazon-ssm-agent.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-10-08 16:19:08 UTC; 5min ago
 Main PID: 1236 (amazon-ssm-agen)
    Tasks: 7
   Memory: 26.2M
  CPU: 308ms
   CGroup: /system.slice/snap.amazon-ssm-agent.amazon-ssm-agent.service
           └─1236 /snap/amazon-ssm-agent/495/amazon-ssm-agent

Oct 08 16:19:08 sql01 systemd[1]: Started Service for snap application amazon-ssm-agent.amazon-ssm-agent.
Oct 08 16:19:10 sql01 amazon-ssm-agent.amazon-ssm-agent[1236]: 2018/10/08 16:19:10 Failed to load instance info from vault. RegistrationKey doe
Oct 08 16:19:10 sql01 amazon-ssm-agent.amazon-ssm-agent[1236]: Error occurred fetching the seelog config file path:  open /etc/amazon/ssm/seelo
Oct 08 16:19:10 sql01 amazon-ssm-agent.amazon-ssm-agent[1236]: Initializing new seelog logger
Oct 08 16:19:10 sql01 amazon-ssm-agent.amazon-ssm-agent[1236]: New Seelog Logger Creation Complete
Oct 08 16:19:10 sql01 amazon-ssm-agent.amazon-ssm-agent[1236]: 2018/10/08 16:19:10 Found config file at /etc/amazon/ssm/amazon-ssm-agent.json.
Oct 08 16:19:10 sql01 amazon-ssm-agent.amazon-ssm-agent[1236]: Applying config override from /etc/amazon/ssm/amazon-ssm-agent.json.
Oct 08 16:19:10 sql01 amazon-ssm-agent.amazon-ssm-agent[1236]: 2018/10/08 16:19:10 processing appconfig overrides
Oct 08 16:21:21 sql01 amazon-ssm-agent.amazon-ssm-agent[1236]: 2018-10-08 16:21:21 INFO Agent enters hibernate mode. Reducing logging...

[alexwbai]

+1

[nehalaws]

Thank you for posting the issue, ERROR Error adding the directory to watcher: no such file or directory is not really related. From the output of "sudo systemctl status snap.amazon-ssm-agent.amazon-ssm-agent" looks like the agent is running. Can you please explain what do you mean by ssm-agent is not visible to AWS?

[nehalaws]

I also noticed from the logs that the agent goes to hibernation, the reason can be that the instance is missing AmazonEc2RoleForSSM and the agent is not able to reach the service. Can you please verify that the role is attached to the instance?

[ghost]

I experienced the same issue when installing ssm manually on Ubuntu 18.04
If you change your iam role on an existing instance it might take some time for the new policies to take effect.

Remove the policies from the role
re-attach them
wait a few minutes and then restart the ssm agent.

check the amazon-ssm-agent.log for success.

nehalaws is correct that this wont work if the AmazonEc2RoleForSSM policy is not attached to the EC2 Instance Role.

Hope it helps

[adautoneto]

Just restarting the service as @ghost mentioned worked for me.

The command executed: sudo snap restart amazon-ssm-agent on Ubuntu 18.04

[rafiqvns]

sudo service sendmail_after_etl_for_LicenceSKU status

sendmail_after_etl_for_LicenceSKU.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)

Please help

[minusdavid]

So amazon-ssm-agent is working fine for me. I've created the required IAM role, attached it to the EC2 instance, set up VPC endpoints for com.amazonaws.region.ssm and com.amazonaws.region.ssmmessages, set up port 443 access for the VPC endpoints, and I'm able to use Session Manager to get a secure browser based shell.

However, I'm still seeing the following:

Apr 15 04:34:19 ip-10-0-0-231 amazon-ssm-agent.amazon-ssm-agent[2055]: 2019/04/15 04:34:19 Failed to load instance info from vault. RegistrationKey does not exist.
Apr 15 04:34:19 ip-10-0-0-231 amazon-ssm-agent.amazon-ssm-agent[2055]: Error occurred fetching the seelog config file path: open /etc/amazon/ssm/seelog.xml: no such file or directory
Apr 15 04:34:19 ip-10-0-0-231 amazon-ssm-agent.amazon-ssm-agent[2055]: Initializing new seelog logger
Apr 15 04:34:19 ip-10-0-0-231 amazon-ssm-agent.amazon-ssm-agent[2055]: New Seelog Logger Creation Complete
Apr 15 04:34:19 ip-10-0-0-231 amazon-ssm-agent.amazon-ssm-agent[2055]: 2019-04-15 04:34:19 ERROR Error adding the directory to watcher: no such file or directory

I'm using the built-in amazon-ssm-agent snap in Ubuntu 18.04...

[minusdavid]

Set up the remaining VPC endpoints as mentioned in https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-setting-up-vpc.html#sysman-setting-up-vpc-create but that didn't help.

Going to look at the source code now...

[adampblack]

+1 also getting this error "Error adding the directory to watcher: no such file or directory"

When I open: /var/log/amazon/ssm/errors.log

The EC2 instance does have the AmazonEc2RoleForSSM role attached.

Running on ubuntu 18.04.

[dorintt]

I got the same problem on Ubuntu 16 (on-premise) installed from .deb package.
Doing cp /etc/amazon/ssm/seelog.xml.template /etc/amazon/ssm/seelog.xml solved the error (at least the latest package created /etc/amazon/ssm dir for me).

Afterwards I was able to do amazon-ssm-agent -register -code $ACTIVATION_CODE -id $ACTIVATION_ID -region $REGION and have the server visible in systems manager.

However I don't know if it's a robust fix, and I'm still wondering why it can't simply work by default.

[ferkhat-aws]

Issue resolved in #192
Closing.