Amazon-ssm-agent: Unable to start AWS ssm-agent service on Win server 2016 Datacenter

[GarbageYard]

I am trying to setup Run command. One of the prerequisite was to check whether ssm-agent is running. I checked services on AWS Windows sever 2016 instance but the service status was "Starting". After few tries, i removed the service and downloaded the latest binary which is version 2.2.916.0. It's also giving the same message "Starting". I tried running the executable on the command prompt with Administrator privilege and i see the following:

C:\>"C:\Program Files\Amazon\SSM\amazon-ssm-agent.exe"
2018/08/06 07:17:30 Failed to load instance info from vault. RegistrationKey does not exist.
Initializing new seelog logger
New Seelog Logger Creation Complete
2018-08-06 07:17:30 INFO Windows Only: Job object creation on SSM agent successful
2018-08-06 07:17:30 INFO Agent enters hibernate mode. Reducing logging...
2018-08-06 07:17:30 INFO Getting IE proxy configuration for current user: The operation completed successfully.
2018-08-06 07:17:30 INFO Getting WinHTTP proxy default configuration: The operation completed successfully.
2018-08-06 07:17:30 INFO Proxy environment variables:
2018-08-06 07:17:30 INFO http_proxy:
2018-08-06 07:17:30 INFO https_proxy:
2018-08-06 07:17:30 INFO no_proxy:
2018-08-06 07:22:31 INFO Backing off health check to every 600 minutes for 1800 minutes. Logging will be reduced to one log per backoff period

I have another Windows server 2016 instance where ssm-agent version 2.2.392.0 is installed. That's running fine. I tried looking for Windows binary of that version but it seems only source code is available to download which has to be compiled.

Any idea what's the issue with the latest version? Also, are older versions' binary available for download anywhere?

[Kitsune-Fox]

Check your persistent routes, there should be 3 routes on your instance that use your default gateway as the gateway for the following IPs:
169.254.169.250
169.254.169.251
169.254.169.254

[GarbageYard]

Sorry, but i didn't get it. Not much into networking. How can i check this and what exactly does it imply?

[Kitsune-Fox]

Sorry about that.

Basically the error you are receiving is stating that it cannot access the meta-data service to pick up the IAM credentials.

This is usually due to the routing table not routing traffic to the correct gateway for your subnet.

To check the current routes on the instance you can open an administrative command prompt window and run the following command:
Route Print

Please Note Below I am assuming that 10.0.0.1 is your Default Gateway address, please replace it with your own instance default gateway address.

You should see 3 persistent routes as below:
Persistent Routes:
Network Address Netmask Gateway Address Metric
169.254.169.254 255.255.255.255 10.0.0.1 15
169.254.169.250 255.255.255.255 10.0.0.1 15
169.254.169.251 255.255.255.255 10.0.0.1 15

If these routes do not exist or are pointing to a different gateway address, please delete the routes by running the following from an Administrative command prompt:
Route Delete 169.254.169.254
Route Delete 169.254.169.250
Route Delete 169.254.169.251

You can then add them in again by running the following (replace 10.0.0.1 with your gateway address):
route -p ADD 169.254.169.254 MASK 255.255.255.255 10.0.0.1
route -p ADD 169.254.169.250 MASK 255.255.255.255 10.0.0.1
route -p ADD 169.254.169.251 MASK 255.255.255.255 10.0.0.1

If you have any Proxy settings that may be applied to the instance. There needs to be proxy exceptions made for the following IP addresses as well: 169.254.169.254, 169.254.169.250 and 169.254.169.251.

Once you have completed these steps the service should be able to access the meta-data service and the instance will then start up normally.

[GarbageYard]

Thanks for the detailed explanation @Kitsune-Fox :) I checked on both the instances (where it's working and where it isn't) and i could see identical persistent routes (under IPv4 Route Table section) on both instances and similar to yours _except_ for the default gateway (172.x.x.x in my case). So based on your explanation, i don't think i need to do anything in this regard, correct?

About the proxy settings, i checked online and few sites mentioned checking browser settings. I checked on both the instances and there was no manual setting in browser's Proxy section although i am still able to access Internet on both the machines. It seems that was the right way to check presence of a proxy server. Please correct me.

[GarbageYard]

Can anyone kindly suggest what's the issue? I am clueless.

[dj-mb]

If these are EC2 instances, ensure that you have the correct IAM role attached. Or you could match the working vs non-working.
Make sure that you are not activating the instance as a hybrid. If you are, then you need to follow this article: https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-managed-win.html
or just attach the IAM role and provide internet access. it should work.

[vchen-panopto]

We are getting the same AmazonSSMAgent service stuck in the "starting" state with version 2.2.916.0. Feels like a regression since our AMI generation work flow is unchanged and the only variable is the newer SSM agent.

REPRO

1. Download and install the latest SSM Agent (AmazonSSMAgentSetup.exe /quiet)
2. Download and install the latest EC2 Launch
3. Capture instance as AMI
4. Launch AMI from packer

Expected:
Packer instance runs normally

Actual:
EC2 Launch InitializeInstance.ps1 is stuck waiting for SSM Agent to launch.
SSM Agent log ends at:

2018-08-06 07:17:30 INFO Proxy environment variables:
2018-08-06 07:17:30 INFO http_proxy:
2018-08-06 07:17:30 INFO https_proxy:
2018-08-06 07:17:30 INFO no_proxy:

[Kitsune-Fox]

I'd recommend downgrading to v 2.2.800.0 for now until this is fixed.

[mmendonca3]

Hello,

Thanks for posting here. We are investigating this issue. In the meanwhile, could you please confirm that you have the right IAM role attached to your instance? Having an instance profile with the AmazonEC2RoleforSSM policy will allow the instance to call the Systems Manager APIs and also start the agent.

[GarbageYard]

@Kitsune-Fox Is compiling the source code of older version only option to downgrade or do we have old binaries available somewhere? I could only find the source code here

@mmendonca3 I'll check with my sys admin as i don't have access to IAM role.

[GarbageYard]

So things have moved a bit further. The IAM role was indeed incorrect. The sys admin fixed it. I can now see the service status changed to "Running" however, the log still says, things aren't really all well:

2018-08-10 16:01:43 ERROR [func1 @ coremanager.go.250] [instanceID=i-04xyzabcas34f3de84] error occurred trying to start core module. Plugin name: StartupProcessor. Error: Internal error occurred by startup processor: runtime error: invalid memory address or nil pointer dereference

[nehalaws]

Thank you for fixing the IAM role, is it possible to share the full agent log and the repro steps by any chance? We are investigating the issue but it would be great to know what triggered it.

[GarbageYard]

I'm not sure what happened in the background but all of a sudden, i was not able to see the one instance that was earlier showing up under "Managed Instances" section of SSM. My EC2 instance list also went away. Later, when i talked to the sys admin, he gave me some access (not sure which accesses) after which the instances started showing in EC2 and multiple instances started showing in "Managed Instances" section of SSM. All is good now but to sum it up, it seems it all had to do with IAM roles. I'm glad the missing instance is now showing in the SSM managed list.

Time to fire some remote commands. ;) Thanks for all the assistance guys! :)

[nehalaws]

Great, glad you were able to get instances up and running. We are still investigating the issue with the agent stuck in starting state and working on a fix.

[antman2]

I know this is a closed issue now but I stumbled across this in Google searches when noticing the error today.

Our Ec2Launch.log shows that it was trying to start AmazonSSMAgent before adding the persistent routes. This only happened once in several instance starts, so potentially it's an intermittent race condition.

2019/03/28 01:41:49Z: Message: Windows is Ready to use
2019/03/28 01:41:58Z: Instance initialization is scheduled successfully
2019/03/28 01:41:58Z: Failed to start AmazonSSMAgent service: Service 'Amazon SSM Agent (AmazonSSMAgent)' cannot be started due to the following error: Cannot start service AmazonSSMAgent on computer '.'.
2019/03/28 01:41:58Z: Initializing instance is done
2019/03/28 01:43:00Z: Initializing instance is started
2019/03/28 01:43:01Z: Opening port (COM1) handle to write to the console
2019/03/28 01:43:01Z: Windows sysprep configuration complete.
2019/03/28 01:43:01Z: Checking primary network interface
2019/03/28 01:43:01Z: Primary network interface found. Adding routes now...
2019/03/28 01:43:02Z: Successfully added the Route: 169.254.169.254/32, gateway: 10.0.4.1, NIC index: 4, Metric: 15
2019/03/28 01:43:02Z: Successfully added the Route: 169.254.169.250/32, gateway: 10.0.4.1, NIC index: 4, Metric: 15
2019/03/28 01:43:02Z: Successfully added the Route: 169.254.169.251/32, gateway: 10.0.4.1, NIC index: 4, Metric: 15
2019/03/28 01:43:02Z: Message: Waiting for meta-data accessibility...
2019/03/28 01:43:02Z: Message: Meta-data is now available.