Amazon-ecs-agent: Metrics for Windows not being sent to ECS

[jamsajones]

Docker stats was added to Windows with the merge of this PR https://github.com/docker/docker/pull/25737 and was back ported to the 1.12.x branch. However it still remains unavailable for calls using pre v1.21 API. However, Agent currently doesn't report metrics for from Windows Container instance nodes. This is in large part due to the fact that docker stats does not return stats in docker API versions pre v1.21 ( docker/docker/daemon/stats.go#77 ).

Agent needs to be updated to the current API version in order to make use of Windows docker stats and re-enable sending of metrics for Windows.

[samuelkarp]

Moving to a new API version for Windows is possible, but we're going to need to maintain a separate implementation for Windows and Linux (the API response structures are different between the two platforms, even if you're using the same version).

[panmanphil]

Autoscaling is impossible without this, has there been any progress made on this issue?

[pkinausleap]

Waiting for this before we can get going with Windows ECS.

[samuelkarp]

This file shows differences between stats exposed on Linux and on Windows.

[ghost]

what is the status of this? I've just today deployed a new AMI to a new cluster with docker version 17.06.2-ee-5 and ECS Agent version v1.15.0 on server 2016 core (based on AMI Windows_Server-2016-English-Core-ContainersLatest-2017.10.13 (ami-6930f611) and included the following user data in the launch config:

[Environment]::SetEnvironmentVariable("ECS_DISABLE_METRICS", "false", "Machine")
[Environment]::SetEnvironmentVariable("ECS_DISABLE_IMAGE_CLEANUP", "true", "Machine")
[Environment]::SetEnvironmentVariable("ECS_IMAGE_MINIMUM_CLEANUP_AGE", "6h", "Machine")
[Environment]::SetEnvironmentVariable("ECS_ENABLE_CONTAINER_METADATA", "true", "Machine")

The instance launched, joined the cluster, is running a service successfully including the CONTAINER_METADATA file, but I still see no metrics being reported despite docker stats working as expected:

PS C:\ProgramData\Amazon\ECS\log> docker version
Client:
 Version:      17.06.2-ee-5
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   508bb92
 Built:        Thu Nov  2 00:51:36 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.06.2-ee-5
 API version:  1.30 (minimum version 1.24)
 Go version:   go1.8.3
 Git commit:   508bb92
 Built:        Thu Nov  2 00:54:07 2017
 OS/Arch:      windows/amd64
 Experimental: false
PS C:\ProgramData\Amazon\ECS\log> docker stats -a --no-stream
CONTAINER           CPU %               PRIV WORKING SET    NET I/O             BLOCK I/O
aebabf3074f9        0.00%               195.7MiB            443kB / 803kB       134MB / 81.6MB

[samuelkarp]

@msheppard-nf That's correct, this is still an open issue. There's additional work to do in the ECS agent beyond getting docker stats to work.

[ghost]

got it, thank you

[samuelkarp]

We've released version 1.16.0 of the ECS agent and a new ECS-optimized Windows AMI that supports metrics. Please see the updated documentation for instructions to get started using the new AMI.