Packer: Support for AWS SSM Session Manager

[e-moshaya]

Please search the existing issues for relevant feature requests, and use the
reaction feature
(https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/)
to add upvotes to pre-existing requests.

https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html

Feature Description

Due to security, we disabled all SSH 22 ports on aws have switched to session manager. We would like to utilise session manager for connections to our packer instances. The way this would work would be to extend ssh_interface to return the instance id, so instead of "private_ip" we would set ssh_interface: "id".

Use Case(s)

https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html

test-kitchen https://github.com/test-kitchen/kitchen-ec2 has already been updated to support session manager so it would be great if packer can support too.

[rickard-von-essen]

This would require a new communicator. A good start is the SSM API Docs - StartSession.

[e-moshaya]

@rickard-von-essen this does not need a new communicator at all. We can still use ssh to connect over ssh. The changes should be very minor.

From the client's side, we just need to add the following to ~/.ssh/config based on aws documentation:

# SSH over Session Manager
host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

Please check the change to the ec2 driver for the test kitchen framework made to support ssm:

https://github.com/test-kitchen/kitchen-ec2/pull/461

My .kitchen.yml template is, with the transport still set to ssh. Only change is I changed the ssh interface from private_ip to id:

driver:
  name: ec2
  aws_ssh_key_id: test-kitchen
  instance_type: t3.small
  security_group_filter:
    tag:   'Name'
    value: 'kitchen-instance'
  region: eu-west-2
  subnet_filter:
    tag:   'Name'
    value: 'syn-ci-cd-private'
  iam_profile_name: test-kitchen
  instance_initiated_shutdown_behavior: terminate
  interface: id
  tags: { created-by: "test-kitchen",
          Owner: "Jenkins" }
  image_search:
    owner-id: "137112412989"
    name: amzn2-ami-hvm-*gp2

transport:
  ssh_key: ~/.ssh/test-kitchen.pem
  connection_timeout: 10
  connection_retries: 5
  username: ec2-user
  keepalive: true
  name: ssh
  max_ssh_sessions: 2

provisioner:
  name: ansible_playbook
  playbook: test/default.yml
  roles_path: ../../roles
  hosts: all
  require_ansible_repo: false
  require_ansible_omnibus: false
  require_chef_omnibus: false
  ansible_verbose: true
  idempotency_test: false
  ansible_diff: true
  max_retries: 2
  require_chef_for_busser: true
  require_pip: true
  enable_yum_epel: true
  requirements_path: "requirements.yml"
  ssh_known_hosts: ["github.com"]

verifier:
  sudo_path: true

platforms:
- name: amazon-linux

suites:
  - name: default

[awiddersheim]

@e-moshaya It isn't that simple I'm afraid. I looked at this as well when doing the test-kitchen updates and found that packer uses go's SSH library to make SSH connections where as test-kitchen uses your systems SSH which made this easy. The SSH library used by packer/go doesn't have anything built-in to read your local SSH config either AFAIK (https://github.com/golang/go/issues/18781). So you'd have to either add support for that and ensure it can also do ProxyCommand things which I'm not sure about or go @rickard-von-essen route of creating a new communicator.

[rickard-von-essen]

What @awiddersheim, Packer doesn't depend on any of OpenSSH, AWS cli, or Python. We would need a new comminucator that issses the API call and then use the web socket to send and receive data.

I don't think this is a very big task but would require someone familiar with Go.

[e-moshaya]

Thanks for the update @awiddersheim @rickard-von-essen would be fantastic if Packer is updated to support ssm seeing as many companies are aborting ssh for session manager on AWS. We use ssm exclusively with the exception of Packer instances. Supporting SSM will allow us to block all port 22 inbound from security groups.

[ericlamus]

Like the original poster, my organization disabled SSH and is requiring Sessions Manager. I'm currently evaluating Packer to build our machine images.

Is there an ETA on when Session Manager will be supported in Packer?

[SwampDragons]

We don't have an ETA, but I'll try to have a team member investigate how feasible it is to add this functionality for the 1.5.3 release.

[j-rat]

This would be totally awesome!

[nywilken]

Hi folks :wave: I am currently looking into how to best integrate with the SSM Session Manager. For those folks using this service could you take a few minutes to answer some questions for me. I'm looking to understand exactly what level of support is needed.

_Note: I'm not diving into the specifics of the websocket connection here as I am still understanding how it works._

General provisioning

1. Packer currently creates temporary security groups on image creation are you using pre-existing security groups with no SSH ports?

2. I would suspect that when using SSM any created security groups should not open up port 22. Please let me know if that is not correct?

   SSM over SSH

3. Are you able to use SSH if it is proxied or tunneled through an SSM Session?

4. If no SSM SSH-Proxy or tunnel is allowed are you connecting through an SSM Shell or Powershell session?

5. Are you connecting some other way? If so, can you explain.

   SSM for Windows Instance (non-ssh)

6. What is the current solution being used for connecting to Windows instances?

7. If its possible to tunnel a WinRM connection through an SSM Tunnel Session would this be an acceptable solution when creating a new AMI via Packer in your environment?
   

   Provisioning and SSM Documents

There looks to be a number of SSM documents for executing remote commands on instances managed through SSM.

1. What are the general expectations for you as a user when using SSM with Packer?
2. Do you anticipate support for provisioning via some run command document?

Thanks in advance for your time and help in answering the above questions.

[kchristensen]

General provisioning

1. We are using pre-defined security groups that do not have SSH open.
2. That is correct.

SSM over SSH

1. Proxied
2. N/A
3. N/A

Provisioning and SSM Documents

1. I can't say we'd use SSM Documents in combination with Packer, aside from the possibly the option outlined in #2. However, Packer's inline provisioning commands work well for us.
2. See above. If all things remained the same, but we had native support for connecting via session manager we'd be fine with not having document support.

[gudlyf]

General provisioning

1. Pre-existing, no SSH
2. Correct. The goal is to not require or use SSH at all -- assume no direct route.

SSM over SSH

1. Either. But the goal would be SSM Document run.
2. N/A
3. N/A

SSM for Windows Instance (non-ssh)

1. N/A
2. N/A

Provisioning and SSM Documents

1. Ideally calling at Document that runs Ansible playbooks
2. That is what we'd want/expect with SSM support -- running custom or AWS-provided Documents

[e-moshaya]

We use ssm extensively please see below:

ssm sessions via the session manager can be established with zero security groups attached to the aws instance. You can also attach a security group with no port 22 inbound rule and it will still work.

However, the session cannot be established without the correct IAM policy attached to the IAM role assumed by the aws instance. In other words, you need packer to attach an IAM role with the following Amazon managed IAM policy "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore" policy summary as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:DescribeAssociation",
                "ssm:GetDeployablePatchSnapshotForInstance",
                "ssm:GetDocument",
                "ssm:DescribeDocument",
                "ssm:GetManifest",
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:ListAssociations",
                "ssm:ListInstanceAssociations",
                "ssm:PutInventory",
                "ssm:PutComplianceItems",
                "ssm:PutConfigurePackageResult",
                "ssm:UpdateAssociationStatus",
                "ssm:UpdateInstanceAssociationStatus",
                "ssm:UpdateInstanceInformation"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssmmessages:CreateControlChannel",
                "ssmmessages:CreateDataChannel",
                "ssmmessages:OpenControlChannel",
                "ssmmessages:OpenDataChannel"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2messages:AcknowledgeMessage",
                "ec2messages:DeleteMessage",
                "ec2messages:FailMessage",
                "ec2messages:GetEndpoint",
                "ec2messages:GetMessages",
                "ec2messages:SendReply"
            ],
            "Resource": "*"
        }
    ]
}

You can use ssh with ssm if the client configures their ssh config with a proxy command for the host using this guide

[pascal-hofmann]

General provisioning

1. Temporary security groups.
2. For SSM no ports would need to be exposed.

SSM over SSH

1. This is possible via host i-* mi-*\nProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'" in ssh config, but we currently don't use it, as it requires SSH keys (which we don't want to manage/use). This would be no issue for packer though, as it could use its temporary SSH keys.
2. We use aws ssm start-session --target <instanceid> to connect to the instances.
3. No.

SSM for Windows Instance (non-ssh)

1. We do not use WinRM at all. We enable SSH on the machines. For Packer we use SSH with a bastion host to provision the instances. Otherwise we use aws ssm start-session…. So there is no difference for us to Linux.
2. N/A as we use SSH anyway.

Provisioning and SSM Documents

• It would be great to have the required, minimal IAM permissions attached automatically. But I could live without this.
• No. Provisioning via SSM Session Manager is all we need.

The information about required IAM policies provided above is not 100% correct. arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore contains a lot of permissions that are not required for SSM Session Manager / SSM RunCommand.

We use the following very basic IAM policies:

SSM Session Manager

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EnableAccessViaSSMSessionManager",
            "Effect": "Allow",
            "Action": [
                "ssmmessages:OpenDataChannel",
                "ssmmessages:OpenControlChannel",
                "ssmmessages:CreateDataChannel",
                "ssmmessages:CreateControlChannel",
                "ssm:UpdateInstanceInformation"
            ],
            "Resource": "*"
        }
    ]
}

SSM RunCommand

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EnableSSMRunCommand",
            "Effect": "Allow",
            "Action": [
                "ssm:UpdateInstanceInformation",
                "ec2messages:SendReply",
                "ec2messages:GetMessages",
                "ec2messages:GetEndpoint",
                "ec2messages:FailMessage",
                "ec2messages:DeleteMessage",
                "ec2messages:AcknowledgeMessage"
            ],
            "Resource": "*"
        }
    ]
}

[nywilken]

Folks, thanks for the quick turnaround on responses to my questions. All super helpful!

I've bumped this feature to the 1.6.0 milestone seeing as 1.5.4 has already been released :wink:

As for the IAM role attachment, I'm aware of the req. and will look at that being an id or name that a user can specify in the configuration as to not have Packer make any IAM assumptions for the user (current thought). With that said, once I have something ready for preview I'll ping the thread for some initial testing.

Thanks again!

[nydalal]

Could you please update with example on how to use with ssm? would be awesome.

[SwampDragons]

@nydalal We are still working on the implementation; currently you can't use SSM.

[aircraft-cerier]

SSM functionality with Packer will be much appreciated! We are aborting SSH as well, and SSM is the future.

[nywilken]

Hey folks, I'm in the process of implementing this. I've tested it from a Linux machine, for both Linux and Windows EBS builds, but haven't tried for Windows. Currently it's only enabled for the SSH communicator, as WinRM is not currently supported. I'd appreciate some testing and feedback; binaries can be found at https://circleci.com/gh/hashicorp/packer/47787#artifacts/containers/0

The instructions for running a build with the Amazon EBS builder can be found within the PR description https://github.com/hashicorp/packer/pull/9082#issue-405081246

Thanks!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.