Amazon-ssm-agent: aws ssm start-session end with **Cannot perform start session: EOF**

[alxsbn]

I try to instanciate a vanilla aws ssm start-session on a Github actions Jobs (without SSH).

The session is well instanciated (i can see it on the AWS Session Manager dashboard) but always finish with a Cannot perform start session: EOF

FYI I can instanciated a sesion outside of GH actions (.e.g. on my laptop) with the same account / role / policy / instances (ami-0ea4a063871686f37).

Here's my job YAML :

name: CI

# Controls when the action will run. 
on:
  # Triggers the workflow on push or pull request events but only for the main branch
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      - name: Configure AWS credentials from Test account
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}

      - name: Initialize a Session Manager session
        shell : sh
        run: aws ssm start-session --target ${{ secrets.TARGET_INSTANCE }}

The results :

Run aws ssm start-session --target ***
  aws ssm start-session --target ***
  shell: /bin/sh -e {0}
  env:
    AWS_DEFAULT_REGION: ***
    AWS_REGION: ***
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***

Starting session with SessionId: gh-ssm-0d50831b64f0xxxxx
$ Cannot perform start session: EOF

Here are the CLI tools packaged with Ubuntu-latest virtual env (20.04.2 LTS) :

• AWS CLI 2.1.24
• AWS CLI Session manager plugin 1.2.54.0

[nitikaaws]

Thanks for reaching out to us!

Could you please share logs for Session Manager plugin for us to investigate this further? More information on logging can be found here - https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html#install-plugin-configure-logs.

[keshavstitcher]

Did you get any resolution on this?

I am also facing the same issue when I tried to login via Jenkins job. while using the command line it's working fine.

[olfway]

I had similar issue with "aws ecs execute-command ..." and it seems it's related to tty being not available when running in ci

I fixed it with "unbuffer aws ecs execute-command ..." to run it with tty available

[alxsbn]

It's linked to tty. I bypass the problem with ssm send-command with remote script execution

[emmahsax]

@alxsbn Can you explain more what you meant by "ssm send-command with remote script execution"? We're running into the same issue with

aws ecs execute-command

I don't know if this makes a difference, but we're using ECS with Fargate, not EC2.

[alxsbn]

@emmahsax Our first reflex was to run a ssh command, then the others commands we need (for us it was linked to CD, so rsync a repo then restart a pm2 service). Since we have this tty problem we just write a shell script with our commands an fire him with ssm send-command

[dgholz]

I have the same problem, I followed https://docs.aws.amazon.com/systems-manager/latest/userguide/session-preferences-shell-config.html to add bash -i to my Linux Shell Profile

I enabled logging and see:

2021-05-11 11:10:09 DEBUG Start File Watcher On: /usr/local/sessionmanagerplugin/seelog.xml
2021-05-11 11:10:09 DEBUG Start Watcher on directory: /usr/local/sessionmanagerplugin
2021-05-11 11:10:09 DEBUG Calling Initialize Datachannel for role: publish_subscribe
2021-05-11 11:10:09 INFO Opening websocket connection to: %!(EXTRA string=wss://ssmmessages.eu-west-1.amazonaws.com/v1/data-channel/botocore-session-1620726103-0af510cb5dd7ad804?role=publish_subscribe)
2021-05-11 11:10:09 INFO Successfully opened websocket connection to: %!(EXTRA string=wss://ssmmessages.eu-west-1.amazonaws.com/v1/data-channel/botocore-session-1620726103-0af510cb5dd7ad804?role=publish_subscribe)
2021-05-11 11:10:09 INFO Sending token through data channel wss://ssmmessages.eu-west-1.amazonaws.com/v1/data-channel/botocore-session-1620726103-0af510cb5dd7ad804?role=publish_subscribe to acknowledge connection
2021-05-11 11:10:09 DEBUG WebsocketChannel: Send ping. Message.
2021-05-11 11:10:10 DEBUG Processing stream data message of type: output_stream_data

and the logs end there.

I saw the same issue when running aws ssm start-session --target INSTANCE_ID --document-name AWS-StartInteractiveCommand --parameters command="bash -i":

Starting session with SessionId: botocore-session-1620726103-06d6cbb9a7fcd54a6
ssm-user@ip-10-129-212-60:/var/snap/amazon-ssm-agent/3552$ Cannot perform start session: EOF

It also messes up the terminal, I have to reset to get what I type to show properly again after it exits.

I'm using version 1.2.54.0

[nitikaaws]

Session manager plugin has been designed to be invoked from a terminal for shell scenarios so it can interpret control sequences returned by pseudo terminal and display appropriately. When a shell session is started, session manager plugin also fetches terminal size on client machine and sends it to pseudo terminal on target server. We suspect one of these actions might be failing for you. Please let us know if you are invoking session manager plugin from a terminal and still seeing error of Cannot perform start session: EOF.

[dgholz]

Yes, I am invoking it from a terminal. I haven't changed anything about the instances I'm connecting to, they're launched from an AMI I baked weeks ago. It's working today, without changing anything in my local environment (same version of session-manager-plugin, same AMI/instances)

When a shell session is started, session manager plugin also fetches terminal size on client machine and sends it to pseudo terminal on target server.

ah, I think I get it. When I try the same command from a terminal with an enormous scrollback, it fails with Cannot perform start session: EOF. Using a new terminal session clears the scrollback, and aws ssm start-session works as expected.

[sruthi-maddineni]

Thanks for letting us know that you were able to get past this issue.
Please let us know in case of further issues.

[zengben-gainfully]

I had similar issue with "aws ecs execute-command ..." and it seems it's related to tty being not available when running in ci

I fixed it with "unbuffer aws ecs execute-command ..." to run it with tty available

@olfway You saved my day! I had the exact same issue and sudo apt-get install expect; aws ecs execute-command .... worked like a charm! Thanks.