Amazon-ssm-agent: Session Manager start-session hangs when root partition full

[iancward]

I dunno if this is the appropriate place for this, but when I attempt to start a session (either via the AWS Console or the CLI--via the session-manager-plugin) onto an EC2 will a full root partition, it just hangs.
In the console I get a blank screen for a long time and then eventually a blinking cursor that doesn't work.

In the CLI (via session-manager-plugin), I get a message that it's starting a session but then it just hangs.
The CLI/plugin doesn't respond to Ctrl+C or Ctrl+D; in fact, I have to start a new terminal on my workstation and kill the CLI command and the plugin command.

[nitikagoyal87]

Thanks for reaching out to us. We will investigate this.

[Millerborn]

Was this ever resolved? I'm having a similar problem where I just see a black screen in the console, if I click, I get the cursor that doesn't do anything.

[TajMahPaul]

I am also having this issue

[IrinaTerlizhenko]

Hi, I'm experiencing this issue as well.
Are there any updates on whether it is going to be fixed? It would be great to at least get a descriptive error message if the connection to instances with no space is impossible. Right now the AWS Console / CLI just hangs without any visible reason.

[xacaxulu]

+1 Using latest Ubuntu 18 AMI with ssm agent running and installed and necessary SSM/CloudWatch policies attached to role. Weirdest thing, it happens on some instances and not on others. Seems like a bug.

[iniinikoski]

Unfortunately this is still happening. AWS, you should really do something here.

[cholletjo]

Hello,

We experience the same issue on Red Hat 7.7.
We couldn't reach the instance through Session Manager once the partition /var/ was full.

On other side, we observed a different behavior, when partition /var/log was full, the machine was still reachable.
Anyway, when we rely on Session Manager to get remote access to server, we would expect to have access to EC2 even if filesystem full.

[rossmckelvie]

In all cases where Session Manager has not been able to successfully open a connection due to disk full, I'm able to use SSH to get access. We would like to remove SSH and switch solely to Session Manager, but that doesn't seem possible with longstanding issues like this and we are leaving SSH as a backup, where we can open the ports and distribute the pem key as needed during emergencies.

[crawforde]

This is a bit of a circular problem given that the way you're supposed to check the available disk space on a volume is by opening a terminal session in the instance. 🤦‍♀️
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-describing-volumes.html

[jackdcasey]

Not sure if it's totally related, I'm running into an issue with the start-session command hanging, when the target instance is offline. As @iancward mentioned, ctrl+c, etc does not exit. Closing and re-opening the terminal is necessary.

I haven't had the opportunity to really dig into it, but I took a quick look at the code for the CLI, and found this:

https://github.com/aws/aws-cli/blob/master/awscli/customizations/sessionmanager.py

        try:
            # ignore_user_entered_signals ignores these signals
            # because if signals which kills the process are not
            # captured would kill the foreground process but not the
            # background one. Capturing these would prevents process
            # from getting killed and these signals are input to plugin
            # and handling in there
            with ignore_user_entered_signals():
                # call executable with necessary input
                check_call(["session-manager-plugin",
                            json.dumps(response),
                            region_name,
                            "StartSession",
                            profile_name,
                            json.dumps(parameters),
                            endpoint_url])
            return 0

Looks like the terminate signals are being swallowed intentionally? I'm not totally sure, but I reckon this ties into things 😄