Amazon-ssm-agent: Custom Shell for `aws ssm start-session`

[kellertobias]

I want to replace our ssh/ldap login with the ssm for all our AWS servers.

However the /bin/sh shell is not my favourite, so I want to use bash as default shell. I already configured that for the user ssm-user, however it still starts /bin/sh on startup.

(also tried to link /bin/sh to /bin/bash, but somehow it still finds /bin/bsh)

[itoperatorguy]

+1 for this feature

[johnnyplaydrums]

Would love for session manager to respect passwd settings (not sure how/why it's not now, is it because it's a non login shell?), like setting the default shell, etc. Is it possible to do this now somehow?

[teambob]

The line of code in the agent is here: https://github.com/aws/amazon-ssm-agent/blob/b9654b268afcb7e70a9cc6c6d9b7d2a676f5b468/agent/session/plugins/shell/shell_unix.go#L53

[JibbyAni]

+1 on this

[HewittJC]

As @johnplaydrums said, I think its sort of _dangerous_ (for lack of a better word) to not respect general Linux behavior.

Some sort of configuration file on the instance or a setting in the AWS console would be acceptable too.

[teambob]

Perhaps the agent could respect the shell setting in /etc/passwd for ssm-user?

[mewelling]

I can get to the shell I want by just calling /bin/bash directly upon login, but that just seems like an extra step. +1 on respecting /etc/passwd.

[Robert-Joe]

Yeah, I have to type in "bash -l -o vi" to get my preferred login shell. It would be so simple for AWS to implement. Talk about being in the dark ages...

[Robert-Joe]

So the fix is this in amazon-ssm-agent/agent/session/plugins/shell/shell_unix.go

cmd := exec.Command("bash -l")

[eyablonowitz]

I'm not sure if this should be a separate feature request, but I'd like to be able to specify the initial command - e.g. aws ssm start-session <command>.

My use case:
I'm writing a tool to start an SSM session on an ECS Container Instance and I want to immediately docker exec to start a shell, rails console, python console... inside one of the containers along the lines of https://engineering.loyaltylion.com/running-an-interactive-console-on-amazon-ecs-c692f321b14d.

Right now my script is sadly forced to print thedocker exec command and advise the user to copy/paste it at the ssm session shell prompt.

[cwarner-mdsol]

Bump; I wanted to provide a patch but also talk this out a bit. It's simple in theory but that's only if it's made a configurable independent option by itself. If we go the passwd route or PAM, this becomes a much more involved thing. If there was a roadmap for this it would be helpful.

[dantech2000]

👍 this pls

[talawahtech]

Has anybody figured out a workaround for this (other than instructing users to manually run a command)?

sh (which is really bash in "sh" mode) doesn't seem to be picking up any of the usual start up scripts like /etc/profile or ~/.profile that could be used to automatically run a command. If the code called exec.Command("sh -l") instead, then at least those could be used.

Apparently if you set an environment variable named ENV with the path to a shell script, then sh will execute that script when it runs, alas it is a chicken and egg problem because there doesn't seem to be a way to set ENV.

What's even worse, the non-primary groups for ssm-user don't seem to get recognized automatically either, so even though I add the ssm-user to the docker group, it can't run docker commands by default.

[fiducioso-dan]

I think custom shell invocations are possible now mostly due to commit d2d2746059ff4dde453c65c028102e25f28d0ee2. Apparently you need to reference an SSM document when calling StartSession. I have not attempted this yet.

[skorfmann]

@fiducioso-dan

I think custom shell invocations are possible now mostly due to commit d2d2746. Apparently you need to reference an SSM document when calling StartSession. I have not attempted this yet.

tried to create a document according to the commit like the following:

{
    "schemaVersion": "1.0",
    "description": "Document to hold regional settings for Session Manager",
    "sessionType": "Standard_Stream",
    "inputs": {
      "s3BucketName": "",
      "s3KeyPrefix": "",
      "s3EncryptionEnabled": true,
      "cloudWatchLogGroupName": "",
      "cloudWatchEncryptionEnabled": true
    },
    "sessionCommands": [{
      "commands": "bash -lc 'echo fooooooooo'"
    }]
  }

Unfortunately, the API is rejecting sessionCommands:

 Error creating SSM document: InvalidDocumentContent: SessionCommands is not supported in Standard_Stream session type
    status code: 400

The only halfway official docs about session documents I found: https://docs.aws.amazon.com/systems-manager/latest/userguide/getting-started-configure-preferences-cli.html

Sort of looks like the agent itself would support it, but there's no way to create such document. Ideas?

[slikk66]

Bump.. my company is evaluating using this to replace our vpn/key access for several hundred machines but this is a dealbreaker. We need logins to load the shell/command of our choice as would be possible in /etc/passwd

[brainstorm]

I'm in the same situation, I then filed an AWS support ticket and here's where it is for now (since 2-3 days ago):

They (SSM AWS internal team) say that the SSM support /bin/sh and the behavior you are noticing is normal and by design, however there is a plan for SSM to support bash preference in the future.
They are currently looking for a workaround to the issue and I will update you as soon as I get feedback.
I will keep this case open until I update you on the workaround. In the mean time please reach out if you have any further questions, I will be glad to assist.

I really hope that they don't come up with a aws ssm --use-bash or similar non-intuitive flag. I do expect and hope that the final solution respects /etc/passwd, transparently, as @slikk66 mentions.

Fingers crossed.

[tdmalone]

Thanks for filing a ticket @brainstorm! Please do let us know what workaround is offered.

[brainstorm]

There's actually an issue from 2017 about this same topic but was left behind, apparently.

And also the possibility to use SSH instead, as pointed out in issue https://github.com/aws/amazon-ssm-agent/issues/188

... let's see what the SSM team comes up with.

[brainstorm]

SSM team came back via AWS Support:

The main reason session manager is not using "bash" as default shell is because of logging. When using "bash" shell, session log files that are generated have formatting issues and gibberish characters. Due the logging issue on bash, session manager launches "sh" as a default shell at the moment.

They have also noted that they are currently working on implementing "bash" but they have to research and look for workarounds to improve logging which I believe might take some time.

[teambob]

SSM team came back via AWS Support:

The main reason session manager is not using "bash" as default shell is because of logging. When using "bash" shell, session log files that are generated have formatting issues and gibberish characters. Due the logging issue on bash, session manager launches "sh" as a default shell at the moment.
They have also noted that they are currently working on implementing "bash" but they have to research and look for workarounds to improve logging which I believe might take some time.

Are there any links to bash bugs which give an idea what the problem is? I haven't had any problems with bash logging

[johnnyplaydrums]

SSM team came back via AWS Support:

The main reason session manager is not using "bash" as default shell is because of logging. When using "bash" shell, session log files that are generated have formatting issues and gibberish characters. Due the logging issue on bash, session manager launches "sh" as a default shell at the moment.
They have also noted that they are currently working on implementing "bash" but they have to research and look for workarounds to improve logging which I believe might take some time.

Are there any links to bash bugs which give an idea what the problem is? I haven't had any problems with bash logging

Same here. No issue with Bash logs.

[phene]

SSM team came back via AWS Support:

The main reason session manager is not using "bash" as default shell is because of logging. When using "bash" shell, session log files that are generated have formatting issues and gibberish characters. Due the logging issue on bash, session manager launches "sh" as a default shell at the moment.
They have also noted that they are currently working on implementing "bash" but they have to research and look for workarounds to improve logging which I believe might take some time.

Are there any links to bash bugs which give an idea what the problem is? I haven't had any problems with bash logging

Wouldn't this also mean that starting bash once you're in the session breaks logging?

[drAlberT]

"breaks logs" means they are messed up with tons of non printable characters .. using colors for example fills the logs with a lot of escape sequences ...

I suppose AWS guys are talking about this

[khuongduybui]

but what the log is going to look like if the first command I run after start-session connects is bash?

[TC-robV]

Already tried that. It still logs.

On Wed, Oct 2, 2019, 7:53 PM Duy K. Bui [e-mail] wrote:

but what the log is going to look like if the first command I run after
start-session connects is bash?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/aws/amazon-ssm-agent/issues/131?email_source=notifications&email_token=AL444QIHZLDOKSZ4R2J4HKDQMVNDZA5CNFSM4F7V4TT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAGZY5Y#issuecomment-537762935,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AL444QMGYJFX6MT65M5V323QMVNDZANCNFSM4F7V4TTQ
.

[khuongduybui]

Yes, but does it produce gibberish log with escape sequences etc?

[mattmcf]

@skorfmann Do you know why sessionCommands didn't take? That seems like a great solution but it looks like it never made it to a released version of the agent.

[CoreySmithRing]

Any progress on this? We recently rolled this out and this is a huge ask. I see someone submitted a PR but it failed or was denied.

[compilingEntropy]

@CoreySmithRing All builds are failing with their build test system right now as far as I can tell; I don't think they've reviewed my PR yet. You can build it from my fork if you like to see if it will fit your needs (I've tested it and it does work), but I don't really like the idea of maintaining a separate branch myself. +1 for getting some attention on this from AWS.

[Midgaze]

By default, SSM invokes a shell as sh. The behavior of Bash under these circumstances is to not execute any shell init scripts. There is one way to override this behavior however: if you can find a way to set the ENV environment variable in the context of the SSM agent, the new shell will execute the script specified there.

The details of how to do this will differ depending on what Linux distribution you use, but for a systemd-based distro like Ubuntu, modifying /etc/systemd/system.conf to include something like this will work:

DefaultEnvironment="ENV=/etc/profile"

Our new shell is still in sh mode. Let's modify /etc/profile to source our user's .profile:

if [ -r ~/.profile ]; then
    . ~/.profile
fi

Now, we can do the part that we don't want to do system-wide in our ssm-user's init. In ~/.profile:

if [ "${0}" = "sh" ]; then
    exec bash
fi

Now when we start a SSM session, we get a properly initialized bash shell. I added a cd ~ to the end of my .bashrc to start out in the ssm-user home directory, and the environment feels better now.

If the behavior of ssm-agent changes in the future, this configuration should not break in bad ways.

[phene]

+1 for SSM respecting /etc/passwd

[cptcrush2]

@Midgaze I have tried your solution for invoking bash on Ubuntu and it is not working for me. Do you have a suggestion of what to try? Thanks

[eshmu]

I found one easy workaround by using an AWS-provided SSM document:

aws ssm start-session --target <INSTANCE_ID> --document-name AWS-StartInteractiveCommand --parameters command="bash -l"

This will put you into a bash shell when the SSM session is started. Obviously, replace "bash -l" with any other command or script you wish to run.

[wwilfinger]

All I was looking for was interactive SSH sessions without standing up and going through a jump host with a public ip.

https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html

Running real ssh with the AWS-StartSSHSession document as a ProxyCommand was a lot closer to a normal ssh session. You actually connect to sshd, so you get your expected login triggers like printing the motd. You log into your home directory instead of /usr/bin. Tmux renders without issues. You can scp files over to the host. Looks and feels like a regular ssh session as far as I can tell.

The ~/.ssh/config config snippet, from the linked documentation page

# SSH over Session Manager
host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

AWS-StartInteractiveCommand, for me, was not rendering vim in vertically split tmux panes correctly. If you go down that path be aware. (Tested) on Amazon Linux 2 you can drop this into /etc/profile.d to put you into your home directory and show motd if you still want to use AWS-StartInteractiveCommand.

PARENT_PROCESS_PATH=$(cat /proc/$PPID/cmdline | cut -d '' -f1)
if [ "${PARENT_PROCESS_PATH}" = "/usr/bin/ssm-session-worker" ]; then
  cd $HOME
  cat /etc/motd
fi

[AndreCouture]

aws ssm start-session --target <INSTANCE_ID> --document-name AWS-StartInteractiveCommand --parameters command="bash -l"

The format has changed in aws2 where the command need to be a list / tuple

aws ssm start-session --target <INSTANCE_ID> --document-name AWS-StartInteractiveCommand --parameters command='["bash -l"]'
This works great from the command line, but anyone knows how to put this in the ~/.ssh/config ?

I tried this

ProxyCommand sh -c "aws2 ssm start-session --target %h --document-name AWS-StartSSHSession --profile sso --document-name AWS-StartInteractiveCommand --parameters command='["bash -l"]' "

basically my issue is how to quote properly the command such that when parsed by ssh it would end as command='["bash -l"]'

here's my trace

debug1: Executing proxy command: exec sh -c "aws2 ssm start-session --target mi-xxxxxxxxx --document-name AWS-StartSSHSession --profile sso --document-name AWS-StartInteractiveCommand --parameters command='["bash -l"]' "
...
debug1: Local version string SSH-2.0-OpenSSH_8.1
-l]' : -c: line 0: unexpected EOF while looking for matching `''
-l]' : -c: line 1: syntax error: unexpected end of file
kex_exchange_identification: Connection closed by remote host

Also you can even use this document-name to directly login to a user by changing the command to something like this
command='["sudo -i -u myuser"]'

[drAlberT]

Try '["bash", "-l"]'

On Sat, May 2, 2020, 15:52 AndreCouture [e-mail] wrote:

aws ssm start-session --target --document-name
AWS-StartInteractiveCommand --parameters command="bash -l"

The format has changed in aws2 where the command need to be a list / tuple

aws ssm start-session --target --document-name
AWS-StartInteractiveCommand --parameters command='["bash -l"]'
This works great from the command line, but anyone knows how to put this
in the ~/.ssh/config ?

I tried this

ProxyCommand sh -c "aws2 ssm start-session --target %h --document-name
AWS-StartSSHSession --profile sso --document-name
AWS-StartInteractiveCommand --parameters command='["bash -l"]' "

basically my issue is how to quote properly the command such that when
parsed by ssh it would end as command='["bash -l"]'

here's my trace

debug1: Executing proxy command: exec sh -c "aws2 ssm start-session --target mi-xxxxxxxxx --document-name AWS-StartSSHSession --profile sso --document-name AWS-StartInteractiveCommand --parameters command='["bash -l"]' "
...
debug1: Local version string SSH-2.0-OpenSSH_8.1
-l]' : -c: line 0: unexpected EOF while looking for matching `''
-l]' : -c: line 1: syntax error: unexpected end of file
kex_exchange_identification: Connection closed by remote host

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aws/amazon-ssm-agent/issues/131#issuecomment-622957074,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AALAHDFU5B35AVQ5ZMGQYI3RPQQQRANCNFSM4F7V4TTQ
.

[AndreCouture]

Actually the issue is to get this string in the ~/.ssh/config

SSH over Session Manager

Host i-* mi-*
parameters command='["bash -l"]' "
ProxyCommand sh -c "aws2 ssm start-session --target %h --document-name AWS-StartSSHSession --profile sso --document-name AWS-StartInteractiveCommand --parameters command='["bash -l"]' "

See, the whole ProxyCommand is wrapped in “ while the parameters command also require “ inside the “

I tried escaping without success

Also splitting won’t do

An error occurred (InvalidParameters) when calling the StartSession operation: Parameter "command" takes exactly one value, got 2.

—
Regards,

Andre Couture
Senior Cloud Solution Architect

andre.[email protected]
https://www.linkedin.com/in/andrecouture

On May 2, 2020, at 1:17 PM, drAlberT [e-mail] wrote:

Try '["bash", "-l"]'

On Sat, May 2, 2020, 15:52 AndreCouture [e-mail] wrote:

aws ssm start-session --target --document-name
AWS-StartInteractiveCommand --parameters command="bash -l"

The format has changed in aws2 where the command need to be a list / tuple

aws ssm start-session --target --document-name
AWS-StartInteractiveCommand --parameters command='["bash -l"]'
This works great from the command line, but anyone knows how to put this
in the ~/.ssh/config ?

I tried this

ProxyCommand sh -c "aws2 ssm start-session --target %h --document-name
AWS-StartSSHSession --profile sso --document-name
AWS-StartInteractiveCommand --parameters command='["bash -l"]' "

basically my issue is how to quote properly the command such that when
parsed by ssh it would end as command='["bash -l"]'

here's my trace

debug1: Executing proxy command: exec sh -c "aws2 ssm start-session --target mi-xxxxxxxxx --document-name AWS-StartSSHSession --profile sso --document-name AWS-StartInteractiveCommand --parameters command='["bash -l"]' "
...
debug1: Local version string SSH-2.0-OpenSSH_8.1
-l]' : -c: line 0: unexpected EOF while looking for matching `''
-l]' : -c: line 1: syntax error: unexpected end of file
kex_exchange_identification: Connection closed by remote host

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aws/amazon-ssm-agent/issues/131#issuecomment-622957074,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AALAHDFU5B35AVQ5ZMGQYI3RPQQQRANCNFSM4F7V4TTQ
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/aws/amazon-ssm-agent/issues/131#issuecomment-622985926, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOXRKTAHBL4MALMZOTGNL7DRPRIRPANCNFSM4F7V4TTQ.

[tomaspinho]

@cptcrush2 probably missing a systemctl daemon-reexec or a reboot. Either should make systemd reload the first file which will make it work.

[conork-datacom]

Looks like this is the 'Official' fix to this, just released...

https://aws.amazon.com/blogs/mt/bring-your-own-cli-session-manager-configurable-shell-profiles/

aws ssm start-session --target i-xxxxxxxxxxxx --document-name param --parameters '{"linuxcmd": ["bash"]}'

[kostyrev]

it doesn't work

> aws ssm start-session --target i-04f3953c3b5f72f1e --document-name param --parameters '{"linuxcmd": ["bash"]}'

An error occurred (InvalidDocument) when calling the StartSession operation: Document with name param does not exist.

> which aws
/usr/local/bin/aws
> aws --version
aws-cli/2.1.25 Python/3.9.1 Darwin/20.3.0 source/x86_64 prompt/off



md5-b6146b0cbce533de88ae468c78ead15c



> session-manager-plugin --version
1.2.30.0

[jk2l]

@kostyrev did you follow the page https://aws.amazon.com/blogs/mt/bring-your-own-cli-session-manager-configurable-shell-profiles/ and create document first?

[kostyrev]

Of course not. Who reads them docs carefully anyway?
p.s. sorry for the fuzz

[nitikaaws]

Session Manager now supports configurable shell profile which can be used to customize preferences within sessions such as shell preferences, environment variables, working directories, and running multiple commands when a session is started. Please check for more details in below documentation and let us know in case of further queries.

https://docs.aws.amazon.com/systems-manager/latest/userguide/session-preferences-shell-config.html

[tfarmerpgr]

How can one make the ssm-user behave like a normal user logging in? I am sourcing /etc/profile via the command ran when shellProfile starts up and invoking /bin/bash, however, it is not running through all login scripts in /etc/profile.d like a regular user would.

As a result, some env vars are not being set correctly.

"parameters": {
  "linuxcmd": {
    "type": "String",
    "default": "base64 -d <<< \"${shellprofile}\" > /home/ssm-user/.bashrc; exec /bin/bash",
    "description": "Decode the shell profile and run bash shell."
  }
}

/home/ssm-user/.bashrc

#!/bin/bash
set +x
clear
export PS1='[\u@\h \w]\$ '
source /etc/profile
cat /etc/motd
cd $HOME