Amazon-ssm-agent: SCP "connection corrupted" on large file transfer

[edwdev]

When trying to SCP from a local machine to and EC2 instance, using session manager (ProxyCommand is setup in SSH config) - it seems that I have issues sending larger files (60MB for example) and when I get to 8624KB uploaded, I get the following error.

ad packet length 2916819926.
ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: Connection corrupted
lost connection

My local connection is stable, so am just wondering if I'm hitting any hard limits ?

[asterikx]

I'm experiencing the same error when SSHing to an EC2 instance to access my database. The error occurs after some period of time (~30 minutes).

Bad packet length 2089525630.
ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: Connection corrupted

@edwdev could you solved the issue?

[rscottwatson]

I would like to know more about this as I am facing a similar issue as well. I am not transferring a file but just using session manager to access my RDS database and I frequently get this. I am connecting from my MAC incase there is something with MACs that is causing this issue.

[asterikx]

@rscottwatson I'm using macOS as well, still experiencing the issue

[rscottwatson]

@asterikx So I added the following to my .ssh/config file and I have not had seen that error yet.

Host i-* mi-*
ServerAliveInterval 300
ServerAliveCountMax 2

No guarantees that this will help but so far it has allowed my connection to stay open longer than it has.

[mikelee3]

This is happening to me as well, usually during or after a large file transfer. It is intermittent.

[nunofernandes]

I also have this issue. Some times it works while other (most of them) doesn't. Already tried @rscottwatson suggestion but got the same results..

[PenelopeFudd]

I'm getting this issue too. I already had ServerAliveInterval & ServerAliveCountMax set. Need to see what data ssh is receiving that is corrupt; I imagine it's a debug or an info message from SSM.

[matt-brewster]

I'm getting this error regularly too, I'm only using SSH (not SCP) and my connections randomly drop after some time with
Bad packet length 2704720976.

[m-norii]

I also have this issue, using SSH (not SCP).

Bad packet length 840196936.
ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: Connection corrupted

ServerAliveInterval & ServerAliveCountMax also set:

    ServerAliveInterval 60
    ServerAliveCountMax 5

[taherbs]

I'm experiencing the same behaviors indirectly when using git, trying to clone a very large repository:

$ git fetch origin develop --depth=1
remote: Enumerating objects: 8778, done.
remote: Counting objects: 100% (8778/8778), done.
remote: Compressing objects: 100% (1933/1933), done.
ssh_dispatch_run_fatal: Connection to IP port 22: message authentication code incorrect
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: index-pack failed

[taherbs]

For my case, the issue was related to instable VPN connection

[archisgore]

I get the same with ssh not necessarily scp.

1. I have one terminal that's connected that stays connected.
2. Other terminals don't connect with the error above:
   ```.bash
   archisgore@Archiss-MacBook keys % ssh -i .pem ec2-user@i-
   Bad packet length 631868568.
   ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: Connection corrupted

I tried all the above: ClientAliveInterval, ClientAliveCountMax, ServerAliveInternal, ServerAliveCountMax. I restarted `amazon-aws-ssm`, I restarted `sshd`.

What's more, I also opened port 22 using a security group and tried connecting over direct ssh, and get this:
```.bash
archisgore@Archiss-MacBook keys % ssh -v -i "<key redacted>.pem" ec2-user@ec2-<ip redacted>.us-west-2.compute.amazonaws.com
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/archisgore/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to ec2-52-11-122-217.us-west-2.compute.amazonaws.com port 22.
debug1: Connection established.
debug1: identity file <key redacted>.pem type -1
debug1: identity file <key redacted>.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug1: Authenticating to ec2-<ip redacted>.us-west-2.compute.amazonaws.com:22 as 'ec2-user'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:IfH362yMWxH3EWdQZOTgsyuq0+jjdJvg0Ag+nFQPjvs
debug1: Host 'ec2-<ip redacted>.us-west-2.compute.amazonaws.com' is known and matches the ECDSA host key.
debug1: Found key in /Users/archisgore/.ssh/known_hosts:109
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: PolyverseDevelopmentKey.pem  explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Trying private key: PolyverseDevelopmentKey.pem
Connection closed by 52.11.122.217 port 22
archisgore@Archiss-MacBook keys % ssh -v -i "<key redacted>.pem" ec2-user@ec2-<ip redacted>.us-west-2.compute.amazonaws.com
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/archisgore/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to ec2-<ip redacted>.us-west-2.compute.amazonaws.com port 22.
debug1: Connection established.
debug1: identity file <key redacted>.pem type -1
debug1: identity file <key redacted>.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug1: Authenticating to ec2-<ip redacted>.us-west-2.compute.amazonaws.com:22 as 'ec2-user'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:IfH362yMWxH3EWdQZOTgsyuq0+jjdJvg0Ag+nFQPjvs
debug1: Host 'ec2-<ip redacted>.us-west-2.compute.amazonaws.com' is known and matches the ECDSA host key.
debug1: Found key in /Users/archisgore/.ssh/known_hosts:109
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: <key redacted>.pem  explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Trying private key: <key redacted>.pem
Connection closed by <ip redacted> port 22

[sruthi-maddineni]

Thanks for reaching out to us!
Please provide SSM Agent and Session Manager Plugin logs for the failed session to help investigate the issue.
You can refer to below documentation to retrieve SSM Agent logs.
https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-agent-logs.html

Documentation for enabling plugin logs.
https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html#install-plugin-configure-logs

[archisgore]

Oh hey sorry I take that back. My issue was unrelated. My SSH config was broken, which I verified by opening a direct SSH port. Once I fixed that, SSM worked too. Apologies for the report above.