I noticed that some TCP sessions forwarded over a meterpreter session were not usable with SSH. Symptoms were hung sessions.
msf auxiliary(socks4a) > route print
Active Routing Table
Subnet Netmask Gateway
------ ------- -------
172.16.0.0 255.255.0.0 Session 4
msf auxiliary(tcp) > use auxiliary/server/socks4a
msf auxiliary(socks4a) > info
[...]
Basic options:
Name Current Setting Required Description
---- --------------- -------- -----------
SRVHOST 0.0.0.0 yes The address to listen on
SRVPORT 1080 yes The port to listen on.
Description:
This module provides a socks4a proxy server that uses the builtin
Metasploit routing to relay connections.
msf auxiliary(socks4a) > run -j
[*] Auxiliary module running as background job
The session is a x86 meterpreter running on a windows2008r2 server.
This is what happens when I try to log in using this socks server:
$ ssh -o 'Proxycommand=nc -X 4 -x 127.0.0.1:1080 172.16.200.43 22' -vvv root@xxx
OpenSSH_7.3p1, OpenSSL 1.0.2j 26 Sep 2016
debug1: Reading configuration data /Users/aris/.ssh/config
debug1: /Users/aris/.ssh/config line 4: Applying options for *
debug1: /Users/aris/.ssh/config line 7: Deprecated option "useroaming"
debug3: kex names ok: [diffie-hellman-group-exchange-sha256]
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug1: Executing proxy command: exec nc -X 4 -x 127.0.0.1:1080 172.16.200.43 22
debug1: permanently_drop_suid: 501
debug1: identity file /Users/aris/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aris/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aris/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aris/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aris/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aris/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aris/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aris/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.3
debug1: ssh_exchange_identification:
debug1: ssh_exchange_identification: \024锟絓233x\033'P\216\220锟絯\023\207IA=
debug1: ssh_exchange_identification: dsa-sha2-nistp256,ssh-ed25519
debug1: ssh_exchange_identification: ator.liu.se
debug1: ssh_exchange_identification: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,h
debug1: ssh_exchange_identification: mac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug1: ssh_exchange_identification: [email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd
debug1: ssh_exchange_identification: [email protected],hmac-sha1-96,hmac-md5-96
^C
The ssh_exchange_identification don't look right. I ran another command to confirm my suspicions:
$ (echo "SSH-2.0-lalalal";cat) | nc localhost 2222 > ssh_banner
$ hexdump -C ssh_banner
00000000 00 00 06 6c 0a 14 6e 41 f6 9c 8b 17 3a a4 87 ea |...l..nA....:...|
00000010 79 f5 22 f4 a1 ae 00 00 00 d4 63 75 72 76 65 32 |y.".......curve2|
00000020 35 35 31 39 2d 73 68 61 32 35 36 40 6c 69 62 73 |5519-sha256@libs|
00000030 73 68 2e 6f 72 67 2c 65 63 64 68 2d 73 68 61 32 |sh.org,ecdh-sha2|
00000040 2d 6e 69 73 74 70 32 35 36 2c 65 63 64 68 2d 73 |-nistp256,ecdh-s|
00000050 68 61 32 2d 6e 69 73 74 70 33 38 34 2c 65 63 64 |ha2-nistp384,ecd|
00000060 68 2d 73 68 61 32 2d 6e 69 73 74 70 35 32 31 2c |h-sha2-nistp521,|
00000070 64 69 66 66 69 65 2d 68 65 6c 6c 6d 61 6e 2d 67 |diffie-hellman-g|
00000080 72 6f 75 70 2d 65 78 63 68 61 6e 67 65 2d 73 68 |roup-exchange-sh|
00000090 61 32 35 36 2c 64 69 66 66 69 65 2d 68 65 6c 6c |a256,diffie-hell|
000000a0 6d 61 6e 2d 67 72 6f 75 70 2d 65 78 63 68 61 6e |man-group-exchan|
000000b0 67 65 2d 73 68 61 31 2c 64 69 66 66 69 65 2d 68 |ge-sha1,diffie-h|
000000c0 65 6c 6c 6d 61 6e 2d 67 72 6f 75 70 31 34 2d 73 |ellman-group14-s|
000000d0 68 61 31 2c 64 69 66 66 69 65 2d 68 65 6c 6c 6d |ha1,diffie-hellm|
000000e0 61 6e 2d 67 72 6f 75 70 31 2d 73 68 61 31 00 00 |an-group1-sha1..|
000000f0 00 2f 73 73 68 2d 72 73 61 2c 73 73 68 2d 64 73 |./ssh-rsa,ssh-ds|
00000100 73 2c 65 63 64 73 61 2d 73 68 61 32 2d 6e 69 73 |s,ecdsa-sha2-nis|
00000110 74 70 32 35 36 2c 73 73 68 2d 65 64 32 35 35 31 |tp256,ssh-ed2551|
00000120 39 00 00 00 e9 61 65 73 31 32 38 2d 63 74 72 2c |9....aes128-ctr,|
...
00000600 36 30 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 2c 68 |[email protected],h|
00000610 6d 61 63 2d 73 68 61 31 2d 39 36 2c 68 6d 61 63 |mac-sha1-96,hmac|
00000620 2d 6d 64 35 2d 39 36 00 00 00 15 6e 6f 6e 65 2c |-md5-96....none,|
00000630 7a 6c 69 62 40 6f 70 65 6e 73 73 68 2e 63 6f 6d |[email protected]|
00000640 00 00 00 15 6e 6f 6e 65 2c 7a 6c 69 62 40 6f 70 |....none,zlib@op|
00000650 65 6e 73 73 68 2e 63 6f 6d 00 00 00 00 00 00 00 |enssh.com.......|
00000660 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000670 53 53 48 2d 32 2e 30 2d 4f 70 65 6e 53 53 48 5f |SSH-2.0-OpenSSH_|
00000680 36 2e 36 2e 31 70 31 20 55 62 75 6e 74 75 2d 32 |6.6.1p1 Ubuntu-2|
00000690 75 62 75 6e 74 75 32 2e 33 0d 0a |ubuntu2.3..|
0000069b
The SSH banner and KEXINIT packets were somehow reversed.
I could reproduce the same behavior with portfwd. It does not happen 100% of the times. I haven't git bisected to isolate the problem by lack of resource.
I got the following messages in ~/.msf4/logs/framework.log, but not exactly during the problem, so I don't think they're 100% relevant:
[10/05/2016 13:20:48] [w(0)] core: monitor_rsock: exception during select: Rex::StreamClosedError Stream #<TCPSocket:0x007f833650e2e8> is closed.
[10/05/2016 13:20:48] [e(0)] core: thread exception: StreamMonitorRemote critical=false error: TypeError no implicit conversion of nil into Integer
source:
/Users/aris/software/metasploit-framework/lib/metasploit/framework/thread_factory_provider.rb:24:in `spawn'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `spawn'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:120:in `monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/stream_abstraction.rb:26:in `initialize_abstraction'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channels/socket_abstraction.rb:94:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:73:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `new'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `create'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:36:in `open'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:104:in `create_tcp_client_channel'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:73:in `create'
/Users/aris/software/metasploit-framework/lib/msf/base/sessions/meterpreter.rb:494:in `create'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket.rb:49:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:37:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:28:in `create'
/Users/aris/software/metasploit-framework/lib/rex/proto/proxy/socks4a.rb:253:in `block in start'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:20:48] [e(0)] core: Call Stack
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:284:in `pack'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:284:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:555:in `block in to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:413:in `each'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:413:in `each'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:554:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:675:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:159:in `send_packet'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:227:in `send_packet_wait_response'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:199:in `send_request'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:104:in `shutdown'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:88:in `close_write'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:188:in `block (2 levels) in monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `loop'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `block in monitor_rsock'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:20:57] [w(0)] core: monitor_rsock: exception during select: Rex::StreamClosedError Stream #<TCPSocket:0x007f833edcd020> is closed.
[10/05/2016 13:20:57] [e(0)] core: thread exception: StreamMonitorRemote critical=false error: TypeError no implicit conversion of nil into Integer
source:
/Users/aris/software/metasploit-framework/lib/metasploit/framework/thread_factory_provider.rb:24:in `spawn'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `spawn'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:120:in `monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/stream_abstraction.rb:26:in `initialize_abstraction'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channels/socket_abstraction.rb:94:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:73:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `new'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `create'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:36:in `open'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:104:in `create_tcp_client_channel'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:73:in `create'
/Users/aris/software/metasploit-framework/lib/msf/base/sessions/meterpreter.rb:494:in `create'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket.rb:49:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:37:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:28:in `create'
/Users/aris/software/metasploit-framework/lib/rex/proto/proxy/socks4a.rb:253:in `block in start'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:20:57] [e(0)] core: Call Stack
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:284:in `pack'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:284:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:555:in `block in to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:413:in `each'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:413:in `each'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:554:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:675:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:159:in `send_packet'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:227:in `send_packet_wait_response'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:199:in `send_request'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:104:in `shutdown'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:88:in `close_write'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:188:in `block (2 levels) in monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `loop'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `block in monitor_rsock'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:21:00] [w(0)] core: monitor_rsock: exception during select: Rex::StreamClosedError Stream #<TCPSocket:0x007f833eaeeca0> is closed.
[10/05/2016 13:21:00] [e(0)] core: thread exception: StreamMonitorRemote critical=false error: TypeError no implicit conversion of nil into Integer
source:
/Users/aris/software/metasploit-framework/lib/metasploit/framework/thread_factory_provider.rb:24:in `spawn'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `spawn'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:120:in `monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/stream_abstraction.rb:26:in `initialize_abstraction'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channels/socket_abstraction.rb:94:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:73:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `new'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `create'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:36:in `open'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:104:in `create_tcp_client_channel'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:73:in `create'
/Users/aris/software/metasploit-framework/lib/msf/base/sessions/meterpreter.rb:494:in `create'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket.rb:49:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:37:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:28:in `create'
/Users/aris/software/metasploit-framework/lib/rex/proto/proxy/socks4a.rb:253:in `block in start'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:21:00] [e(0)] core: Call Stack
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:284:in `pack'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:284:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:555:in `block in to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:413:in `each'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:413:in `each'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:554:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:675:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:159:in `send_packet'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:227:in `send_packet_wait_response'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:199:in `send_request'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:104:in `shutdown'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:88:in `close_write'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:188:in `block (2 levels) in monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `loop'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `block in monitor_rsock'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:21:07] [w(0)] core: monitor_rsock: exception during select: Rex::StreamClosedError Stream #<TCPSocket:0x007f833640c188> is closed.
[10/05/2016 13:21:07] [e(0)] core: thread exception: StreamMonitorRemote critical=false error: TypeError no implicit conversion of nil into Integer
source:
/Users/aris/software/metasploit-framework/lib/metasploit/framework/thread_factory_provider.rb:24:in `spawn'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `spawn'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:120:in `monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/stream_abstraction.rb:26:in `initialize_abstraction'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channels/socket_abstraction.rb:94:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:73:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `new'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `create'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:36:in `open'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:104:in `create_tcp_client_channel'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:73:in `create'
/Users/aris/software/metasploit-framework/lib/msf/base/sessions/meterpreter.rb:494:in `create'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket.rb:49:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:37:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:28:in `create'
/Users/aris/software/metasploit-framework/lib/rex/proto/proxy/socks4a.rb:253:in `block in start'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:21:07] [e(0)] core: Call Stack
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:284:in `pack'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:284:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:555:in `block in to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:413:in `each'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:413:in `each'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:554:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:675:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:159:in `send_packet'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:227:in `send_packet_wait_response'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:199:in `send_request'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:104:in `shutdown'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:88:in `close_write'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:188:in `block (2 levels) in monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `loop'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `block in monitor_rsock'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:21:33] [w(0)] core: monitor_rsock: exception during select: Rex::StreamClosedError Stream #<TCPSocket:0x007f83362da030> is closed.
[10/05/2016 13:21:33] [e(0)] core: thread exception: StreamMonitorRemote critical=false error: TypeError no implicit conversion of nil into Integer
source:
/Users/aris/software/metasploit-framework/lib/metasploit/framework/thread_factory_provider.rb:24:in `spawn'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `spawn'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:120:in `monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/stream_abstraction.rb:26:in `initialize_abstraction'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channels/socket_abstraction.rb:94:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:73:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `new'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `create'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:36:in `open'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:104:in `create_tcp_client_channel'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:73:in `create'
/Users/aris/software/metasploit-framework/lib/msf/base/sessions/meterpreter.rb:494:in `create'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket.rb:49:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:37:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:28:in `create'
/Users/aris/software/metasploit-framework/lib/rex/proto/proxy/socks4a.rb:253:in `block in start'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:21:33] [e(0)] core: Call Stack
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:284:in `pack'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:284:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:555:in `block in to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:413:in `each'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:413:in `each'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:554:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:675:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:159:in `send_packet'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:227:in `send_packet_wait_response'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:199:in `send_request'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:104:in `shutdown'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:88:in `close_write'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:188:in `block (2 levels) in monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `loop'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `block in monitor_rsock'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:21:47] [w(0)] core: monitor_rsock: the remote socket is nil, exiting loop
[10/05/2016 13:22:01] [w(0)] core: monitor_rsock: exception during select: Rex::StreamClosedError Stream #<TCPSocket:0x007f83352f4990> is closed.
[10/05/2016 13:22:01] [e(0)] core: thread exception: StreamMonitorRemote critical=false error: Rex::Post::Meterpreter::RequestError stdapi_net_socket_tcp_shutdown: Operation failed: The handle is invalid.
source:
/Users/aris/software/metasploit-framework/lib/metasploit/framework/thread_factory_provider.rb:24:in `spawn'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `spawn'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:120:in `monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/stream_abstraction.rb:26:in `initialize_abstraction'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channels/socket_abstraction.rb:94:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:73:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `new'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `create'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:36:in `open'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:104:in `create_tcp_client_channel'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:73:in `create'
/Users/aris/software/metasploit-framework/lib/msf/base/sessions/meterpreter.rb:494:in `create'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket.rb:49:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:37:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:28:in `create'
/Users/aris/software/metasploit-framework/lib/rex/proto/proxy/socks4a.rb:253:in `block in start'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:22:01] [e(0)] core: Call Stack
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:104:in `shutdown'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:88:in `close_write'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:188:in `block (2 levels) in monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `loop'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `block in monitor_rsock'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:22:15] [w(0)] core: monitor_rsock: exception during select: Rex::StreamClosedError Stream #<TCPSocket:0x007f833a70b240> is closed.
[10/05/2016 13:22:16] [e(0)] core: thread exception: StreamMonitorRemote critical=false error: Rex::Post::Meterpreter::RequestError stdapi_net_socket_tcp_shutdown: Operation failed: The handle is invalid.
source:
/Users/aris/software/metasploit-framework/lib/metasploit/framework/thread_factory_provider.rb:24:in `spawn'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `spawn'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:120:in `monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/stream_abstraction.rb:26:in `initialize_abstraction'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channels/socket_abstraction.rb:94:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:73:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `new'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `create'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:36:in `open'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:104:in `create_tcp_client_channel'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:73:in `create'
/Users/aris/software/metasploit-framework/lib/msf/base/sessions/meterpreter.rb:494:in `create'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket.rb:49:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:37:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:28:in `create'
/Users/aris/software/metasploit-framework/lib/rex/proto/proxy/socks4a.rb:253:in `block in start'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:22:16] [e(0)] core: Call Stack
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:104:in `shutdown'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:88:in `close_write'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:188:in `block (2 levels) in monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `loop'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `block in monitor_rsock'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:22:25] [w(0)] core: monitor_rsock: exception during select: Rex::StreamClosedError Stream #<TCPSocket:0x007f833a3ef1d8> is closed.
[10/05/2016 13:22:25] [e(0)] core: thread exception: StreamMonitorRemote critical=false error: TypeError no implicit conversion of nil into Integer
source:
/Users/aris/software/metasploit-framework/lib/metasploit/framework/thread_factory_provider.rb:24:in `spawn'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `spawn'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:120:in `monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/stream_abstraction.rb:26:in `initialize_abstraction'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channels/socket_abstraction.rb:94:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:73:in `initialize'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `new'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/channel.rb:121:in `create'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:36:in `open'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:104:in `create_tcp_client_channel'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket.rb:73:in `create'
/Users/aris/software/metasploit-framework/lib/msf/base/sessions/meterpreter.rb:494:in `create'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket.rb:49:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:37:in `create_param'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-socket-0.1.0/lib/rex/socket/tcp.rb:28:in `create'
/Users/aris/software/metasploit-framework/lib/rex/proto/proxy/socks4a.rb:253:in `block in start'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[10/05/2016 13:22:25] [e(0)] core: Call Stack
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:284:in `pack'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:284:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:555:in `block in to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:413:in `each'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:413:in `each'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:554:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet.rb:675:in `to_r'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:159:in `send_packet'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:227:in `send_packet_wait_response'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:199:in `send_request'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:104:in `shutdown'
/Users/aris/software/metasploit-framework/lib/rex/post/meterpreter/extensions/stdapi/net/socket_subsystem/tcp_client_channel.rb:88:in `close_write'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:188:in `block (2 levels) in monitor_rsock'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `loop'
/usr/local/lib/ruby/gems/2.3.0/gems/rex-core-0.1.2/lib/rex/io/socket_abstraction.rb:121:in `block in monitor_rsock'
/Users/aris/software/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/Users/aris/software/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
This is metasploit from git master, running on osx El Capitan.
msf auxiliary(socks4a) > version
Framework: 4.12.30-dev-548efc3
Console : 4.12.30-dev-548efc3
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15] (installed from homebrew)Thanks,
Aris
Worked with Aris and could not reproduce: http://pastebin.com/FAnDpzcm
http://pastebin.com/fcU9UbmY
Maybe some environment issue on his side. Will leave this open in case other people can reproduce.
@dmaloney-r7 , please take a look - this may be more regressions due to Rex gem split-ups. Exception is happening close to rex-core
@lvarela-r7 often to simulate real scenarios, you need to separate framework from the payload via some sort of real network. Often bugs that show up talking to real machines across the internet cannot be reproduced on low-latency, high-throughput VMs and such. I recommend turning on 'Network Link Conditioner' on OS X to configure more realistic networking.
I just re-trigged this bug with windows/meterpreter/reverse_tcp and windows/x64/meterpreter/reverse_tcp.
I was able to reproduce this pretty easily.
To rule out meterpreter itself reordering the packets, I modified the channel_read/write routines from running asynchronously to running inline, which means that the packets should be sent in the order they are received. This had no effect.
I also went back about 4 months in code and the same thing is happening (including the unhandled exceptions listed above), so not a new issue or related directly to rex code refactoring. So, it's likely a pre-existing issue somewhere on the Metasploit framework side of things.
No doubt this will be a fabulous one to fix :)
This is looking more like a flushing issue than an ordering issue. I suspect that somewhere between socks4a reading a packet and forwarding it to the meterpreter channel, it gets stuck in a queue. Then on the subsequent connection, the last packet from the last connection is sent first, followed by the first packet on the new proxy connection.
Verified that the first time the proxy is started, behavior seems to be different than all subsequent runs. Granted, it also looks like meterpreter has a race condition since it also handles packets out of order potentially, but that doesn't seem to be the major factor here.
@bcook-r7 Would this be more accurately titled something like "SOCKS protocol handler buffers incorrectly" ?
Sure, though keep in mind that I may have also been incorrect in my assessment. Trust but verify!
@egypt is there an easy way to test if modules other than SOCKS are impacted ? I have many metasploit modules that don't work properly over routed connections, but I am not sure it's the same issue.
I confirm that I see that behavior 19/20 times when trying to ssh through proxychains over a soks4a proxy in metasploit.
I'm facing with this issue on this setup. Kali is accessible from windows machine which has access to ssh server. windows/x64/meterpreter/reverse_tcp session is established to Kali then port is forwarded with portfwd add -L 127.0.0.1 -l 8866 -p 22 10.10.2.30. I've noticed that more time passes since session establishment less error occur upon ssh connection establishment. Pictures taken on the same session with several attempts.
Normal order:

Reversed order:

Oooh yeah nice one. I think I might have to have a crack at fixing this one. It looks fun. Thanks for the detailed information.
@busterb is anyone working on this at the moment? It might just be worth throwing a sequence number in ?
I've made several test attempts. Changing Windows machine to Linux with linux/x64/meterpreter/reverse_tcp same result. Changing portfwd to autoroute and issuing internal metasploit script like ssh_login same result. Then i changed to python/meterpreter/reverse_tcp and no issues at all, ssh logins like in should on all attempts. So it seems like problem is in original meterpreter. Hope this info might be helpful to fix existing issue.
Finally getting around to looking at this now. Bear with me :)
PR is up for testing. Would love you all to have a play if you can: https://github.com/rapid7/metasploit-framework/pull/13465 Thanks!
Congrats for solving this bug, it was a tough one!
Thanks! It was certainly painful to track down :)
Most helpful comment
PR is up for testing. Would love you all to have a play if you can: https://github.com/rapid7/metasploit-framework/pull/13465 Thanks!