Metasploit-framework: Meterpreter Migrate

[akefallonitis]

When i try migrate on metepreter the operation timeouts and then my shell breaks.
Running in high integrate context and as system.

MSF latest version
meterpreter x64 running shellcode on Windows 10 latest build
No AV No Firewall and No EDR running

Anyone having the same issue ?

[timwr]

I think this is a known "issue"

[akefallonitis]

I cannot find any new issue regarding the latest 5.0.50-dev and migrate please assist

[bwatters-r7]

After bisecting this, it looks like a regression introduced in 47afa21b5ff983a4ce137dc9f23b628e1e1b7aa7

Looking at that briefly, I don't understand why? I need to dig some more, though, so I assigned this to myself.

[OJ]

I have no idea how that could be the culprit either!

On Wed, 9 Oct 2019, 04:17 Brendan, [e-mail] wrote:

After bisecting this, it looks like a regression introduced in 47afa21
https://github.com/rapid7/metasploit-framework/commit/47afa21b5ff983a4ce137dc9f23b628e1e1b7aa7

Looking at that briefly, I don't understand why? I need to dig some more,
though, so I assigned this to myself.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/rapid7/metasploit-framework/issues/12390?email_source=notifications&email_token=AAAHBYFAZ5E7UORR6YRSOQLQNTFE3A5CNFSM4I5EEGC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAVDVLA#issuecomment-539638444,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAHBYDN2Z6QKQ3UN4JDA5TQNTFE3ANCNFSM4I5EEGCQ
.

[bwatters-r7]

I mean..... I'm hopping between commits 540e2699a64494265dbd4fc3f4591aba0d66212d and 47afa21b5ff983a4ce137dc9f23b628e1e1b7aa7. The the results are consistent....

540e2699a64494265dbd4fc3f4591aba0d66212d migrates successfully

47afa21b5ff983a4ce137dc9f23b628e1e1b7aa7 times out, but occasionally (10%-20%) calls back after a timeout, and it is running in the migrated process.

https://media.giphy.com/media/BcpsfmGhjAwcE/giphy.gif

[OJ]

I'll see if I have time to take a look later today.

But if that commit is responsible it may just fry the little bit of sanity
I have left.

On Wed, 9 Oct 2019, 06:49 Brendan, [e-mail] wrote:

I mean..... I'm hopping between commits 540e269
https://github.com/rapid7/metasploit-framework/commit/540e2699a64494265dbd4fc3f4591aba0d66212d
and 47afa21
https://github.com/rapid7/metasploit-framework/commit/47afa21b5ff983a4ce137dc9f23b628e1e1b7aa7.
The the results are consistent....

540e269
https://github.com/rapid7/metasploit-framework/commit/540e2699a64494265dbd4fc3f4591aba0d66212d
migrates successfully

47afa21
https://github.com/rapid7/metasploit-framework/commit/47afa21b5ff983a4ce137dc9f23b628e1e1b7aa7
times out, but occasionally (10%-20%) calls back after a timeout, and it is
running in the migrated process.

https://media.giphy.com/media/BcpsfmGhjAwcE/giphy.gif

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rapid7/metasploit-framework/issues/12390?email_source=notifications&email_token=AAAHBYF2XQVHYRPHTW3C47DQNTW4BA5CNFSM4I5EEGC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAVSHVY#issuecomment-539698135,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAHBYGNPTF6MO4V5J7JNCDQNTW4BANCNFSM4I5EEGCQ
.

[bwatters-r7]

OK... I feel a little better; it turns out that this might be a intermittent problem, rather than a definitive problem, which explains why bisect might have landed on this commit (though I tested it three times manually on each side, and it was repeatable manually)

I set up automated testing and got these results (commit hashes listed at the top)

Commit before 47afa21b5ff983a4ce137dc9f23b628e1e1b7aa7

Commit 47afa21b5ff983a4ce137dc9f23b628e1e1b7aa7

I am pretty sure this is a regression, as all tests fail on master...

Commit a6ec5131ad8e557b213b5b45a3ea588b1b4d70de (Current Master):

[bwatters-r7]

OK.... after running a ton of tests, I think I may have found the real slim shady.

Commit 760defd218157ccc8a7a56d0a2f6cc47ee3f4b81 AKA: https://github.com/rapid7/metasploit-framework/pull/11184

Commit 880e5976bf30d2386eee03fdf9e136b38c0ccbbe AKA: https://github.com/rapid7/metasploit-framework/pull/11176 (Preceding land)

I also tested the other two lands in either direction. Everything after 760defd218157ccc8a7a56d0a2f6cc47ee3f4b81 had complete failures, everything before had partial success. Now to figure out why the Ruby bump made the difference?

[OJ]

I wonder if, for some reason, the change in version resulted in the
migration stubs being compiled incorrectly by metasm?

On Sat, 12 Oct 2019, 05:56 Brendan, [e-mail] wrote:

OK.... after running a ton of tests, I think I may have found the real
slim shady.
Commit 760defd
https://github.com/rapid7/metasploit-framework/commit/760defd218157ccc8a7a56d0a2f6cc47ee3f4b81
AKA: #11184 https://github.com/rapid7/metasploit-framework/pull/11184

[image: image]
https://user-images.githubusercontent.com/17987018/66680714-c04a6180-ec36-11e9-8eab-ea32ca9f36cb.png
Commit 880e597
https://github.com/rapid7/metasploit-framework/commit/880e5976bf30d2386eee03fdf9e136b38c0ccbbe
AKA: #11176 https://github.com/rapid7/metasploit-framework/pull/11176
(Preceding land)

[image: image]
https://user-images.githubusercontent.com/17987018/66680774-de17c680-ec36-11e9-9329-ff4bf79a4e46.png

I also tested the other two lands in either direction. Everything after
760defd
https://github.com/rapid7/metasploit-framework/commit/760defd218157ccc8a7a56d0a2f6cc47ee3f4b81
had complete failures, everything before had partial success. Now to figure
out why the Ruby bump made the difference?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rapid7/metasploit-framework/issues/12390?email_source=notifications&email_token=AAAHBYE6HQMF4C53LA67DU3QODK5FA5CNFSM4I5EEGC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBBBUEY#issuecomment-541202963,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAHBYB2G3BMLO52ESQASXLQODK5FANCNFSM4I5EEGCQ
.

[bwatters-r7]

Here's the log:

[10/11/2019 15:10:18] [e(0)] meterpreter: Error running command migrate: Rex::TimeoutError Operation timed out.
[10/11/2019 15:10:18] [d(0)] meterpreter: Call stack:
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:177:in `send_request'
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/client_core.rb:747:in `negotiate_tlv_encryption'
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/client_core.rb:691:in `migrate'
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console/command_dispatcher/core.rb:1222:in `cmd_migrate'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:523:in `run_command'
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console.rb:105:in `run_command'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:474:in `block in run_single'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in `each'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in `run_single'
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console.rb:68:in `block in interact'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/shell.rb:146:in `run'
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console.rb:66:in `interact'
/home/tmoose/rapid7/metasploit-framework/lib/msf/base/sessions/meterpreter.rb:576:in `_interact'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/interactive.rb:51:in `interact'
/home/tmoose/rapid7/metasploit-framework/lib/msf/ui/console/command_dispatcher/core.rb:1364:in `cmd_sessions'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:523:in `run_command'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:474:in `block in run_single'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in `each'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in `run_single'
/home/tmoose/rapid7/metasploit-framework/lib/msf/ui/console/command_dispatcher/exploit.rb:215:in `cmd_exploit'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:523:in `run_command'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:474:in `block in run_single'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in `each'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in `run_single'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/shell.rb:151:in `run'
/home/tmoose/rapid7/metasploit-framework/lib/metasploit/framework/command/console.rb:48:in `start'
/home/tmoose/rapid7/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'
./msfconsole:49:in `<main>'

Though not particularly helpful, TBH.

[OJ]

Yeah my guess is that the stub in the target process is busted.

Did the target process crash?

On Sat, 12 Oct 2019, 06:11 Brendan, [e-mail] wrote:

Here's the log:

[10/11/2019 15:10:18] [e(0)] meterpreter: Error running command migrate: Rex::TimeoutError Operation timed out.
[10/11/2019 15:10:18] [d(0)] meterpreter: Call stack:
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:177:in send_request' /home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/client_core.rb:747:innegotiate_tlv_encryption'
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/client_core.rb:691:in migrate' /home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console/command_dispatcher/core.rb:1222:incmd_migrate'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:523:in run_command' /home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console.rb:105:inrun_command'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:474:in block in run_single' /home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:ineach'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in run_single' /home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console.rb:68:inblock in interact'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/shell.rb:146:in run' /home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console.rb:66:ininteract'
/home/tmoose/rapid7/metasploit-framework/lib/msf/base/sessions/meterpreter.rb:576:in _interact' /home/tmoose/rapid7/metasploit-framework/lib/rex/ui/interactive.rb:51:ininteract'
/home/tmoose/rapid7/metasploit-framework/lib/msf/ui/console/command_dispatcher/core.rb:1364:in cmd_sessions' /home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:523:inrun_command'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:474:in block in run_single' /home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:ineach'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in run_single' /home/tmoose/rapid7/metasploit-framework/lib/msf/ui/console/command_dispatcher/exploit.rb:215:incmd_exploit'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:523:in run_command' /home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:474:inblock in run_single'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in each' /home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:inrun_single'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/shell.rb:151:in run' /home/tmoose/rapid7/metasploit-framework/lib/metasploit/framework/command/console.rb:48:instart'
/home/tmoose/rapid7/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in start' ./msfconsole:49:in

'

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rapid7/metasploit-framework/issues/12390?email_source=notifications&email_token=AAAHBYDCXRA4X3B6WAOED6LQODMVLA5CNFSM4I5EEGC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBBCX3Y#issuecomment-541207535,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAHBYF3FVDQYZNASY5S5ITQODMVLANCNFSM4I5EEGCQ
.

[OJ]

Also might be worth testing migrating with different transports given that
the stubs are different for each.

On Sat, 12 Oct 2019, 06:11 Brendan, [e-mail] wrote:

Here's the log:

[10/11/2019 15:10:18] [e(0)] meterpreter: Error running command migrate: Rex::TimeoutError Operation timed out.
[10/11/2019 15:10:18] [d(0)] meterpreter: Call stack:
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:177:in send_request' /home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/client_core.rb:747:innegotiate_tlv_encryption'
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/client_core.rb:691:in migrate' /home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console/command_dispatcher/core.rb:1222:incmd_migrate'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:523:in run_command' /home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console.rb:105:inrun_command'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:474:in block in run_single' /home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:ineach'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in run_single' /home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console.rb:68:inblock in interact'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/shell.rb:146:in run' /home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console.rb:66:ininteract'
/home/tmoose/rapid7/metasploit-framework/lib/msf/base/sessions/meterpreter.rb:576:in _interact' /home/tmoose/rapid7/metasploit-framework/lib/rex/ui/interactive.rb:51:ininteract'
/home/tmoose/rapid7/metasploit-framework/lib/msf/ui/console/command_dispatcher/core.rb:1364:in cmd_sessions' /home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:523:inrun_command'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:474:in block in run_single' /home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:ineach'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in run_single' /home/tmoose/rapid7/metasploit-framework/lib/msf/ui/console/command_dispatcher/exploit.rb:215:incmd_exploit'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:523:in run_command' /home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:474:inblock in run_single'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in each' /home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:inrun_single'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/shell.rb:151:in run' /home/tmoose/rapid7/metasploit-framework/lib/metasploit/framework/command/console.rb:48:instart'
/home/tmoose/rapid7/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in start' ./msfconsole:49:in

'

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rapid7/metasploit-framework/issues/12390?email_source=notifications&email_token=AAAHBYDCXRA4X3B6WAOED6LQODMVLA5CNFSM4I5EEGC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBBCX3Y#issuecomment-541207535,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAHBYF3FVDQYZNASY5S5ITQODMVLANCNFSM4I5EEGCQ
.

[bwatters-r7]

The target process has never crashed.

[OJ]

Interesting. Maybe the thread creation failed.

I'll try to look at this a bit later today.

On Sat, 12 Oct 2019, 07:31 Brendan, [e-mail] wrote:

The target process has never crashed.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rapid7/metasploit-framework/issues/12390?email_source=notifications&email_token=AAAHBYAKEE6FRMEEVN2KTT3QODWDLA5CNFSM4I5EEGC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBBIQAY#issuecomment-541231107,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAHBYDBHIPWHST7MLRWGM3QODWDLANCNFSM4I5EEGCQ
.

[an1mehacker]

I had the EXITFUNC set to thread and this error happened. When I changed it to process it's fine. Not sure if this helps in any way tho.

[jeffmcjunkin]

This still affects Windows 10 Enterprise (version 1903) and Metasploit 5.0.55-dev:

meterpreter > run migrate -n spoolsv.exe

[!] Meterpreter scripts are deprecated. Try post/windows/manage/migrate.
[!] Example: run post/windows/manage/migrate OPTION=value [...]
[*] Current server process: Icecast2.exe (1264)
[+] Migrating to 1940

[-] Could not migrate in to process.
[-] Operation timed out.
meterpreter > ps
[-] Error running command ps: Rex::TimeoutError Operation timed out.
meterpreter > exit
[*] Shutting down Meterpreter...

[*] 10.10.10.20 - Meterpreter session 1 closed.  Reason: User exit

msf5 exploit(windows/http/icecast_header) > version
Framework: 5.0.55-dev-
Console  : 5.0.55-dev-

[OJ]

I'll try to fix this soon!

[OJ]

Quick investigation shows that http-based transports are fine. Migration works as expected.

reverse_tcp is busted. The migration happens behind the scenes, and the first packet to go out over the wire once the socket has been taken over by the new process is the TLV encryption packet. The new session in the target process correctly creates an AES key and writes the response to the socket, however on the MSF side, no data is received. MSF just sits there and waits for data on the socket, and nothing happens.

Not sure why yet.

[OJ]

Definitely something strange going on with the Ruby side. I honestly don't know what it is. This is a pic of the Windows box. We can see that the TLV encryption response packet is actually going out on the wire and we get an ACK from MSF:

But on the MSF side, I can see that select is being called on the socket as part of the typical wait/dispatch loop, but _no_ data is coming through on it.

Going to have to call on some Ruby gurus to help out with this one @wvu-r7 @bcook-r7 and whoever else is a master of the Rubies. I honestly don't know what's going on. The socket is still open, data is sent to it from Meterp, and it has been ACK'd. But as far as MSF is concerned the select call doesn't return with any data and so the wait just times out.

Does anyone know of anything that may have changed in Ruby 2.6+ that could cause this kind of behaviour?

[OJ]

rv = Rex::ThreadSafe.select([ self.sock.fd ], nil, nil, PING_TIME)

This is the line, in packet_dispatcher.rb which times out when it shouldn't. I haven't dived into Rex::ThreadSafe.select, but perhaps there's something in there that's causing the problem?

[OJ]

I changed the above line to use ::IO.select instead, and it didn't help. So something else is up.

[OJ]

Also, FWIW, if you have a handler open and listening, exiting the current session will result in Meterp reconnecting and from there everything works fine.

[OJ]

Confirmed that reverting to Ruby 2.5.1 results in everything working perfectly.

[OJ]

Confirmed broken in 2.5.7 ! Doing the painful binary search to see which version killed it.

[OJ]

OK after a painful binary search, I can say that:

• Ruby 2.5.4 works on my machine
• Ruby 2.5.5 does not work on my machine

So I'm going to say that changes introduced in Ruby 2.5.5 are causing the problem. The thing is, it doesn't really look like anything changed in the source could have caused it. The Release Notes don't really imply much either.

So I'm at a bit of a loss. HALP PLZ!

[OJ]

It is interesting that the issue manifests in a case where multiple entities are working on the socket, indicating that it could be related to threading. I'd be keen to see if patching out that line in the latest ruby would result in things working.

But yeah, not sure where to go from here.

[bwatters-r7]

Yeah. I took a look into it the other day and I asked a few rubier people than me, and there's nothing that stuck out as obvious.

[busterb]

I started taking a look at this last night, and have a working diff on Ruby 2.6.5. Looks like some assumptions about thread lifetimes need stronger guarantees; MSF might have been relying on undefined behavior for a while.