Webmin: Webmin backdoor in versions 1.882 through 1.921

[art-lucas]

I initially posted this comment on this post - #947 which is dated July 26, 2018, but since it was closed long ago, I've opened a new issue.

I've just upgraded to 1.930 this morning after receiving this report from SecurityWeek titled _Webmin Backdoored for Over a Year_ that stated:

To exploit the malicious code, your Webmin installation must have Webmin -> Webmin Configuration -> Authentication -> Password expiry policy set to Prompt users with expired passwords to enter a new one. This option is not set by default, but if it is set, it allows remote code execution

Right after reading this, I checked my config and saw that Webmin -> Webmin Configuration -> Authentication -> Password was set to Prompt users with expired passwords to enter a new one. This was by default, since I never changed that value.

How do I know my system has never been tempered with through this backdoor? Is there a way to verify/trace that? Needless to say that this is very worrying as per this report as well as this post on reddit a root shell can/could have been launched very easily.

[art-lucas]

Incidentally, this post explains:

Yes, it shows up in the access log as an access of password_change.cgi; though that won't necessarily indicate a successful exploit. Any version other than 1.890 requires that password expiry/change option to be enabled, so accesses would fail.
When I was testing a couple of different methods of exploit (for 1.890 and for 1.920) it shows up in miniserv.error, as well, though it depends on what is in the input (and which version of Webmin is running) as to what shows up. But, there maybe a perl error looking something like Perl execution failed - Your password has expired [...] at /usr/share/webmin/password_change.cgi.
Seeing that error probably indicates the system was exploited in some way, as reaching that code indicates the option is enabled or it was running under version 1.890.

So I did:

# cd /var/webmin/
# rgrep password_change .
#

As well as:

# rgrep -i "Your password has expired" .
#

Anything else should I check?

[jcameron]

That should cover it - unless password_change.cgi was called on your system (which will show up in miniserv.log and/or miniserv.error), you're safe.

[art-lucas]

Thanks @jcameron.

[swelljoe]

Then again, if the attacker obtained root, they may have tampered with the logs.

One of our machines (a blog web server, nothing important) was running 1.890, and I plan to replace it, as there's no way to be absolutely certain it wasn't compromised and then covered up...it's kinda unlikely the attacker would think to clean up everything, as most of the people talking about the exploit don't have a high level of understanding of how it works or what it does and the traces it leaves behind (though now that we've talked about it in a couple of public places, some will).

When I have helped people clean up or try to establish whether an attacker got root, I usually try running rkhunter, and an rpm -Va (this verifies the consistency of files installed with RPM) looking for suspicious stuff (e.g. changes to passwd/shadow, changes to important system binaries related to logins, etc....the resulting list is very long, so it can take a while to go over and understand what you're seeing, but many attackers don't bother covering up this particular trail of breadcrumbs), in addition to looking at the various system logs, the root .bash_history, etc. You can figure out if something definitively was compromised using these techniques, but you can never really figure out if it definitively wasn't compromised. That's the bad news, the good news is that the attack didn't start showing up widely until the 17th (with some early pings during the week prior). So, if you upgraded quickly, your window of risk was smallish.

I've been trying to find an active drop site for one of the payloads being seen in the wild (they download some kind of root kit and run it with the attack), but so far, none have been active by the time I try to download them. But, there could be as many different payloads as there are attackers, so that may or may not be all that useful in figuring out what attackers are doing.

Regarding the option in question, I guess it may have been on if Webmin was first installed with version 1.890, as upgrades don't replace config files. It was a short-lived release, I think, due to the bugginess of that particular exploit, which led to the ticket you referenced and that vector of attack being closed (accidentally, since it wasn't noticed as an exploit at the time).

[MurzNN]

I have found in my logs:

# rgrep password_change /var/webmin/
./miniserv.log:96.126.114.121 - - [22/Aug/2019:07:54:54 +0300] "POST /password_change.cgi HTTP/1.1" 500 131
./miniserv.error:[22/Aug/2019:07:54:54 +0300] [96.126.114.121] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/share/webmin/password_change.cgi line 12.

Does this mean, that attack is not successful?

[swelljoe]

That particular error is, I think, indicative of a failed attack. It comes from this line (which is before the exploit, and the die means it never makes it further in the code):

$miniserv{'passwd_mode'} == 2 || die "Password changing is not enabled!";

[nolimitdev]

What about webmin with IP access limited to some IP? From not allowed IP is it possible to use this attack?

[nolimitdev]

@art-lucas

So I did:
rgrep password_change .
As well as:
rgrep -i "Your password has expired" .
Anything else should I check?

@jcameron

That should cover it - unless password_change.cgi was called on your system (which will show up in miniserv.log and/or miniserv.error), you're safe.

Anyone with Webmin version 1.890 OR anyone with Versions 1.900 to 1.920 and Password expiry policy set to Prompt users with expired passwords to enter a new one should consider his system as compromised no matter what is written in log files. You can not rely on log files - attacker with root access could modify it (and probably did it) and he could install permanent rootkit or custom backdoor. You can try some rootkit hunters tools but really consider your system compromised and you should left it and deploy new one (do not copy any files without auditing it).

[swelljoe]

@nolimitdev I think IP access restrictions would prevent this, as well. Those happen in miniserv.pl before any CGIs can be called, so I'm pretty confident it would shut it down before calling the exploitable code.

[swelljoe]

I'm closing this, as it isn't really actionable. We've covered the detection question in a few places (though, again, a rooted system may not have trustworthy logs).