Webmin: Request: Forced Auto Self Upgrade, for critical security bugs.

[chris001]

It'd be wise to upgrade Webmin/Virtualmin's self-update algorithm, in light of the recent type of severe flaw in the code, which make take millions of unique human sys admins, years to fully come around to noticing and upgrading their servers.

The new forced auto self upgrade code would force auto-self-upgrade for any critical security bugs, which would be decided on by the maintainers, (such as for example, this remote code execution vulnerability that makes webmin easily available to participate in a botnet, without the knowledge of the server admin).

To implement this is simple, each day webmin runs a cron job, at a random time of day, during which it pings the root webmin update server, and if it receives back a value of TRUE or FORCE-UPGRADE or any pre determined signal, then webmin/virtualmin will immediately run apt/yum/pacman/dnf/the relevant package manager, to upgrade the software directly from the official webmin/virtualmin repo.

This would put the botnet out of business very fast, and not rely at all upon the vigilance of sys admins, which is highly unpredictable, as they're far too busy doing other things to chase after patching yet another system software, when the software can easily and efficiently patch/upgrade itself.

[jcameron]

We certainly could do this very easily technically ... but I expect that most sysadmins would be very unhappy about forced upgrades. What we're planning instead is a prominent notification on the system information page in this kind of situation.

[chris001]

We certainly could do this very easily technically ... but I expect that most sysadmins would be very unhappy about forced upgrades.

The DEFAULT setting should be "Allow auto upgrade for critical security fixes" set to ON. This is because the the health and welfare of the entire internet depends on allowing fixes from maintainers which close these critical bugs which would otherwise allow widespread criminal botnets.

Allowing critical bug fix upgrades by default, is the only way to more or less guarantee a near total quick end to bugs which allowed botnets and organized DDoS/cryptomining/other malicious criminal activity on webmin/virtualmin servers, allowed when arbitraty remote root code execution bugs unfortunately make their way into the code base.

It happened once before, I guarantee it'll happen again, and therefore, it's best to be prepared by adding this feature, and allow auto upgrade for critical security fixes by default.

Picky sys admins could always set the setting to OFF. But they would be negligent to disable this feature designed to only upgrade their system when it becomes vulnerable to full takeover by criminal gangs, Those fussy sys admings must therefore accept the liability for their decision which would arguably judge them responsible for damaging the internet and causing data loss in other organizations servers, with their compromised servers prevented from upgrading to close the critical bugs.

[jcameron]

For comparison, Linux distros like CentOS and Debian don't auto-install even security-related package updates either. Webmin already offers a one-click button to upgrade when you login, but auto-upgrading has it's own risks - if we were to push out a new version that was buggy, it could lead to crashing millions of systems with Webmin installed.

[chris001]

For comparison, Linux distros like CentOS and Debian don't auto-install even security-related package updates either. Webmin already offers a one-click button to upgrade when you login, but auto-upgrading has it's own risks - if we were to push out a new version that was buggy, it could lead to crashing millions of systems with Webmin installed.

Good point, this is yet another great reason to activate and cultivate automated unit tests and acceptance tests of the webmin code base, here on Github, using the free and awesome Travis-CI tool. Every time a pull request is created, Travis would run the CI build script on the code base with the change included, and see if compilation succeeds or fails, if unit tests run OK or fail, etc. Worth it in the long run because, as you say, this would largely assure us of code that does not crash millions of systems with Webmin installed.

Furthermore, it's possible, and best, for a webmin system, with "crashing" code, to STILL be able to self-auto-update. In other words, how this is done is, if/when a buggy update was pushed out to millions of systems and caused a crash, then the self-auto-updater is so bulletproof and independent, that it will still continue to run every X minutes, and find the next update, which fixes the previous crash, and downloads the fixed up code, and auto fixes the system, without any intervention from the sys admin.

[iliarostovtsev]

if/when a buggy update was pushed out to millions of systems and caused a crash, then the self-auto-updater is so bulletproof and independent.

What if Webmin server/website/repo is being hacked, and malicious code would be automatically pushed as part of auto-update scenario, infecting millions of machines? Besides, do you remember when Jamie made a release, that would have a "crashing code" in it? :wink:

[swelljoe]

This isn't our role. On a system that has good update tools (RHEL/Fedora have yum/dnf, Debian/Ubuntu have apt), it's the sysadmin's decision. It is already common for administrators to setup automatic updates of some or all software. It's not our place to say which software should be automatically upgraded, though we do strongly recommend folks have a process in place for upgrading world-facing systems regularly.

There is already a mechanism to auto-upgrade Webmin: Configure your package manager to automatically upgrade available packages (maybe only specific packages) on a schedule.

I'm strongly against expanding into spaces already filled by good tools. Package management on Linux is good and reliable and powerful and flexible. Use it.

[chris001]

@swelljoe
I appreciate the feeling of limiting our role in auto updates, and letting well learned sys admins schedule their own auto updates.
However, auto self update wouldn't be meant for normal updates and upgrades, as a way of supplanting the role of a highly educated and well trained sys admin.
This auto self update feature would be aimed at the following users:

1. Users who are newbs, just getting into hosting their own web services on cheap yet powerful world facing VPSs. Probably 100,000 systems/users fall into this category.
2. Users/professionals who try out webmin/virtualmin, maybe get distracted by serious work or school responsibilities, and forget that they left it up and running and facing the world internet, for weeks, months, etc. Probably another 100,000 systems/users fall into this category...

When suddenly, a zero day appears.... The black hat hacking community discovers/plants a zero-day remote root code execution vulnerability in webmin such as the one this year which received world press coverage, (#1093 , #947 , #1105 , #1106 ), and 200,000 webmin servers (newb systems and neglected systems) with the flawed code suddenly become ripe to get enslaved as members of botnets, costing users tens of billions of dollars in business interruption, serious financial losses, ransomware data fees, etc.

Auto self update would let webmin maintainers close the hole by simply fixing the bug, pushing the update out and webmin servers would automatically take it, because it's a highly critical security fix marked as a mandatory update.

This auto self update would, again, only ever update webmin, in cases of emergency critical security bug/remote root full system compromise that need patching immediately.

[swelljoe]

I don't think we have the trust of people for that kind of process. After all, it was an exploited build server that cause the problem; if we had the ability to force-push a new update, that attacker could have used that feature to push a backdoor to everyone, and expanded the number of exploitable hosts, and they could have removed our ability to upgrade them again...making the situation even worse. We're doing what we can to make sure that kind of exploit never happens again, but lots of people are still mad and distrustful about it (and there is never a guarantee), and had there been such a mechanism in place when it happened, it could have been much worse.

Nobody wants there to be security bugs in the wild. But, a force-upgrade just isn't tenable.

There are, I think, better directions for improving security going forward with fewer unintended consequences. Security is hard and there is no silver bullet.

[chris001]

Did you do an investigation of the build server and determine the attack vector that the hackers used to inject the root backdoor into the source code?

[chris001]

This catastrophic remote root vulnerability calls for getting ahead of the problem before it becomes a problem again.

I highly recommend migrating from the "build servers", and from whatever systems they're running on now, to github and github's fantastic Travis-CI.
You can run all of the tests, on a matrix of ALL supported OS'es.
You can have it lint all your code, and recommend edits for code quality, etc.
When a build succeeds, you can have it run your script to build the RPM and DEB (and BSD) system packages, and tag them with the version number.
It's really great way to run your development operations, and going the github/Travis route would restore a huge amount of trust from the user/admin community toward Webmin, Virtualmin, Cloudmin, and toward the entire team of maintainers.
As it's pretty much impossible to hack into the code hosted on github, and inject a root backdoor, when you compare to the existing infrastructure of standalone external systems which aren't audited constantly by a pro security team, like is done at github.