Couchdb: cpu usage 100% caused by silence and fs-manager

[barbalex]

Expected Behavior

Usually CPU usage should be below 10%.

Current Behavior

CPU usage is at 100% 95% of the time.

On 5 different production servers I have noticed that cpu usage has been at 100% for most of the time during at least the last 30 days.

From running top it seems that most often Silence is the main culprit, using 98% of the cpu. Sometimes fs-manager and Silence both use 49% each. And on one server it seems that fs-manager uses 98%.

There are no active tasks running.

Two of these servers couches contain very low number of docs (a few hundred). The others have about 30'000 docs.

What could cause this behaviour?
How problematic is it?
How can I prevent it?

Your Environment

• Version used: 1.6.1
• Operating System and version (desktop or mobile): ubuntu 14.04, ubuntu 16.04.02
• Link to your project: https://github.com/FNSKtZH/artendb_v1, https://github.com/barbalex/bb2, https://github.com/barbalex/brexit
  
  
  
  

[garrensmith]

From the other reports I've seen on the dev mailing list, your couchdb instance has been hacked and someone has installed a bitcoin mining instance on your service.

[barbalex]

@garrensmith Yep, you were right. These were crypto miners. I found a mailing list mail from Sinan Gabel explaining how to turn off the miner. Unfortunately I do not know how to link to Sinan's mail so here is the gist:

I needed to remove a cron job created from user couchdb:

sudo -i
su couchdb
crontab -e

// Then remove the line in the cron
// save and reboot

So thanks a lot to you and Sinan Gabel.

Of course I will also have to rebuild my servers now.

My biggest concern now is: How to prevent this from happening again, as these couch instances were not in admin mode.

[garrensmith]

If you upgrade to the latest CouchDB you will be fine. We released Security advisories about this late last year.

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Alexander Gabriel [e-mail]
Sent: Thursday, January 4, 2018 6:11:42 PM
To: apache/couchdb
Cc: garren smith; Mention
Subject: Re: [apache/couchdb] cpu usage 100% caused by silence and fs-manager (#1088)

Closed #1088https://github.com/apache/couchdb/issues/1088.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/apache/couchdb/issues/1088#event-1409824165, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAK9AovuXTP7F09XGuuWab6s341FPvtGks5tHPg-gaJpZM4RS4TQ.

[barbalex]

@garrensmith: thanks a lot for your great help!

[paulocoghi]

I was also hit by this invasion. I am using CouchDB 1.6.1 from stable PPA https://launchpad.net/~couchdb/+archive/ubuntu/stable

How can I update it to solve this security vulnerability, since there are no updates in the PPA? Or there is an available workaround?

[paulocoghi]

Found a solution: https://github.com/apache/couchdb/issues/787

[barbalex]

I would not update. Eliminating the cron job is not enough. In my case the
problem reappeared inside minutes.

I rebuilt the servers with v2.1.1, then synced my DB. Don't know if syncing
could even be risky...

But so far it works.

vom Handy

Am 12.01.2018 9:23 nachm. schrieb "Paulo Coghi" [e-mail]:

I was also hit by this invasion. I am using CouchDB 1.6.1 from stable PPA
https://launchpad.net/~couchdb/+archive/ubuntu/stable
https://launchpad.net/%7Ecouchdb/+archive/ubuntu/stable

How can I update it to solve this security vulnerability, since there are
no updates in the PPA? Or there is an available workaround?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/apache/couchdb/issues/1088#issuecomment-357333260,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAo4V_GHG68fBQ-qEIwg8ztIjHQVSJ9Bks5tJ74DgaJpZM4RS4TQ
.

[paulocoghi]

In my case the problem reappeared inside minutes.

@barbalex , this happened in which version?

I upgraded to version 2.1.1 after using 1.6.1 and experiencing this problem. I used couchup to migrate data from the old version to the new one.

So far, everything working perfectly. The only detail was that couchup did not migrate the users, and I had to re-create them manually.

[paulocoghi]

@barbalex , just removing the cron jobs is certainly not enough since the vulnerability remains open if you do not upgrade.

[barbalex]

@paulocoghi

This was during the short period before I rebuilt the servers. So still with version 1.6.1.

And you may be better off not migrating the users: The "bad guys" added a few users. And also a few db's (with only one document each).

[wotaen]

Just for reference if anyone is looking for the process name. It's supsplk in my case. It's stored in /var/tmp/

[AlbertDavid94]

I've had the same problem. A Monero miner was installed through a curl request to 192.99.142.232:8220/logo3.jpg which is in turn a bash script. When it starts with 192., doesn't it refer to a local network?

[wohali]

@AlbertDavid94 No, the private IP space is only 192.168.0.0 - 192.168.255.255. Addresses outside this range are publicly routeable, including 192.99.142.232.

The full set of private IP space blocks is documented in RFC 1918.