Cockroach: OOM panic while inserting data into node with 512MB of RAM

[kaeverens]

I'm doing an insert of a few thousand rows of data. One of the nodes has crashed (not the node I'm inserting through). It threw up a load of Go error messages on the way down.

I've attached a copy of the INFO log
dump.txt

[kaeverens]

just adding that a few seconds later, the rest of the nodes in the cluster went down as well

[kaeverens]

ah - spotted it... (in another node's logs)

"fatal error: runtime: out of memory"

is there a minimum RAM recommended? any plans for memory management?

I'll add swap for now, but it would be nice if the system did something other than just crash.

[knz]

@kaeverens thanks for your report.

Could you please remind us:

• which version of CockroachDB you are currently using
• how many nodes you're running
• how you are inserting this data into the database: is that one client or multiple clients? Do you use a single transaction for everything or do you split the data into multiple transactions? How many tables are you loading data into?

[kaeverens]

beta-20160908
3 nodes, each on separate servers
one client, no explicit transactions (so multiple transactions ;-) ), all into the one table.

I am certain the issue was lack of memory. the nodes were all running on relatively small virtual machines (512M RAM each).

I think the real issue here is the lack of overt memory management. MySQL, for example, has a load of different options you can set to help stop the database from eagerly destroying itself.

[knz]

I think there are separate issues at hand. Certainly like you suggest we could do a better job at making memory usage more configurable, or perhaps documenting our existing options better.

However it is also the case that we're trying hard to set automatic defaults that fit your particular hardware configuration. It is surprising to me that the database "exploded" so readily.

Nevertheless I think we may know enough to investigate further.

[a-robinson]

Ah, yeah, I don't know that we run properly yet with only 512MB of memory. @nvanbenschoten had to explicitly enable swap to get cockroach running on a 512MB raspberry pi.

[knz]

For the docs: we absolutely need to quantify then document properly reasonable minimum platform requirements for running a node safely.

[a-robinson]

Agreed. We have to figure them out, but cc @jseldess @sploiselle

[sploiselle]

Once we have them figured out, we can easily add them to our recommended production settings.

CC: @a-robinson @knz @jseldess

[jseldess]

@knz, @a-robinson, there are some ways to updated the docs even before you're gotten to the bottom of this:

• Update theCache Size section of recommended production settings to mention that you might want to manually set the cache size either if you're running multiple apps on the same machine or if the machine does not have a "reasonable" amount of memory.
• Update the troubleshooting page to show what to do if you get such an OOM error.

Thoughts?

[spencerkimball]

@jseldess I know minimum config is something planned for 1.0 documentation. I'm closing this particular issue in favor of https://github.com/cockroachdb/docs/issues/965