Metabase: Memory quota exceeded on Heroku

[lorenzosantos]

Running v0.19.3 on Heroku - 2x (1GB) Dyno. (Used the one click install)

Metabase exceeds memory usage on both the 1x (512MB) Dyno and the 2x (1GB) Dyno. Memory usage immediately climbs after startup. Consistent R14 errors.

Screenshot of metabase DB on heroku:

I've connected only one Redshift database. With roughly 300,000 records. One dashboard and a few saved questions.

Seemed like this issue was previously fixed with a version update (#2704). Any thoughts/suggestions?

[camsaul]

@tlrobinson it sounds like we need to be passing a map heap size param to the JVM when running Metabase, e.g. -Xmx512m, since the JVM isn't doing the right thing automatically.

Ideally we'd want to set that number based on the whether it's a 1x or 2x dyno, if there's some way that could be determined

[lorenzosantos]

@camsaul That's what I thought as well based on Heroku docs (https://devcenter.heroku.com/articles/java-support#adjusting-environment-for-a-dyno-size). JAVA_TOOL_OPTIONS config variable are set to "-Xmx350m -Xss512k".

(I'm currently back on the standard 1x with 512MB RAM)

[camsaul]

@lorenzosantos, I didn't realize Heroku let you set that. It should be helpful in debugging this 👍 .

Can you try doing adding these options to JAVA_TOOL_OPTIONS:

-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC

That will tell the GC to collect classes when they're no longer needed. Clojure tends to create lots of one-off classes which can cause memory issues on low-memory instances.

If that doesn't work my next suggestion is to try lowering the max heap size a bit:

-Xmx300m

Metabase should still run fine with that amount of memory. It seems like for whatever reason Heroku might be telling the JVM it can use more memory than it's actually allowed to use.

Let me know if either one works.

[kcm]

Seeing this as well on 0.19.3 on hobby/hobby-basic, using the more restrictive JVM options:

2016-09-19T23:24:39.693672+00:00 heroku[web.1]: Restarting
2016-09-19T23:24:39.694186+00:00 heroku[web.1]: State changed from up to starting
2016-09-19T23:24:43.528503+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2016-09-19T23:24:44.291326+00:00 app[web.1]: 09-19 23:24:44 INFO metabase.core :: Metabase Shutting Down ...
2016-09-19T23:24:44.296859+00:00 app[web.1]: 09-19 23:24:44 INFO metabase.core :: Metabase Shutdown COMPLETE
2016-09-19T23:24:44.874922+00:00 heroku[web.1]: Process exited with status 143
2016-09-19T23:24:50.690906+00:00 heroku[web.1]: Starting process with command `./bin/start`
2016-09-19T23:24:52.399358+00:00 app[web.1]: Setting JAVA_TOOL_OPTIONS defaults based on dyno size. Custom settings will override them.
2016-09-19T23:24:52.432857+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx300m -Xss512k -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC
2016-09-19T23:25:09.079654+00:00 app[web.1]: 09-19 23:25:09 INFO metabase.core :: Starting Metabase in STANDALONE mode
2016-09-19T23:25:09.130718+00:00 app[web.1]: 09-19 23:25:09 INFO metabase.core :: Launching Embedded Jetty Webserver with config:
2016-09-19T23:25:09.130721+00:00 app[web.1]:  {:port 38762}
2016-09-19T23:25:09.130722+00:00 app[web.1]: 
2016-09-19T23:25:09.213313+00:00 app[web.1]: 09-19 23:25:09 INFO metabase.core :: Starting Metabase version v0.19.3 (796613e release-0.19.3) ...
2016-09-19T23:25:09.216098+00:00 app[web.1]: 09-19 23:25:09 INFO metabase.core :: System timezone is 'Etc/UTC' ...
2016-09-19T23:25:09.609451+00:00 heroku[web.1]: State changed from starting to up
2016-09-19T23:25:10.373292+00:00 app[web.1]: 09-19 23:25:10 DEBUG metabase.driver :: Registered driver :bigquery 🚚
2016-09-19T23:25:10.392591+00:00 app[web.1]: 09-19 23:25:10 DEBUG metabase.driver :: Registered driver :crate 🚚
2016-09-19T23:25:10.447492+00:00 app[web.1]: 09-19 23:25:10 DEBUG metabase.driver :: Registered driver :druid 🚚
2016-09-19T23:25:10.476741+00:00 app[web.1]: 09-19 23:25:10 DEBUG metabase.driver :: Registered driver :h2 🚚
2016-09-19T23:25:10.770234+00:00 app[web.1]: 09-19 23:25:10 DEBUG metabase.driver :: Registered driver :mongo 🚚
2016-09-19T23:25:10.792514+00:00 app[web.1]: 09-19 23:25:10 DEBUG metabase.driver :: Registered driver :mysql 🚚
2016-09-19T23:25:10.810006+00:00 app[web.1]: 09-19 23:25:10 DEBUG metabase.driver :: Registered driver :postgres 🚚
2016-09-19T23:25:10.820759+00:00 app[web.1]: 09-19 23:25:10 DEBUG metabase.driver :: Registered driver :redshift 🚚
2016-09-19T23:25:10.834858+00:00 app[web.1]: 09-19 23:25:10 DEBUG metabase.driver :: Registered driver :sqlite 🚚
2016-09-19T23:25:10.850553+00:00 app[web.1]: 09-19 23:25:10 DEBUG metabase.driver :: Registered driver :sqlserver 🚚
2016-09-19T23:25:11.422963+00:00 app[web.1]: 09-19 23:25:11 INFO metabase.db :: Verifying postgres Database Connection ...
2016-09-19T23:25:11.800481+00:00 app[web.1]: 09-19 23:25:11 INFO metabase.db :: Verify Database Connection ... ✅
2016-09-19T23:25:26.428136+00:00 heroku[web.1]: Process running mem=536M(104.8%)
2016-09-19T23:25:26.428252+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)

[kcm]

And the Metrics console snapshot

[lorenzosantos]

@camsaul Tried adding those options but it didn't help. A few minutes after startup memory usage was already exceeding the quota.

Going to try lowering the max heap size to -Xmx300m now.

[lorenzosantos]

@camsaul Lowered the max heap size to -Xmx300m didn't fix the issue either. Do you have any other ideas?

[camsaul]

@lorenzosantos it seems like for whatever reason the JAVA_TOOL_OPTIONS aren't being respected or are being overriden somewhere.

@tlrobinson are we setting custom JVM options when running Metabase on Herkou?

[kcm]

I looked at this a little more, and it does seem like the memory options are being respected. The metabase-deploy project added custom settings in bin/start, then removed them in favor of the jvm buildpack (see https://github.com/metabase/metabase-deploy/pull/5).

The heroku-buildpack-jvm-common sets memory per dyno size, or picks up custom settings, in /opt/jvmcommon.sh - indeed, this seems to happen:

2016-09-26T16:55:44.775298+00:00 heroku[web.1]: Starting process with command `./bin/start`
2016-09-26T16:55:46.755831+00:00 app[web.1]: Setting JAVA_TOOL_OPTIONS defaults based on dyno size. Custom settings will override them.
2016-09-26T16:55:46.795288+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx300m -Xss512k -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC

My next step might be to try deploying the main project directly, although I can't see why that would make a difference if both metabase and metabase-deploy stable branches are 0.19.3, and the JVM options are being set already.

[camsaul]

ping @tlrobinson

[kcm]

Still happening with metabase-deploy currently on v0.20.2. Didn't look through the changelog for applicable commits, just FYI.

[symbolist]

Just discovered that if instead of heroku-buildpack-jvm-common, null-buildpack is used (the reverse of https://github.com/metabase/metabase-deploy/pull/6), the JAVA_OPTS config (which is read here https://github.com/metabase/metabase-deploy/blob/master/bin/start#L85) is respected. Otherwise neither JAVA_OPTS nor JAVA_TOOL_OPTIONS is.

In testing -Xmx300m seems to be the upper bound to keep memory usage within the limits of a 1X dyno. So it looks like the default Xmx settings in heroku-buildpack-jvm-common are too high for Metabase.

[jkutner]

This can be fixed with https://github.com/metabase/metabase-deploy/pull/5

[kanexren]

Currently still experiencing this on 0.26 with Pro 1x. There doesn't seem to be a resolution clearly stated anywhere. Is this still an ongoing issue?

[jsharpe-mr]

This appears to be ongoing. We're currently on a standard 1x dyno, so presumably hitting this code:
https://github.com/jsharpe-mr/metabase-deploy/blob/master/bin/start#L93

...which, I think is forcing the limit to 248... and leaving me no option to lower it. If I set JAVA_OPTS to -Xmx230m -Xss512k -Dfile.encoding=UTF-8 (230m + the other default jvm buildpack opts), then the resulting JAVA_OPTS is this:

JAVA_OPTS: -Xmx230m -Xss512k -Dfile.encoding=UTF-8 -Xmx248m ...
...a munging of my JAVA_OPTS and metabase-deploys. I see no evidence that it's respecting the value that I've passed it (230) instead it's sticking with the 2nd argument there (248).

Given how bin/start is currently written any ideas how I can lower the memory limits to see if these R14 errors go away?

[jsharpe-mr]

After struggling with a 1X instance, I've upgraded to a 2x instance but continued having the same problems. The errors (and subsequent crashing) tend to crop up sometime within a day of restarting the instance.... so, not completely broken, but also not truly usable.

I tried reducing the available memory with these settings:
-Xmx686m -Xss1000k

I'm now trying -Xmx670m -Xss1000k

Stay tuned. Any help appreciated.

[jkutner]

I'd recommend something like -Xmx500m -Xss512k. Most JVM apps don't need a ton of heap these days, because they use off-heap mem for stuff like IO buffers.

If you still get R14 errors with -Xmx500m, then i would bet there is an actually native memory leak in the app. I've done some blog posts about this kind of this:

• http://jkutner.github.io/2018/01/03/mystery-memory-leaks-and-jni.html
• http://jkutner.github.io/2017/04/28/oh-the-places-your-java-memory-goes.html

[jsharpe-mr]

I can't recall, but I think I tried settings even lower than -Xmx500m.

That all said, I've moved us off heroku as it just isn't cutting it - so I don't have a way to continue troubleshooting this thread.

[keyuls]

@jkutner As you explained here " the total process memory (measure.mem.linux.rss) will grow with each iteration"

I am facing a similar problem with Java Spring Application. How do I found which library is creating a problem?

[te-chris]

Has anyone had a solution for this? I've just run into it but it seems the JVM is self determining memory limits app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx300m -Xss512k -Dfile.encoding=UTF-8 We love metabase but can't continue with it if it is going to keep running out of memory on our instances.

[jkutner]

@te-chris the -Xmx300m in JAVA_TOOL_OPTIONS will be overridden by anything you put in your java command, or JAVA_OPTS even though the JVM logs the "Picked up" line. If you really want to be sure you can heroku config:set JAVA_TOOL_OPTIONS="xxx" to whatever you want.

What dyno size are you using?

You might also want to try these options as described in Dev Center:

-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap

Lots of JVM users have had luck with that.

[AndreaGiardini]

@jkutner Those two options are the way to go, but they are completely ignored:

Inside the metabase container:

bash-4.3# java -version
Picked up JAVA_TOOL_OPTIONS: 
-XX:+UnlockExperimentalVMOptions
-XX:+UseCGroupMemoryLimitForHeap
-XshowSettings:vm

Unrecognized VM option 'UseCGroupMemoryLimitForHeap'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

For some reason metabase is started with the flag -XX:+IgnoreUnrecognizedVMOptions which ignores this option silently. Why aren't we upgrading to a newest version of java?

I created a pull request for it since I tested it and everything looks good to me: https://github.com/metabase/metabase/pull/8459

[holtkamp]

Also encountered this with new 0.31, can https://github.com/metabase/metabase/pull/8459 be considered for merge?

[leenasn]

I am getting OutofMemory error after upgrading to 0.31. It was working fine until I upgraded the metabase version.

The app is deployed in Heroku, in free tier. I've been using this for quite some time now. Please let me know how to fix the same [preferably without upgrading to a paid version in Heroku]

Here is the error from the log, hope it helps.

[jkutner]

I think an upgrade to Java 11 would help. I'm not sure if that's possible, but other apps have been reporting a much better memory profile.

[leenasn]

I tried upgrading to Java 11, that didn't work either.

The issue started once I upgraded to the latest version. Is there a way I can downgrade to the previous version?

[tlrobinson]

We're working on a fix to reduce memory usage. Stay tuned.

[tlrobinson]

The issue started once I upgraded to the latest version. Is there a way I can downgrade to the previous version?

@leenasn You should be able to git clone your Heroku app's git repo, then do git reset --hard v0.30.4 and git push -f heroku master.

[leenasn]

The issue started once I upgraded to the latest version. Is there a way I can downgrade to the previous version?

@leenasn You should be able to git clone your Heroku app's git repo, then do git reset --hard v0.30.4 and git push -f heroku master.

@tlrobinson I tried that, but no luck :(.

Noticed one thing in the log that its still downloading 0.31.0 version of the Metabase. Pasting the log for your reference. Am I missing anything or doing anything wrong?

remote: -----> Metabase app detected
remote: -----> Downloading Metabase... from http://downloads.metabase.com/v0.31.0/metabase.jar to /tmp/build_7420be9059a100208eb270bbed9ec112/target/uberjar/metabase.jardone
remote: -----> Discovering process types
remote:        Procfile declares types -> web

[tlrobinson]

@leenasn Sorry, my instructions were wrong, our Heroku deploys use a custom buildpack so that they're faster. You need to change the buildpack like this:

 heroku buildpacks:remove https://github.com/metabase/metabase-buildpack
 heroku buildpacks:add https://github.com/metabase/metabase-buildpack#0.30.4

However, people have reported memory problems with 0.30.4 as well so I'd recommend going back to whatever version you were on previously until we fix the memory issue (likely this week)

You'll also need to trigger a new deploy by adding a commit and pushing:

git commit --allow-empty -m "Trigger deploy"
git push heroku master

[leenasn]

@tlrobinson the above also didn't work out, because when I changed the buildpack to use a specific version, it stopped downloading the Jar.

I forked the buildpack and changed the version to 0.30.4 and then deployed and it worked. And for me, 0.30.4 is working fine without memory issues.

Thank you for the support, looking forward to the new version with the fix for memory issues.

[camsaul]

Fixed by https://github.com/metabase/metabase-deploy/pull/22

Try deploying again and it should work pick up the new deploy script (I think) in which case it should be working without issue

[leenasn]

@camsaul I tried it. But still the same result, OutofMemory error. I tried with free, hobby and Std 1X dynos.

It works in the free dyno for version 0.30.4.

Hope this helps.

[kritop]

@camsaul this hasn't been fixed. i get the same issues since i updated metabase-deploy and update to v0.31 and now v0.31.1. v0.30.4 worked fine before. even the PL instance doesn't fix it. OOM basically happens few minutes after every reboot.

[kritop]

@camsaul thanks for opening it up again. do you have an ETA on this? if we need to deal with it for another week. i'd rather start looking for alternative hosting.

[tlrobinson]

@leenasn @kritop I pushed a change to metabase-deploy which may fix this: https://github.com/metabase/metabase-deploy/commit/c8147a7af6d135e72f00e4c4ed7d34193d4829a4 You can do the following to update your app:

heroku git:clone -a YOUR_HEROKU_APP_NAME
cd YOUR_HEROKU_APP_NAME
git remote add origin https://github.com/metabase/metabase-deploy.git
git pull origin master
git push heroku master

If that doesn't work please post the exact OOM logs you're seeing.

[camsaul]

Closing as fixed by metabase/metabase-deploy@c8147a7

If anyone is still having issues after trying @tlrobinson's fix please chime in

[leenasn]

I tried it and now am not getting OOM for 0.31 version. Yaay!!

Thank you @tlrobinson @camsaul

[holtkamp]

Same here, https://github.com/metabase/metabase/issues/3360#issuecomment-441840597 did the trick perfectly and all runs on the free Heroku container again. Thanks!

[kritop]

same here!

On Thu, 29 Nov 2018 at 11:51, Menno Holtkamp [e-mail]
wrote:

Same here, #3360 (comment)
https://github.com/metabase/metabase/issues/3360#issuecomment-441840597
did the trick perfectly and all runs on the free Heroku container again.
Thanks!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/metabase/metabase/issues/3360#issuecomment-442790354,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB8LIq9n997J1BhQvpSsByxfg4R8qLtkks5uz7wggaJpZM4J9Thu
.

--
Christoph André Müller
CTO
coffeecircle.com | Richtig. Gut. Wir ♥ Kaffee.

T: [phone]
E: [email protected]

Newsletter https://www.coffeecircle.com/de/e/newsletter | Facebook
https://www.facebook.com/coffeecircle | Youtube
https://www.youtube.com/coffeecircle | Instagram
https://www.instagram.com/coffeecircle | LinkedIn
https://www.linkedin.com/company/circle-products-gmbh/| Jobs
https://www.coffeecircle.com/de/e/jobs-und-praktika

Sitz der Gesellschaft: Lindower Str. 18, 13347 Berlin | Amtsgericht
Charlottenburg, HRB 128185 B | Geschäftsführer: Martin Elwert | USt-IdNr.
DE272941751

[te-chris]

I've just updated and am still getting this:

2018-11-29T14:47:58.543942+00:00 heroku[web.1]: Process running mem=512M(100.0%)

2018-11-29T14:47:58.544199+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)

Is there anything else I needed to do?

[leenasn]

I have seen this error in the log, but ignored it as it never affected the performance of the application so far :)

[camsaul]

Heroku's docs say R14 means some memory might get paged out to disk, which is fine. There are a lot of classes Metabase loads up that can be paged out to disk without affecting performance. You can ignore that error.

[raajeshk-2020]

Hi
I'm getting the same error from heroku.
2020-11-01T16:53:52.532510+00:00 heroku[web.1]: Process running mem=943M(184.4%)
2020-11-01T16:53:52.534138+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)

When I try to clone the metadata as explained in one of the comment above, it doesn't allow me. It throws error below. Can anyone help me? Thanks!

To https://git.heroku.com/raj-movie-recommendationsystem.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://git.heroku.com/raj-movie-recommendationsystem.git'