Rocketmq: how to decrease broker memory

[reachfish]

our rocketmq cluster contains 3 group brokers, each group contains 1 master 2 slaves. we have 5 producer client and 5 consumer client. total produce qps is 5k/s, and consumer qps is the same. every msg size is 1KB.

using "top" command, we found some broker's memory cost 48G * 70% = 33G.
my java option is "-Xms8g -Xmx8g -Xmn4g -XX:MaxDirectMemorySize=12g ...", so Xmx + MaxDirectMemorySize is just 20G

my question is:

1. why broker's memory exceed "Xms + MaxDirectMemorySize", is this java option have no effect ?
2. how to decrease broker memory ?

sincerely hope for your answer, thx very much ~

[reachfish]

after dump memory, i found commit log cost much memory. there are 20+ map file, each size is 1G.

when does the map file release memory ?

[finalcola]

log使用的是MappedByteBuffer，不受MaxDirectMemorySize的限制

[reachfish]

@finalcola 那java中有没有办法限制MappedByteBuffer的内存 ？

[finalcola]

不知道，估计只能把日志过期时间(默认是72小时)设置得小些，避免加载的日志文件过多

[reachfish]

@finalcola 好的，谢谢了！ ps. 我尝试了使用cgroup来限制， 对于二进制可执行程序，cgroup可以限制回收mmap内存，但对java，cgroup就无效了

[ahuazhu]

@finalcola
Does this cause any problem?

1. MappedByteBuffer creates Virtual-memory mapping with JVM memory thus READ/WRITE operations do not need System-Call. And it does not occupy the JVM heap but virtual memory space only.

1. When you use top to diagnose the memory usage, a large RES may be displayed. Just feel free, you could use free -m or free -g to check the actual memory usage by used. You may also find an amount of memory used by buff/cache, you could treat this part of memory as free.

[finalcola]

@ahuazhu thanks. ：）

[duhenglucky]

@reachfish the release process of pagecache is managed by os, or you can use echo 3 > /proc/sys/vm/drop_caches to release, but not recommend to do this. and issue only used be bug report and feature request, so, it would be nice if we can discuss these question in our mailing list(https://rocketmq.apache.org/about/contact/)