Generator-jhipster: Memory goes up with docker and microservices

[danielbrutti]

Hi,

Opening this ticket because of this other

Overview of the issue

I created the projects and trying to execute it over a linux machine. All containers are created just fine, I can see them with "docker ps" and look at usage with "docker stats".

However, memory keeps going up and up.. eventualy my laptop (core i5, 10gb RAM) stops responding and die!

I tried with mem_limit on each service definition on docker_compose.yml, and I can see the limit working on docker stats... however memory keeps going up and when the container gets > 95% usage it does dead... container stops working and is removed. So.. this setup just prevents my pc to die.. but nothing else.

I also tried with a JHipster project and set JAVA_OPTS for -Xmx and other memory parameters.. but the result is the same.

The registry, gateway, business microservices and UAA need at least 1.5GB to not die and be able to start, other images (mysql, elasticsearch, etc) can work with 512m. However having 4 x 1.5GB and several 512mb is consuming all my memory. JHipster Registry should be ok with 512mb.. I checked that with registry deployed on Heroku and this is what I found more confusing! why is consuming 1.5Gb on my laptop??

I will check ehcache configuration based on @jdubois advice.

I'm running elementary OS (ubuntu 14.04 based), 64 bits, Java 1.8, Docker version 1.11.2 build b9f10c9, and I did the cgroups configuracion on grub for memory (from docker installation guide). Here is my docker-compose.yml (extract for gateway service so you can check memory configs).

gateway-app:
        mem_limit: 1280m
        image: gateway
        external_links:
            - gateway-mysql:mysql
            - gateway-elasticsearch:elasticsearch
            - jhipster-registry:registry
        environment:
            - SPRING_PROFILES_ACTIVE=prod
            - SPRING_CLOUD_CONFIG_URI=http://admin:admin@registry:8761/config
            - SPRING_DATASOURCE_URL=jdbc:mysql://mysql:3306/gateway?useUnicode=true&characterEncoding=utf8&useSSL=false
            - SPRING_DATA_CASSANDRA_CONTACTPOINTS=cassandra
            - JHIPSTER_SLEEP=30
            - SPRING_DATA_ELASTICSEARCH_CLUSTER_NODES=elasticsearch:9300
            - EUREKA_CLIENT_SERVICEURL_DEFAULTZONE=http://admin:admin@registry:8761/eureka
            - JAVA_OPTS="-Xmx1280m -Xss512k"
        ports:
            - 8080:8080

JHipster Version(s)

3.4.2

JHipster configuration, a .yo-rc.json file generated in the root folder

And this is my .yo-rc.json for gateway app.. let me know if you need the other ones.

{
  "generator-jhipster": {
    "jhipsterVersion": "3.4.2",
    "baseName": "gateway",
    "packageName": "com.jhipster",
    "packageFolder": "com/jhipster",
    "serverPort": "8080",
    "authenticationType": "uaa",
    "uaaBaseName": "uaa",
    "hibernateCache": "hazelcast",
    "clusteredHttpSession": "hazelcast",
    "websocket": "spring-websocket",
    "databaseType": "sql",
    "devDatabaseType": "mysql",
    "prodDatabaseType": "mysql",
    "searchEngine": "elasticsearch",
    "buildTool": "maven",
    "jwtSecretKey": "efcf63fc245010dc39e58d4fcbcdd09ef8149bca",
    "useSass": true,
    "applicationType": "gateway",
    "testFrameworks": [
      "gatling"
    ],
    "jhiPrefix": "jhi",
    "enableTranslation": true,
    "nativeLanguage": "en",
    "languages": [
      "en",
      "es"
    ]
  }
}
{
  "generator-jhipster": {
    "jhipsterVersion": "3.4.2",
    "baseName": "gateway",
    "packageName": "com.jhipster",
    "packageFolder": "com/jhipster",
    "serverPort": "8080",
    "authenticationType": "uaa",
    "uaaBaseName": "uaa",
    "hibernateCache": "hazelcast",
    "clusteredHttpSession": "hazelcast",
    "websocket": "spring-websocket",
    "databaseType": "sql",
    "devDatabaseType": "mysql",
    "prodDatabaseType": "mysql",
    "searchEngine": "elasticsearch",
    "buildTool": "maven",
    "jwtSecretKey": "efcf63fc245010dc39e58d4fcbcdd09ef8149bca",
    "useSass": true,
    "applicationType": "gateway",
    "testFrameworks": [
      "gatling"
    ],
    "jhiPrefix": "jhi",
    "enableTranslation": true,
    "nativeLanguage": "en",
    "languages": [
      "en",
      "es"
    ]
  }
}

Browsers and Operating System

Elementary OS, based on Ubuntu 14.04

Reproduce the error

1. create a microservice project with 1 entity
2. create a gateway with 1 entity (from microservice)
3. create UAA project
4. create docker-compose
5. start it with docker-compose up -d
   

   Related issues

https://github.com/jhipster/jhipster-registry/issues/63

Thanks!

[pascalgrimaud]

Hello,

Very strange. I didn't manage to reproduce your issue.
Here my docker stats $(docker ps --format={{.Names}}):

CONTAINER                 CPU %               MEM USAGE / LIMIT     MEM %               NET I/O               BLOCK I/O             PIDS
jhuaa-mysql               0.04%               221 MB / 7.966 GB     2.77%               66.69 kB / 36.78 kB   8.802 MB / 302.5 MB   37
00compose_msmongo-app_1   0.09%               965.6 MB / 7.966 GB   12.12%              121.3 kB / 54.64 kB   89.84 MB / 14.88 MB   35
00compose_gateway-app_1   0.39%               1.106 GB / 7.966 GB   13.89%              227.5 kB / 484 kB     117 MB / 28.84 MB     92
jhipster-registry         0.74%               1.018 GB / 7.966 GB   12.78%              120.2 kB / 126.4 kB   91.12 MB / 139.3 kB   63
gateway-elasticsearch     0.27%               249.1 MB / 7.966 GB   3.13%               42.57 kB / 21.33 kB   48.16 MB / 4.096 kB   58
00compose_jhuaa-app_1     0.29%               1.042 GB / 7.966 GB   13.08%              101.8 kB / 78.84 kB   70.08 MB / 13.5 MB    68
msmongo-mongodb           0.34%               44.8 MB / 7.966 GB    0.56%               49.72 kB / 48.08 kB   33.97 MB / 811 kB     18
gateway-mysql             0.03%               202.7 MB / 7.966 GB   2.54%               60.84 kB / 31.22 kB   27.03 MB / 297 MB     37

My OS: Ubuntu 16.04 - Linux pgrxps 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Ram: 8go
Docker version: 1.11.1
Compose version: 1.7.1
Version of JHipster: master

Can you try to reproduce your issue with our devbox ?

[gmarziou]

It's not clear whether the problem is on java side or on docker side.
Have you tried to dump your java process and use a profiler to analyze memory usage?

WIthout Docker, I have some OutOfMemory errors in registry after few days if I limit it to 192 MB.
I plan to profile it but haven't got time yet.

[danielbrutti]

Hi,

Thanks for looking into this. I will try both alternatives (devbox and run it with ./mvnw).

I must say your memory usage is similar to mine.. maybe mine a bitter higher.. but registry, uaa, gateway are about 1GB like yours. Which is weird since when deploying on heroku it works just fine with a -Xmx of 300 something MB.

I will get back to you with more information.. but I'm thinking could be a docker/linux configuration because I'm having the same issues trying these projects.

Thanks again!

Daniel

[danielbrutti]

Hi.. here again..

We believe the issue is not with JHipster, but with docker containers.

I started a registry, gateway and microservice app using ./mvnw -Pprod and it works great.

• Registry has 430mb assigned, 130mb (33%) used after start.
• Gateway is using 818M.. and increased registry memory usage by 40mb
• Microservice is consuming 818M too... and increased registry memory by 70mb.

So at the end I have 240mb on registry + 1.6gb in gateway and microservice. This doesn't sound too bad to me and also probably it can be better after adjust jvm memory parameters for gateway and microservice.

I will continue working on this and post here if is something with docker in order to help other people with this issue.

Thanks,

Daniel

[danielbrutti]

Hi,

I found it.. the issue is with memory parameters for the JVM.

I cloned the jhipster registry project and worked with different docker configurations, images generation, image optimization, etc, nothing worked. Running the registry with ./mvnw command was just beautiful.. but I need it on docker.

Finally, realize on modify src/main/docker/dev/Dockerfile and modify the java CMD for starting the app adding 2 simple parameters: -Xmx512m, -Xms256mb

Now, the container works perfectly under 512mb... registry app shows stats of:

Total Available Memory  323mb
Current Memory Usage    130mb (40%)

I can't ask for something else!

My Dockerfile looks like this now:

FROM java:openjdk-8-jdk-alpine

ENV JAVA_HOME=/usr/lib/jvm/default-jvm
RUN apk add --no-cache openjdk8 && \
    ln -sf "${JAVA_HOME}/bin/" "/usr/bin/"

# add directly the war
ENV SPRING_PROFILES=dev
ADD *.war /jhipster-registry.war

RUN sh -c 'touch /jhipster-registry.war'
EXPOSE 8761
VOLUME /tmp
CMD ["java","-Xmx512m", "-Xms256m", "-Djava.security.egd=file:/dev/./urandom","-jar","/jhipster-registry.war","--spring.profiles.active=${SPRING_PROFILES}"]

Running on your vagrant devbox, the memory seems to be ok.. any process goes higher than 600mb... however after a few minutes the VM starts to work with some lag.

Now, my last question for you guys:

1. Do you have any java configuration on the vagrant devbox? how is possible that projects just work fine there without any JVM memory param on Dockerfile?
2. Is there a way to setup these parameters outside of Dockerfile, maybe on docker-compose.yml?

Again, thanks for your help!

[pascalgrimaud]

Thanks for feedback, @danielbrutti
Different points:

• In the jhipster-registry project, src/main/docker/dev/Dockerfile is old (the based image is alpine:3.3 instead of java:openjdk-8-jdk-alpine / java:openjdk-8-jre-alpine). This Dockerfile is used only for development -> we had to update it to be consistent with the main Dockerfile
• if it works fine with our vagrant devbox, maybe an issue with your OS? I have no issue with my personal ubuntu 16.04 too, and same for Windows7 at work

About your questions
1) yes, try something like that, in your Dockerfile:

ENV JAVA_OPTS="-Xmx512m -Xmx256m"

2) My prefered solution, in your docker-compose file:

environment:
  (...)
  - JAVA_OPTS="-Xmx512m -Xmx256m"
  (...)

[danielbrutti]

Hi Pascal,

Thanks for the feedback, your help was really helpful guys. Closing the ticket now.

Keep the hard work!

Daniel

[PierreBesson]

@danielbrutti Do you think adding some information to https://jhipster.github.io/docker-compose/ about memory tweaking and docker stats would help ? If so would you consider to PR a short section to https://github.com/jhipster/jhipster.github.io

[danielbrutti]

Sure, no problem. If the hipster guys are ok about having me modifying the site :)

[PierreBesson]

It will be alright, this is a community project. We actually need your help ! Just be concise. I think it might be important for some people. Thanks.

[danielbrutti]

done! https://github.com/jhipster/jhipster.github.io/pull/283

[PierreBesson]

Thanks !