Generator-jhipster: Cassandra backend tests failures

Created on 24 Jan 2020  ·  28Comments  ·  Source: jhipster/generator-jhipster
Overview of the issue

In our CI, the backend tests with Cassandra failed regularly and randomly...
It needs to be fixed.

See https://github.com/jhipster/generator-jhipster/runs/406645977
See the discussion in our mailing list: https://groups.google.com/forum/#!topic/jhipster-dev/DdPPWTlIk0c

Motivation for or Use Case

Here an extract of logs:

[ERROR] io.github.jhipster.sample.web.rest.UserResourceIT.testUserToUserDTO  Time elapsed: 5.8 s  <<< ERROR!
com.datastax.driver.core.exceptions.TransportException: [localhost/127.0.0.1:32771] Connection has been closed
    at io.github.jhipster.sample.web.rest.UserResourceIT.initTest(UserResourceIT.java:127)
Caused by: com.datastax.driver.core.exceptions.TransportException: [localhost/127.0.0.1:32771] Connection has been closed

[INFO] Running io.github.jhipster.sample.web.rest.errors.ExceptionTranslatorIT
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.359 s - in io.github.jhipster.sample.web.rest.errors.ExceptionTranslatorIT
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   UserResourceIT.createUser:149 Status expected:<201> but was:<500>
[ERROR] Errors: 
[ERROR]   CassTestServiceImplEntityResourceIT.initTest:270 » CassandraConnectionFailure ...
[ERROR]   CassTestServiceImplEntityResourceIT.initTest:270 » CassandraConnectionFailure ...
[ERROR]   CassTestServiceImplEntityResourceIT.initTest:270 » CassandraConnectionFailure ...
[ERROR]   CassTestServiceImplEntityResourceIT.initTest:270 » CassandraConnectionFailure ...
[ERROR]   CassTestServiceImplEntityResourceIT.initTest:270 » CassandraConnectionFailure ...
[ERROR]   UserResourceIT.initTest:127 » OperationTimedOut [localhost/127.0.0.1:32771] Ti...
[ERROR]   UserResourceIT.initTest:127 » Transport [localhost/127.0.0.1:32771] Connection...
[ERROR]   UserResourceIT.initTest:127 » OperationTimedOut [localhost/127.0.0.1:32771] Ti...
[INFO] 
[ERROR] Tests run: 150, Failures: 1, Errors: 8, Skipped: 0
##### **Reproduce the error** ##### **Related issues** ##### **Suggest a Fix** ##### **JHipster Version(s)** Current master ##### **JHipster configuration** Welcome to the JHipster Information Sub-Generator ##### **JHipster Version(s)** 
[email protected] /home/runner/app
└── [email protected]
##### **JHipster configuration, a `.yo-rc.json` file generated in the root folder**
.yo-rc.json file 
{
  "generator-jhipster": {
    "applicationType": "monolith",
    "baseName": "sampleReactCassandraSessionRedis",
    "packageName": "io.github.jhipster.sample",
    "packageFolder": "io/github/jhipster/sample",
    "serverPort": "8080",
    "authenticationType": "session",
    "cacheProvider": "redis",
    "enableHibernateCache": false,
    "websocket": false,
    "databaseType": "cassandra",
    "devDatabaseType": "cassandra",
    "prodDatabaseType": "cassandra",
    "searchEngine": false,
    "messageBroker": false,
    "serviceDiscoveryType": false,
    "buildTool": "maven",
    "enableSwaggerCodegen": false,
    "rememberMeKey": "YourJWTSecretKeyWasReplacedByThisMeaninglessTextByTheJHipsterInfoCommandForObviousSecurityReasons",
    "embeddableLaunchScript": false,
    "useSass": true,
    "clientPackageManager": "npm",
    "clientFramework": "react",
    "clientTheme": "none",
    "clientThemeVariant": "",
    "testFrameworks": ["gatling", "protractor"],
    "jhiPrefix": "jhi",
    "entitySuffix": "",
    "dtoSuffix": "DTO",
    "otherModules": [],
    "enableTranslation": true,
    "nativeLanguage": "en",
    "languages": ["en", "fr"],
    "blueprints": [],
    "jhipsterVersion": "6.6.0",
    "creationTimestamp": 1579850992835
  }
}

JDL for the Entity configuration(s) entityName.json files generated in the .jhipster directory


JDL entity definitions

entity CassBankAccount {
  userId UUID,
  name String required,
  bankNumber Integer,
  agencyNumber Long,
  lastOperationDuration Float,
  meanOperationDuration Double,
  balance BigDecimal required,
  lastOperationDate Date,
  active Boolean,
  picture ByteBuffer required,
  operationsFile ByteBuffer,
  accountType AccountTypeEnum
}
entity CassTestEntity {
  stringField String,
  stringRequiredField String required,
  stringMinlengthField String minlength(0),
  stringMaxlengthField String maxlength(20),
  stringPatternField String pattern(/^[a-zA-Z0-9]*$/),
  integerField Integer,
  integerRequiredField Integer required,
  integerMinField Integer min(0),
  integerMaxField Integer max(100),
  longField Long,
  longRequiredField Long required,
  longMinField Long min(0),
  longMaxField Long max(100),
  floatField Float,
  floatRequiredField Float required,
  floatMinField Float min(0),
  floatMaxField Float max(100),
  doubleRequiredField Double required,
  doubleMinField Double min(0),
  doubleMaxField Double max(100),
  bigDecimalRequiredField BigDecimal required,
  bigDecimalMinField BigDecimal min(0),
  bigDecimalMaxField BigDecimal max(100),
  dateField Date,
  dateRequiredField Date required,
  zonedDateTimeField ZonedDateTime,
  zonedDateTimeRequiredField ZonedDateTime required,
  booleanField Boolean,
  booleanRequiredField Boolean required,
  picture ByteBuffer required,
  operationsFile ByteBuffer
}
entity CassTestMapstructEntity {
  stringField String,
  stringRequiredField String required,
  stringMinlengthField String minlength(0),
  stringMaxlengthField String maxlength(20),
  stringPatternField String pattern(/^[a-zA-Z0-9]*$/),
  integerField Integer,
  integerRequiredField Integer required,
  integerMinField Integer min(0),
  integerMaxField Integer max(100),
  longField Long,
  longRequiredField Long required,
  longMinField Long min(0),
  longMaxField Long max(100),
  floatField Float,
  floatRequiredField Float required,
  floatMinField Float min(0),
  floatMaxField Float max(100),
  doubleRequiredField Double required,
  doubleMinField Double min(0),
  doubleMaxField Double max(100),
  bigDecimalRequiredField BigDecimal required,
  bigDecimalMinField BigDecimal min(0),
  bigDecimalMaxField BigDecimal max(100),
  dateField Date,
  dateRequiredField Date required,
  zonedDateTimeField ZonedDateTime,
  zonedDateTimeRequiredField ZonedDateTime required,
  booleanField Boolean,
  booleanRequiredField Boolean required,
  picture ByteBuffer required,
  operationsFile ByteBuffer
}
entity CassTestServiceClassEntity {
  stringField String,
  stringRequiredField String required,
  stringMinlengthField String minlength(0),
  stringMaxlengthField String maxlength(20),
  stringPatternField String pattern(/^[a-zA-Z0-9]*$/),
  integerField Integer,
  integerRequiredField Integer required,
  integerMinField Integer min(0),
  integerMaxField Integer max(100),
  longField Long,
  longRequiredField Long required,
  longMinField Long min(0),
  longMaxField Long max(100),
  floatField Float,
  floatRequiredField Float required,
  floatMinField Float min(0),
  floatMaxField Float max(100),
  doubleRequiredField Double required,
  doubleMinField Double min(0),
  doubleMaxField Double max(100),
  bigDecimalRequiredField BigDecimal required,
  bigDecimalMinField BigDecimal min(0),
  bigDecimalMaxField BigDecimal max(100),
  dateField Date,
  dateRequiredField Date required,
  zonedDateTimeField ZonedDateTime,
  zonedDateTimeRequiredField ZonedDateTime required,
  booleanField Boolean,
  booleanRequiredField Boolean required,
  picture ByteBuffer required,
  operationsFile ByteBuffer
}
entity CassTestServiceImplEntity {
  stringField String,
  stringRequiredField String required,
  stringMinlengthField String minlength(0),
  stringMaxlengthField String maxlength(20),
  stringPatternField String pattern(/^[a-zA-Z0-9]*$/),
  integerField Integer,
  integerRequiredField Integer required,
  integerMinField Integer min(0),
  integerMaxField Integer max(100),
  longField Long,
  longRequiredField Long required,
  longMinField Long min(0),
  longMaxField Long max(100),
  floatField Float,
  floatRequiredField Float required,
  floatMinField Float min(0),
  floatMaxField Float max(100),
  doubleRequiredField Double required,
  doubleMinField Double min(0),
  doubleMaxField Double max(100),
  bigDecimalRequiredField BigDecimal required,
  bigDecimalMinField BigDecimal min(0),
  bigDecimalMaxField BigDecimal max(100),
  dateField Date,
  dateRequiredField Date required,
  zonedDateTimeField ZonedDateTime,
  zonedDateTimeRequiredField ZonedDateTime required,
  booleanField Boolean,
  booleanRequiredField Boolean required,
  picture ByteBuffer required,
  operationsFile ByteBuffer
}
enum AccountTypeEnum {
  STANDARD,
  PREMIUM
}

dto CassBankAccount, CassTestMapstructEntity with mapstruct
service CassBankAccount, CassTestMapstructEntity, CassTestServiceImplEntity with serviceImpl
service CassTestServiceClassEntity with serviceClass
angularSuffix CassBankAccount with mySuffix
filter CassTestServiceClassEntity, CassTestServiceImplEntity

Environment and Tools

openjdk version "11.0.6" 2020-01-14 LTS
OpenJDK Runtime Environment Zulu11.37+17-CA (build 11.0.6+10-LTS)
OpenJDK 64-Bit Server VM Zulu11.37+17-CA (build 11.0.6+10-LTS, mixed mode)

git version 2.25.0

node: v12.14.0

npm: 6.13.4

yarn: 1.21.1

Docker version 3.0.8, build 2355349d

docker-compose version 1.22.0, build f46880fe

INFO! Congratulations, JHipster execution is complete!

$$ bug-bounty $$ $300 area Continuous Integration cassandra
Source
picture pascalgrimaud
🎉1

All 28 comments

@SudharakaP : if it can help, it seems to be more stable with Azure, less failures there
See https://dev.azure.com/hipster-labs/jhipster-daily-builds/_build?definitionId=11&_a=summary

pascalgrimaud picture pascalgrimaud on 25 Jan 2020
👍1

It is something related to our Continuous Integration, as I couldn't reproduce locally

pascalgrimaud picture pascalgrimaud on 26 Jan 2020
👍1

@pascalgrimaud : Yes, I agree it might be related to the CI itself. I could reproduce this in my fork. I was trying out various different things yesterday, but unfortunately couldn't find a proper fix to this yet. Will try out some things today as well. :thinking:

SudharakaP picture SudharakaP on 26 Jan 2020

@SudharakaP :

pascalgrimaud picture pascalgrimaud on 26 Jan 2020
👍1

Here my results:

I stop my investigation for today, and will maybe retry it later if I'm motivated enough with this issue...

pascalgrimaud picture pascalgrimaud on 26 Jan 2020
👍1

@pascalgrimaud : I agree it comes from deleteAll as in my CI also I noticed it always fails there. Let me think about making a refactoring for that :thinking:

SudharakaP picture SudharakaP on 26 Jan 2020

@SudharakaP : to help reproducing in the CI, you can use this https://github.com/pascalgrimaud/generator-jhipster/blob/cassandra-backend-tests/.github/workflows/react.yml#L48-L63

So no need to commit / push, each time you tried something.

pascalgrimaud picture pascalgrimaud on 27 Jan 2020
👍1

@pascalgrimaud : Ah, thanks; let me try that out. I am close to finishing the refactoring, hopefully can get the PR done in the next couple of days. :smile:

SudharakaP picture SudharakaP on 27 Jan 2020

I've done a PR with the refactoring of the UserResourceIT and EntityResourceIT classes so that the deleteAll calls are removed from the BeforeEach block. I think the other PR(#11182) is still needed but from my tests removing the deleteAll before every single unit test seem to make the test suit move a bit faster and this refactoring should help to remove any remaining failures with the Cassandra builds.

I am not absolutely sure if this is the best way to refactor the unit tests and the EntityResourceIT.java.ejs file seemed very complex. I'll keep thinking of a way to do it better. :thinking: Feel free to let me know if you see any issues or suggestions. :smile:

SudharakaP picture SudharakaP on 29 Jan 2020

I remember having this kind of issue.
To fix it, I used DURABLE_WRITES = false (instead of true), for the keyspace creation only for integration testing.

ctamisier picture ctamisier on 29 Jan 2020
👍1

@ctamisier : Thanks for the suggestion. 👍

We can try #11204, since I believe it fixes the Cassandra builds. If we see the issues again we can try the DURABLE_WRITE thing as well. 😄

SudharakaP picture SudharakaP on 29 Jan 2020

I tried :

pascalgrimaud picture pascalgrimaud on 30 Jan 2020
👍1

@pascalgrimaud : You are right. I did another bunch of tests and some of
them seems to fail. Back to square one.... Let me think about some
different approachs to this. :thinking:

On Wed, Jan 29, 2020, 3:20 PM Pascal Grimaud notifications@github.com
wrote:

I tried :

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jhipster/generator-jhipster/issues/11169?email_source=notifications&email_token=AC64D7OUFDXKTKTZUTH4JBDRAIFLDA5CNFSM4KLCLN6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKJD2HY#issuecomment-580009247,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC64D7LXZJMU2PQR5LAEJ4DRAIFLDANCNFSM4KLCLN6A
.

SudharakaP picture SudharakaP on 30 Jan 2020

To track the problem, it started with these commits:

jhipster lib:

*** jhipster: JHI_LIB_REPO=https://github.com/jhipster/jhipster.git with JHI_LIB_BRANCH=master
Cloning into 'jhipster'...
* 63450d3 - (HEAD -> master, origin/master, origin/HEAD) Change version for development 3.3.0-SNAPSHOT (10 hours ago) <Pascal Grimaud>
* f695a33 - (tag: 3.2.0) Release 3.2.0 (10 hours ago) <Pascal Grimaud>
*   70ba482 - Merge pull request #544 from jhipster/travis (13 hours ago) <Daniel Franco>
|\  
| * 8e4e560 - (origin/travis) Update Angular workflow: related with https://github.com/jhipster/generator-jhipster/pull/11100 (13 hours ago) <Daniel Franco>
| * 62507d0 - Remove travis and substitute by github actions (13 hours ago) <Daniel Franco>
| * 00b087f - Remove travis (13 hours ago) <Daniel Franco>
|/  
*   af89c5b - Merge pull request #542 from jhipster/spring-boot_2.2.3 (13 hours ago) <Daniel Franco>
|\  
| * 97b5931 - Switch back to master branch (13 hours ago) <Daniel Franco>
| * 2d0458e - Switch back to master branch (13 hours ago) <Daniel Franco>
| * 3987d78 - Switch back to master branch (13 hours ago) <Daniel Franco>

generator-jhipster:

*** generator-jhipster: JHI_GEN_REPO=https://github.com/jhipster/generator-jhipster.git with JHI_GEN_BRANCH=master
Cloning into 'generator-jhipster'...
*   c1911441d - (HEAD -> master, origin/master, origin/HEAD) Merge pull request #11125 from murdos/sb22-extra (9 hours ago) <Aurélien Mino>
|\  
| * 1943a90d9 - Spring Boot now lets Hibernate chose the dialect to use rather than applying a default dialect based on the detected database (30 hours ago) <Aurélien Mino>
| * a481617fd - Spring Boot now lets Hibernate chose the dialect to use rather than applying a default dialect based on the detected database (3 days ago) <Aurélien Mino>
* |   1dca2ea51 - Merge pull request #11138 from qmonmert/feature/improve-code (9 hours ago) <Aurélien Mino>
|\ \  
| * | 779f66884 - Simplify statement (9 hours ago) <qmonmert>
| * | b1e8bcc1e - The declared exception is never thrown (10 hours ago) <qmonmert>
| * | 330a3aef7 - Declaration can have final modifier (10 hours ago) <qmonmert>
| * | 9325c38cd - Unnecessary semicolon (10 hours ago) <qmonmert>
* | |   0f549eb87 - Merge pull request #11137 from jhipster/jhipster-bom_3.2.0 (9 hours ago) <Daniel Franco>
|\ \ \  
| |/ /  
|/| |   
| * | 56c78e5f3 - Switch jhipster lib to release (10 hours ago) <Daniel Franco>

So maybe something related to Spring Boot 2.2 ?

pascalgrimaud picture pascalgrimaud on 30 Jan 2020

@SudharakaP : I found this https://github.com/spring-projects/spring-boot/issues/19802 -> do you think it can be related ?

pascalgrimaud picture pascalgrimaud on 30 Jan 2020
👍1

@pascalgrimaud This is what I try to implement, using the CqlSession... but did not have enough knowledge to finalise it.

DanielFran picture DanielFran on 30 Jan 2020
👍1

@DanielFran :

?

pascalgrimaud picture pascalgrimaud on 30 Jan 2020
👍1

@pascalgrimaud Yes and the commits I did rollback:
https://github.com/jhipster/generator-jhipster/pull/11175/commits/03158ce80ef675acdacf7f66583f814a16e3c082
https://github.com/jhipster/generator-jhipster/pull/11175/commits/93b7cd08631f0b571b32ed116588d130a01a81aa
https://github.com/jhipster/generator-jhipster/pull/11175/commits/73fecfa2d781ee0bee67f6962434447492cd58ca
https://github.com/jhipster/generator-jhipster/pull/11175/commits/6cf3b6ba279abe3f8a7d93ac7ffc91d8532777dd

This is also related with the update to cassandra-unit v4.3.1.0 to sync with last cassandra driver: https://github.com/jsevellec/cassandra-unit/compare/cassandra-unit-parent-3.11.2.0...cassandra-unit-parent-4.3.1.0

DanielFran picture DanielFran on 30 Jan 2020
👍1

@SudharakaP : I found this spring-projects/spring-boot#19802 -> do you think it can be related ?

@pascalgrimaud : Yes, I have a feeling that that one is related. However I've tried a similar solution; https://github.com/jsevellec/cassandra-unit/issues/184#issuecomment-516615678; trying to increase the timeouts, but that didn't work. There might be some differences in that thread you mentioned; let me carefully go through it and try again 😄

SudharakaP picture SudharakaP on 30 Jan 2020

Here some more tests, and it doesn't change anything:

  • remove Redis from config for testing only Cassandra
  • play with PoolOptions
  • play with SocketOptions
  • remove --batch-mode
  • change sysctl values
pascalgrimaud picture pascalgrimaud on 3 Feb 2020
👍1

@pascalgrimaud : Yes, I tried those as well; but adding .withSharedMemorySize((long) 8 * 1024 * 1024) has some improvement I think with one of my tests passing every single run (https://github.com/SudharakaP/generator-jhipster/actions/runs/34087687). But on average it fails for two out of the ten. :disappointed:

SudharakaP picture SudharakaP on 3 Feb 2020

It seems this is a deep rooted problem with the spring boot update (https://github.com/spring-projects/spring-boot/issues/19802). The solution suggested in that thread is for 2.3.0.M2 and I think if we need to implement a solution using the CqlSession we have to migrate to v2.3.0.M1 or higher since that's where spring-boot starts supporting Cassandra 4 driver. But given that v2.3.0.M1 is a pre-release I am not sure if we should do that now itself; I think we should wait for v2.3.1 and then do a Cassandra driver upgrade.

According to my testing, if we downgrade the cassandra-unit-spring dependency to the previous major version (2.2.2.1) the tests seems to pass without any problem. I've done 15 tests for both gradle and maven and all seem to pass the backend stage.

So for the time being, I've created a pull request to downgrade the cassandra-unit-spring dependency. Feel free to let me know your thoughts. :smile:

SudharakaP picture SudharakaP on 4 Feb 2020

I'm becoming crazy with this ticket
Thanks a lot for this deep analyze. If downgrading the cassandra-unit-spring dependencies resolve the issue, then let's do it !

With this fix, I'll be able to do a release !

pascalgrimaud picture pascalgrimaud on 4 Feb 2020
👍1

@pascalgrimaud : Sure thing; I've submitted the PR. Feel free to give it a test and let me know if you see the issue again. I did three rounds of testing as indicated above and didn't see a single failure both on maven or on gradle so I think we are good. :smile:

SudharakaP picture SudharakaP on 4 Feb 2020

you used the same modified React.yml than I suggested (which I could rename to CrazyCassandraBug.yml) so it's ok !!!

pascalgrimaud picture pascalgrimaud on 4 Feb 2020
👍1

@pascalgrimaud : Yes indeed. It's a very insidious bug 👿

SudharakaP picture SudharakaP on 4 Feb 2020

@SudharakaP don't forget to claim the bounty, very well deserved

pascalgrimaud picture pascalgrimaud on 5 Feb 2020
👍1

@pascalgrimaud : Thanks for reminding me. Done at https://opencollective.com/generator-jhipster/expenses/13624

SudharakaP picture SudharakaP on 5 Feb 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Steven-Garcia picture Steven-Garcia  ·  3Comments
shivroy121 picture shivroy121  ·  3Comments
sdoxsee picture sdoxsee  ·  4Comments
edvjacek picture edvjacek  ·  3Comments
marcelinobadin picture marcelinobadin  ·  3Comments