Generator-jhipster: Compatibility for Elasticsearch 7.1.1

[mvasil15]

Overview of the issue

Me and my team are trying to setup and EFK stack for the monitoring of our application and were trying to put a security up for the Elasticsearch (version 7.1.1). However when attempting to authenticate into the elasticsearch through the spring-data-jest configuration available in jhipster we would get a 401 unauthorized or 403 forbidden despite being sure that the accounts used had the right roles and access. I soon started to suspect an incompatability between ES 7.1.1 and jhipster.

In order to try and isolate the issue i did the following:

I've generated a basic application with the jhipster generator, connected it to an Elasticsearch 7.1.1 and it fails with the following error:

Caused by: com.github.vanroy.springdata.jest.exception.JestElasticsearchException: Cannot execute jest action , response code : 400 , error : {"root_cause":[{"type":"parse_exception","reason":"unknown key [index.refresh_interval] for create index"}],"type":"parse_exception","reason":"unknown key [index.refresh_interval] for create index"} , message : null
at com.github.vanroy.springdata.jest.mapper.DefaultErrorMapper.mapError(DefaultErrorMapper.java:30)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.execute(JestElasticsearchTemplate.java:1126)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.executeWithAcknowledge(JestElasticsearchTemplate.java:1136)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.createIndex(JestElasticsearchTemplate.java:197)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.createIndexWithSettings(JestElasticsearchTemplate.java:1345)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.createIndexIfNotCreated(JestElasticsearchTemplate.java:1330)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.createIndex(JestElasticsearchTemplate.java:178)
at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.createIndex(AbstractElasticsearchRepository.java:96)
at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.<init>(AbstractElasticsearchRepository.java:87)
at org.springframework.data.elasticsearch.repository.support.NumberKeyedRepository.<init>(NumberKeyedRepository.java:36)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:172)
... 46 common frames omitted

Motivation for or Use Case

I'm trying to setup an EFK stack for the monitoring of my application and the elasticsearch version i wanted to use was 7.1.1. As no particular version of elasticsearch in requested whilst generating the app i guessed that it would be compatible with any version.

Reproduce the error

_Prerequisites_:

• docker
• jhipster

_Step one_: Generate a jhipster application with elasticsearch option with jhipster generate
_Step two_: Create an elasticsearch, docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.1.1
_Step three_: ./mvnw

And the bug should appear.

Note that if you change the image tag from 7.1.1 to 6.4.3, the error doesn't appear and elasticsearch works without any issue

Related issues

I did not find any similar issues.

Suggest a Fix

I'm not sure at all, i'm guessing it has to do with the ES version and an incompatability of the spring data jest resource. On my side i tried to create my application's indices manually through ES' rest api in order to see if the application would work. However i noticed that the mapping required to initialize an index is different from ES 6.4.3 to 7.1.1. So even if i do create the index in es 7.1.1 the application then throws a different error:

Caused by: org.springframework.data.elasticsearch.ElasticsearchException: Failed to build mapping for user:user
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.putMapping(JestElasticsearchTemplate.java:265)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.putMapping(JestElasticsearchTemplate.java:277)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.putMapping(JestElasticsearchTemplate.java:237)
at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.putMapping(AbstractElasticsearchRepository.java:100)
at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.<init>(AbstractElasticsearchRepository.java:88)
at org.springframework.data.elasticsearch.repository.support.NumberKeyedRepository.<init>(NumberKeyedRepository.java:36)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:172)
... 46 common frames omitted
Caused by: com.github.vanroy.springdata.jest.exception.JestElasticsearchException: Cannot execute jest action , response code : 400 , error : {"root_cause":[{"type":"illegal_argument_exception","reason":"Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."}],"type":"illegal_argument_exception","reason":"Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."} , message : null
at com.github.vanroy.springdata.jest.mapper.DefaultErrorMapper.mapError(DefaultErrorMapper.java:30)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.execute(JestElasticsearchTemplate.java:1126)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.executeWithAcknowledge(JestElasticsearchTemplate.java:1136)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.putMapping(JestElasticsearchTemplate.java:262)
... 57 common frames omitted

The error seems to be indicating that the indexes are initialized in a way that is compatible with ES 6.4.3 and not 7.1.1.

JHipster Version(s)

[email protected] /Users/MARTIN/workspace/testapp
└── [email protected]

JHipster configuration

{
  "generator-jhipster": {
    "promptValues": {
      "packageName": "com.mycompany.myapp",
      "nativeLanguage": "en"
    },
    "jhipsterVersion": "6.1.2",
    "applicationType": "monolith",
    "baseName": "testapp",
    "packageName": "com.mycompany.myapp",
    "packageFolder": "com/mycompany/myapp",
    "serverPort": "8080",
    "authenticationType": "jwt",
    "cacheProvider": "ehcache",
    "enableHibernateCache": true,
    "websocket": false,
    "databaseType": "sql",
    "devDatabaseType": "h2Disk",
    "prodDatabaseType": "mysql",
    "searchEngine": "elasticsearch",
    "messageBroker": false,
    "serviceDiscoveryType": false,
    "buildTool": "maven",
    "enableSwaggerCodegen": false,
    "jwtSecretKey": "NGEyN2VhNDQ4MGY5NDE1ZjViNWVhNjk0ZmQxNzM0NGNiMzg3ODlmNDhiYjMzMmIwYmFjZjQ2MTNiMzZiMGNlYTljYTQ5ZDg5ZWU3MjgzOGJjMzEzNzAyZjRkNTExMmFiMGI3OTNiYWEzMTJlNzRhN2FiNzM0ODNmNjVhMmRlYmQ=",
    "useSass": true,
    "clientPackageManager": "npm",
    "clientFramework": "angularX",
    "clientTheme": "none",
    "clientThemeVariant": "",
    "testFrameworks": [],
    "jhiPrefix": "jhi",
    "entitySuffix": "",
    "dtoSuffix": "DTO",
    "otherModules": [],
    "enableTranslation": true,
    "nativeLanguage": "en",
    "languages": [
      "en"
    ]
  }

Entity configuration(s) entityName.json files generated in the .jhipster directory

{
    "fluentMethods": true,
    "clientRootFolder": "",
    "relationships": [],
    "fields": [
        {
            "fieldName": "verbose",
            "fieldType": "String"
        }
    ],
    "changelogDate": "20190627140754",
    "dto": "no",
    "searchEngine": "elasticsearch",
    "service": "no",
    "entityTableName": "sentencce",
    "databaseType": "sql",
    "jpaMetamodelFiltering": false,
    "pagination": "infinite-scroll"
}

Browsers and Operating System

MacOS Mojave 10.14.5

• [x] Checking this box is mandatory (this is just to show you read everything)

[pascalgrimaud]

For now, we use Elasticsearch 6.4.3
See https://github.com/jhipster/generator-jhipster/blob/master/generators/generator-constants.js#L33

We should migrate to Elasticsearch 7+
So thanks for opening this ticket with details. Don't hesitate to PR if you find the fix

[pascalgrimaud]

Maybe related to https://github.com/VanRoy/spring-data-jest/issues/116

[jdubois]

I'm adding a bug bounty here, this is important and should be upgraded

[DanielFran]

This is not yet even added in SDE, see https://github.com/spring-projects/spring-data-elasticsearch/pull/284, and will be added only to v4, so it will take some time, problably available only at the end of the year.
Spring-data Moore has been delayed to september because Spring Framework 5.2 has moved its GA date to September 4 and is supporting only 6.7 (https://github.com/spring-projects/spring-data-elasticsearch/blob/master/pom.xml#L22).

[DanielFran]

Next SDE Moore 3.2.RC2 will now support ES 6.8.1! See https://github.com/spring-projects/spring-data-elasticsearch/commit/a9d57936c32b6e1ab811ad768c6dec5c2dc85e18

[DanielFran]

@mvasil15 Good news, next SDE version will support elastic stack v.6.8.1 that also include free security support.
See https://github.com/spring-projects/spring-data-elasticsearch/pull/284#issuecomment-509563309

[rhoiyds]

I am currently experiencing this issue whilst following the JHipster 6 introduction video .
If I generate the JHipster project (selecting Elasticsearch for searching), in the final step of the tutorial when you deploy to Heroku - You receive the error message from the original issue post.
As stated, this seems to be because of a compatibility issue with indexing on Elasticsearch 7.1.1+.

During deployment, Heroku will automatically configure a Bonsai add-on running the latest version of Elasticsearch (7.2.0) - The free tier of Bonsai only supports running the latest version of elasticsearch - You can downgrade to Elasticsearch 6.5.4 (which should fix the error, as suggested in the comments above) but only at a cost ($10USD/month), which isn't great for tinkerers starting out on their first project.

Edit: Also attempted to use SearchBox as an alternative to Bonsai which allows version selection on it's free tier - However, encountering a different problem.

Are there any updates as to when/how this problem can be fixed?

[DanielFran]

@rhoiyds Actually Spring-boot 2.2 is only managing ES till version 6.8.4 (Spring-data Moore), so there is no way using spring-boot for now to use ES 7+:

• https://docs.spring.io/spring-data/elasticsearch/docs/3.2.3.RELEASE/reference/html/#preface.versions
• https://github.com/spring-projects/spring-data-elasticsearch/blob/3.2.x/pom.xml#L22

Only with Spring-data Neumann version we will have support for ES 7:

• https://github.com/spring-projects/spring-data-elasticsearch/blob/master/pom.xml#L22

[martwykotek]

That sould be also taken in account when using jhipster aws generator

[gordolio]

For those coming here looking for how this is done, this is the command...
heroku addons:create bonsai:staging -a heroku-app-name --version=6.5.4

[pascalgrimaud]

@murdos : is it done in Spring Boot 2.3 branch ? In this case, we can close this.

[murdos]

It will be done with Spring Boot 2.3. My intent was to close it once Spring Boot 2.3 is merged in master (I think it will be done automatically since it's mentioned in a commit)

[andrejpodgorsek]

Hi, any news yet?

[devrajgiri]

i used RestHighLevelClient for now , here is the configuration

` @Configuration
@EnableConfigurationProperties(ElasticsearchProperties.class)
public class ElasticsearchConfiguration {

private ObjectMapper mapper;

public ElasticsearchConfiguration(ObjectMapper mapper) {
    this.mapper = mapper;
}

@Bean
public EntityMapper getEntityMapper() {
    return new CustomEntityMapper(mapper);
}

@Bean
@Primary
public ElasticsearchOperations elasticsearchTemplate(RestHighLevelClient jestClient,
                                                     ElasticsearchConverter elasticsearchConverter,
                                                     EntityMapper entityMapper) {
    return new ElasticsearchRestTemplate(
        jestClient,
        elasticsearchConverter,
        entityMapper);
}

public class CustomEntityMapper implements EntityMapper {

    private ObjectMapper objectMapper;

    public CustomEntityMapper(ObjectMapper objectMapper) {
        this.objectMapper = objectMapper;
        objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
        objectMapper.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, true);
        objectMapper.configure(SerializationFeature.WRITE_DATE_TIMESTAMPS_AS_NANOSECONDS, true);
        objectMapper.configure(SerializationFeature.INDENT_OUTPUT, false);          objectMapper.configure(DeserializationFeature.READ_DATE_TIMESTAMPS_AS_NANOSECONDS, true);
    }
    @Override
    public String mapToString(Object object) throws IOException {
        return objectMapper.writeValueAsString(object);
    }
    @Override
    public <T> T mapToObject(String source, Class<T> clazz) throws IOException {
        return objectMapper.readValue(source, clazz);
    }
    @Override
    public Map<String, Object> mapObject(Object source) {
        try {
            return objectMapper.readValue(mapToString(source), HashMap.class);
        } catch (IOException e) {
            throw new MappingException(e.getMessage(), e);
        }
    }
    @Override
    public <T> T readObject(Map<String, Object> source, Class<T> targetType) {
        try {
            return mapToObject(mapToString(source), targetType);
        } catch (IOException e) {
            throw new MappingException(e.getMessage(), e);
        }
    }
}

}`