Elasticsearch: java.util.UUID objects not recognized by XContentBuilder in TermQuery

[raymundane]

Elasticsearch version (bin/elasticsearch --version): 6.3.0

Plugins installed: []

JVM version (java -version): 1.8.0_172

OS version (uname -a if on a Unix-like system): macOS 10.13.5 High Sierra

Description of the problem including expected versus actual behavior:
When doing a TermQuery and passing a UUID as an object, XContentBuilder throws an IllegalArgumentException stating that it cannot write xcontent for unknown value of type class java.util.UUID. This does not happen in 6.2.4 and I do not see any mention of the change in 6.3.0 release notes, so I thought it likely to be a bug. It would be very helpful to keep this feature for 6.3.1! Thanks for your time.

Steps to reproduce:

1. Start up an elasticsearch instance with elasticsearch -Ecluster.name=fakebrain
2. In IntelliJ Idea or similar IDE, create a Spring Boot project (e.g. with Spring Initializr) with package group com.philosobyte
3. Copy in the gradle file provided below and the Spring Web controller provided below.
4. Run the Spring Boot program, then go to a browser and invoke the REST endpoints provided below.

gradle file:

buildscript {
    ext {
        springBootVersion = '2.0.3.RELEASE'
    }
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}")
    }
}

apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'org.springframework.boot'
apply plugin: 'io.spring.dependency-management'

group = 'com.philosobyte'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = 1.8

repositories {
    mavenCentral()
}


dependencies {
    compile('org.apache.logging.log4j:log4j-core')
    compile('org.springframework.boot:spring-boot-starter-actuator')
    compile('org.springframework.boot:spring-boot-starter-aop')
    compile("org.springframework.boot:spring-boot-starter-web")
    compile("org.elasticsearch:elasticsearch:6.3.0")
    compile("org.elasticsearch.client:elasticsearch-rest-high-level-client:6.3.0")
    compile("org.elasticsearch.client:elasticsearch-rest-client:6.3.0")
    compile("com.fasterxml.jackson.core:jackson-databind")
    compile("com.fasterxml.jackson.core:jackson-annotations")
    compileOnly('org.projectlombok:lombok')
    testCompile('org.springframework.boot:spring-boot-starter-test')
}

Controller class (you must provide the package name you used):

import java.io.IOException;
import java.util.Collections;
import java.util.Map;
import java.util.UUID;

import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.*;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.entity.ContentType;
import org.apache.http.message.BasicHeader;
import org.apache.http.nio.entity.NStringEntity;
import org.elasticsearch.action.admin.indices.create.CreateIndexRequest;
import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.springframework.web.bind.annotation.*;

@org.springframework.stereotype.Controller
@Slf4j
public class Controller {

    public static final String INDEX = "foobar";
    public static final String TYPE = "_doc";
    private ObjectMapper objectMapper = new ObjectMapper();
    private RestHighLevelClient highLevelClient = new RestHighLevelClient(
        RestClient.builder(
            new HttpHost("localhost", 9200, "http")
        ).setDefaultHeaders(
            new Header[]{
                new BasicHeader("cluster.name", "fakebrain")
            }
        )
    );
    private Document document;

    @GetMapping("/create_index")
    @ResponseBody
    public String create_index() throws IOException {
        highLevelClient.indices().create(new CreateIndexRequest(INDEX));
        RestClient lowLevelClient = highLevelClient.getLowLevelClient();
        Map<String, String> params = Collections.emptyMap();
        String mapping = "{" +
                "\"properties\": {" +
                    "\"id\": {" +
                        "\"type\": \"keyword\"" +
                    "}" +
                "}" +
            "}";
        HttpEntity entity = new NStringEntity(mapping, ContentType.APPLICATION_JSON);
        return lowLevelClient.performRequest("PUT", String.format("/%s/_mapping/%s", INDEX, TYPE), params, entity).toString();
    }

    @GetMapping("/index_item")
    @ResponseBody
    public String index_item() throws IOException {
        // Create our document
        document = new Document();
        document.setId(UUID.randomUUID());

        // Index our document
        return highLevelClient.index(new IndexRequest(INDEX, TYPE, document.getId().toString()).source(objectMapper.writeValueAsString(document), XContentType.JSON)).toString();
    }

    @GetMapping("/search")
    @ResponseBody
    public String search() throws IOException {
        BoolQueryBuilder query = QueryBuilders.boolQuery().filter(QueryBuilders.termQuery("id", document.getId()));

        SearchRequest searchRequest = new SearchRequest(INDEX).types(TYPE);

        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder()
            .query(query);

        searchRequest.source(searchSourceBuilder);

        return highLevelClient.search(searchRequest).getHits().getHits()[0].getId();
    }

    @GetMapping("/delete_index")
    @ResponseBody
    public boolean deleteIndex() throws IOException {
        return highLevelClient.indices().delete(new DeleteIndexRequest(INDEX)).isAcknowledged();
    }

    @Data
    class Document {
        private UUID id;
    }
}

REST endpoints:

http://localhost:8080/create_index
http://localhost:8080/index_item
http://localhost:8080/search

Provide logs (if relevant):
This is the error thrown for me:

java.lang.IllegalArgumentException: cannot write xcontent for unknown value of type class java.util.UUID
    at org.elasticsearch.common.xcontent.XContentBuilder.unknownValue(XContentBuilder.java:755) ~[elasticsearch-x-content-6.3.0.jar:6.3.0]
    at org.elasticsearch.common.xcontent.XContentBuilder.value(XContentBuilder.java:726) ~[elasticsearch-x-content-6.3.0.jar:6.3.0]
    at org.elasticsearch.common.xcontent.XContentBuilder.field(XContentBuilder.java:711) ~[elasticsearch-x-content-6.3.0.jar:6.3.0]
    at org.elasticsearch.index.query.BaseTermQueryBuilder.doXContent(BaseTermQueryBuilder.java:154) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.index.query.AbstractQueryBuilder.toXContent(AbstractQueryBuilder.java:82) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.index.query.BoolQueryBuilder.doXArrayContent(BoolQueryBuilder.java:276) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.index.query.BoolQueryBuilder.doXContent(BoolQueryBuilder.java:258) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.index.query.AbstractQueryBuilder.toXContent(AbstractQueryBuilder.java:82) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.common.xcontent.XContentBuilder.value(XContentBuilder.java:779) ~[elasticsearch-x-content-6.3.0.jar:6.3.0]
    at org.elasticsearch.common.xcontent.XContentBuilder.value(XContentBuilder.java:772) ~[elasticsearch-x-content-6.3.0.jar:6.3.0]
    at org.elasticsearch.common.xcontent.XContentBuilder.field(XContentBuilder.java:764) ~[elasticsearch-x-content-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.builder.SearchSourceBuilder.toXContent(SearchSourceBuilder.java:1156) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.common.xcontent.XContentHelper.toXContent(XContentHelper.java:349) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.common.xcontent.XContentHelper.toXContent(XContentHelper.java:335) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.client.Request.createEntity(Request.java:632) ~[elasticsearch-rest-high-level-client-6.3.0.jar:6.3.0]
    at org.elasticsearch.client.Request.search(Request.java:507) ~[elasticsearch-rest-high-level-client-6.3.0.jar:6.3.0]
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:519) ~[elasticsearch-rest-high-level-client-6.3.0.jar:6.3.0]
    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.0.jar:6.3.0]
    at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:404) ~[elasticsearch-rest-high-level-client-6.3.0.jar:6.3.0]
    at com.philosobyte.elastic630test.Controller.search(Controller.java:87) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_172]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_172]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_172]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_172]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877) ~[spring-webmvc-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783) ~[spring-webmvc-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) ~[spring-webmvc-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866) ~[spring-webmvc-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) ~[spring-webmvc-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.5.31.jar:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:158) ~[spring-boot-actuator-2.0.3.RELEASE.jar:2.0.3.RELEASE]
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:126) ~[spring-boot-actuator-2.0.3.RELEASE.jar:2.0.3.RELEASE]
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:111) ~[spring-boot-actuator-2.0.3.RELEASE.jar:2.0.3.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90) ~[spring-boot-actuator-2.0.3.RELEASE.jar:2.0.3.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) ~[spring-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) ~[spring-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468) [tomcat-embed-core-8.5.31.jar:8.5.31]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.31.jar:8.5.31]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_172]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_172]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.31.jar:8.5.31]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_172]
    Suppressed: java.lang.IllegalStateException: Failed to close the XContentBuilder
        at org.elasticsearch.common.xcontent.XContentBuilder.close(XContentBuilder.java:924) ~[elasticsearch-x-content-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.xcontent.XContentHelper.$closeResource(XContentHelper.java:113) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.xcontent.XContentHelper.toXContent(XContentHelper.java:354) ~[elasticsearch-6.3.0.jar:6.3.0]
        ... 71 common frames omitted
    Caused by: java.io.IOException: Unclosed object or array found
        at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:444)
        at org.elasticsearch.common.xcontent.XContentBuilder.close(XContentBuilder.java:922)
        ... 73 common frames omitted

[elasticmachine]

Pinging @elastic/es-core-infra

[nik9000]

I expect we broke this because we recently tightened up what xcontent would write without complaining and didn't realize that we were relying on xcontent to serialize this one. We'll take a look.

I expect you can work around the issue for now by using the toString when building the query.

[dakrone]

@nik9000 yes that's correct, we did tighten this us as part of the XContent work.

However, this isn't something I think we should relax. The reason that we tightened it is that previously we called Objects.toString(..) on an object we didn't know how to encode, however, this can lead to encoding JSON with strings like "[B@724af044" because the object didn't have a good toString() method. Rather than allow this we purposefully throw this exception. Calling toString() (or whatever method) on UUID when building the query is the expected solution for this.

[raymundane]

I see. Thanks for clearing that up, @nik9000 and @dakrone!

[nik9000]

@dakrone, yeah, I don't think we should loosen the method. I wonder if we should do anything in this case and/or if we should push some more docs about this case.

[gmarshall56]

I have the same issue with a Java Date field:
java.lang.IllegalArgumentException: cannot write xcontent for unknown value of type class java.sql.Timestamp

[dakrone]

@gmarshall56 you'll need to strigify the Timestamp before passing it to the builder method, as I mentioned in https://github.com/elastic/elasticsearch/issues/31377#issuecomment-398102292

[ZLinFeng]

java.lang.IllegalArgumentException: cannot write xcontent for unknown value of type class java.sql.Timestamp

[yargna]

The reasoning for moving away from Objects.toString() is clear. However, since UUID is a widely used class and a part of java.util, could you please reconsider adding support for this particular class? It's as easy as adding

writers.put(UUID.class, (b, v) -> b.value(v.toString()));

to XContentBuilder.

Alternatively, if the above request is not suitable, please consider providing an easy way to add custom implementations for XContentBuilder.Writer. Currently I can implement XContentBuilderExtension, but I couldn't find an easy way to register it. Building a separate JAR just for this purpose seems to be an overkill.

Thank you!