Elasticsearch: Template not getting applied with ES version 1.4.1 - Works for 1.4.0
Hi,
I have the following template
{
"_source": {
"enabled": false
},
"_all": {
"enabled": false
},
"_timestamp": {
"enabled": true,
"store": true
},
"dynamic_templates": [
{
"template_timestamp": {
"match": "timestamp",
"mapping": {
"store": false,
"index": "not_analyzed",
"type": "date"
}
}
},
{
"template_no_store_analyzed": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"store": false,
"index": "not_analyzed",
"fields": {
"analyzed": {
"store": false,
"type": "string",
"index": "analyzed"
}
}
}
}
},
{
"template_no_store": {
"match_mapping_type": "date|boolean|double|long|integer",
"match_pattern": "regex",
"path_match": ".*",
"mapping": {
"store": false,
"index": "not_analyzed"
}
}
}
]
}
This template is applicable for all indexes and type - "document".
I am running an embedded instance in my java process for testing purposes.
When I am trying to index a document in the following method, above template is not getting applied -
connection.getClient()
.prepareIndex()
.setIndex("abc")
.setType("document")
.setId(document.getId())
.setTimestamp(Long.toString(timestamp))
.setSource(mapper.writeValueAsBytes(document.getData()))
.setConsistencyLevel(WriteConsistencyLevel.QUORUM)
.execute()
.get(2, TimeUnit.SECONDS);
However, when I am using BulkRequestBuilder and indexing same document, mappings are getting applied correctly.
The same mapping works fine for ES 1.1.1, 1.2.4, 1.3.6 and 1.4.0.
Am I missing something here ? Has any setting changed from 1.4.0 to 1.4.1 ?
r0goyal
All 14 comments
Hi @r0goyal
You've given us incomplete information, eg how you set the template up, what the document contains, what you mean by "the mapping is not applied" (ie all of it? or just some field mappings?) etc.
A simple recreation of the problem would be appreciated.
clintongormley
on 5 Apr 2015
Hi @clintongormley
Following is the code snippet
public static void main(String[] args) throws IOException {
ImmutableSettings.Builder elasticsearchSettings = ImmutableSettings.settingsBuilder()
.put("http.enabled", "false")
.put("path.data", "target/" + UUID.randomUUID().toString());
Node esNode = nodeBuilder()
.local(true)
.settings(elasticsearchSettings.build())
.node();
Client client = esNode.client();
PutIndexTemplateRequest templateRequest = getClusterTemplateMapping(client.admin().indices());
client.admin().indices().putTemplate(templateRequest).actionGet();
String index = "foxtrot-abcd";
String type = "document";
Map<String, String> data = Collections.singletonMap("abcd", "abcd");
client.prepareIndex()
.setIndex(index)
.setType(type)
.setId(UUID.randomUUID().toString())
.setTimestamp(Long.toString(System.currentTimeMillis()))
.setConsistencyLevel(WriteConsistencyLevel.QUORUM)
.setSource(new ObjectMapper().writeValueAsString(data))
.execute()
.actionGet();
GetMappingsResponse mappingsResponse = client.admin().indices()
.prepareGetMappings(index)
.execute().actionGet();
MappingMetaData metaData = mappingsResponse.getMappings().get(index).get(type);
}
public static PutIndexTemplateRequest getClusterTemplateMapping(IndicesAdminClient indicesAdminClient) {
PutIndexTemplateRequestBuilder builder = new PutIndexTemplateRequestBuilder(indicesAdminClient, "generic_template");
builder.setTemplate("foxtrot-*");
builder.addMapping("document", "{\n" +
" \"_source\" : { \"enabled\" : false },\n" +
" \"_all\" : { \"enabled\" : false },\n" +
" \"_timestamp\" : { \"enabled\" : true, \"store\" : true },\n" +
"\n" +
" \"dynamic_templates\" : [\n" +
" {\n" +
" \"template_timestamp\" : {\n" +
" \"match\" : \"timestamp\",\n" +
" \"mapping\" : {\n" +
" \"store\" : false,\n" +
" \"index\" : \"not_analyzed\",\n" +
" \"type\" : \"date\"\n" +
" }\n" +
" }\n" +
" },\n" +
" {\n" +
" \"template_no_store_analyzed\" : {\n" +
" \"match\" : \"*\",\n" +
" \"match_mapping_type\" : \"string\",\n" +
" \"mapping\" : {\n" +
" \"store\" : false,\n" +
" \"index\" : \"not_analyzed\",\n" +
" \"fields\" : {\n" +
" \"analyzed\": {\n" +
" \"store\" : false,\n" +
" \"type\": \"string\",\n" +
" \"index\": \"analyzed\"\n" +
" }\n" +
" }\n" +
" }\n" +
" }\n" +
" },\n" +
" {\n" +
" \"template_no_store\" : {\n" +
" \"match_mapping_type\": \"date|boolean|double|long|integer\",\n" +
" \"match_pattern\": \"regex\",\n" +
" \"path_match\": \".*\",\n" +
" \"mapping\" : {\n" +
" \"store\" : false,\n" +
" \"index\" : \"not_analyzed\"\n" +
" }\n" +
" }\n" +
" }\n" +
" ]\n" +
" }");
return builder.request();
}
With ES 1.1.1 to 1.4.0 in classpath, serializing metaData gives
{
"dynamic_templates": [
{
"template_timestamp": {
"mapping": {
"index": "not_analyzed",
"store": false,
"type": "date"
},
"match": "timestamp"
}
},
{
"template_no_store_analyzed": {
"mapping": {
"index": "not_analyzed",
"store": false,
"fields": {
"analyzed": {
"index": "analyzed",
"store": false,
"type": "string"
}
}
},
"match": "*",
"match_mapping_type": "string"
}
},
{
"template_no_store": {
"mapping": {
"index": "not_analyzed",
"store": false
},
"match_mapping_type": "date|boolean|double|long|integer",
"match_pattern": "regex",
"path_match": ".*"
}
}
],
"_all": {
"enabled": false
},
"_timestamp": {
"enabled": true,
"store": true
},
"_source": {
"enabled": false
},
"properties": {
"abcd": {
"type": "string",
"index": "not_analyzed",
"fields": {
"analyzed": {
"type": "string"
}
}
}
}
}
However with ES 1.4.1 and above, only following mapping is present -
{"properties":{"abcd":{"type":"string"}}}
Please let me know if you need anything else as well.
r0goyal
on 5 Apr 2015
@clintongormley Any update on this ?
r0goyal
on 14 Apr 2015
Can anybody who can read Java give some feedback here?
clintongormley
on 14 Apr 2015
I had a look and can confirm this behaviour. You should wrap the whole mapping object, including dynamic_templates etc. into a document object in the json, which is what your type is called. Not sure why this used to work with 1.4.0 to be honest (I confirm it does), do you know @clintongormley ?
javanna
on 20 Apr 2015
@javanna I am already wrapping this whole mapping in "document" object in the Json. Look at the getClusterTemplateMapping() method.
r0goyal
on 21 Apr 2015
Hi @r0goyal I ran your code, I see you are passing in the type as an argument, but the mapping itself doesn't contain the document type: builder.addMapping("document", "{\n" + \"_source\" : { \"enabled\" : false },...." should be builder.addMapping("document", "{ \"document\" : { \n" + \"_source\" : { \"enabled\" : false },....}", that is what I meant. If you do that you should work around the problem you are seeing.
javanna
on 22 Apr 2015
@javanna that seems very odd. Is addMapping the equivalent of the update mapping REST API?
In the REST layer, we support passing the mapping with the type name and without. In fact, I thought we preferred the version without the type name, but I see the docs still show the example with type name: http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html
No idea what changed in 1.4
clintongormley
on 25 Apr 2015
agreed @clintongormley it is odd, I have no idea either what changed in 1.4 to be honest, only tested and acknowledged the problem.
javanna
on 28 Apr 2015
@javanna Yes it started working with the fix suggested by you. Thanks a lot.
r0goyal
on 28 Apr 2015
In summary, an addMapping request in the Java API requires the mapping to be wrapped in a JSON object with the type name as a key, while the REST API allows you to specify the mapping directly without this extra layer.
Should we change the Java API to work in the same way as the REST layer?
clintongormley
on 15 Jan 2016
Another person just bitten by this in #17886
We should make the Java API consistent with the REST api. Should be easy to do as using a type name at the top level will throw an exception (bad mapping)
clintongormley
on 20 Apr 2016
I got bitten today on 5.3.1
billnbell
on 30 Apr 2017
I can't reproduce this problem any more.
Java code
public class Application {
public static void main(String[] args) throws Exception {
Settings settings = Settings.builder()
.put("cluster.name", "distribution_run").build();
try (TransportClient client = new PreBuiltTransportClient(settings)
.addTransportAddress(new TransportAddress(InetAddress.getByName("localhost"), 9300))) {
PutIndexTemplateRequest templateRequest = getClusterTemplateMapping(client.admin().indices());
client.admin().indices().putTemplate(templateRequest).actionGet();
String index = "foxtrot-abcd";
String type = "document";
Map<String, String> data = Collections.singletonMap("abcd", "abcd");
client.prepareIndex()
.setIndex(index)
.setType(type)
.setId(UUID.randomUUID().toString())
.setSource(data)
.execute()
.actionGet();
GetMappingsResponse mappingsResponse = client.admin().indices()
.prepareGetMappings(index)
.execute().actionGet();
MappingMetaData metaData = mappingsResponse.getMappings().get(index).get(type);
}
}
private static PutIndexTemplateRequest getClusterTemplateMapping(IndicesAdminClient indicesAdminClient) {
PutIndexTemplateRequestBuilder builder = indicesAdminClient.preparePutTemplate("generic_template");
builder.setPatterns(Collections.singletonList("foxtrot-*"));
builder.addMapping("document", "{\n" +
" \"_source\" : { \"enabled\" : false },\n" +
"\n" +
" \"dynamic_templates\" : [\n" +
" {\n" +
" \"template_timestamp\" : {\n" +
" \"match\" : \"timestamp\",\n" +
" \"mapping\" : {\n" +
" \"store\" : false,\n" +
" \"index\" : \"true\",\n" +
" \"type\" : \"date\"\n" +
" }\n" +
" }\n" +
" },\n" +
" {\n" +
" \"template_no_store_analyzed\" : {\n" +
" \"match\" : \"*\",\n" +
" \"match_mapping_type\" : \"string\",\n" +
" \"mapping\" : {\n" +
" \"type\" : \"text\"\n" +
" }\n" +
" }\n" +
" },\n" +
" {\n" +
" \"template_no_store\" : {\n" +
" \"match_mapping_type\": \"string\",\n" +
" \"match_pattern\": \"regex\",\n" +
" \"path_match\": \".*\",\n" +
" \"mapping\" : {\n" +
" \"store\" : false,\n" +
" \"index\" : \"true\"\n" +
" }\n" +
" }\n" +
" }\n" +
" ]\n" +
" }", XContentType.JSON);
return builder.request();
}
}
and this resulted in the expected mapping:
{
"foxtrot-abcd" : {
"mappings" : {
"document" : {
"_source" : {
"enabled" : false
},
"dynamic_templates" : [
{
"template_timestamp" : {
"match" : "timestamp",
"mapping" : {
"index" : "true",
"store" : false,
"type" : "date"
}
}
},
{
"template_no_store_analyzed" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text"
}
}
},
{
"template_no_store" : {
"path_match" : ".*",
"match_mapping_type" : "string",
"match_pattern" : "regex",
"mapping" : {
"index" : "true",
"store" : false
}
}
}
],
"properties" : {
"abcd" : {
"type" : "text"
}
}
}
}
}
}
martijnvg
on 13 Mar 2018
Related issues
ppf2
路
3Comments
rjernst
路
3Comments
jpountz
路
3Comments
makeyang
路
3Comments
rpalsaxena
路
3Comments
Most helpful comment
Another person just bitten by this in #17886
We should make the Java API consistent with the REST api. Should be easy to do as using a type name at the top level will throw an exception (bad mapping)