Elasticsearch: v5.3.1 Entire Bulk request is rejected if containing a single invalid index request

[oririechman]

Elasticsearch version: 5.3.1

When using Bulk API, if one of the index-requests is missing ‘type’, then ES reject all the bulk, although other index-requests in the bulk are valid.
I would expect that the invalid index-request will be rejected but the rest of the valid index-requests will be executed successfully.

Steps to reproduce:

curl -XPOST 'localhost:9200/_bulk?pretty' -H 'Content-Type: application/json' -d'
{ "index" : { "_index" : "test", "_type":"type1" } }
{ "field1" : "value1" }
{ "index" : { "_index" : "test" } }
{ "field2" : "value2" }
{ "index" : { "_index" : "test", "_type":"type1" } }
{ "field3" : "value3" }'

The response

{
  "error" : {
    "root_cause" : [
      {
        "type" : "action_request_validation_exception",
        "reason" : "Validation Failed: 1: type is missing;"
      }
    ],
    "type" : "action_request_validation_exception",
    "reason" : "Validation Failed: 1: type is missing;"
  },
  "status" : 400
}

[jasontedor]

Duplicates #24028, closed by #24048

[jasontedor]

This will be fixed in 5.3.2.

[jasontedor]

Oh, sadly, this is not addressed by #24048, this looks like an always-existing issue (at least since 1.x and I don't think it's worth checking before that). Your expectation that we should not fail the entire request is correct though.

Reopening this, it will not be addressed in 5.3.2, but we will work on a fix.

[jasontedor]

I've asked @nik9000 to take a look at this one.

[nik9000]

I want to make sure that we should make this change because there are two ways of doing things competing here:

1. For the past few years Elasticsearch has tried to be more strict about things. Failing the entire request if you add a URL parameter that isn't recognized is my favorite example. The current behavior is more in line with this way of doing things.
2. The _bulk API by its nature must allow partial failure because we don't have distributed transactions. It takes pride in being one of our only API that'll do what it can and report on what it can't. The suggested behavior change is more in line with _bulk's individual spirit.

This is one of the few places where _bulk can be strict, so maybe it ought to be?

[jasontedor]

I think that the bulk API (and other APIs like mget, msearch, etc.) are best thought of as consisting of multiple requests that are batched together which saves on transport costs, and allow for coalescing expensive side-effects like refreshes and fsyncs. If these requests were sent individually, they would succeed and fail individually, and I think that's the semantics that these batch-request oriented APIs should maintain too.

[nik9000]

So I can do this pretty easily for the REST layer but it gets a lot harder if I want it to work in the transport layer. We rely on the sub-requests being valid in transport protocol. I don't like to diverge the REST and transport behavior very much but _bulk is already pretty different. I'll put this down for a bit and think about it some more.

[qwerty4030]

Should we still expect the entire request to fail if it is malformed, e.g. invalid JSON? It would be tricky to determine what is "valid" and what is not. Here's another tricky example:

curl -XPOST 'localhost:9200/_bulk?pretty' -H 'Content-Type: application/json' -d'
{ "index" : { "_index" : "test", "_type":"type1" } }
{ "index" : { "_index" : "test" } }
{ "field2" : "value2" }
{ "index" : { "_index" : "test", "_type":"type1" } }
{ "field3" : "value3" }'

What is the expected behavior here?

[nik9000]

Should we still expect the entire request to fail if it is malformed, e.g. invalid JSON?

I suspect the entire request would fail in your case. I imagine it is possible to do the rest of the request in cases where the framing is correct but I imagine at this point the entire request fails.

[olcbean]

@nik9000 did you have a chance to agree on future _bulk behavior (https://github.com/elastic/elasticsearch/issues/24319#issuecomment-297138123) ?

I am looking at #9379 and it is not really clear if the entire _bulk request should fail if an invalid-yet-parseable JSON is sent or just the invalid sub-request.

[aishwarya-nandapurkar]

Facing a similar issue with msearch end point.
Someone has mentioned above in the trail that this error is because of some malformed json request but for me I am able to see the same request being processed on Kibana with the same payload and when I try it from Postman it fails with the below error :

"root_cause": [
{
"type": "action_request_validation_exception",
"reason": "Validation Failed: 1: no requests added;"
}
],

[dadoonet]

That's not the same problem. I think you are missing the \n character at the end. If you can't fix it, please open a question on discuss.elastic.co where we can help.

[aimeos]

We prefer the "all or nothing" approach. Otherwise, it's not possible to emulate a transaction

[aishwarya-nandapurkar]

Got it now! Thanks for the response.
Also are there any Rest API’s for accessing data through Kibana if we dont
want to direct our queries to Elasticsearch?

On Tue, Jul 31, 2018 at 10:02 AM Aimeos [e-mail] wrote:

We prefer the "all or nothing" approach. Otherwise, it's not possible to
emulate a transaction

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/elastic/elasticsearch/issues/24319#issuecomment-409294155,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Akj1M2MiUKGxNhtsnk6Jb-aXnf3wW9oQks5uMI2xgaJpZM4NH28f
.

[nik9000]

It looks like I let this one fall through the cracks. I haven't looked at this issue in over a year so I'm unassigning myself. I'm still happy to pick it up if we think it is important but I wonder if we really do given that I let this sit for a year and no one poked me about it.

[elasticmachine]

Pinging @elastic/es-core-infra

[fdartayre]

It would be great to see this one fixed. It took me quite some time to realize why an entire bulk was failing (one id was longer than 512 bytes)

Elasticsearch version: 7.2.0

cc @nik9000 @elastic/es-core-infra

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)