Elasticsearch-js: bulk insert not working - newline separated JSON body not working

[ORESoftware]

🐛 Bug Report

Cannot insert newline separate JSON

To Reproduce

  const v = lst.map(v => JSON.stringify({doc: v.value})).join('\n') + '\n';

  client.bulk({
    index: 'foo',
    body: v
  })
   .catch(e => {
     log.error(e);
   });

I get this error:

      ResponseError: illegal_argument_exception
      at IncomingMessage.<anonymous> (/Users/alex/codes/interos/@interos/elastic-search/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25)
      at IncomingMessage.emit (events.js:208:15)
      at endReadableNT (_stream_readable.js:1154:12)
      at processTicksAndRejections (internal/process/task_queues.js:77:11)

Why doesn't the stack trace give me more to work with btw? ResponseError: illegal_argument_exception with no further information does not give me much to work with :(

Your Environment

• node version: 12
• "@elastic/elasticsearch": "^7.3.0",
• os: Mac

This however does work fine:

  client.index({
      index: 'foo',
      body: {
        doc: v.value
      }
    })
     .catch(e => {
       log.error(e);
     });

[delvedor]

Hello!
If you inspect the error object you should be able to see many other fields. Unfortunately, they are not well documented in the docs, I'll take care of this.
In the meantime, you can find the error classes here:
https://github.com/elastic/elasticsearch-js/blob/f7be49f2bafa3b9ddc90b0caac5b70932e6db35b/lib/errors.js#L71-L94

Regarding the bulk body, I think you are not creating the body correctly. I would suggest you take a look at this example :)

[ORESoftware]

@delvedor it probably needs a fix on your end, I am on Node.js 12 and as you can see:

  client.index({
      index: 'foo',
      body: {
        doc: v.value
      }
    })
     .catch(e => {
       log.error(e);  // I am logging the whole error object
     });

and it doesn't log the meta properties etc

[delvedor]

What is log in your code?

Take this example to reproduce the error:

'use strict'

const { Client } = require('@elastic/elasticsearch')
const client = new Client({ node: 'http://localhost:9200' })

client.indices.create({
  index: 'test',
  body: {
    mappings: {
      properties: {
        foo: { type: 'integer' }
      }
    }
  }
}, { ignore: [400] }, (err, result) => {
  if (err) return console.log(err)
  client.index({
    index: 'test',
    body: { foo: 'bar' }
  }, (err, result) => {
    if (err) return console.log(err)
  })
})

The code above produces the following log on Node.js v12:

ResponseError: mapper_parsing_exception
    at IncomingMessage.<anonymous> (.../elasticsearch-js/lib/Transport.js:287:25)
    at IncomingMessage.emit (events.js:201:15)
    at endReadableNT (_stream_readable.js:1130:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:9) {
  name: 'ResponseError',
  meta: {
    body: { error: [Object], status: 400 },
    statusCode: 400,
    headers: {
      'content-type': 'application/json; charset=UTF-8',
      'content-length': '459'
    },
    warnings: null,
    meta: {
      context: null,
      request: [Object],
      name: 'elasticsearch-js',
      connection: [Object],
      attempts: 0,
      aborted: false
    }
  }
}

As you can see, the full error object is logged and the properties are there.
I think you are using a custom logger that does not know how to serialize the Error objects, and by default, it logs only the error message and stacktrace.

[ORESoftware]

@delvedor you are correct - my logger was deficient - it was only logging the stack/message of an error. But that brings up an important point, some people may only be logging the stack:

console.error(err && err.stack)`

a lot of older code does this, since Node.js improved their native logging in newer versions. So maybe add the meta info to the stack or something?

err.stack = err.stack + '\n' + String(meta)

just a suggestion. Or wrap it, like so:

const error = {
   error:  {stack, message},   // the original error etc
   meta: 'whatever'
}

wrapping the error makes more sense.

[delvedor]

Thank you for the suggestion, but I fear that adding the metadata to the stacktrace would add a lot of confusion for two reasons:

1. in the stacktrace I would expect to find just the stacktrace, and not HTTP related details.
2. The meta-object can be quite big, and log it in a single place as a string would make it quite hard to read.

What I think should be done, is to document better what an error object contains and how to access it easily. In this regard, I've opened https://github.com/elastic/elasticsearch-js/issues/964. Would you like to work on it?

[stale[bot]]

We understand that this might be important for you, but this issue has been automatically marked as stale because it has not had recent activity either from our end or yours.
It will be closed if no further activity occurs, please write a comment if you would like to keep this going.

Note: in the past months we have built a new client, that has just landed in master. If you want to open an issue or a pr for the legacy client, you should do that in https://github.com/elastic/elasticsearch-js-legacy