Elasticsearch-net: Introduce Attachment type for attachment mapping

Created on 10 May 2016 · 16Comments · Source: elastic/elasticsearch-net

The mapper-attachment plugin allows indexing and searching of documents in various different formats such as PDF, Word, etc.

This is exposed in NEST with AttachmentProperty, which can be used to map the properties of a CLR object to the attachment field mapping.

In its simplest form, indexing an attachment need pass a base64 encoded string of the document as the attachment field:

PUT http://localhost:9200/docs/document/1?pretty=true&refresh=true 
{
  "id": 1,
  "title": "Some document",
  "file": "some base64 encoded string"
}

If the attachment mapping has specified other metadata fields to indexed such as content_type, language, etc. these will be extracted from the content field and indexed as requested (NOTE: they do not exist in source, only in the index).

The plugin also allows explicit metadata field values to be passed when indexing, by using the name of the metadata field prefixed with an underscore

PUT http://localhost:9200/docs/document/2?pretty=true&refresh=true 
{
  "id": 2,
  "title": "Another document",
  "file": {
    "_content":  "some base64 encoded string",
    "_content_type": "text/plain"
  }
}

Having explicit metadata fields affects the structure of the source returned from results e.g.

POST http://localhost:9200/docs/document/_search?pretty=true 
{
  "query": {
    "match": {
      "file.content": {
        "query": "NEST mapper"
      }
    }
  }
}

Status: 200
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 0.01125201,
    "hits" : [ {
      "_index" : "docs",
      "_type" : "document",
      "_id" : "2",
      "_score" : 0.01125201,
      "_source" : {
        "id" : 2,
        "title" : "Another document",
        "file" : {
          "_content" : "some base64 encoded string",
          "_content_type" : "text/plain"
     }
      }
    }, {
      "_index" : "docs",
      "_type" : "document",
      "_id" : "1",
      "_score" : 0.01125201,
      "_source" : {
        "id" : 1,
        "title" : "Some document",
        "file" : "some base64 encoded string"
      }
    } ]
  }
}

Here, the first document has an explicit metadata field for _content_type, and has therefore passed the attachment content to be indexed as _content. The second document passed the attachment content against the name of the attachment field.

_(using explicit metadata fields does not affect the attachment mapping i.e. new fields beginning with underscore are not added to the mapping)_

The extracted values are available in the hits

POST http://localhost:9200/docs/document/_search?pretty=true 
{
  "fields": [
    "file.content_type"
  ],
  "query": {
    "match": {
      "file.content_type": {
        "query": "pdf"
      }
    }
  }
}

Status: 200
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 0.19178301,
    "hits" : [ {
      "_index" : "docs",
      "_type" : "document",
      "_id" : "1",
      "_score" : 0.19178301,
      "fields" : {
        "file.content_type" : [ "application/pdf" ]
      }
    } ]
  }
}

I think we should introduce an Attachment type to make using the mapper-attachment plugin easier with NEST to

Handle the serialization of properties when indexing
Handle the deserialization of content with any additional explicit meta fields
Handle using any of the Attachment type properties to for strongly typed access to fields.

Thoughts?

/cc @Mpdreamz, @gmarz

Discuss Feature v2.3.3 v5.0.0-alpha2

Source

russcam

All 16 comments

@russcam Hi Russ. thanks for the detailed explanation on attachment plugin.
Even I have the same problem as Issue 1828..

nasreekar on 10 May 2016

@russcam Hi Russ.. Is there any plugin or method to check if any new files(either doc or pdf or ppt) are added to a folder and if added, they should be indexed to the ES server as an attachment. TIA :)

nasreekar on 10 May 2016

@nasreekar that question would be better asked on discuss.elastic.co as it will have more visibility

russcam on 10 May 2016

👍1

@russcam sure.. thanks.. Asked on the forum. :)

nasreekar on 10 May 2016

-1 as is because we are deprecating this plugin in 5.0.
It's replaced by ingest attachment which you might need to support with a similar approach you wrote.

dadoonet on 13 May 2016

@dadoonet Deprecated but not removed in 5?

This is aimed more at NEST 2.x, although I think it would also be useful to have in NEST 5.x, marked as deprecated. This would help those using the mapper-attachments plugin in 2.x to migrate to using ingest in 5.x more smoothly.

russcam on 13 May 2016

Indeed. My comment was just here to highlight the fact that this code might be removed in the near future so it might not worth investing too much on it.

dadoonet on 13 May 2016

Gotchya. I think it's a small piece of work to have something here to help with using the plugin with NEST.

russcam on 13 May 2016

@russcam Hi Russ, is the issue resolved for #2085?

piyushgiri on 13 May 2016

@piyushgiri It's not an issue but an enhancement/feature :) There's a PR open with an Attachment type that you may want to do something similar to in your own project.

This doesn't mean that it's going to make it into NEST though, it still needs review and discussion.

russcam on 14 May 2016

Is there any way in which i can search for phrases with special charecters in the content of the attached file? for eg: i want to search for [email protected] in the contents of all the indexed files.

I actually tried it on my code and it seems to be like '@' is not considered and instead it is searching for asn, gmail and com separately and generating results. How can I solve this problem. TIA

nasreekar on 24 May 2016

Bumped to 2.3.3...but this change could arguably target 2.4 instead.

gmarz on 25 May 2016

@russcam Hi Russ. I have a doubt in using attachments plugin. when we are attaching any document(say pdf or doc) as an attachment , are the meta fields considered by default or should we set them explicitly? (NEST)

for eg: previously i was doing something like below.

Attachment attach = new Attachment
                {
                    Name = Path.GetFileNameWithoutExtension(file),
                    Content = encodedData,
                    ContentType = Path.GetExtension(file)

                };

                doc = new Document()
                {
                    Title = Path.GetFileNameWithoutExtension(file),
                    FilePath = Path.GetFullPath(file), //added to get the path of the file
                    CreatedDate = DateTime.Now,
                    File = attach
                };

class Document
    {

        public string Title { get; set; } 

        public string FilePath { get; set; }

        public Attachment File { get; set; }

    }

class Attachment
    {
            [String(Name = "_name")]
            public string Name { get; set; } //name of the document

            [String(Name = "_content", TermVector = TermVectorOption.WithPositionsOffsets)]
            public string Content { get; set; } //content of the document

            [String(Name = "_content_type")]
            public string ContentType { get; set; } //pdf or doc or etc
    }

Is there any other way in which I can index a document with some more metafields (like I want to add author and modified date of the document )as an attachment??

TIA

nasreekar on 10 Jun 2016

@nasreekar, the mapper attachments plugin extracts metadata fields from the file _where they are available (and the plugin can extract them)_; some of the metadata fields can be explicitly sent when indexing the document and as far as I know, these are the following (in addition to content which is sent as _content).