Haystack: ValueError: badly formed hexadecimal UUID string exception

[nsankar]

Question
ValueError: badly formed hexadecimal UUID string exception

Additional context
I created a document store in elasticsearch by tokenizing text from a pdf document

when I used the DensePassageRetriever,

from haystack.retriever.dense import DensePassageRetriever
retriever = DensePassageRetriever(document_store=document_store, embedding_model="dpr-bert-base-nq",do_lower_case=True, use_gpu=True)
document_store.update_embeddings(retriever)

I get the following ValueError . What could be the reason ? and would like to know how to fix this issue?

07/31/2020 16:12:59 - INFO - haystack.retriever.dpr_utils -   Loading saved model from models/dpr/checkpoint/retriever/single/nq/bert-base-encoder.cp
07/31/2020 16:12:59 - INFO - haystack.retriever.dense -   Loaded encoder params:  {'do_lower_case': True, 'pretrained_model_cfg': 'bert-base-uncased', 'encoder_model_type': 'hf_bert', 'pretrained_file': None, 'projection_dim': 0, 'sequence_length': 256}
07/31/2020 16:13:09 - INFO - haystack.retriever.dense -   Loading saved model state ...
07/31/2020 16:13:09 - INFO - haystack.retriever.dense -   Loading saved model state ...
07/31/2020 16:13:09 - INFO - elasticsearch -   POST https://df4bc7e5f2a54314ac10223dd343fe94.us-central1.gcp.cloud.es.io:9243/document/_search?scroll=5m&size=1000 [status:200 request:0.139s]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-22-9372aaabee19> in <module>()
     11 # At query time, we only need to embed the query and compare it the existing doc embeddings which is very fast.
     12 
---> 13 document_store.update_embeddings(retriever)
     14 
     15 # ES retreivar

5 frames
/usr/lib/python3.6/uuid.py in __init__(self, hex, bytes, bytes_le, fields, int, version)
    138             hex = hex.strip('{}').replace('-', '')
    139             if len(hex) != 32:
--> 140                 raise ValueError('badly formed hexadecimal UUID string')
    141             int = int_(hex, 16)
    142         if bytes_le is not None:

**ValueError: badly formed hexadecimal UUID string**

[karimjp]

Hi @nsankar I obtained the same error by posting to the API endpoint /models/{model_id}/doc-qa with an Elasticsearch Document Store. Hopefully my findings can help you out. Digging further into the issue I noticed that this line https://github.com/deepset-ai/haystack/blob/d90435efd60d94034a3b64b711e9d055c3495ea0/haystack/database/base.py#L35 was causing the issue. It looks like the returned results by Elasticsearch are passed to https://github.com/deepset-ai/haystack/blob/d90435efd60d94034a3b64b711e9d055c3495ea0/haystack/database/elasticsearch.py#L391-L399 which by definition as seen here the field id https://github.com/deepset-ai/haystack/blob/d90435efd60d94034a3b64b711e9d055c3495ea0/haystack/database/base.py#L8 can be a string or a UUID, however because my elasticsearch ids are not hex based strings then it fails like previously mentioned on line https://github.com/deepset-ai/haystack/blob/d90435efd60d94034a3b64b711e9d055c3495ea0/haystack/database/base.py#L35

I got around the issue on my local codebase by doing the following changes:

• modified line 35 from base.py to self.id = id

• adding to https://github.com/deepset-ai/haystack/blob/d90435efd60d94034a3b64b711e9d055c3495ea0/rest_api/controller/search.py#L4 the type Union

• Correcting https://github.com/deepset-ai/haystack/blob/d90435efd60d94034a3b64b711e9d055c3495ea0/rest_api/controller/search.py#L119 to match the type flexibility defined in https://github.com/deepset-ai/haystack/blob/d90435efd60d94034a3b64b711e9d055c3495ea0/haystack/database/base.py#L8 by modifying line 119 of search.py to document_id: Optional[Union[str, UUID]] = None

[tholor]

however because my elasticsearch ids are not hex based strings [...]

@karimjp @nsankar Do I get this right that you both use your own "custom" elasticsearch indices that have been created without using DocumentStore.write_documents()?

@tanaysoni Can you please investigate this further and work on a fix? I think we didn't cover this use case in the last PR that introduced UUIDs (#243 )

[karimjp]

@tholor I used DocumentStore.write_documents() to ingest my documents in elasticsearch 7.8.1. The latest commit on master at that time was https://github.com/deepset-ai/haystack/commit/52a805be86414ea987cf6bf5be6bb5f6e2bf8f53. Later, I pulled the latest changes and tried to run the API when I started seeing the errors related the uuid. However, the case of custom indices is possible and if so an Elasticsearch document would need to be compatible with an _id field of at most 512 bytes

[tholor]

Ok got it. In that older commit we were still indexing with arbitrary IDs. The switch to UUID is now causing the issue.

@tanaysoni what do you think? Shall we allow also arbitrary string IDs in the Document class, even though we loose assurance of uniqueness? Otherwise support of already existing, custom ES Indices might be difficult.

[nsankar]

@tholor I didn't use any custom ES indexing when I reported the error. I was using the standard Haystack APIs. My Elastic search instance is an externally hosted one .that's it.

[nsankar]

@tholor How about Numeric hashes instead of UUIDs ?

https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch

[tanaysoni]

Hi @nsankar and @karimjp, thank you for raising the issue.

Shall we allow also arbitrary string IDs in the Document class

@tholor from how I see, there are some constraints for the document IDs:

• the Document object is generic across different document store implementations, so the IDs should be compatible across document stores.
• for the creation of new documents via write_documents(), the IDs must be unique.

For supporting existing document stores not created with Haystack, we cannot rely on the availability of UUIDs.

As a workaround, we can change the Document.id to be _always_ of str type. When we create a new document, we assign a UUID string under the hood, otherwise, we read string(or numeric) values from an existing index as str.

[tholor]

@tanaysoni Yeah, I think changing the id to str type would be good. It will come at some costs of risking duplicates if user provide their own id and don't check uniqueness by themselves, but I think the flexibility we gain (e.g support for custom indices and existing IDs from QA datasets) outweighs this.

[nsankar]

@tanaysoni @tholor Sounds good!

[tanaysoni]

Hi @nsankar and @karimjp, this should now be resolved with #284. I am closing this issue but please feel free to re-open if you still face any errors.

[karimjp]

@tanaysoni thank you for taking a look at this issue. Not sure if this should go into another issue, but here is another use case for generating a unique ID with write_documents() beyond the use of UUIDs.

• Document Hash as ID case: A user has a directory with multiple documents to write documents from and keeps adding documents to it. In order to prevent duplicate documents from being inserted with a different id the document id needs to have a hash of the document itself.

I haven't tested what happens if I have an index with a custom mapping and my document to be inserted using write_documents() already has an _id field with any string I define but if that works, it could be the right answer for the use case above.

[tanaysoni]

Hi @karimjp, for the document hash case, you can use the newly added(#285) update_existing_documents parameter for the ElasticsearchDocumentStore. When the parameter is set to True, the documents with the same id gets updated. Otherwise, when set to False, the document store throws an error if a document with the given id already exists.