Uwazi: Automatically assign a unique "record number" to each entity added to instance

[kjantin]

As of 11 March 2020, there are at least 6 partner organisations that are requesting the ability to add a unique record number to a particular entity type, and to have this unique record number automatically assigned by the system, and to be able to search the collection by this number.

The reason why this feature is being requested is because organisations need a way to easily, systematically, and accurately refer to information in the system (whether that is a case, or a person, or an event, etc).

Regarding the format of this unique number:

• The format of this unique number is not very important.
• One approach is to use the ~20-digit number/code that the system already automatically assigned to each entity and is displayed in the URL of the entity
• Another approach is to have a 5 digit number/code that is generated separately by the system.

Regarding the UI of this feature:

• My own 2 cents: different organisations are going to want different types of entities with this unique record number, so I think it would make sense to offer a new property type when an admin is creating an entity template. This new property type would be "Unique ID generator". That way, they can control which entity templates this unique ID is generated for.

---

OLD VERSION OF THIS ISSUE:

__As a user, I would like to have a unique code assigned to each document, page, image, etc that I upload to Uwazi. I need this code for legal purposes - this process is called "Bates numbering"._

Definition of Bates numbering (https://en.wikipedia.org/wiki/Bates_numbering):

Bates numbering is used in the legal, medical, and business fields to place identifying numbers and/or date/time-marks on images and documents as they are scanned or processed, for example, during the discovery stage of preparations for trial or identifying business receipts. This process provides identification, protection, and automatic consecutive numbering of the images._

[txau]

This is complex and cumbersome. Are we sure we want to support that? I don't see the value:

"So come on folks. Don’t accept a blanket statement that you need a Bates number. There’s a better way to handle document identification that is faster, more efficient and cheaper."

http://www.advanceddiscovery.com/2016/01/18/are-we-really-still-talking-about-bates-numbers/

[RafaPolit]

I have to agree with the complaint.

The problem is really somewhere else. I remember saying: we already have unique IDs, what are you going to use them for?

The answer was: we refer to the documents by this numbers. Hence the issue: they are used to it. Their process is already familiar with them and they are very swift using it. So it would involve a really clever solution plus a lot of good will and effort for them to change this way of thinking.

What wasn't at all clear from the talk, was that you should be able to STAMP each page with it and see it in the final document. This would involve actually modifying the original PDF to include the stamps so that they are included onwards on the downloadable PDF. That is certainly really complex, and would require PDF manipulation which we have avoided up until now.

I agree that, if that is the case, we should probably say no.

[txau]

Can you guys provide examples of the codes they are using? Can this be solved with a text property of the document?

[RafaPolit]

From what I understood, this is a PER-PAGE code, not per-document. So probably not. Maybe @kjantin has more info.

[kjantin]

User story: "As an administrator of an Uwazi instance being used to document and analyse information about human rights violations, I need to have a unique record number automatically assigned to each new record so that I can easily distinguish between one record and another. There are many entity titles that are identical or very similar (e.g. John Smith). Currently, our workflow is to create the entity, then copy the unique number in the URL of that entity, edit the entity, and paste that unique number into the entity title."

[nwyu]

Another user story:

_As a researcher interviewing victims of human rights abuses, I need a unique reference number to be automatically assigned to every statement, document and material received so that I can anonymise identities and document sources, plus simplify cross-referencing and indexing._

[kjantin]

From what I understood, this is a PER-PAGE code, not per-document. So probably not. Maybe @kjantin has more info.

Just to be clear, this issue was originally about a "PER-PAGE code" but has changed to a per-entity code, and the proposal is the use the 10-digit code at the end of the URL of each entity as this code.

[kjantin]

Today I rec'd a msg from a partner:

we have still not found a satisfactory solution to the unique identifier
issue that we have highlighted before. If Huridocs would be able to program
in unique identifiers into Uwazi, we would be happy to pay for this service.

[txau]

This is relevant:

our workflow is to create the entity, then copy the unique number in the URL of that entity, edit the entity, and paste that unique number into the entity title

So in this particular case is not only about the unique-id field, but also about the title field since they want to have some sort of "alias" so that they don't put names on the title.

[kjantin]

User story:

As an editor, when I want to create a relationship between an ACT entity type and a PERSON entity type, it is impossible to know the difference between one PERSON entity and another PERSON entity if they have the same name, so there needs to be something unique that is connected to each entity to distinguish it from others. I may have only the first name, or I may have only the last name, and I need to be able to see some of the other properties of each entity that comes up in the search results to know which entity is the right one. But it would also be very helpful to have a unique ID assigned automatically so a person could easily search by that ID number instead of having to search by descriptive properties.

[simonfossom]

This is asking partners to design feature for us. This is not the way to solve it. Lets talk to these users to find underlying problem.

My suspicion is that this conversation will go something like this:

• users: _We want unique identifier!_
• us: why?
• users: _We want to more quickly find entity in search._
• us: why?
• users: _We don't see full title and sometimes multiple entities have same titles._
• us: what would be your workflow if you had unique identifier?
• users: _In one tab we copy unique ID and in the other tab paste in search in relationships sidebar._

the underlying problem is probably that they can't see the card or an entity in full view to confirm they are connecting the right entities. Their proposal is not what we should be building, but it is a hint of what problem we should be solving.

[kjantin]

the underlying problem is probably that they can't see the card or an entity in full view to confirm they are connecting the right entities.

Yes this is very true, and I hope we'll be able to address this issue as we work on this issue https://github.com/huridocs/uwazi/issues/2750 (improved workflow for adding entities).

But what I hear from partners is that it is also still helpful to have a unique identifier, so that they can easily and quickly find certain entities really quickly (without having to do a search and then inspect the other properties to check to see if it's the right one. For example, at any given time, there might be a list of people or acts or detention centers that you need to access often for some research, or case management, or a court case.

[RafaPolit]

Another clients use case:

• they have this policy of having a manual way of binding records so that they do not depend on a specific platform: a simple export will show this value referenced throughout the different templates.

This actually makes sense.

[simonfossom]

I was thinking that this number doesn't have to be a field but maybe just a selectable number in the corner that you can copy-paste and search

[Predragh]

Hi, I was asked by @kjantin to comment on behalf HRDI Iraq team - this solution is just fine! If we can copy-paste and search selectable (uniquely generated) entity's numbers from the corner - while creating connections between entities in edit mode - and regullary, through the Search box on the top left - then case closed.

[whyfrycek]

Just to mention this has come up again over the last two days with three organisations I spoke with. Something like @simonfossom would make a lot of sense. Use cases are:

• quickly find a record in the database
• communicate outside the database with a colleague, e.g. "please have a look at #6789"
• use the number in (external) communication to avoid using personal information, e.g. "in the complaint received by us under the #67687 the victim #7656575678 alleges the following..."

[txau]

If these requirements are true:

• Record numbers need to be unique
• The format varies depending on the organization preferences
• It is desirable that it is auto-generated, but also allow for manual input
• The format can be based in other existing data in the same template (ie. type, date, etc)

...a good approach would be some sort of field that supports hooks and is aware of the rest of the values in the format. Also this field can request to/from server. So the hook events could be something like:

• Another field has changed value
• A user has typed or deleted some text in the input
• User hit save
  etc

So the field is configured with some custom code that can be injected via UI (only admins and with proper validation) and responds to the hook's contract:

• the field is blank
• [the date field changed] -> I'll grab the year as sufix for the record number
• [the type field has changed] - > I'll create a particular acronym to represent that type in the code
• And so on so forth.

This could be useful for other scenarios as well, ie. some of our users are creating titles based in existing metadata as well such as "Galindo Cardenas y otros. Order of the IACourt. October 8, 2020".

Or to automatically calculate certain data:

• Date of birth of a person = X
• Date of a particular event in that person's life = Y
• Age when the event happened = f(X,Y)

[simonfossom]

On the product call, we have settled on simplest version for now. It will have auto-generated code containing 2 letters and 4 numbers. On the edit side, it is a text field. This allows the admin/editor to append or override this id, if required.
This is how template builder property looks like (Design link):

[RafaPolit]

To consider:

• [ ] On public form, this should not be a field that gets rendered, but assigned a value upon creation. Perhaps zero-out on the public endpoint and then auto-fill on backend if field is missing? (discuss)

[nwyu]

This has been implemented in #3579. Other related issues to be addressed: #3666 and #3672.