Paperless: Skip Language Check

[V3L]

I've just recently installed paperless in docker on my Synology and I've managed to set it up to consume documents, add correspondents and add tags automatically.

However I have encountered a few issues, but the main one and the purpose of this thread is that I have a document where the language is being identified as "CY", however the document is actually English, this isn't a huge issue, but due to the fact "CY" isn't a supported document it fails to import the document.

In my current deployment I only plan to load documents that are in English, is there a way to skip the whole language check as at present the OCR checks seem to be taking up to 10mins anyway, which seems quite long, is there any advice on how this could be sped up?

Also, I saw a really weird issue with dates, the date was 12th Oct 2018, paperless read the Oct 2018 and set the date to 01/10/2018 which is obviously incorrect, is there something I need to change to make sure it reads the dates correctly going forward?

I am not sure if you need anymore specific information to assist with any of these items, if you do just let me know.

Thanks,
Ben

[ddddavidmartin]

This seems odd. I thought that it would only be able to detect languages that are installed. What error do you see in the logs?

In terms of available settings have a look at the paperless.conf.example. You can set these values as environment variables for the docker container. I thought there was a setting to disable OCR, but it does not seem to be the case.

From what I remember the dates are parsed manually in the code. It may not be smart enough to handle the day ending in letters (12th, 1st, 3rd and so on). Have a look at get_date in parsers.py for the respective bits in the code.

[V3L]

@ddddavidmartin I added this line to my parsers.py file and I checked one document and it correctly recognised 26th June 2018 as the 26/06/2018.

r'(\b|(?!=([_-])))([0-9]{1,2})(st|nd|rd|th)([. ]+[^ ]{3,9} ([0-9]{4}|[0-9]{2}))(\b|(?=([_-])))|'

I've not added it is a PR, as I am fairly new to GitHub, and don't fancy breaking anything else, also this is the first regex statement I've ever written before, and I don't know if its position in the among the other statements is important.

Also, regarding the OCR issue, I worked out that it was caused by having two pages in one scan that have different orientations. I fixed this by removing one of the pages and the document loaded as expected.

Additionally, I set the PAPERLESS_FORGIVING_OCR parameter to true, I realised that setting it to "true" was causing it not to be picked up in docker, but removing the quotes resolved that and helped with some of my other documents that were failing to load.

[danielquinn]

The date handling is probably the most complicated thing running under the hood because of the multitude of locales using this project. If you're curious, have a look at parsers.py and just search for date to see what's going on.

The good news though is that this should be as easy as setting PAPERLESS_DATE_ORDER in your paperless.conf file or your environment. Setting it to DMY should get you what you want.

[danielquinn]

Oh, and as for the language check, that's an annoying one that could maybe use some improvement.

We use a Python library called langdetect to guess the language of a file based on a sampling of text pulled from the middle page of a document using Tesseract's English reading of your file. If that middle page contains a sufficient amount of non-English text however, that library may misidentify the file's language and then tell Tesseract to read the whole document in some other language. Unfortunately, langdetect doesn't have a means of knowing what languages are currently available on your system.

A better model might be to do something like:

• Guess the language
• Check if that language is available for Tesseract
• If not, fall back to settings.OCR_LANGUAGE and hope that's correct.

[robsdedude]

I second this idea as I'm having problems too with German documents that are being detected as Swedish for what ever reason.

pyocr.get_available_tools()[0].get_available_languages() would be a way to go

I'm thinking about making a PR but I'm afraid it would take me much longer to set up the dev env (mainly for testing the changes) for such a small change than it would take someone with an existing env to just patch it.

[robsdedude]

Another idea to improve this: @danielquinn why did you decide to choose the middle page and not use the text of all pages (e.g. using a majority vote or concatenating all the text before guessing the language)?

[danielquinn]

It was mostly for performance I guess. Imagine the case of a 60-page document: do you really want to OCR all 60 pages twice? Sampling the first page often left too little content to make use of, so the middle page was selected. I'm not opposed however to opting for your suggestion though. For my use-case anyway, most documents are typically < 4 pages and the async nature of Paperless means that it'll all get done eventually.

I'm just concerned that there are a lot of people using Paperless on low-powered systems, so I want to keep the labour required to a minimum.