Spark-nlp: EntityExtractor().setRequireSentences(True) throws error.

[clayms]

EntityExtractor().setRequireSentences(True)

throws this error:

Py4JJavaError: An error occurred while calling o461.getParam.
: java.util.NoSuchElementException: Param requireSentences does not exist.

From the EntityExtractor.scala file, I found the sentences parameter defined on lines 99 to 105, but when I changed lines 191 and 192 in annotator.py from:

def setRequireSentences(self, value):
    return self._set(requireSentences=value)

to:

def setRequireSentences(self, value):
    return self._set(sentences=value)

The error was basically the same:

Py4JJavaError: An error occurred while calling o474.getParam.
: java.util.NoSuchElementException: Param sentences does not exist.

[aleksei-ai]

It seems that EntityExtractor works totally incorrect. Will rewrite it in a day.

[aleksei-ai]

@clayms Thank you for the report.
I've recreated EntityExtractor in branch https://github.com/JohnSnowLabs/spark-nlp/tree/rewrite_entity_extractor
Here is an example notebook how to use it https://github.com/JohnSnowLabs/spark-nlp/blob/rewrite_entity_extractor/python/example/entities-extractor/extractor.ipynb

Will release it soon

[sofianeh]

I can't have the EntityExtractor() match with any of my entities even though my documents contain those. Here's a simple example I've built in Databricks, as you can see _entitites_ column is an empty list
https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/3306898732834076/1537424650694939/3546260386063631/latest.html

[aleksei-ai]

@sofianeh It seems that file /dbfs/testfile.txt is local file not an hdfs file.

Can you try file:///dbfs/testfile.txt instead?

[sofianeh]

Already tried with both file:///dbfs/testfile.txt and file:/dbfs/testfile.txt. They end up with this error:
Caused by: java.io.FileNotFoundException: file:/dbfs/testfile.txt (No such file or directory)

[saif-ellafi]

@sofianeh there is a param called entitiesFormat. By default, this param is set to "TXT" which means it only reads from local files without any prefix (we could add parsing of the prefix). To use with prefix and/or to read from hdfs with prefix, you should set this param to "TXTDS"

This will use spark to read such file, and spark allows file:// or hdfs:// prefix. Either that or you remove the file:// prefix (this will usually be faster than TXTDS if the amount of data input is small)

use setEntitiesFormat("TXTDS")

Let us know how it goes! Thanks

[sofianeh]

Yes, using TXTDS prevents the error, but I still end up with empty entities. Is there something wrong in my example (content of testfile.txt, my documents, my Pipeline, ...)?
Also, the documentation mentions a _setMaxLen(number)_ parameter but it seems no longer working.

[saif-ellafi]

@sofianeh what is the content of testfile.txt?? and thanks, setMaxLen doesn't exist anymore.

I have just tried this and worked for me (this is one of the unit tests we have there, I have changed it to TXTDS though)

    val entityExtractor = new EntityExtractor()
      .setInputCols("sentence", "normalized")
      .setInsideSentences(false)
      .setEntitiesFormat(TXTDS)
      .setEntitiesPath("./src/test/resources/entity-extractor/test-phrases.txt")
      .setOutputCol("entity")

en this example test-phrases looks like this:

dolore magna aliqua
Lorem ipsum dolor sit
laborum
anim id est laborum

so it basically searches this lines in the text

[sofianeh]

testfile.txt is created on the fly within the example notebook I've provided above

[saif-ellafi]

Right, sorry about that. Let me check i'll try the same thing

[saif-ellafi]

@sofianeh wow. We found the bug. Actually it is not a bug its a bad decision. We are normalizing the input file you provide (testfile.txt) and thus, it lower cases all its content. Hence it is not matching 'Hello World' but it would match 'hello world'

We'll fix this quick. Not normalize the input unless the user uses RecursivePipelines (a new Pipeline object that allows transforming the input with the same Pipeline the user is using)

For now, your pipeline will work if you include a normalizer

normalizer = Normalizer()\
  .setInputCols(["token"])\
  .setOutputCol("normal")

thank you

[sofianeh]

Thanks @saifjsl. I will use the workaround you've suggested for now. By the way, setInsideSentences() param is not documented anywhere

[saif-ellafi]

Will update the website asap as well. setInsideSentences decides whether you want to respect or not the sentence bounds. Is set to True by default.

For example, if you search for a piece of text 'This is a sentence. This is another', if InsideSentences is True, this will never match since the target belongs to different sentences. Most of the times it is not the case.

Feel free to share feedback anytime

[sofianeh]

I've published my working example on github in case people will need to understand the issue I was having.