Spark-nlp: [Question] How to get back an in-place NER annotated document?

[santoshbs]

Consider the following sentence:
Elon Reeve Musk was born on the 28th of June of 1972 in Pretoria, South Africa.

Is there a way I can get the following onto_recognize_entities_lg annotated sentence?
Elon_Reeve_Musk|PERSON was born on the_28th_of_June_of_1972|DATE in Pretoria|GPE, South_Africa|GPE.

[maziyarpanahi]

The NerDLModel will give you single labels for each token. On the other hand, NerDLConverter will give you the entities (I-PER and B-PER for IOB and IOB2 schema).
Combining the Tokens and Entities will give you the result you are looking for:

documentAssembler = DocumentAssembler()\
    .setInputCol("text")\
    .setOutputCol("document")

tokenizer = Tokenizer() \
    .setInputCols(["document"]) \
    .setOutputCol("token")

glove_embeddings = WordEmbeddingsModel.pretrained('glove_100d').\
  setInputCols(["document", 'token']).\
  setOutputCol("embeddings")

onto_ner = NerDLModel.pretrained("onto_100", 'en') \
          .setInputCols(["document", "token", "embeddings"]) \
          .setOutputCol("ner")

ner_converter = NerConverter() \
  .setInputCols(["document", "token", "ner"]) \
  .setOutputCol("ner_chunk")

nlpPipeline = Pipeline(stages=[
 documentAssembler, 
 tokenizer,
 word_embeddings,
 onto_ner,
 ner_converter
 ])

result = nlpPipeline.fit(df).transform(df)

result.select(F.explode(F.arrays_zip('ner_chunk.result', 'ner_chunk.metadata')).alias("cols")) \
.select(F.expr("cols['0']").alias("chunk"),
        F.expr("cols['1']['entity']").alias("ner_label")).show(truncate=False)
# this is just an example from CoNLL 2003 results
+---------------------+---------+
|chunk                |ner_label|
+---------------------+---------+
|Turner   Newall      |PER      |
|Federal Mogul        |ORG      |
|TORONTO              |MISC     |
|Canada               |LOC      |
|A                    |ORG      |
|Ansari X Prize       |PER      |
|University           |ORG      |
|Louisville           |MISC     |
|amino                |MISC     |
|Mike                 |PER      |
|Fitzpatrick          |LOC      |
|Southern California's|MISC     |
|emissions            |MISC     |
|The                  |ORG      |
|British              |MISC     |
|Education            |PER      |
|Skills               |PER      |
|DfES                 |ORG      |
|Music                |LOC      |
|Manifesto            |PER      |
+---------------------+---------+
only showing top 20 rows

There are many examples here:
https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Public/3.SparkNLP_Pretrained_Models.ipynb

Here: https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Public/4.NERDL_Training.ipynb

And lots of other training notebooks here: https://github.com/JohnSnowLabs/spark-nlp-workshop/tree/master/tutorials/Certification_Trainings/Public

[santoshbs]

Many thanks!

[santoshbs]

@maziyarpanahi , in the above code you shared, when I try to replace glove_100d with either glove_6B_300 or glove_840B_300 I get the following error:

glove_6B_300 download started this may take some time.
Can not find the model to download please check the name!
Traceback (most recent call last):

  File "<ipython-input-20-8c1a1e422d77>", line 1, in <module>
    glove_embeddings = WordEmbeddingsModel.pretrained("glove_6B_300", "en").\

AttributeError: 'NoneType' object has no attribute 'setInputCols'

This happens even if I explicitly include lang="en"

[maziyarpanahi]

These are all documented: https://nlp.johnsnowlabs.com/2020/01/22/glove_6B_300.html

Those are not English Embeddings, they are multi-lingual.

[maziyarpanahi]

PS:onto_100 uses glove_100d for instance. To know which embeddings should be used, you can use:

ner_model = NerDLModel.pretrained("onto_100", 'en') \
          .setInputCols(["document", "token", "embeddings"]) \
          .setOutputCol("ner")

print(ner_model.getStorageRef)

This will give you a name which is the name of the embeddings. You can find it here with a full description:

• https://nlp.johnsnowlabs.com/models
• or here: https://github.com/JohnSnowLabs/spark-nlp-models