Spark-nlp: bert_large_cased_en_2.4.0_2.4_1580580251298 Embedding doesn't work

[simon-oz]

bert_large_cased_en_2.4.0_2.4_1580580251298 doesn't work

Description

There are four pre-trained variants of BERT: bert_base_uncased , bert_base_cased, bert_large_uncased, bert_large_cased, when I tested the above models, only bert_base_cased model works for me.

Following is the code snippets

  val documentAssembler = new DocumentAssembler()
    .setInputCol("text")
    .setOutputCol("document")

  val sentenceDetector = new SentenceDetector().
    setInputCols(Array("document")).
    setOutputCol("sentence")

  val tokenizer = new Tokenizer()
    .setInputCols(Array("sentence"))
    .setOutputCol("token")

  val normalizer = new Normalizer()
    .setInputCols("token")
    .setOutputCol("normal")

  val bertEmbeddings = BertEmbeddings.load(path_bert_base_cased)
    .setInputCols(Array("sentence", "token"))
    .setOutputCol("bertEmbd")
    .setCaseSensitive(true)

  val nerBertModel = NerDLModel.load(modelPath)
    .setInputCols(Array("sentence", "token", "bertEmbd"))
    .setOutputCol("nerBert")

  val nerConverter = new NerConverter()
    .setInputCols(Array("document", "token", "nerBert"))
    .setOutputCol("ner_converter")

  val finisher = new Finisher()
    .setInputCols("nerBert", "ner_converter")
    .setIncludeMetadata(true)
    .setOutputAsArray(false)
    .setCleanAnnotations(false)
    .setAnnotationSplitSymbol("@")
    .setValueSplitSymbol("#")

  val stages = (Array(
    documentAssembler,
    sentenceDetector,
    tokenizer,
    bertEmbeddings,
    nerBertModel,
    nerConverter,
    finisher))

  val pipeline = new Pipeline().setStages(stages)

  val result = pipeline.fit(Seq.empty[String].toDS.toDF("text")).transform(testData)

  result.printSchema()

Expected Behavior

All the four pre-trained variants of BERT should work

Current Behavior

If use any model other than bert_base_cased, get the following exception:

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Found input column with storage metadata. But such ref does not match to the ref this annotator requires. Make sure you are loading the annotator with ref: bert_base_cased
at scala.Predef$.require(Predef.scala:224)
at com.johnsnowlabs.storage.HasStorageRef$class.validateStorageRef(HasStorageRef.scala:27)
at com.johnsnowlabs.nlp.annotators.ner.dl.NerDLModel.validateStorageRef(NerDLModel.scala:19)
at com.johnsnowlabs.nlp.annotators.ner.dl.NerDLModel.beforeAnnotate(NerDLModel.scala:95)
at com.johnsnowlabs.nlp.AnnotatorModel._transform(AnnotatorModel.scala:49)
at com.johnsnowlabs.nlp.AnnotatorModel.transform(AnnotatorModel.scala:79)
at org.apache.spark.ml.PipelineModel$$anonfun$transform$1.apply(Pipeline.scala:305)
at org.apache.spark.ml.PipelineModel$$anonfun$transform$1.apply(Pipeline.scala:305)
at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57)
at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66)
at scala.collection.mutable.ArrayOps$ofRef.foldLeft(ArrayOps.scala:186)
at org.apache.spark.ml.PipelineModel.transform(Pipeline.scala:305)

Possible Solution

Context

Test NER performance by using different BERT models

Your Environment

• Spark NLP version: 2.4.4
• Apache NLP version: 2.4.4
• Java version (java -version): 1.8.0_162
• Setup and installation (Pypi, Conda, Maven, etc.): Maven
• Operating System and version: CentOS 7.7

[maziyarpanahi]

This is not a bug, our NerDLModel pretrained model ner_dl_bert was trained by using bert_base_cased, therefore it can be only used by bert_base_cased, not any others. To use other BERT models, you need to train your own:

Blog: https://towardsdatascience.com/named-entity-recognition-ner-with-bert-in-spark-nlp-874df20d1d77

Notebook: https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/blogposts/3.NER_with_BERT.ipynb

[simon-oz]

Thanks maziyarpanahi to make it clear. Very much appreciate.

Will follow the link above to train a new model based on bert_large_cased embedding

[simon-oz]

Hi maziyarpanahi,

Thanks for your suggestion and I followed the links above implemented my own bert_ner model.
However, when I use the trained model to do predictions, I found that the NER is not of BIO format, it always starts with I-....

I found it seems the output in the following link has the similar issue, for example, in the last part of the linked page, the output of an entity does not start with B-PER, but as following:
Peter NNP NNP I-PER
Parker NNP NNP I-PER

https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/blogposts/3.NER_with_BERT.ipynb.

I also tested to transform training data by using
bert.transform(trainingData)
seems there is no different.

Am I wrong somewhere?

Thanks in advance

Following is the code I used to train the BERT ner model.

val bert = BertEmbeddings.load(bertEmbdPath)
  .setInputCols("sentence", "token")
  .setOutputCol("bert")
  .setCaseSensitive(false)
  .setPoolingLayer(0)

val dataTrans = bert.transform(testData)
dataTrans.write.mode(SaveMode.Overwrite).parquet(testDataPath)

val trainTrans = bert.transform(trainingData)

val nerTagger = new NerDLApproach()
  .setInputCols(Array("sentence", "token", "bert"))
  .setLabelColumn("label")
  .setOutputCol("ner")
  .setMaxEpochs(3) 
  .setBatchSize(8) 
  .setRandomSeed(0)
  .setVerbose(1)
  .setEnableOutputLogs(false)
  .setIncludeConfidence(true) 
  .setEvaluationLogExtended(false)
  .setValidationSplit(0.2.toFloat) 
  .setTestDataset(testDataPath)

val ner_pipeline = new Pipeline()
  .setStages(Array(bert, nerTagger))

val bertNer = ner_pipeline.fit(trainTrans)

bertNer.write.overwrite().save(modelSavePath)

[maziyarpanahi]

You can train NerDLModel with any tagging style. It will learn no matter what and will predict. The quality of the tagging it's up to the tagging style based on research etc. However, NerConverter to extract entities only support IOB and IOB2.

Now that is being said, what you are seeing is IOB:

The CoNLL 2003 dataset is using that, it's not something that NerDL is doing to it, the data appears to be in that format, and that is how it learns. You can find the IOB2 style of CoNLL2003 and use that or BIOES format and convert it into IOB2 then use that or just use BIOES and not use NerConverter since it only accepts IOB/IOB2.

It's not an issue, what goes in, exactly comes out if the prediction is correct (there is a margin for error so one or two in a sentence might be false positive)

[simon-oz]

you are a legend maziyarpanahi !

Thanks tons !