Spark-nlp: NGramGenerator crashes after StopWordsCleaner

[maziyarpanahi]

It appears that NGramGenerator crashes when it sees empty strings or inputs after StopWordsCleaner.

Caused by: java.lang.NumberFormatException: For input string: "1.0"

val documentAssembler = new DocumentAssembler()
      .setInputCol("text")

val sentenceDetector = new SentenceDetector().
      setInputCols(Array("document")).
      setOutputCol("sentence")

val tokenizer = new Tokenizer()
      .setInputCols("sentence")
      .setOutputCol("token")

val normalizer = new Normalizer()
    .setInputCols("token")
    .setOutputCol("normalizedTokens")

val symSpellChecker = new NorvigSweetingApproach()
      .setInputCols("normalizedTokens")
      .setOutputCol("checkedTokens")
      .setDictionary(spell, "[a-zA-Z]+")

val stopWordsCleaner = new StopWordsCleaner()
      .setInputCols("checkedTokens")
      .setOutputCol("cleanTokens")
      .setStopWords(stopWordsList)
      .setCaseSensitive(false)

val nGrams = new NGramGenerator()
      .setInputCols("cleanTokens")
      .setOutputCol("ngrams")
      .setN(2)
      .setEnableCumulative(false)

[luigidisotto]

Here's the snippet relative to the fitting process

val flow = new Pipeline()
    .setStages(
        Array(
            documentAssembler, 
            sentenceDetector, 
            tokenizer, 
            normalizer, 
            symSpellChecker, 
            stopWordsCleaner,
            nGrams
        )
    )

val dataFlow = flow
      .fit(corpus)
      .transform(corpus)

The error comes in not during the fitting process, but when trying to show the content of the dataFlow DataFrame content as follows

dataFlow.show()

The error vanishes if NorvigSweetingApproach and stopWordsClener are removed.

[luigidisotto]

It seems also that with the following subset of the previous pipeline

val symSpellChecker = new NorvigSweetingApproach()
      .setInputCols("normalizedTokens")
      .setOutputCol("checkedTokens")
      .setDictionary(spell, "[a-zA-Z]+")

val stopWordsCleaner = new StopWordsCleaner()
      .setInputCols("checkedTokens")
      .setOutputCol("cleanTokens")
      .setStopWords(stopWordsList)
      .setCaseSensitive(false)

with no NGramGenerator, we have the NorvigSweetingApproach fail with

Caused by: java.util.NoSuchElementException: Failed to find a default value for frequencyPriority

[maziyarpanahi]

@luigidisotto I think I found the problem, it's NorvigSweetingApproach output.