Spark-nlp: How to tokenize and split sentences?

[cigrainger]

Description

I can't get tokenized sentences out of a document and have been driving myself crazy with the documentation to try to figure it it out.

Expected Behavior

I'd like to take a paragraph such as:

This is a sentence. This is a another sentence. I have many sentences.

and use the sentence detector and tokenizer to get a column with a list of lists like:

[['this', 'is', 'a', 'sentence', '.'], ['this', 'is', 'another', 'sentence', '.'], ['i', 'have', 'many', 'sentences', '.']]

Current Behavior

Passing the sentence detector to the tokenizer and through the finisher yields only tokenization. e.g.

documentAssembler = (
    DocumentAssembler().setInputCol("description").setOutputCol("document")
)

sentenceDetector = (
    SentenceDetector()
    .setInputCols(["document"])
    .setOutputCol("sentence")
)

tokenizer = Tokenizer().setInputCols(["sentence"]).setOutputCol("tokens")

finisher = (
    Finisher()
    .setInputCols(["tokens"])
    .setOutputCols(["sent"])
)

pipeline = Pipeline(stages=[documentAssembler, sentenceDetector, tokenizer, finisher])

Context

Your Environment

• Version used: 2.0.8 in PySpark
• Browser Name and version:
• Operating System and version (desktop or mobile): Amazon Linux 2 (EMR)
• Link to your project:

[CyborgDroid]

Here you go

finisher = Finisher() \
    .setInputCols(["sentence", "tokens"]) \
    .setIncludeMetadata(True) #optional

And if you want the sentences in multiple rows:

sentenceDetector = (
    SentenceDetector()
    .setInputCols(["document"])
    .setOutputCol("sentence")
    .setExplodeSentences(True)
)

All of this is in the documentation and examples:

SentenceDetector:
https://nlp.johnsnowlabs.com/docs/en/annotators

Finisher:
https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/17b977927c0ea78584d2e2203cfe18e7e6d862cc/jupyter/annotation/english/model-downloader/dl-ner.ipynb

[cigrainger]

This doesn't work. Thank you for pointing me to those (but I have already gone through them). Including both input columns just gives me a finished_sentence column and a finished_tokens column. Not a column with a list of tokenised sentences. Exactly the problem stated above. I'd like a column of tokenised sentences like:

[['this', 'is', 'a', 'sentence', '.'], ['this', 'is', 'another', 'sentence', '.'], ['i', 'have', 'many', 'sentences', '.']]

[maziyarpanahi]

Hi @cigrainger

The result of Tokenizer is standard to what other annotators need. However, it is possible with a little bit of Spark manipulation.

You have the metadata of tokenizer:

[[token, 0, 3, Pour, [sentence -> 0, chunk -> 0]

As you can see it has the id of the sentence. All you need to do is to create a simple UDF and feed tokens and sentence columns. Then you can create an array of arrays where you map through your sentences and push tokens related to that sentence id.

Or, if your documents have IDs you can simply use .setExplodeSentences(true) in SentenceDetector where it creates one row per sentence. That way you can do a groupBy IDs and push the results into an array.

[maziyarpanahi]

Just out of curiosity, is there a specific reason you would like to have the tokens in that format? I am just asking to see if there is any Spark ML/SQL/Graph function that accepts that format. Or is this situation unique to your use case?

Many thanks @cigrainger

[mathematiguy]

One reason you might do this is if you're counting ngrams and you don't want to link words that are in different sentences.

"This is a sentence. This is another sentence." - i.e. you don't want to include the bigram 'sentence this' based on this string.

[maziyarpanahi]

For that reason, I think using SentenceDetector which detects sentences and then right after you have Tokenizer and now that we have NGramGenerator should be fine. Tokens are being extracted from each sentence, and NGrams will be generated within each sentence without overlapping with others.