Text: FEATURE REQUEST: Remove undesired words from vocabulary

Created on 23 Jul 2018  路  5Comments  路  Source: pytorch/text

When building the vocabulary, it would be great to be able to pass a list of undesired words (e.g. punctuation marks, stop words, etc) and they will not belong to the vocabulary (hence, will be '' when "numericalizing").

enhancement help wanted
Source
picture manuelsh

All 5 comments

@manuelsh Not sure if this helps you, but you could probably pass your own preprocessing/postprocessing pipeline to the Field constructor that would handle the removal of unwanted tokens.

domaala picture domaala on 30 Jul 2018

Yes, I actually did that, thanks!

However, being such a common thing to do in NLP, possibly makes sense to add it in the library?

manuelsh picture manuelsh on 30 Jul 2018
👍1

Possibly, yes!

domaala picture domaala on 31 Jul 2018
👍1

Added in #397

mttk picture mttk on 21 Sep 2018

Amazing, thanks!

manuelsh picture manuelsh on 21 Sep 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

pasinit picture pasinit  路  3Comments
wpfnlp picture wpfnlp  路  5Comments
lipingbj picture lipingbj  路  6Comments
tnlin picture tnlin  路  5Comments
felipebravom picture felipebravom  路  3Comments