Keras: Text tokenizer no attribute: 'oov_token'

[Mrugankakarte]

Hi,

My code was written for keras version <2.1.3, I was getting error no attribue :'oov_token'. This post said to set the 'oov_token = NULL'.

I saw the preprocess.R file which has the oov_token set to NULL, but the error still persists.

I tried doing the following steps,

args <- list(
      num_words = 30000,
      filters = "\"#$%&()*+-./:<=>@[\\]^_`{|}~\t\n",
      lower = F,
      split = " ",
      char_level = F,
      oov_token = NULL
)

do.call(keras$preprocessing$text$Tokenizer, args)

but I get the error, Error in do.call(keras$preprocessing$text$Tokenizer, args) : object 'keras' not found. Also couldn't find the keras_version() function. Do I have to train the model again?

[skeydan]

With the current version of keras, oov_token works fine, see e.g.
https://blogs.rstudio.com/tensorflow/posts/2018-09-17-eager-captioning/

[Mrugankakarte]

Thanks for the awesome example!!!

I am sure for current version it works, but what I meant was since the oov_token was introduced in keras 2.1.3 and my text tokenizer was created for keras version<2.1.3, I think it was 2.0.2, so when loading the tokenizer now in keras 2.2.0, it is not assigning the NULL value to oov_token as expected and that is why I am getting the error. Is there any way I can see the value assigned to oov_token or maybe manually update the value of oov_token without recreating a new tokenizer? Because creating a new tokenizer will destroy my trained model as words in dictionary may shuffle.

[skeydan]

I think that with text_tokenizer, and especially when using it across versions, it's best to inspect it and possibly write your own lookup table around it (as there seem to have been various bugs in the underlying Python implementation).

For example, you could experiment / inspect and create your own lookup like this:

sample_text <- list("The cat sat on the mat")

top_k <- 4

tokenizer <- text_tokenizer(
  num_words = top_k,
  oov_token = "<unk>")

tokenizer$fit_on_texts(sample_text)

tokenizer$word_index

text_tokenized <-
  tokenizer %>% texts_to_sequences(sample_text)

text_tokenized

word_index_df <- data.frame(
  word = tokenizer$word_index %>% names(),
  index = tokenizer$word_index %>% unlist(use.names = FALSE),
  stringsAsFactors = FALSE
)

word_index_df

Here you see that unk has been mapped to 1 (in current keras).

If you construct small examples like this, you can also directly investigate how arguments like num_words work (the above small example suggests you might want to add +1 if you care about the exact number).

[Mrugankakarte]

I get it. Thanks a lot for helping me dubug.

There was no oov_token when I loaded my tokenizer, so I added it as follows and could get rid of the error by doing

tokenizer$oov_token <- '<unk>'
fit_text_tokenizer(tokenizer, '<unk>')

I really appreciate the quick responses by you and your team to any errors. Thank you.

[skeydan]

You're very welcome!