Keras: How to save tokenizer/dictionary for use on unseen data

[aliarsalankazmi]

I have trained a model on 'seen' data (trained on a training set, tested on a test set). But I now need to apply the model to new, unseen data.

My understanding is that if I apply a tokenizer to this new data and then supply the sequenced data to my model, it will likely contain differing sequences for same words that were trained earlier.

For this purpose, I was wondering how to save the existing tokenizer and apply it on new data.

[aliarsalankazmi]

Just an update: Tried saving the entire environment, hoping that all objects (tokeniser, model, etc.) will be saved too.

Loaded the environment, but since the said objects are not R objects, attempt to access them just prints this:

<pointer: 0x0>

[jjallaire]

Thanks for reporting this~ Yes, Keras objects are under the hood Python objects which of course don't automatically serialize. The solution is to use pickle to save and load the tokenizer (see example code below). I will wrap this code in higher level save_tokenizer() load_tokenizer() functions soon:

library(reticulate)
pickle <- import("pickle")
utils <- import_builtins()

# saving
with(utils$open("tokenizer.pickle", "wb") %as% handle, {
  pickle$dump(tokenizer, handle, protocol = pickle$HIGHEST_PROTOCOL)
})

# loading
with(utils$open("tokenizer.pickle", "rb") %as% handle, {
  tokenizer <- pickle$load(handle)
})

[aliarsalankazmi]

Thanks ever so much @jjallaire :+1: !

[jjallaire]

Just added save_text_tokenizer() and load_text_tokenizer() functions. Docs are here: https://keras.rstudio.com/reference/save_text_tokenizer.html