Flair: Exception in Model Optimization

[adgsantos]

Describe the bug
I'm trying to optimize the hyper parameters for a sequence tagger. To get started I followed the instructions of the tutorial presented in Model Optimization but to no avail. The error message is: AttributeError: 'tuple' object has no attribute 'embedding_length'. I have also tried with the example (apart from the dataset) in the tutorial, for which the error message is: ValueError: Expected input batch_size (32) to match target batch_size (0).

To Reproduce
Download the CONLL_03 dataset and put it in the default location: $HOME/.flair/datasets

Run the following code:

from flair.data import TaggedCorpus
from flair.data_fetcher import NLPTaskDataFetcher, NLPTask

# load your corpus
corpus: TaggedCorpus = NLPTaskDataFetcher.load_corpus(NLPTask.CONLL_03)

# Embeddings and training the classifier
from hyperopt import hp
from flair.hyperparameter.param_selection import SearchSpace, Parameter
from flair.embeddings import FlairEmbeddings

search_space = SearchSpace()
search_space.add(Parameter.EMBEDDINGS, hp.choice, options=[
    [ FlairEmbeddings('news-forward'), FlairEmbeddings('news-backward') ]
])
search_space.add(Parameter.HIDDEN_SIZE, hp.choice, options=[32, 64, 128])
search_space.add(Parameter.RNN_LAYERS, hp.choice, options=[1, 2])
search_space.add(Parameter.DROPOUT, hp.uniform, low=0.0, high=0.5)
search_space.add(Parameter.LEARNING_RATE, hp.choice, options=[0.05, 0.1, 0.15, 0.2])
search_space.add(Parameter.MINI_BATCH_SIZE, hp.choice, options=[8, 16, 32])

from flair.hyperparameter.param_selection import SequenceTaggerParamSelector, OptimizationValue

# create the parameter selector
param_selector = SequenceTaggerParamSelector(
    corpus, 
    'ner', 
    'resources/taggers/optimization', 
    max_epochs=50, 
    training_runs=3,
    optimization_value=OptimizationValue.DEV_SCORE
)

# start the optimization
param_selector.optimize(search_space, max_evals=100)

It yields the following stack trace:

(py36) ags@ds-vm:~/notebooks/notebooks$ python a.py
2019-01-07 12:19:52,246 Reading data from /home/ags/.flair/datasets/conll_03
2019-01-07 12:19:52,247 Train: /home/ags/.flair/datasets/conll_03/eng.train.openNLP
2019-01-07 12:19:52,247 Dev: /home/ags/.flair/datasets/conll_03/eng.testa
2019-01-07 12:19:52,247 Test: /home/ags/.flair/datasets/conll_03/eng.testb.openNLP
2019-01-07 12:19:59,671 ----------------------------------------------------------------------------------------------------
2019-01-07 12:19:59,671 Evaluation run: 1
2019-01-07 12:19:59,671 Evaluating parameter combination:
2019-01-07 12:19:59,671         dropout: 0.009483228554103051
2019-01-07 12:19:59,671         embeddings: /home/ags/.flair/embeddings/lm-news-english-forward-v0.2rc.pt,/home/ags/.flair/embeddings/lm-news-english-backward-v0.2rc.pt
2019-01-07 12:19:59,671         hidden_size: 64
2019-01-07 12:19:59,671         learning_rate: 0.2
2019-01-07 12:19:59,671         mini_batch_size: 8
2019-01-07 12:19:59,671         rnn_layers: 1
2019-01-07 12:19:59,671 ----------------------------------------------------------------------------------------------------
2019-01-07 12:19:59,809 ----------------------------------------------------------------------------------------------------
2019-01-07 12:19:59,810 Training run: 1
Traceback (most recent call last):
  File "a.py", line 35, in <module>
    param_selector.optimize(search_space, max_evals=100)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/flair/hyperparameter/param_selection.py", line 147, in optimize
    best = fmin(self._objective, search_space, algo=tpe.suggest, max_evals=max_evals)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/hyperopt/fmin.py", line 385, in fmin
    rval.exhaust()
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/hyperopt/fmin.py", line 244, in exhaust
    self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/hyperopt/fmin.py", line 218, in run
    self.serial_evaluate()
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/hyperopt/fmin.py", line 137, in serial_evaluate
    result = self.domain.evaluate(spec, ctrl)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/hyperopt/base.py", line 840, in evaluate
    rval = self.fn(pyll_rval)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/flair/hyperparameter/param_selection.py", line 87, in _objective
    model = self._set_up_model(params)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/flair/hyperparameter/param_selection.py", line 193, in _set_up_model
    **sequence_tagger_params)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/flair/models/sequence_tagger_model.py", line 115, in __init__
    rnn_input_dim: int = self.embeddings.embedding_length
AttributeError: 'tuple' object has no attribute 'embedding_length'

Replacing the parameter selector for the following example in the tutorial:

from flair.hyperparameter.param_selection import TextClassifierParamSelector, OptimizationValue

# create the parameter selector
param_selector = TextClassifierParamSelector(
    corpus, 
    False, 
    'resources/results', 
    'lstm',
    max_epochs=50, 
    training_runs=3,
    optimization_value=OptimizationValue.DEV_SCORE
)

# start the optimization
param_selector.optimize(search_space, max_evals=100)

It yields the following stack trace:

(py36) ags@ds-vm:~/notebooks/notebooks$ python a.py
2019-01-07 12:23:49,781 Reading data from /home/ags/.flair/datasets/conll_03
2019-01-07 12:23:49,782 Train: /home/ags/.flair/datasets/conll_03/eng.train.openNLP
2019-01-07 12:23:49,782 Dev: /home/ags/.flair/datasets/conll_03/eng.testa
2019-01-07 12:23:49,782 Test: /home/ags/.flair/datasets/conll_03/eng.testb.openNLP
2019-01-07 12:23:56,953 ----------------------------------------------------------------------------------------------------
2019-01-07 12:23:56,953 Evaluation run: 1
2019-01-07 12:23:56,953 Evaluating parameter combination:
2019-01-07 12:23:56,953         dropout: 0.23348604057524586
2019-01-07 12:23:56,953         embeddings: /home/ags/.flair/embeddings/lm-news-english-forward-v0.2rc.pt,/home/ags/.flair/embeddings/lm-news-english-backward-v0.2rc.pt
2019-01-07 12:23:56,953         hidden_size: 32
2019-01-07 12:23:56,953         learning_rate: 0.15
2019-01-07 12:23:56,953         mini_batch_size: 8
2019-01-07 12:23:56,953         rnn_layers: 1
2019-01-07 12:23:56,953 ----------------------------------------------------------------------------------------------------
2019-01-07 12:23:57,122 ----------------------------------------------------------------------------------------------------
2019-01-07 12:23:57,122 Training run: 1
2019-01-07 12:23:57,400 ----------------------------------------------------------------------------------------------------
2019-01-07 12:23:57,400 Evaluation method: MICRO_F1_SCORE
2019-01-07 12:23:57,401 ----------------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "a.py", line 36, in <module>
    param_selector.optimize(search_space, max_evals=100)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/flair/hyperparameter/param_selection.py", line 147, in optimize
    best = fmin(self._objective, search_space, algo=tpe.suggest, max_evals=max_evals)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/hyperopt/fmin.py", line 385, in fmin
    rval.exhaust()
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/hyperopt/fmin.py", line 244, in exhaust
    self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/hyperopt/fmin.py", line 218, in run
    self.serial_evaluate()
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/hyperopt/fmin.py", line 137, in serial_evaluate
    result = self.domain.evaluate(spec, ctrl)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/hyperopt/base.py", line 840, in evaluate
    rval = self.fn(pyll_rval)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/flair/hyperparameter/param_selection.py", line 98, in _objective
    **training_params)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/flair/trainers/trainer.py", line 150, in train
    loss = self.model.forward_loss(batch)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/flair/models/text_classification_model.py", line 151, in forward_loss
    return self._calculate_loss(scores, sentences)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/flair/models/text_classification_model.py", line 202, in _calculate_loss
    return self._calculate_single_label_loss(scores, sentences)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/flair/models/text_classification_model.py", line 240, in _calculate_single_label_loss
    return self.loss_function(label_scores, self._labels_to_indices(sentences))
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 904, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/torch/nn/functional.py", line 1970, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/data/anaconda/envs/py36/lib/python3.6/site-packages/torch/nn/functional.py", line 1788, in nll_loss
    .format(input.size(0), target.size(0)))
ValueError: Expected input batch_size (8) to match target batch_size (0).

Environment (please complete the following information):

• GNU/Linux
• flair-0.4.0

Edit 1: grammar

[alanakbik]

Thanks for reporting this!

I can reproduce the error and will take a closer look!

[alanakbik]

The list of embeddings is causing the problem since the SequenceTagger expects one Embedding class object and not a list. So if you want to use a combination of several embeddings (which is always good :)), you should pass a StackedEmbeddings object.

You can fix this by changing the line:

search_space.add(Parameter.EMBEDDINGS, hp.choice, options=[
    [FlairEmbeddings('news-forward'), FlairEmbeddings('news-backward')]
])

to

search_space.add(Parameter.EMBEDDINGS, hp.choice, options=[
    StackedEmbeddings([FlairEmbeddings('news-forward'), FlairEmbeddings('news-backward')])
])

Hope this helps!

[adgsantos]

Thank you, it's working now!

[iamyihwa]

Hi guys, I got the same error, and
replaced the list of embeddings into StackedEmbeddings.
I

However now I get error that says
'TypeError: 'StackedEmbeddings' object is not iterable'

Since my first error was corresponding to the one that started this thread, I decided to continue on this thread.

I am using master branch (that I got last week).

from flair.embeddings import StackedEmbeddings
#search_space.add(Parameter.EMBEDDINGS, hp.choice, options = [document_embeddings])
search_space.add(Parameter.EMBEDDINGS, hp.choice, options = [
    StackedEmbeddings([ WordEmbeddings('en-twitter-glove'), FlairEmbeddings('mix-forward'), FlairEmbeddings('mix-backward')])
])
from flair.hyperparameter.param_selection import TextClassifierParamSelector, OptimizationValue

#create the parameter selector
param_selector = TextClassifierParamSelector(
    corpus, 
    False, 
    'resources/results', 
    'lstm', 
    max_epochs = 50, 
    training_runs = 3, 
    optimization_value = OptimizationValue.DEV_SCORE)

#start the optimization 
param_selector.optimize(search_space, max_evals = 100)

This is the error I get

2019-01-17 16:27:44,611 ----------------------------------------------------------------------------------------------------
2019-01-17 16:27:44,612 Evaluation run: 1
2019-01-17 16:27:44,612 Evaluating parameter combination:
2019-01-17 16:27:44,613     dropout: 0.4871168310822357
2019-01-17 16:27:44,613     embeddings: StackedEmbeddings [/home/taiger/ner-flair-20181105/flair/embeddings/glove-twitter/glove.twitter.27B.200d/glove.twitter.27B.200d.txt.gensim,/home/taiger/.flair/embeddings/lm-mix-english-forward-v0.2rc.pt,/home/taiger/.flair/embeddings/lm-mix-english-backward-v0.2rc.pt]
2019-01-17 16:27:44,613     hidden_size: 512
2019-01-17 16:27:44,614     learning_rate: 0.05
2019-01-17 16:27:44,614     mini_batch_size: 16
2019-01-17 16:27:44,614     rnn_layers: 2
2019-01-17 16:27:44,615 ----------------------------------------------------------------------------------------------------
2019-01-17 16:27:44,646 ----------------------------------------------------------------------------------------------------
2019-01-17 16:27:44,646 Training run: 1
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-30-2f139a9a5978> in <module>
      1 #start the optimization
----> 2 param_selector.optimize(search_space, max_evals = 100)

~/flair-master-branch/flair/flair/hyperparameter/param_selection.py in optimize(self, space, max_evals)
    145     def optimize(self, space: SearchSpace, max_evals=100):
    146         search_space = space.search_space
--> 147         best = fmin(self._objective, search_space, algo=tpe.suggest, max_evals=max_evals)
    148 
    149         log_line(log)

~/miniconda3/envs/flair-master-branch/lib/python3.7/site-packages/hyperopt/fmin.py in fmin(fn, space, algo, max_evals, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin, points_to_evaluate, max_queue_len)
    383                     max_queue_len=max_queue_len)
    384     rval.catch_eval_exceptions = catch_eval_exceptions
--> 385     rval.exhaust()
    386     if return_argmin:
    387         return trials.argmin

~/miniconda3/envs/flair-master-branch/lib/python3.7/site-packages/hyperopt/fmin.py in exhaust(self)
    242     def exhaust(self):
    243         n_done = len(self.trials)
--> 244         self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
    245         self.trials.refresh()
    246         return self

~/miniconda3/envs/flair-master-branch/lib/python3.7/site-packages/hyperopt/fmin.py in run(self, N, block_until_done)
    216             else:
    217                 # -- loop over trials and do the jobs directly
--> 218                 self.serial_evaluate()
    219 
    220             if stopped:

~/miniconda3/envs/flair-master-branch/lib/python3.7/site-packages/hyperopt/fmin.py in serial_evaluate(self, N)
    135                 ctrl = base.Ctrl(self.trials, current_trial=trial)
    136                 try:
--> 137                     result = self.domain.evaluate(spec, ctrl)
    138                 except Exception as e:
    139                     logger.info('job exception: %s' % str(e))

~/miniconda3/envs/flair-master-branch/lib/python3.7/site-packages/hyperopt/base.py in evaluate(self, config, ctrl, attach_attachments)
    838                 memo=memo,
    839                 print_node_on_error=self.rec_eval_print_node_on_error)
--> 840             rval = self.fn(pyll_rval)
    841 
    842         if isinstance(rval, (float, int, np.number)):

~/flair-master-branch/flair/flair/hyperparameter/param_selection.py in _objective(self, params)
     85             log.info(f'Training run: {i + 1}')
     86 
---> 87             model = self._set_up_model(params)
     88 
     89             training_params = {key: params[key] for key in params if key in TRAINING_PARAMETERS}

~/flair-master-branch/flair/flair/hyperparameter/param_selection.py in _set_up_model(self, params)
    227 
    228         if self.document_embedding_type == 'lstm':
--> 229             document_embedding = DocumentLSTMEmbeddings(**embdding_params)
    230         else:
    231             document_embedding = DocumentPoolEmbeddings(**embdding_params)

~/flair-master-branch/flair/flair/embeddings.py in __init__(self, embeddings, hidden_size, rnn_layers, reproject_words, reproject_words_dimension, bidirectional, dropout, word_dropout, locked_dropout)
   1372         super().__init__()
   1373 
-> 1374         self.embeddings: StackedEmbeddings = StackedEmbeddings(embeddings=embeddings)
   1375 
   1376         self.reproject_words = reproject_words

~/flair-master-branch/flair/flair/embeddings.py in __init__(self, embeddings, detach)
    102 
    103         # IMPORTANT: add embeddings as torch modules
--> 104         for i, embedding in enumerate(embeddings):
    105             self.add_module('list_embedding_{}'.format(i), embedding)
    106 

TypeError: 'StackedEmbeddings' object is not iterable`

[tabergma]

You are optimizing a TextClassifier instead of a SequenceTagger, thus the input is different. A TextClassifier just expect a list of word embeddings, not a StackedEmbedding.

So, instead of

search_space.add(Parameter.EMBEDDINGS, hp.choice, options = [
    StackedEmbeddings([ WordEmbeddings('en-twitter-glove'), FlairEmbeddings('mix-forward'), FlairEmbeddings('mix-backward')])
])

do

search_space.add(Parameter.EMBEDDINGS, hp.choice, options = [
    [ WordEmbeddings('en-twitter-glove'), FlairEmbeddings('mix-forward'), FlairEmbeddings('mix-backward')]
])

[iamyihwa]

After replacing it with

search_space.add(Parameter.EMBEDDINGS, hp.choice, options = [
    [ WordEmbeddings('en-twitter-glove'), FlairEmbeddings('mix-forward'), FlairEmbeddings('mix-backward')] 
])

I get original error again.

ValueError: Expected input batch_size (32) to match target batch_size (83).

[iamyihwa]

The issue of mine was solved. I had a multi label dataset, and forgot to set the multi_label = True in TextClassifierParamSelector.