Flair: TypeError: classification_report() got an unexpected keyword argument 'zero_division'

[eh-93]

Describe the bug
Just want to report a bug with master, upon the first epoch completing the classification report fails with the following exception: TypeError: classification_report() got an unexpected keyword argument 'zero_division'

Full stack trace and some context:

2020-06-27 12:27:53,241 ----------------------------------------------------------------------------------------------------
2020-06-27 12:27:53,241 Device: cuda:0
2020-06-27 12:27:53,241 ----------------------------------------------------------------------------------------------------
2020-06-27 12:27:53,241 Embeddings storage mode: none
2020-06-27 12:27:53,241 ----------------------------------------------------------------------------------------------------
2020-06-27 12:29:29,176 epoch 1 - iter 76/760 - loss 0.93225607 - samples/sec: 12.69
2020-06-27 12:31:06,572 epoch 1 - iter 152/760 - loss 0.87678470 - samples/sec: 12.49
2020-06-27 12:32:47,559 epoch 1 - iter 228/760 - loss 0.83017487 - samples/sec: 12.05
2020-06-27 12:34:24,617 epoch 1 - iter 304/760 - loss 0.80174087 - samples/sec: 12.54
2020-06-27 12:36:02,725 epoch 1 - iter 380/760 - loss 0.78552397 - samples/sec: 12.40
2020-06-27 12:37:40,569 epoch 1 - iter 456/760 - loss 0.76393380 - samples/sec: 12.46
2020-06-27 12:39:22,314 epoch 1 - iter 532/760 - loss 0.74449620 - samples/sec: 11.97
2020-06-27 12:41:06,882 epoch 1 - iter 608/760 - loss 0.73161619 - samples/sec: 11.71
2020-06-27 12:42:55,085 epoch 1 - iter 684/760 - loss 0.71728993 - samples/sec: 11.35
2020-06-27 12:44:40,769 epoch 1 - iter 760/760 - loss 0.70415614 - samples/sec: 11.68
2020-06-27 12:44:41,984 ----------------------------------------------------------------------------------------------------
2020-06-27 12:44:41,984 EPOCH 1 done: loss 0.7042 - lr 0.1000000
Traceback (most recent call last):
  File "flairx/train_model.py", line 95, in <module>
    trainer.train(output_path,
  File "/home/lib/python3.8/site-packages/flair/trainers/trainer.py", line 426, in train
    dev_eval_result, dev_loss = self.model.evaluate(
  File "/home/lib/python3.8/site-packages/flair/models/text_classification_model.py", line 325, in evaluate
    classification_report = metrics.classification_report(y_true, y_pred, digits=4,
TypeError: classification_report() got an unexpected keyword argument 'zero_division'

To Reproduce
Steps to reproduce the behavior (e.g. which model did you train? what parameters did you use? etc.).

corpus: Corpus = ClassificationCorpus(data_folder)
label_dict = corpus.make_label_dictionary()
word_embeddings = [
    WordEmbeddings('en'),
    FlairEmbeddings('forward'),
    FlairEmbeddings('backward'),
]

DrnnEmbeddings = DocumentRNNEmbeddings(word_embeddings,
                                       hidden_size=hidden_size,
                                       reproject_words=reproject_words,
                                       reproject_words_dimension=reproject_words_dimension,
                                       rnn_type=rnn_type)

document_embeddings: DocumentRNNEmbeddings = DrnnEmbeddings
classifier = TextClassifier(document_embeddings, label_dictionary=label_dict)
trainer = ModelTrainer(classifier, corpus)

trainer.train(output_path,
              learning_rate=learning_rate,
              mini_batch_size=mini_batch_size,
              anneal_factor=anneal_factor,
              patience=patience,
              max_epochs=max_epochs,
              embeddings_storage_mode='none')

Environment (please complete the following information):

• OS: Linux Manjaro
• Version flair-0.5 - but installed from master as of today

[alanakbik]

Hello @eh-93 thanks for reporting this. Could it be that you have an older version of sklearn installed?

[anjani-dhrangadhariya]

I have the same issue.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-161-cea1624cfce4> in <module>
      6 trainer.train('resources/taggers/example-ner',
      7               train_with_dev=True,
----> 8               max_epochs=15)

~/anaconda3/envs/lib/python3.6/site-packages/flair/trainers/trainer.py in train(self, base_path, learning_rate, mini_batch_size, mini_batch_chunk_size, max_epochs, scheduler, anneal_factor, patience, initial_extra_patience, min_learning_rate, train_with_dev, monitor_train, monitor_test, embeddings_storage_mode, checkpoint, save_final_model, anneal_with_restarts, anneal_with_prestarts, batch_growth_annealing, shuffle, param_selection_mode, num_workers, sampler, use_amp, amp_opt_level, eval_on_train_fraction, eval_on_train_shuffle, **kwargs)
    573         # test best model if test data is present
    574         if self.corpus.test:
--> 575             final_score = self.final_test(base_path, mini_batch_chunk_size, num_workers)
    576         else:
    577             final_score = 0

~/anaconda3/envs/lib/python3.6/site-packages/flair/trainers/trainer.py in final_test(self, base_path, eval_mini_batch_size, num_workers)
    621             num_workers=num_workers,
    622             out_path=base_path / "test.tsv",
--> 623             embedding_storage_mode="none",
    624         )
    625 

~/anaconda3/envs/lib/python3.6/site-packages/flair/models/sequence_tagger_model.py in evaluate(self, sentences, out_path, embedding_storage_mode, mini_batch_size, num_workers)
    567         for i in range(len(labels)):
    568             target_names.append(labels.get_item_for_index(i))
--> 569         classification_report = metrics.classification_report(y_true, y_pred, digits=4, target_names=target_names, zero_division=1)
    570 
    571         # get scores

TypeError: classification_report() got an unexpected keyword argument 'zero_division'

[alanakbik]

@someAdjectiveNoun what versions of Flair and scikit learn are you using?

[gbian007]

I'm Having the same Issue, someone have a solution for it?
Flair Version -- 0.5.1
scikit learn Version -- 0.21.3

[anjani-dhrangadhariya]

@someAdjectiveNoun what versions of Flair and scikit learn are you using?

@alanakbik flair 0.5.1 and scikit-learn 0.21.3

[pwcahyo]

i have same error

File "pos_tag.py", line 27, in <module>
    max_epochs=10)
  File "/Users/pwcahyo/anaconda3/envs/de_flask_37/lib/python3.7/site-packages/flair/trainers/trainer.py", line 430, in train
    embedding_storage_mode=embeddings_storage_mode,
  File "/Users/pwcahyo/anaconda3/envs/de_flask_37/lib/python3.7/site-packages/flair/models/sequence_tagger_model.py", line 569, in evaluate
    classification_report = metrics.classification_report(y_true, y_pred, digits=4, target_names=target_names, zero_division=1)
TypeError: classification_report() got an unexpected keyword argument 'zero_division'

i'm using:
flair==0.5.1
scikit-learn==0.23.2

[anjani-dhrangadhariya]

@someAdjectiveNoun what versions of Flair and scikit learn are you using?

Just wanted to check if there is any update on this issue? I would really like to check how FLAIR works with my data.

Thank you in advance for looking into it.

[alanakbik]

Could you try updating to Flair 0.6 to see if the error persists?

[iamjanvijay]

@pwcahyo @someAdjectiveNoun @gbian007 I solved this issue for myself by upgrading to following versions: flair==0.6
scikit-learn==0.23.2

[pwcahyo]

@iamjanvijay still same error. i used flair==0.6 scikit-learn==0.23.2 and Python 3.7

[iamjanvijay]

@pwcahyo I strongly believe that the scikit-learn version you are using isn't 0.23.2. Please check that again.

There's one more option. Go to line 569 in /Users/pwcahyo/anaconda3/envs/de_flask_37/lib/python3.7/site-packages/flair/models/sequence_tagger_model.py, and edit line classification_report = metrics.classification_report(y_true, y_pred, digits=4, target_names=target_names, zero_division=1) to classification_report = metrics.classification_report(y_true, y_pred, digits=4, target_names=target_names).

Let me know if something works for you.

[pwcahyo]

thank you @iamjanvijay for your response, but i'm using scikit-learn 0.23.2.
i solved my error with update all package in conda using conda update --all and then flair can running normally.

[iamjanvijay]

@pwcahyo Great!

Actually metrics.classification_report is a scikit-learn's function call. And in scikit-learn version <=0.21.3 metrics.classification_report wasn't having zero_division param. Thats why I was speculating that your scikit-learn version is <=0.21.3.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.