Ludwig: Possible bug with calculation of jaccard similarity

[jenishah]

Firstly, thanks a lot for this wonderful library and also for adding transformers in the latest release.

Describe the bug
I trained a model for multi-label classification. After training, when I compute the jaccard similarity on predictions on test set, by calculating intersection over union, the score does not match with the results given by the library. test_statistics reported jaccard of 0.6661, but I calculated it to be 0.5587. This issue was not there in the previous version.

To Reproduce
Steps to reproduce the behavior:

1. Download train_data.csv from link and test_data.csv from link and place them in data dir.

1. Copy

training:
    epochs: 1
    validation_field: labels
    validation_measure: jaccard
input_features:
    -
        name: comment_text
        type: sequence
        sequence_length_limit: 128
        representation: dense
        lowercase: true
        embedding_size: 256
        cell_type: lstm
        reduce_output: null
        num_layers: 1
        bidirectional: true

output_features:
    -
        name: labels
        type: set
        validation_field: jaccard_index

as model_definition.yaml

1. Run ludwig experiment -rs 42 --training_set data/train_data.csv \ --test_set data/test_data.csv --data_format csv -cf model_definition.yaml

2. After one epoch, in order to calculate the jaccard score, run

import pandas as pd
import csv
def compute_jaccard():

    # read test file
    df_test = pd.read_csv("data/test_data.csv")
    true_labels = list(df_test["labels"])

    # read predicted labels
    pred_labels = []
    with open("results/experiment_run/labels_predictions.csv") as csvfile:
        label_reader = csv.reader(csvfile, delimiter=',')
        for row in label_reader:
            all_sectors = ' '.join(row)
            pred_labels.append(all_sectors)

    # compute jaccard similiarity
    list_jaccard = []
    for str_true, str_pred in zip(true_labels, pred_labels):
        set_true = set(str_true.split())
        set_pred = set(str_pred.split())
        tp = len(set_true.intersection(set_pred))
        union = len(set_true.union(set_pred))
        list_jaccard.append(tp / union)

    jaccard = sum(list_jaccard) / len(list_jaccard)
    return jaccard

print(compute_jaccard())

Environment :

• OS: MacOS Mojave
• Version: 10.14.16
  
  
  
  • Python version: 3.7.3
  
  
  • Ludwig version: 0.3
  
  

Additional context
I used the same logic to calculate jaccard with ludwig version 0.2.2.8, but this issue is not there.

[w4nderlust]

Thank you for the report. We indeed modified the calculation of the jaccard index in v0.3. You example will help us determine if we introduced an error.
Let us look into this.
Maybe @jimthompson5802 can help with this as he implemented the current calculation.

[jimthompson5802]

@w4nderlust got it...let me take a look.

[jimthompson5802]

@jenishah Thank you for providing complete and detailed description of the issue. I was able to use the code and data you provided to reproduce the issue on my side. I'm still in the process of looking into the root cause. One thing I can relate. Instead of manually calculating the metric, Ludwig is using the keras metric class MeanIOU in v0.3. Right now I'm looking to assess how the difference comes about. This may take me a day or two for this work.

[w4nderlust]

We merged the PR #979 . We figured out that the notion of MeanIoU in TF2 is different from our previous calculation of the jaccard inde (which was the same as yours), so we re-implemented the old calculations, and added a test that uses some of your outputs to test the calculation is correct. We should be good to go, please test it by installing Ludwig from master and let us know if there are still issues. Closing in the meantime.

[jenishah]

I checked and it's giving correct results. Thank you very much for this quick fix! :)