Ignite: ValueError: Targets should be binary (0 or 1).

[etetteh]

I am facing this issue.
ValueError: For binary cases, y must be comprised of 0's and 1's.

The task is multilabel, and I am converting to binary with:

def custom_prepare_batch(batch, device, non_blocking):
    x, y = batch["img"], batch["lab"]
    return (
        convert_tensor(x, device=device, non_blocking=non_blocking),
        convert_tensor(y, device=device, non_blocking=non_blocking),
    )


### model update function
def process_function(engine, batch):
    model.train()

    images, targets = custom_prepare_batch(batch, device=device, non_blocking=True)

    optimizer.zero_grad()
    outputs = model(images)

    for task in range(targets.shape[1]):
        task_output = outputs[:,task]
        task_target = targets[:,task]
        mask = ~torch.isnan(task_target)
        task_output = task_output[mask]
        task_target = task_target[mask]
        if len(task_target) > 0:
            if agreement_threshold > 0.0:
                mean_loss, masks = and_mask_utils.get_grads(
                    agreement_threshold=agreement_threshold,
                    batch_size=1,
                    loss_fn=criterion,
                    n_agreement_envs=batch_size,
                    params=optimizer.param_groups[0]['params'],
                    output=task_output,
                    target=task_target,
                    method="and_mask",
                    scale_grad_inverse_sparsity=scale_grad_inverse_sparsity,
                )
            else:
                mean_loss = criterion(y_pred, y)
                mean_loss.backward()


    optimizer.step()

    return {
#         "batchloss": mean_loss.item()
    }

I used this from the ignite docs:

def activated_output_transform(output):
    y_pred, y = output
    y_pred = torch.sigmoid(y_pred)
    return y_pred, y

metrics = {
        "roc_auc": ROC_AUC(activated_output_transform),
     }

And, now I am getting
ValueError: Targets should be binary (0 or 1).

[vfdev-5]

Hi @etetteh, the error tells you that target should be binary. If your target is plain labels from 0 to N-1 you can use ignite.utils.to_onehot to encode the target as binary tensor of shape (batch_size, N) :

from ignite.utils import to_onehot


def activated_output_transform(output):
    y_pred, y = output
    y_pred = torch.sigmoid(y_pred)
    # here we assume that y_pred is (batch_size, num_classes)
    y_ohe = to_onehot(y, num_classes=y_pred.shape[1])
    return y_pred, y_ohe

metrics = {
    "roc_auc": ROC_AUC(activated_output_transform),
}

What kind of task you are learning a model on, what kind of target do you have and which metrics you think to compute ?

[etetteh]

Thanks for the help.
Is the function

Hi @etetteh, the error tells you that target should be binary. If your target is plain labels from 0 to N-1 you can use ignite.utils.to_onehot to encode the target as binary tensor of shape (batch_size, N) :

from ignite.utils import to_onehot


def activated_output_transform(output):
    y_pred, y = output
    y_pred = torch.sigmoid(y_pred)
    # here we assume that y_pred is (batch_size, num_classes)
    y_ohe = to_onehot(y, num_classes=y_pred.shape[1])
    return y_pred, y

metrics = {
    "roc_auc": ROC_AUC(activated_output_transform),
}

What kind of task you are learning a model on, what kind of target do you have and which metrics you think to compute ?

Is the function supposed to return y_pred and y or y_pred and y_ohe. Running
y_ohe = to_onehot(y, num_classes=out.shape[1])
generates the following error

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-13-b23c86b6f523> in <module>
----> 1 y_ohe = to_onehot(tag, num_classes=out.shape[1])

~/.conda/envs/mila/lib/python3.8/site-packages/ignite/utils.py in to_onehot(indices, num_classes)
     56     """
     57     onehot = torch.zeros(indices.shape[0], num_classes, *indices.shape[1:], dtype=torch.uint8, device=indices.device)
---> 58     return onehot.scatter_(1, indices.unsqueeze(1), 1)
     59 
     60 

IndexError: scatter_(): Expected dtype int64 for index.

[vfdev-5]

Thanks, fixed y->y_ohe. What are your targets y, type, datatype etc ?

[etetteh]

My targets are dtype=torch.float64

[vfdev-5]

OK, I see, and what do they represent ? It is unnatural to produce one-hot encoding from float target. I'm a bit confused why you'd like to compute roc_auc with float target... what is the task you are training ?

[etetteh]

I am working on multilabel images.
Each image contains 18 targets, where some of the targets are nan.

optimizer.zero_grad()
    outputs = model(images)

    for task in range(targets.shape[1]):
        task_output = outputs[:,task]
        task_target = targets[:,task]
        mask = ~torch.isnan(task_target)
        task_output = task_output[mask]
        task_target = task_target[mask]

My loss is to be computed on task_output and task_target, and they both have sizes of 64 (which is the batch_size)

[vfdev-5]

So, to confirm, for a given image there is a target represented as a integer number, right ?
or multilabel as multiple tags for a single image, the target is, for example, y=[0, 0, 1, 1, 0, 1] ?
Anyway, I do not understand why your target is float and not torch.int64. Any reason for that ?

EDIT: OK, I see, as there are nans, target is type promoted to float64

[vfdev-5]

I think I understand better the question and the problem. So, you would like to compute roc_auc metric for each task separately or considering all the tasks somehow ?

[etetteh]

Great. Compute it for each task

[vfdev-5]

Right, in this case, you have to split the target with output_transform function and create 16 metrics.
Maybe, something like that could work in your case:

from functools import partial
import torch

from ignite.utils import to_onehot
from ignite.engine import Engine
from ignite.contrib.metrics import ROC_AUC


torch.manual_seed(0)
num_tasks = 16
batch_size = 4
roc_auc_per_task = {}


def ot_per_task(output, task_index):
    y_pred, y = output
    # Remove NaNs with mask
    task_output = y_pred[:, task_index]
    task_target = y[:,task_index]
    mask = ~torch.isnan(task_target)
    task_output = torch.sigmoid(task_output[mask])
    task_target = task_target[mask].long()
    return task_output, task_target


for i in range(num_tasks):
    roc_auc_per_task["auc_{}".format(i)] = ROC_AUC(output_transform=partial(ot_per_task, task_index=i))


def processing_fn(e, b):
    # Let's generate predictions and targets
    y_true = torch.randint(0, 2, size=(batch_size, num_tasks)).double()
    # add nans
    for _ in range(int(batch_size * num_tasks * 0.33)):
        i = torch.randint(0, batch_size, size=(1, )).item()
        j = torch.randint(0, num_tasks, size=(1, )).item()
        y_true[i, j] = float("nan")
    y_preds = (torch.rand(batch_size, num_tasks) - 0.5) * 10    
    return y_preds, y_true

evaluator = Engine(processing_fn)

# Add metrics
for n, m in roc_auc_per_task.items():
    m.attach(evaluator, name=n)

evaluator.run([0, 1, 3, 4, 5])
evaluator.state.metrics

HTH

[etetteh]

Thank you so much. This is exactly what my task looks like, and I can work with this.

[vfdev-5]

@etetteh great ! Glad that it could help :)
I close the issue as solved, feel free to reopen it if need more support on this topic. If you have other questions, please open another issue or use Discussions.

[etetteh]

Is there an easier way to compute the roc_auc for all the tasks as well

[vfdev-5]

Is there an easier way to compute the roc_auc for all the tasks as well

I'm not sure if there is an easy way to do that... I can think of something like to replace target nans with 0 and set outputs also as 0 predicted. Such that metric is not penalized by nans...