Ignite: SSIM Metric

[ydcjeff]

🚀 Feature

I would like to request to have SSIM metric in ignite. Since it has a few steps to implement on your own when you want to use in your project/paper implementation, I thought it's good to have OOTB in ignite and supported by the community.

I would like to implement this and if I am allowed, under which module/sub-module should my implementation go in?

Here's my current implementation.

[vfdev-5]

@ydcjeff thanks for the feature request ! Yes, I agree that it would be good to have it in Ignite !

I would like to implement this and if I am allowed, under which module/sub-module should my implementation go in?

Sure, contributions are welcome and highly appreciated !
Normally, all metrics we put into ignite/metrics/ folder: e.g. ignite/metrics/ssim.py. Let me check your implementation and we can discuss ignite's implementation details here and then you can send a PR. Does this sound good for you ?

[vfdev-5]

@ydcjeff looking to the implementation under the link I'm wondering whether we can compute this metric in online manner:

ssim = SSIM()
ssim.update(y_preds1, y1)  # batch 1
ssim.update(y_preds2, y2) # batch 2
...
ssim_index = ssim.compute()
print(ssim_index)

?

[ydcjeff]

@vfdev-5 sure, i will keep align with other metrics implementation.
The included link is just plain ssim implementation, haven't brought to the ignite style.
so the final api will look like this?

ssim = SSIM()
ssim.update(y_preds1, y1)  # batch 1
ssim.update(y_preds2, y2) # batch 2
...
ssim_index = ssim.compute()
print(ssim_index)

[vfdev-5]

@ydcjeff yes, basically, we implement those methods for a metric. Metric API can be used either like that or with an engine using attach() method like in the examples here : https://pytorch.org/ignite/metrics.html#ignite-metrics

[ydcjeff]

okay, can I now start implementing with update, compute method first? attach() will come after that.

[vfdev-5]

Yes, let's start with update/compute and attach is automatically provided (see here: https://pytorch.org/ignite/metrics.html#how-to-create-a-custom-metric)

[vfdev-5]

Actually, I'm still wondering how to compute SSIM in online manner. What kind of temporary updatable data we need to store before computing the final metric...
The most intuitive and not optimized at all method is to store all prediction and target images and then compute SSIM on all these data (and can get OOM etc).
So, probably, we can precompute something on updates and use this precomputed info to compute the final metric. So, do you know how can it be done ?
This can be also important to understand in order to adapt the metric to distributed configuration...

[ydcjeff]

how do you mean by the "online manner"?
for the computation, i will need some time to check.

[vfdev-5]

how do you mean by the "online manner"?

See above https://github.com/pytorch/ignite/issues/1213#issuecomment-662425049 :)

[vfdev-5]

What is the shape of ssim_idx here : https://gist.github.com/ydcjeff/ae94ef310cdf269353400aba7e65aae1#file-ssim-py-L109 for a batch of predictions and targets of shape (4, 3, 32, 32) ?

[ydcjeff]

Sorry for the inconvenience. now i get it.

What is the shape of ssim_idx here : https://gist.github.com/ydcjeff/ae94ef310cdf269353400aba7e65aae1#file-ssim-py-L109 for a batch of predictions and targets of shape (4, 3, 32, 32) ?

It would be (4, 3, 22, 22)

[vfdev-5]

Sorry for the inconvenience. now i get it.

@ydcjeff no problems at all :)

OK, if ssim_idx is of the shape (4, 3, 22, 22) and final metric probably can be seen as a mean of 4 mean ssim indices of shape (4, ) where we just averaged C, H, W dimensions. So, we can then just compute final ssim_idx as an average of "batchwise" ssim idx (of shape (B, )).

I think, the metric can be implemented as :

• on every update: we compute batchwise ssim idx as torch.mean(ssim_idx, dim=(1, 2, 3)) and sum up the result with an internal accumulator _batch_ssim_idx (initialized by 0) and count the number of images seen _num_samples.
• on compute: we just devide _batch_ssim_idx / _num_samples.

What do you think ?

[ydcjeff]

yep, i also thought of that way

[vfdev-5]

yep, i also thought of that way

OK, so, please, let's work this inside a PR. Could you please send one ?

[ydcjeff]

Okay 👌 I will send one