Pillow: `Pillow` vs `cv2` resize

[TropComplique]

Results of reading and resizing can be different in cv2 and Pilllow. This creates problems when you want to reuse a model (neural network) trained using cv2 with Pillow.

Case 1

Different results of resizing an image.

import cv2
from PIL import Image
import numpy as np

# read images
a = cv2.imread('images/dogs-and-cats.jpg', cv2.IMREAD_COLOR)
b = Image.open('images/dogs-and-cats.jpg')

# they are equal
print((np.asarray(b)[:, :, [2, 1, 0]] == a).all())
# True

# resize images
b = b.resize((300, 300), Image.NEAREST)
a = cv2.resize(a, (300, 300), interpolation=cv2.INTER_NEAREST)

# results of resizing are different, but visually they are equal
b = np.asarray(b)[:, :, [2, 1, 0]]
print((b - a).mean())
# 71.1218444444

Case 2

Different results of reading an image.

import cv2
from PIL import Image
import numpy as np

a = cv2.imread('images/cat_train.jpg', cv2.IMREAD_COLOR)

b = Image.open('images/cat_train.jpg')
b = np.asarray(b)[:, :, [2, 1, 0]]

# results of reading are not equal
print((b == a).all())
# False
print((b == a).mean())
# 0.999996744792

I use: Python 3.6, Pillow 4.2.1, opencv-python 3.3

[wiredfool]

Case 1: Nearest neighbor is a fast, low quality, best effort interpolation. That said, I believe that our tests show our implementation is reasonably correct. a mean difference of 71 would fall into the range of seriously visually different, and probably the result of either a rotation or channel swapping.

Case 2. That appears to be 1 pixel level difference in 10^6 pixels. That's pretty close, and well within what I'd consider reasonable for a lossy format with potentially different implementations and settings on the decoder.

Feel free to point out where you feel that the implementation is incorrect.

[TropComplique]

For case 1 Image.fromarray(a) and Image.fromarray(b) show the same image.
Also, their values are kinda close:

print(a[41:55, 60, :])
# [[255 255 255]
#  [117 124 144]
#  [ 96 117 145]
#  [ 70  90 121]
#  [ 68  88 119]
#  [ 78  96 127]
#  [ 70  88 119]
#  [ 71  87 116]
#  [ 71  89 118]
#  [ 77  98 126]
#  [116 151 185]
#  [135 167 202]
#  [142 170 205]
#  [126 162 200]]
print(b[41:55, 60, :])
# [[225 215 215]
#  [113 120 140]
#  [ 83 105 141]
#  [ 74  95 133]
#  [ 80 102 138]
#  [ 53  68  94]
#  [ 52  65  91]
#  [ 42  49  69]
#  [ 43  52  72]
#  [ 69  86 107]
#  [114 149 183]
#  [132 168 198]
#  [137 170 203]
#  [128 160 196]]

I use this image:
https://github.com/TropComplique/ssd-pytorch/blob/master/images/dogs-and-cats.jpg

[wiredfool]

Which one do you think is correct?

[TropComplique]

I do not know. I am just stating a fact that there is a difference.

[homm]

visually they are equal

Visually they are also significantly different:

Please, look at the sample:

This image with the uniform gradient (from 100% white to 100% black) allows us to find out which pixels are used by each library. This is the same image after resizing to (3, 3). Left is CV2, right is Pillow:

OpenCV uses the topmost left white pixel from the source image, but the bottommost right pixel on the result is too bright. Here are maps of pixels on the source image:

It's clear to me there are some problems with rounding in OpenCV there.

[TropComplique]

Hi. Nice visualization.

But I find it strange that sometimes neural network's predictions
change a lot because of an image resizing method.

[homm]

I'm not familiar with neural network's, but as I understand this is called "overlearning". This is the case when the network shows good results on the test data, but doesn't accept any deviations.

[biaojiang]

I tested that Pillow resize has the same result as Matlab function "resize", The output image size is a little bit difference, maybe Matlab and Pillow use different rounding operation from float to int. But this doesn't matter.
So, it is Skimage and Opencv that have weird resized outputs.

I guess Pillow used an anti-aliasing filter together with down-sampling filter, because by default Matlab will apply an anti-aliasing filter.
I reported this issue to Skimage, their document said it will include anti-aliasing filter in new version.

[shahvi]

Just for completeness, as described at https://stackoverflow.com/questions/21997094/why-opencv-cv2-resize-gives-different-answer-than-matlab-imresize/21998119, Matlab's imresize by default performs antialiasing when downsampling, which is why you see different results.

[wanxinzzz]

In case 1

print((b - a).mean())
# 71.1218444444

This is because the numpy array's dtype is uint8

a = np.array([1], dtype='uint8')
b = np.array([2], dtype='uint8')
a - b
# array([255], dtype=uint8)
# 1 - 2 = -1 = 255 in uint8

So, we can transform it to int16

a_i = np.int16(a)
b_i = np.int16(b)
print(np.abs(b_i - a_i).mean())
# 0.028748148148148148
# which is slight difference

[BarCodeReader]

I tested that Pillow resize has the same result as Matlab function "resize", The output image size is a little bit difference, maybe Matlab and Pillow use different rounding operation from float to int. But this doesn't matter.
So, it is Skimage and Opencv that have weird resized outputs.

I guess Pillow used an anti-aliasing filter together with down-sampling filter, because by default Matlab will apply an anti-aliasing filter.
I reported this issue to Skimage, their document said it will include anti-aliasing filter in new version.

for me PIL BICUBIC downsampling and MatLab imresize gives different result, though the difference is not very big.
but I believe for those people working on image restoration tasks, this small error is big enough to impact their model accuracy.

above image only has PSNR of 57.54...and you can see from the image, the error seems more random instead of having some structure related properties...and I think even with neural network, it may not be able to learn this extra "error"