Keras-yolo3: PIL and OpenCV letterbox produce different result

[laihaotao]

Hello,

I try to replace the letterbox() functions in yolo3/utils.py. But I can't get the same or closed result. Can someone give some hints what may be the problem?

I noticed that in PIL image.size will give width and height and in OpenCV image.shape will give height and width, already take them into consideration.

The output I got for the same image

PIL (which is good)
person 0.98 (163.9247, 90.553734) (354.22302, 345.3709)

OpenCV (not as expected)
person 0.95 (161.36316, 82.50816) (374.4994, 357.61987)

the original code:

def letterbox_image(image, size):
    '''resize image with unchanged aspect ratio using padding'''
    iw, ih = image.size
    w, h = size
    scale = min(w/iw, h/ih)
    nw = int(iw*scale)
    nh = int(ih*scale)

    image = image.resize((nw,nh), Image.BICUBIC)
    new_image = Image.new('RGB', size, (128,128,128))
    new_image.paste(image, ((w-nw)//2, (h-nh)//2))
    return new_image

My OpenCV implementation:

import cv2 as cv
def letterbox_image1(image, expected_size):
    ih, iw, _ = image.shape
    eh, ew = expected_size
    scale = min(eh / ih, ew / iw)
    nh = int(ih * scale)
    nw = int(iw * scale)

    image = cv.resize(image, (nw, nh), interpolation=cv.INTER_CUBIC)
    new_img = np.full((eh, ew, 3), 128, dtype='uint8')
    # fill new image with the resized image and centered it
    new_img[(eh - nh) // 2:(eh - nh) // 2 + nh,
            (ew - nw) // 2:(ew - nw) // 2 + nw,
            :] = image.copy()
    return new_img

Thanks.

[GeHongpeng]

@laihaotao
Hi,

Actually, You have Two things to take them into consideration.

1. The difference of "RGB" and "BGR in the Pillow and Opencv.
2. The size(w, h) and expected_size(h, w) in letterbox_image and letterbox_image1 have difference order.

Did you use them correctly in the pre-processing step?

[laihaotao]

@GeHongpeng
Hi,
Thanks for your reply. I can't manage to implement directly with OpenCV. But what I figure out is that, if I transfer the result image from OpenCV's letterbox_image function to PIL's resulted image. It is worked.

image_opencv = letterbox1_image(image, (416, 416))
image_pil = Image.fromarray(cv2.cvtColor(image_opencv, cv.COLOR_BGR2RBG))

Obviously, it is not what I want. Since the following code in yolo.py, it direclty transfer PIL image to Numpy ndarray using:

image_data = np.array(letterbox_image_pil_format, dtype='float32')
image_data = np.array(letterbox_image_opencv_format, dtype='float32') # this line produce the wrong reslut

Do you have any idea, how I can achieve the same result, if I have an image in OpenCV format.

[gouthamvgk]

def letterbox_image(image, size):
    '''resize image with unchanged aspect ratio using padding'''
    iw, ih = image.shape[0:2][::-1]
    w, h = size
    scale = min(w/iw, h/ih)
    nw = int(iw*scale)
    nh = int(ih*scale)
    image = cv2.resize(image, (nw,nh), interpolation=cv2.INTER_CUBIC)
    new_image = np.zeros((size[1], size[0], 3), np.uint8)
    new_image.fill(128)
    dx = (w-nw)//2
    dy = (h-nh)//2
    new_image[dy:dy+nh, dx:dx+nw,:] = image
    return new_image

This cv2 code works correctly as the alternative to PIL

[laihaotao]

The following is my solution to using only OpenCV

def letterbox_image(image, expected_size):
  ih, iw, _ = image.shape
  eh, ew = expected_size
  scale = min(eh / ih, ew / iw)
  nh = int(ih * scale)
  nw = int(iw * scale)

  image = cv2.resize(image, (nw, nh), interpolation=cv2.INTER_CUBIC)
  new_img = np.full((eh, ew, 3), 128, dtype='uint8')
  # fill new image with the resized image and centered it
  new_img[(eh - nh) // 2:(eh - nh) // 2 + nh,
          (ew - nw) // 2:(ew - nw) // 2 + nw,
          :] = image.copy()
  return new_img

When using the above method:

boxed_image = letterbox_image(image, tuple(self.model_image_size))
boxed_image_ = cv2.cvtColor(boxed_image, cv.COLOR_BGR2RGB)
image_data = np.array(boxed_image_, dtype='float32')

[kevintkoi]

@laihaotao Did this opencv implementation increased the speed of detection for you?