Neural-style: preprocess image for vgg_net (BGR or RGB)?

[euwern]

I notice there are two different ways to preprocess image before feeding in the data to vgg_net:

May I know which is correct? Please advise.

In jcjohnson's neural style: (using VGG_ILSVRC_19_layers.caffemodel)
image is converted to BGR before substracting the mean_pixel
https://github.com/jcjohnson/neural-style/blob/be42a12bc9f852596fbf37ea9d25b0139f66cfe6/neural_style.lua

function preprocess(img)
  local mean_pixel = torch.DoubleTensor({103.939, 116.779, 123.68})
  local perm = torch.LongTensor{3, 2, 1}
  img = img:index(1, perm):mul(256.0)
  mean_pixel = mean_pixel:view(3, 1, 1):expandAs(img)
  img:add(-1, mean_pixel)
  return img
end

In karpathy's neural talk 2: (using: VGG_ILSVRC_16_layers.caffemodel)
image is not converted to BGR before substracting the mean_pixel
https://github.com/karpathy/neuraltalk2/blob/3c81602809b8b9e5bd3e9e213bf955986485dda7/misc/net_utils.lua

function net_utils.prepro(imgs, data_augment, on_gpu)
  assert(data_augment ~= nil, 'pass this in. careful here.')
  assert(on_gpu ~= nil, 'pass this in. careful here.')

  local h,w = imgs:size(3), imgs:size(4)
  local cnn_input_size = 224

  -- cropping data augmentation, if needed
  if h > cnn_input_size or w > cnn_input_size then 
    local xoff, yoff
    if data_augment then
      xoff, yoff = torch.random(w-cnn_input_size), torch.random(h-cnn_input_size)
    else
      -- sample the center
      xoff, yoff = math.ceil((w-cnn_input_size)/2), math.ceil((h-cnn_input_size)/2)
    end
    -- crop.
    imgs = imgs[{ {}, {}, {yoff,yoff+cnn_input_size-1}, {xoff,xoff+cnn_input_size-1} }]
  end

  -- ship to gpu or convert from byte to float
  if on_gpu then imgs = imgs:cuda() else imgs = imgs:float() end

  -- lazily instantiate vgg_mean
  if not net_utils.vgg_mean then
    net_utils.vgg_mean = torch.FloatTensor{123.68, 116.779, 103.939}:view(1,3,1,1) -- in RGB order
  end
  net_utils.vgg_mean = net_utils.vgg_mean:typeAs(imgs) -- a noop if the types match

  -- subtract vgg mean
  imgs:add(-1, net_utils.vgg_mean:expandAs(imgs))

  return imgs
end

[jcjohnson]

Both VGG-16 and VGG-19 were trained using Caffe, and Caffe uses OpenCV to load images which uses BGR by default, so both VGG models are expecting BGR images.

In neural-style we swap images from RGB to BGR when reading them from disk; neuraltalk2 instead swaps the first-layer filters of the CNN so that it expects RGB images:

https://github.com/karpathy/neuraltalk2/blob/master/misc/net_utils.lua#L26

[euwern]

thank you for explaining the difference between both projects :) really appreciate your quick reply.