neural-style 🚀 - Implementing features from the "Controlling Perceptual Factors in Neural Style Transfer" research paper

Ok, I think I have gotten the new -reflectance parameter working, though I don't know what it does: https://github.com/ProGamerGov/neural-style/blob/master/neural_style.lua

Though it seems to alter the output.

ProGamerGov on 11 Feb 2017

Multires without -reflectance: https://i.imgur.com/LvpXgaW.png

Multires with -reflectance: https://i.imgur.com/YIiqsOx.png

The -reflectance command increases the GPU usage.

Content image: https://i.imgur.com/sgLtFDi.png

Style image: https://i.imgur.com/PsXIJLM.jpg

ProGamerGov on 12 Feb 2017

It seems to me that your code inserts the new padding layer after the convolution layer which already has done padding, so that padding is done twice (first with zeroes in nn.SpatialConvolution and the by reflection in nn.SpatialReflectionPadding). It is like first adding an empty border and the another one which acts as if a mirror. It would seem to me that the mirror then only reflects the empty border that was added first.

If you look closely at Gatys' code in https://github.com/leongatys/NeuralImageSynthesis/blob/master/ImageSynthesis.lua#L85-L94 you'll notice that the new padding layer is inserted first, and then the convolution layer without padding.

Your code also increases the size of the layer output, as padding is done twice, which might give size mismatch errors.

htoyryla on 12 Feb 2017

In my previous comment, I overlooked the fact that it is possible to change the layer parameters after the layer has been added to the model. Thus the lines https://github.com/ProGamerGov/neural-style/blob/master/neural_style.lua#L140-L141 in fact remove the padding from the already inserted convolution layer, so the double padding does not happen and the size of the output is not changed.

Thus the main difference between your code and Gatys' is that you do padding after the convolution, while the normal practice is to do padding before convolution.

htoyryla on 12 Feb 2017

@htoyryla

Thus the main difference between your code and Gatys' is that you do padding after the convolution, while the normal practice is to do padding before convolution.

So the reflectance padding works correctly, though I have placed it in the wrong location?

This code here is the convolution: https://github.com/ProGamerGov/neural-style/blob/master/neural_style.lua#L131-L142 ?

ProGamerGov on 12 Feb 2017

And for implementing the masks, Gatys' implementation uses hdf5 files, though Neural-Style does not:

cmd:option('-mask_file', 'path/to/HDF5file', 'Spatial mask to constrain the gradient descent to specific region')

    -- Load mask if specified
    local mask = nil
    if params.mask_file ~= 'path/to/HDF5file' then
        local f = hdf5.open(params.mask_file, 'r')
        mask = f:all()['mask']
        f:close()
        mask = set_datatype(mask, params.gpu)
    end

I have been trying to figure out how to modify the above code for Neural-Style masks, but non of my attempts to replace the hdf5 requirement have worked thus far. Any ideas?

ProGamerGov on 12 Feb 2017

The code you now linked looks better, now the padding is inserted (line #127) before the convolution (line #141). Most of what you have highlighted is NOT the convolution but related to selecting between max and avg pooling. But if you follow the if logic, if the layer is convolution it will be inserted to the model in line 141 of your present code.

I cannot guarantee that it now works but now the padding and convolution come in the correct order.

htoyryla on 12 Feb 2017

"I have been trying to figure out how to modify the above code for Neural-Style masks, but non of my attempts to replace the hdf5 requirement have worked thus far. Any ideas?"

The code you cited does not implement any mask functionality, it only loads a mask from an existing hdf5 file.

htoyryla on 12 Feb 2017

I ran a quick test with the -reflectance option. The change is not particularly obvious at first glance, but it does appear to cause a change. More testing, and different parameter combinations could be needed to farther understand it's affect on artistic outputs.

On the left is the control test with -reflectance false, and on the right is -reflectance true:

Direct link to the comparison: https://i.imgur.com/YGCOCiu.png

False: https://i.imgur.com/0oQNsxl.png

True: https://i.imgur.com/a7fQTLb.png

Command used:

th neural_style.lua -seed 876 -reflectance -num_iterations 1500 -init image -image_size 640 -print_iter 50 -save_iter 50 -content_image examples/inputs/hoovertowernight.jpg -style_image examples/inputs/starry_night.jpg -backend cudnn -cudnn_autotune

ProGamerGov on 12 Feb 2017

Are Gatys' Grad related functions different that Neural-Styles? I'm looking for where the style masks come into play. Or should I be looking at different functions for implementing these features like masks?

ProGamerGov on 12 Feb 2017

From what I can see, luminescence style transfer requires the LUV color space, which unlike YUV, it has no easy to use function in the image library.

Style masks seem to require a modifying deeper levels of the Neural-Style code.

For the independent style_scale control with multiple style images, it seems like we only need a way to disable content loss:

From the research paper:

We initialise the optimisation procedure with the coarse-scale image and omit the content loss entirely, so that the fine-scale texture from the coarse-style image will be fully replaced.

And then a simple sh script similar to multires.sh should do the trick. That runs your style images through Neural-Style first should do the trick, but such a script needs a way to disable content loss.

I am thinking that a parameter like:

cmd:option('-content_loss', true, 'if set to false, content loss will be disabled')

if params.reflectance then 

content loss code

end

@htoyryla Which part of the content loss code should this be implemented on to achieve the desired effect?

https://github.com/ProGamerGov/neural-style/blob/master/neural_style.lua#L461-L497

Or: https://github.com/ProGamerGov/neural-style/blob/master/neural_style.lua#L109

Edit: I figured it out and now the content loss module can be disabled.

Currently testing different parameters alongside the new -content_loss parameter: https://gist.github.com/ProGamerGov/7f3d2b6656e02a7a4a23071bd0999b31

I edited this part of the neural_style.lua script: https://gist.github.com/ProGamerGov/7f3d2b6656e02a7a4a23071bd0999b31#file-neural_style-lua-L148-L151

Though I think that I need to find a way to transfer the color from the intended content image, to this first Neural-Style run with the two style images. Seeing as -init image includes, content as well, maybe I need to add another new parameter, or maybe using -original_color 1 on step two will solve this problem?

Second Edit:

It seems that -content_layers relu1_1,relu2_1 and the default style layers work the best, Though the research paper only specified layers relu1_1 and relu2_1, not whether you should use those values for content or style layers.

ProGamerGov on 13 Feb 2017

I must be missing something when trying to replicate the "Naive scale combination" from here: https://github.com/leongatys/NeuralImageSynthesis/blob/master/ExampleNotebooks/ScaleControl.ipynb

Following the steps on the research paper:

Should result in something like this output that I made running Gatys' iPython code: https://i.imgur.com/boz8PhW.jpg

And the styled style image from his code: https://i.imgur.com/6xEumk0.jpg

But instead I get this:

The styled style image: https://i.imgur.com/30HUeOH.png

And here is the final output: https://i.imgur.com/SWhzMn0.png

I tried this code to create the styled style image: https://gist.github.com/ProGamerGov/53979447d09fe6098d4b00fc8e924109

And then ran:

th neural_style_c.lua -original_colors 1 -output_image out.png -num_iterations 1000 -content_image fig4_content.jpg -style_image out7.png -image_size 640 -save_iter 50 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

The final content image: https://raw.githubusercontent.com/leongatys/NeuralImageSynthesis/master/Images/ControlPaper/fig4_content.jpg

The two style images:

https://raw.githubusercontent.com/leongatys/NeuralImageSynthesis/master/Images/ControlPaper/fig4_style3.jpg

https://raw.githubusercontent.com/leongatys/NeuralImageSynthesis/master/Images/ControlPaper/fig4_style2.jpg

What am I doing wrong here?

ProGamerGov on 13 Feb 2017

Ok, so analyzing the styled style image from Gatys' code:

The outputs have the parameters used, and the values used, in the name:

[scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_hrpt_layer_relu4_1_hrsz_1024_model_norm_pad_ptw_1.0E+05]

I think was used to make this: https://i.imgur.com/6xEumk0.jpg

From another experiment using his code:

cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_Amazing-Nature_3840x2160.jpg_simg_raime.jpg_pt_layer_relu2_1_sz_512_model_norm_pad_sw_2.0E+08_cw_1.0E+05_naive_scalemix.jpg

The enlarged version (I think 1 step multires?):

cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_raime.jpg_simg_Amazing-Nature_3840x2160.jpg_pt_layer_relu2_1_sz_512_hrsz_1024_model_norm_pad_sw_2.0E+08_cw_1.0E+05_naive_scalemix.jpg.filepart

And:

cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_raime.jpg_simg_Amazing-Nature_3840x2160.jpg_pt_layer_relu2_1_sz_512_model_norm_pad_sw_2.0E+08_cw_1.0E+05_naive_scalemix.jpg

The layers used are: relu2_1 and relu4_1

Style weight is: sw_2.0E+08

Content weight is: cw_1.0E+05

The Normalized VGG-19 model is used: model_norm

Not sure what this is: ptw_1.0E+05

Naive Scale mix is the best version, and also the styled style image: naive_scalemix.jpg

Not sure if pt_layer refers to both style_layers and content_layers, or just one of them?

ProGamerGov on 13 Feb 2017

On the subject of Gram Matrices (Leon Gatys said this would be important for transferring features to Neural-Style):

Neural-Style is normalising the Gram Matrices differently, as it additionally divides by the number of features, when compared with Gatys' code. This means that the style loss weights for the different layers in Neural-Style and Gatys' code are a little different:

In a layer l with n_l = 64 features, a style loss weight of 1 in Neural-Style, is a style loss weight of 1/64^2 in Gatys' code.

ProGamerGov on 13 Feb 2017

"Neural-Style is normalising the Gram Matrices differently, as it additionally divides by the number of features, when compared with Gatys' code. This means that the style loss weights for the different layers in Neural-Style and Gatys' code are a little different:

In a layer l with n_l = 64 features, a style loss weight of 1 in Neural-Style, is a style loss weight of 1/64^2 in Gatys' code."

I am not familiar with Gatys's code, but what you wrote is confusing. First you say that Neural_style divides the Gram matrix by the number of features, but in your example you don't do this division.

If Gatys' normalizes by 1/C^2 where C is the number of features, it makes sense to me as the size of the Gram matrix is CxC.

In neural_style, the gram matrix is normalized for style loss in the line https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L534
Here, input:nElements() is not C but CxHxW, where C,H,W are the dimensions of the layer to which the Gram matrix is added, so that in practice neural-style ends up with a smaller value for the normalized style loss than 1/C^2.

Dividing instead by self.G:nElements() would implement division by C^2 so if that's what you want, try it.

I don't know if this use of input:nElement() instead of self.G:nElements() here is intentional or an accident. @jcjohnson ?

There has been an earlier discussion about this division but there was nothing on this in particular: https://github.com/jcjohnson/neural-style/issues/90

PS. I checked the corresponding code in fast-neural-style https://github.com/jcjohnson/fast-neural-style/blob/master/fast_neural_style/GramMatrix.lua#L46-L49 which also normalizes the Gram matrix by 1/(CHW), so I guess this is done on purpose. After all, normalizing by 1/C^2 would favor the lower layers too much.

htoyryla on 13 Feb 2017

I ran a quick test with the -reflectance option. The change is not particularly obvious at first glance, but it does appear to cause a change.

As padding only means adding a few pixels around the image I wouldn't expect large changes. Mostly this should be visible close to the edges, and indeed there appears to be a difference along the left hand side.

htoyryla on 13 Feb 2017

Changing line https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L534 to divide by self.G:nElement(), I ran neural-style with defaults and got this.

outcxc

whereas with the original the resulting image was

outchw

Now, they are obviously different but as the style weight has been effectively increased, we should not read too much into this difference. Anyway, this is worth more testing and the idea of normalizing this way makes intuitively sense to me.

htoyryla on 13 Feb 2017

Concerning YUV... I was under the impression that Y is the luminance.

When you want to disable content_loss, why not simply set content_weight to 0?

htoyryla on 13 Feb 2017

It looks like the 1/C^2 style normalization favors the lowest layers which have smaller C (64 for conv1 as opposed to 512 for conv5). The original neural-style behavior 1/(CxHxW) penalizes less the higher layers because H and W decrease when going to higher layers.

htoyryla on 13 Feb 2017

When you want to disable content_loss, why not simply set content_weight to 0?

I will try that as well later today. I think my settings from before were to different from Gatys' settings.

The other issue is that I think transferring the color from a third image, might be needed, as I would imagine that Gatys' would have used something similar to -original_colors 1 if it were the better solution.

ProGamerGov on 13 Feb 2017

I think I figure out the style combination:

The styled style image: https://i.imgur.com/G1eZerW.png

This was used to produce the final image:

th neural_style.lua -original_colors 1 -style_weight 10000 -output_image out3.png -num_iterations 1000 -content_image fig4_content.jpg -style_image out1_200.png -image_size 512 -save_iter 0 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

And this was used to produce the styled style image:

th neural_style_c.lua -content_weight 0 -style_weight 10000 -output_image out1.png -num_iterations 200 -content_image fig4_style3.jpg -style_image fig4_style1.jpg -image_size 2800 -content_layers relu2_1 -style_layers relu2_1 -save_iter 50 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

I wonder if something similar could be accomplished by being able to control the layers each style image uses?

I am unable to produce a larger version like Gatys was able to do, Any larger images seem to be blurry, and the shapes begin to fade. The darkness of Seated Nude seems to make this harder as the dark areas seem to take over areas on the new style image in my experiments.

ProGamerGov on 14 Feb 2017

A note on 1/C^2 gram matrix normalization: this line also needs to be changed https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L553 so that the backward pass too will use the normalized matrix.

This will require quite different weights, like content_weight 1e3 and style_weight 1, it can take some 300 iterations before the image starts really to develop, but to me the results look good. I am talking about plain neural_style with modifed Gram matrix normalization. Haven't really looked deeper into the Gatys project.

htoyryla on 14 Feb 2017

ProGamerGov, just a little suggestion: since GPU handling is already implemented in "function setup_gpu(params)" (_line 324_), maybe it's possible to use that function instead of new "set_datatype(data, gpu)"?

It could make the code more maintainable – in case of any changes someone will have to modify only one function instead of two.

For example: pad_layer = nn.SpatialReflectionPadding(padW, padW, padH, padH):type(dtype)
(see how nn.SpatialAveragePooling(kW, kH, dW, dH):type(dtype) is added in line 136).

Currently I can not test it on GPU, but I can confirm that it does work on CPU.

VaKonS on 14 Feb 2017

@VaKonS

I'll take a look. I originally pasted in Gatys GPU handling code at the time because I couldn't get the reflection function to work with this line of code:

pad_layer = set_datatype(pad_layer, params.gpu)

As I couldn't figure out how to use function setup_gpu with the code.

Are you saying to change this line:

https://github.com/ProGamerGov/neural-style/blob/6814479c8ebcc11498b7c123ee2ba7ef9f0fe09f/neural_style.lua#L125

to this:

local pad_layer = nn.SpatialReflectionPadding(padW, padW, padH, padH):type(dtype)

And then delete this line:

pad_layer = set_datatype(pad_layer, params.gpu)

?

ProGamerGov on 14 Feb 2017

@ProGamerGov, yes.
And to delete function set_datatype(data, gpu) at line 611, as it will not be needed anymore.

VaKonS on 14 Feb 2017

@VaKonS , I made a version that contains other padding types: https://gist.github.com/ProGamerGov/0e7523e221935442a6a899bdfee033a8

When using -padding, you can try 5 different types of padding: default, reflect, zero, replication, or pad. In my testing, the pad option seems to leave untouched edges on other either side of the image.

Edit: Modified version with htoyryla's suggestions: https://gist.github.com/ProGamerGov/5b9c9f133cfb14cf926ca7b580ea3cc8

The modified version only has two 3 options, default, reflect, or replicate.

ProGamerGov on 15 Feb 2017

Types 'reflect' and 'replication' make sense, although with the typical padding width = 1 as in VGG19 the result is identical.

Type 'zero' is superfluous as the convolution layer already pads with zeroes.

Type 'pad' only pads in one dimension so it hardly makes sense.

You should read nn documentation when using the nn layers. The nn.Spatial.... layers are meant to work with two-dimensional data like images. nn.Padding provides a lower level access for padding of tensors, you need to specify which dimension, which side, which value, and if one wants to use it to pad an image one needs to apply it several times with different settings.

But frankly, with the 1-pixel padding in VGG there are not so many ways to pad. We should also remember that the main reason for padding in the convolution layers is to get the correct output size. Without padding convolution tends to shrink the size.

htoyryla on 15 Feb 2017

👍1

The code could also be structured like this (to avoid duplicating code and making the same checks several times). Here I used 'reflect' and 'replicate' as they are shorter, you may prefer 'replication' and 'reflection' as in the layer names. But having one as a verb and the other as a noun is maybe not a good idea.

local is_convolution = (layer_type == 'cudnn.SpatialConvolution' or layer_type == 'nn.SpatialConvolution')   
if is_convolution and params.padding ~= 'default' then
    local padW, padH = layer.padW, layer.padH
    if params.padding == 'reflect' then
        local pad_layer = nn.SpatialReflectionPadding(padW, padW, padH, padH):type(dtype)
    elseif params.padding == 'replicate' then 
        local pad_layer = nn.SpatialReplicationPadding(padW, padW, padH, padH):type(dtype)
    else
        error('Unknown padding type')
   end  
   net:add(pad_layer)
   layer.padW = 0
   layer.padH = 0
end

htoyryla on 15 Feb 2017

👍1

@htoyryla, reflective padding probably takes pixels starting from 1 pixel distance: [ x-2, x-1, x ] _[ x-1, x-2 ]_.
And replication duplicates the edge: [ x-2, x-1, x ] _[ x, x ]_.

VaKonS on 15 Feb 2017

Yes, I just realized that when I did a small test. That explains why it made a difference also with padding of one row/column. The documentation is a bit unclear so I believed reflection would result in [ x-2, x-1, x ] [ x, x-1 ] when it only says 'reflection of the input boundary'. But obviously this is more useful.

htoyryla on 15 Feb 2017

I have been trying to get this python script to work for the linear color feature found in Gatys' code here: https://github.com/leongatys/NeuralImageSynthesis/blob/master/ExampleNotebooks/ScaleControl.ipynb

https://gist.github.com/ProGamerGov/5fc5ef9035edc9a026e41925f733a45c

The idea is that making this feature into a simple python script will be easier and less messy than implementing into neural_style.lua. But I can't figure out the python parameters so that the image is fed into the function properly.

Edit:

Trying to reverse engineer the code that feeds into the function:

https://gist.github.com/ProGamerGov/32b7d68a098f8b0655d71a08eb3ba050

So far it doesn't output the converted images.

ProGamerGov on 16 Feb 2017

About your first script https://gist.github.com/ProGamerGov/5fc5ef9035edc9a026e41925f733a45c

To make it process the images and save the result you need something like this. You did not pass the images to your function and you did not use the resulting image returned by the function. Remember that the function parameters target_img and source_img are totally separate from the variables with the same names, usually it is a good practice to avoid using the same names for both.

The numpy imports were needed, on the other hand I had to use skimage.io instead of PIL for reading and saving the image, probably they use a different format for the image inside python. Anyway, Gatys used imread() and not Image.open().

This works in principle but the resulting image is probably not what one would expect. It could be that some kind of pre/deprocessing is needed which was not obvious to me (not being familiar with the process you are trying to duplicate).

PS. imread returns an image where the data is between 0 and 255 as integers, while match_color expects 0..1 floats. Thats why the result is not good yet.

import scipy
import h5py
import skimage
import os
from skimage import io,transform,img_as_float
from skimage.io import imread,imsave
from collections import OrderedDict
#from PIL import Image, ImageFilter
import numpy as np
from numpy import eye 
import decimal
#import click

target_img = imread('to.png')
source_img = imread('from.png')

def match_color(target_img, source_img, mode='pca', eps=1e-5):
    ....
    return matched_img

output_img = match_color(target_img, source_img)
imsave('result.png', output_img)

htoyryla on 16 Feb 2017

OK, by still changing the two imread lines to

target_img = imread('to.png').astype(float)/256
source_img = imread('from.png').astype(float)/256

from these two images
from

I get this (don't know if this is what is expected but it looks ok)

result

htoyryla on 16 Feb 2017

Just noticed that there was already an import for img_as_float so these work as well

target_img = img_as_float(imread('to.png'))
source_img = img_as_float(imread('from.png'))

But anyway, I hope this illustrates that one cannot simply cut and paste code but needs also to examine it and make sure the pieces fit together.

htoyryla on 16 Feb 2017

The script seems to work like the outputs Gatys' code produced in the iPython interface now:

The source image:

The target images:

The images I used can be found in Gatys respository here, and in my Imgur album here: https://imgur.com/a/PrKtg.

Before the Gatys' Scale Control code tried to transfer the brush strokes onto the circular pattern image, it created images like these with the linear transfer color function. So I guess the next step is to test how well these modified style images work.

The working script: https://gist.github.com/ProGamerGov/73e6c242abc00777e4e8cf05cf39dc70

This code here:

target_img = img_as_float(imread('to.png'))
source_img = img_as_float(imread('from.png'))

Did not seem to work for me, though that could be a Virtualbox related issue like some ImageMagick scripts can cause.

ProGamerGov on 16 Feb 2017

👍1

If img_as_float does not work, check that you have

from skimage import io,transform,img_as_float

(Just noticed that you have it. Don't know what is going on there if you have skimage installed in your python and can import it.)

And by the way, assuming you want to try all options, you can change the match_color mode and eps like this:

output_img = match_color(target_img, source_img, mode='chol', eps=1e-4)

htoyryla on 16 Feb 2017

Python interpreter is useful for testing small things (just like th in lua):

Python 2.7.6 (default, Oct 26 2016, 20:30:19)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import skimage
>>> from skimage import io,transform,img_as_float
>>> from skimage.io import imread,imsave
>>> img_as_float
<function img_as_float at 0x7f3bc9cad230>
>>> img = imread('to.png')
>>> img
array([[[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [133, 119, 112],
        [101,  84,  85],
        [ 54,  45,  44]],
>>> img_as_float(img)
array([[[ 1.        ,  1.        ,  1.        ],
        [ 1.        ,  1.        ,  1.        ],
        [ 1.        ,  1.        ,  1.        ],
        ...,
        [ 0.52156863,  0.46666667,  0.43921569],
        [ 0.39607843,  0.32941176,  0.33333333],
        [ 0.21176471,  0.17647059,  0.17254902]],

htoyryla on 16 Feb 2017

I got the script to accept user specified parameters: https://gist.github.com/ProGamerGov/d0917848a728bceb4131272734f61e8b

Only the target and source image are required, but you can also control the eps value and the transfer mode. Though the --eps parameter currently only accepts values in scientific notation.

I also cleaned up the unused lines of code.

I currently testing different parameters for scale control.

ProGamerGov on 17 Feb 2017

It seems you don't understand how functions work. When one defines a function like match_color one specifies the parameters that are input to the function when it is called.

When one calls the function one gives the actual values for those parameters. One can then call the function as many times as needed with different values.

What you are doing now is defining a function so that the default values of transfer_mode and eps are defined from user input. It works when you only run the function once but it is confusing. That is not the way to pass values into a function.

You should change the def line as it was and add the actual values of transfer_mode and eps to the line where the function is called (like I already suggested).

output_img = match_color(target_img, source_img, mode=transfer_mode, eps=int(float(eps_value)))

BTW, I don't understand the int() for eps... first we give something like 1e-5, then float it and finally int which gives 0. So you limit eps to integer values only? Why the int? Float(eps_value) should be enough to convert the input string into a number.

htoyryla on 17 Feb 2017

It seems you don't understand how functions work.

It works when you only run the function once but it is confusing. That is not the way to pass values into a function.

I went for making the code work, without putting a lot of focus on how. Which is a terrible way to go about coding.

You should change the def line as it was and add the actual values of transfer_mode and eps to the line where the function is called (like I already suggested).

Yea, I see that now. Not sure what I was thinking at time when I made such an embarrassing and obvious mistake.

BTW, I don't understand the int() for eps... first we give something like 1e-5, then float it and finally int which gives 0. So you limit eps to integer values only? Why the int?

It was the first that worked, which I now think was because I fixed a bracket placement error. I have removed the integer limitation.

Thanks for helping me correct the issues!

ProGamerGov on 17 Feb 2017

I think I am getting close to the research paper's results:

Layers relu2_1,relu4_2:

Direct link to full image: https://i.imgur.com/Vo9p96O.png

I know the research paper talks about only using layers relu1_1 and relu2_1, but the fine brush strokes from the paint style image seem to work best with relu2_1 and relu4_2, or just relu4_2, at least with this coarse style image. I'm not sure if I am missing something, or if this is due to a different between Gatys' and JcJohnson's code?

This was my content image: https://i.imgur.com/eoX7f3I.jpg

Control test without scale control:

Screenshot from the research paper:

I used this command to create my "stylemix" image:

 th neural_style.lua -tv_weight 0 -content_weight 0 -style_weight 10000 -output_image out5.png -num_iterations 550 -content_image result.png -style_image result_3.png -image_size 1536 -content_layers relu2_1,relu4_2 -style_layers relu2_1,relu4_2 -save_iter 50 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

Then I used this two step set of commands to create the final output:

th neural_style.lua -style_weight 10000 -output_image out_final.png -num_iterations 1000 -content_image fig4_content.jpg -style_image out5_pca.png -image_size 512 -save_iter 0 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

th neural_style.lua -style_weight 10000 -output_image out_final_hr.png -num_iterations 550 -content_image fig4_content.jpg -init_image out_final.png -style_image out5_pca.png -image_size 1536 -save_iter 0 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

I used the default linear-color-transfer.py script on my stylemix image before using it to create my final output, so the colors are more vivid than Gatys' version in the research paper. The default linear-color-transfer.py script was also used on both style images before I added the fine style to the coarse style. Both times used the final content image with the city lights, as the source image.

ProGamerGov on 17 Feb 2017

👍1

Can you give the commands how to run the whole process, would like to test.

htoyryla on 17 Feb 2017

@htoyryla

Images used:

fig4_content.jpg: https://github.com/leongatys/NeuralImageSynthesis/blob/master/Images/ControlPaper/fig4_content.jpg

Fine style: https://github.com/leongatys/NeuralImageSynthesis/blob/master/Images/ControlPaper/fig4_style1.jpg

Course style: https://github.com/leongatys/NeuralImageSynthesis/blob/master/Images/ControlPaper/fig4_style2.jpg

Step 1:

python linear-color-transfer.py --target_image coarse_style.png --source_image fig4_content.jpg --output_image coarse_pca.png

python linear-color-transfer.py --target_image fine_style.png --source_image fig4_content.jpg --output_image fine_pca.png

Step 2 (Gatys called the output from this step, "stylemix", but I used a generic name from a the list of experiments I was running):

th neural_style.lua -tv_weight 0 -content_weight 0 -style_weight 10000 -output_image out5.png -num_iterations 550 -content_image coarse_pca.png -style_image fine_pca.png -image_size 1536 -content_layers relu2_1,relu4_2 -style_layers relu2_1,relu4_2 -save_iter 50 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

Step 2.5 (I don't think Gatys' code does this, but I thought it would make the colors look better):

python linear-color-transfer.py --target_image out5.png --source_image fig4_content.jpg --output_image out5_pca.png

Step 3:

Then I tried to mimic Gaty's two step process where the first image is generated at 512px:

th neural_style.lua -style_weight 10000 -output_image out_final.png -num_iterations 1000 -content_image fig4_content.jpg -style_image out5_pca.png -image_size 512 -save_iter 0 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

th neural_style.lua -style_weight 10000 -output_image out_final_hr.png -num_iterations 550 -content_image fig4_content.jpg -init_image out_final.png -style_image out5_pca.png -image_size 1536 -save_iter 0 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

Those commands in that order should give you the exact same output as I got.

ProGamerGov on 17 Feb 2017

After making more tests with different models, I was wrong: the noise is not added by padding.
It's a quality of some models: vgg19 from crowsonkb's repository makes clean images with or without padding, and images made with Illustration2Vec, for example, have noisy borders even with default padding.

noise

VaKonS on 17 Feb 2017

Examining the outputs produced by ScaleControl.ipynb:

Gatys' Scale Control code produce 3 different outputs, each follows a two step Multires process. I am not sure if these are 3 different ways of doing Scale Control, or if 1 or 2 of them are meant to showcase ways that don't work?

For each of the 3 options in the iPython script, I ran the code and generated the images. Each produced a low resolution 648x405 image and then a 1296x810 "hr" resolution image. Though the image names say that the first image has a resolution of 512px and the second image has a resolution of 1024px, which means there may be some else going on here (maybe downsampling?). I have included both images for each example, and they can be viewed in full in the Imgur link below each example.

Gatys' iPython code names the images with the parameters used to create them, and as such I have included the image file names.

"Stylemix images" are what Gatys calls the resulting combination style image made of both the coarse and fine style images.

Combine 2 images with fine and coarse scale:

low res and hr res: https://imgur.com/a/D7AcK

File names:

low res:

cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_sz_512_model_org_pad_sw_1.0E+03_cw_1.0E+00.jpg

hr:

cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_hrpt_layer_relu4_1_sz_512_hrsz_1024_model_org_pad_sw_1.0E+03_cw_1.0E+00.jpg

iPython terminal output: https://gist.github.com/ProGamerGov/a613c42514b9059ebc8230d2c1cd0fd1

norm net:

low res and hr res: https://imgur.com/a/oTB1k

File names:

low res:

cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_sz_512_model_norm_pad_sw_2.0E+08_cw_1.0E+05.jpg

hr res:

cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_hrpt_layer_relu4_1_sz_512_hrsz_1024_model_norm_pad_sw_2.0E+08_cw_1.0E+05.jpg

iPython terminal output: https://gist.github.com/ProGamerGov/3d8f8ffdbde5f8ec69c46f3076fa3f2d

Naive scale combination:

low res and hr res: https://imgur.com/a/LbqJQ

File names:

low res:

cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_sz_512_model_norm_pad_sw_2.0E+08_cw_1.0E+05_naive_scalemix.jpg

hr res:

cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_sz_512_hrsz_1024_model_norm_pad_sw_2.0E+08_cw_1.0E+05_naive_scalemix.jpg

iPython terminal output: https://gist.github.com/ProGamerGov/71eda3b16793835bbe142d902c480fe7

The code in addition to creating the two images for each example, also created 4 stylemix images:

Stylemix image: https://i.imgur.com/m7nRgKP.jpg

Name:

scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_hrpt_layer_relu4_1_hrsz_1024_model_norm_pad_ptw_1.0E+05.jpg

Stylemix image: https://i.imgur.com/XPt6N52.jpg

Name:

scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_sz_512_model_norm_pad_ptw_1.0E+05

Stylemix image: https://i.imgur.com/Vf3mg2n.jpg

Name:

spimg_fig4_style2.jpg_simg_fig4_style3.jpg_hrpt_layer_relu4_1_hrsz_1024_model_org_pad_ptw_1.0E+03.jpg

Stylemix image: https://i.imgur.com/c1ZNDoZ.jpg

Name:

spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_sz_512_model_org_pad_ptw_1.0E+03.jpg

I am not sure why there are 3 Examples of Scale Control, and 4 stylemix images. But I assume one of the examples must use 2 stylemix images?

ProGamerGov on 18 Feb 2017

Ok, so trying both models from Gatys repository which are the normalized VGG19, and the VGG-19 Conv model, I can't seem to get the parameters right. Up until now I was using the default VGG-19 model.

wget -c --no-check-certificate https://bethgelab.org/media/uploads/deeptextures/vgg_normalised.caffemodel
wget -c --no-check-certificate https://bethgelab.org/media/uploads/stylecontrol/VGG_ILSVRC_19_layers_conv.caffemodel

I assume the default Neural-Style VGG-19 prototxt may not work with these models?

wget -c https://gist.githubusercontent.com/ksimonyan/3785162f95cd2d5fee77/raw/bb2b4fe0a9bb0669211cf3d0bc949dfdda173e9e/VGG_ILSVRC_19_layers_deploy.prototxt

Edit: It seems that the models are special versions created by Leon Gatys: https://github.com/jcjohnson/neural-style/issues/7

I don't know why, but I can't seem to get either model to work.

ProGamerGov on 18 Feb 2017

Using Gatys' weights for Scale Control in Neural-Style seems to work pretty well:

Also, the sym option on the match_color function is for luminescence style transfer.

ProGamerGov on 18 Feb 2017

👍1

@ProGamerGov, by the way, the code can probably be reimplemented in Torch and maybe even will not be too different:

VaKonS on 18 Feb 2017

👍2

@VaKonS Thanks, I'll take a look at that.

@htoyryla I have started trying to extract the python code responsible for luminescence style transfer: https://gist.github.com/ProGamerGov/08c5d25bb867e4313821a45b2e3b2978

As I understand it, the research paper basically describes converting your content/style images to LUV or YIQ, before running them through the style transfer network. In his python code, Gatys appears to use LUV, so I'll start with that.

Testing those 3 functions:

rgb2luv creates this:

luv2rgb creates this:

lum_transform results in this error whenever I try to use it:

ubuntu@ip-Address:~/neural-style$ python lum2.py --input_image fig4_content.jpg
Traceback (most recent call last):
  File "lum2.py", line 47, in <module>
    output_img = lum_transform(input_img)
  File "lum2.py", line 32, in lum_transform
    img = tile(lum[None,:],(3,1)).reshape((3,image.shape[0],image.shape[1]))
NameError: global name 'tile' is not defined
ubuntu@ip-Address:~/neural-style$

I don't know what "tile" is from and I can't figure out whether it belongs to a package, or is related to another specific variable.

Edit: "tile" is part of numpy, and just like "eye" from linear-color-transfer.py, it needs a "np." appended to it's front.

lum_transform creates this:

These functions come from Gatys' Color Control code here: https://github.com/leongatys/NeuralImageSynthesis/blob/master/ExampleNotebooks/ColourControl.ipynb

I am not sure if lum_transform is needed to perform the LUV change to and from for style transfer.

ProGamerGov on 18 Feb 2017

Edit: "tile" is part of numpy, and just like "eye" from linear-color-transfer.py, it needs a "np." appended to it's front.

Good you found it. I just woke up and was about to comment :)

htoyryla on 18 Feb 2017

How do the Gatys models fail when you try them?

I can load the conv model using the prototxt, both from the links you gave. Also runs fine in neural_style.

th> require "loadcaffe"
{
  load : function: 0x416b6098
  C : userdata: 0x41383dc8
}
th> model_file = "VGG_ILSVRC_19_layers_conv.caffemodel"
                                                                      [0.0001s]
th> proto_file = "VGG_ILSVRC_19_layers_deploy.prototxt"
                                                                      [0.0001s]
th> cnn = loadcaffe.load(proto_file, model_file, "nn")
Successfully loaded VGG_ILSVRC_19_layers_conv.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv3_4: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv4_4: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
conv5_4: 512 512 3 3
                                                                      [0.2703s]
th> cnn
nn.Sequential {
  [input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> (12) -> (13) -> (14) -> (15) -> (16) -> (17) -> (18) -> (19) -> (20) -> (21) -> (22) -> (23) -> (24) -> (25) -> (26) -> (27) -> (28) -> (29) -> (30) -> (31) -> (32) -> (33) -> (34) -> (35) -> (36) -> (37) -> output]
  (1): nn.SpatialConvolution(3 -> 64, 3x3, 1,1, 1,1)
  (2): nn.ReLU
  (3): nn.SpatialConvolution(64 -> 64, 3x3, 1,1, 1,1)
  (4): nn.ReLU
  (5): nn.SpatialMaxPooling(2x2, 2,2)
  (6): nn.SpatialConvolution(64 -> 128, 3x3, 1,1, 1,1)
  (7): nn.ReLU
  (8): nn.SpatialConvolution(128 -> 128, 3x3, 1,1, 1,1)
  (9): nn.ReLU
  (10): nn.SpatialMaxPooling(2x2, 2,2)
  (11): nn.SpatialConvolution(128 -> 256, 3x3, 1,1, 1,1)
  (12): nn.ReLU
  (13): nn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
  (14): nn.ReLU
  (15): nn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
  (16): nn.ReLU
  (17): nn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
  (18): nn.ReLU
  (19): nn.SpatialMaxPooling(2x2, 2,2)
  (20): nn.SpatialConvolution(256 -> 512, 3x3, 1,1, 1,1)
  (21): nn.ReLU
  (22): nn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
  (23): nn.ReLU
  (24): nn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
  (25): nn.ReLU
  (26): nn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
  (27): nn.ReLU
  (28): nn.SpatialMaxPooling(2x2, 2,2)
  (29): nn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
  (30): nn.ReLU
  (31): nn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
  (32): nn.ReLU
  (33): nn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
  (34): nn.ReLU
  (35): nn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
  (36): nn.ReLU
  (37): nn.SpatialMaxPooling(2x2, 2,2)
}

htoyryla on 18 Feb 2017

@htoyryla

How do the Gatys models fail when you try them?

The style loss function does not work (The values basically stay the same) with them for me with any variation of this command for both models:

th neural_style.lua -content_weight 0 -style_weight 10000 -image_size 1024 -output_image out_norm_hr.png -num_iterations 500 -content_image result_2.png -style_image result.png -content_layers relu2_1,relu4_1 -style_layers relu2_1,relu4_1 -model_file models/vgg_normalised.caffemodel -proto_file models/VGG_ILSVRC_19_layers_deploy.prototxt -save_iter 50 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

ProGamerGov on 18 Feb 2017

For me this

th neural_style.lua -model_file ../test/pg/VGG_ILSVRC_19_layers_conv.caffemodel -proto_file ../test/pg/VGG_ILSVRC_19_layers_deploy.prototxt

works on a clean copy of neural-style and produces this image at 400 iterations.

out_400

I see you tried the normalized... I have almost never used the normalized models, they probably require different weight values.

I now tried using -init image -content_weight 0 and it works too.

htoyryla on 18 Feb 2017

Normalized works too, but the losses are very small (which I think is typical for a normalized model), and the does not produce the expect result image --probably one would need a much higher style weight.

Iteration 50 / 1000
  Content 1 loss: 0.000000
  Style 1 loss: 4.361065
  Style 2 loss: 0.464805
  Style 3 loss: 0.069460
  Style 4 loss: 0.017796
  Style 5 loss: 0.015553
  Total loss: 4.928679
Iteration 100 / 1000
  Content 1 loss: 0.000000
  Style 1 loss: 3.661895
  Style 2 loss: 0.397874
  Style 3 loss: 0.057522
  Style 4 loss: 0.017153
  Style 5 loss: 0.015032
  Total loss: 4.149477

htoyryla on 18 Feb 2017

Looking into where the functions are located in Gatys' code here: https://github.com/leongatys/NeuralImageSynthesis/blob/master/ExampleNotebooks/ColourControl.ipynb

lum_transform seems to come before the style transfer process:

 if cp_mode == 'lum':
        org_content = imgs['content'].copy()
        for cond in conditions:
            imgs[cond] = lum_transform(imgs[cond])
        imgs['style'] -= imgs['style'].mean(0).mean(0)
        imgs['style'] += imgs['content'].mean(0).mean(0)
        for cond in conditions:
            imgs[cond][imgs[cond]<0] = 0
            imgs[cond][imgs[cond]>1] = 1

And then rgb2luv and luv2rgb are used:

#execute script 
    !{script_name}
    output = deprocess(get_torch_output(output_file_name))
    if cp_mode == 'lum':
        org_content = rgb2luv(org_content)
        org_content[:,:,0] = output.mean(2)
        output = luv2rgb(org_content)
        output[output<0] = 0
        output[output>1]=1
    imshow(output);gcf().set_size_inches(8,14);show()
    imsave(result_dir + result_image_name, output)

But I am not sure exactly what the code is doing. I think it's doing something with the style transfer output file, and the input file. Though I haven't been able to get the code that uses rgb2luv and luv2rgb to work properly yet.

ProGamerGov on 18 Feb 2017

@htoyryla If they are working for you, then it could have been something weird with the instance I was running, or a corrupted/incorrect prototxt file with the same name.

ProGamerGov on 18 Feb 2017

I downloaded the models and the prototxt from the links you gave, otherwise a fresh copy of neural-style. Maybe you have some modifications in your neural-style? The conv model worked just as usual, the normalized one didn't produce a good image with the default content and style weights, but I've seen that before too.

htoyryla on 18 Feb 2017

Difficult to say what a code snippet is doing without being familiar with the whole of it.

But from the paper the color control looks basically simple.

"The modification is simple. The luminance channels LS and LC are first extracted from the style and content images. Then the Neural Style Transfer algorithm is applied to these images to produce an output luminance image Lˆ. Using the YIQ colour space, the colour information of the content image is represented by the I and Q channels; these are combined with Lˆ to produce the final colour output image (Fig. 3(d))."

In other words, one makes luminance-only versions of content and style images, runs them through neural-style and finally applies the color information from the content image to it. I am not too familiar with the these color schemes, but it would be simple to try how this works using YUV and neural-style.

Gatys then goes deeper into using histogram matching for the cases where the histograms of the two images are quite different.

htoyryla on 18 Feb 2017

Speaking of "vgg_normalised.caffemodel": together with much higher style weight, it probably _requires_ "normalize_gradients". In above examples with this model, the content weight was 100, style weight 300000, and gradients were "semi-normalized" with scale 0.7.

VaKonS on 18 Feb 2017

Related to the discussion above, I made a quick try to run neural-style on luminance (Y) only when -original_colors == 1, then add color (UV) from the content image https://gist.github.com/htoyryla/38a4d6b2280ed5b4e47fc8d67b304f9f

Using my modified code, Gatys VGG19 conv model and neural-style defaults, style transfer with luminance only, followed by transferring color from the content image (Gatys' basic color control)

out_900

For comparison, original_colors == 0 (style transfer with color):

out2_800

For comparison, unmodified neural_style with original_colors == 1 (style transfer with color, followed by transferring color from the content image)

out3_900

Looks like luminance-only style transfer makes a visible difference in the sky.

htoyryla on 18 Feb 2017

In above examples with this model, the content weight was 100, style weight 300000, and gradients were "semi-normalized" with scale 0.7.

I can't see which examples you are referring to, but never mind. I was only suggesting that probably there is nothing wrong with the models.

htoyryla on 18 Feb 2017

I further tried to add the first histogram adjustment by Gatys (formula 10 in the paper), adjusting the luminance-only style image (before preprocessing) so that its mean and variance match the luminance-only content image:

   local cmean = torch.mean(content_image)
  local cvar = torch.var(content_image)
  for _, img_path in ipairs(style_image_list) do
    local img = image.load(img_path, 3)
    img = image.scale(img, style_size, 'bilinear')
    if params.original_colors == 1 then
      -- use luminance only
      img = image.rgb2yuv(img)[{{1, 1}}]
     -- match historgram
      local smean = torch.mean(img)
      local svar = torch.var(img)
      img = img:add(-smean):mul(cvar/svar):add(cmean)   
    end
    local img_caffe = preprocess(img):float()
    table.insert(style_images_caffe, img_caffe)
  end

I guess something went wrong, but somehow I like how this looks:

out3d

htoyryla on 18 Feb 2017

In this version I separated style transfer (color or luminance), histogram matching (none, whole tensor, channel-wise) and original_colors, to allow trying different approaches (for instance doing style transfer with color, histogram matching per channel and finally restoring original colors. Histogram matching is by mean and var only.

Note that I have not really looked at Gatys' code other than some snippets posted here, so this is simply based on reading a page of Gatys' paper. Do not assume that my params work like Gatys'.

-- quick hack to make neural-style do
-- 1 either luminance-only or color based style transfer (--transfer lum|color)
-- 2 match style histogram to content (--histogram no|all|rgb)
-- no: no histogram matching
-- all: match whole tensor (use this with --transfer lum)
-- rgb: match each channel separately
-- 3 restore original colors can be combined with the above
-- NOT ALL COMBINATIONS ARE GUARANTEED TO WORK

https://gist.github.com/htoyryla/af9de7a712d74d12f5d3acc7725e6229

PS. I am quite happy with this now. Made this picture from my own portrait and Picasso's Seated nude.

hannu5e

htoyryla on 18 Feb 2017

👍1

So using this terrible and messy code: https://gist.github.com/ProGamerGov/ba9a9d54bae53e84ebf0116262df6758

I think I have achieved luminescence style transfer without editing neural_style.lua:

Images used:

https://github.com/leongatys/NeuralImageSynthesis/blob/master/Images/ControlPaper/fig3_style1.jpg

https://github.com/leongatys/NeuralImageSynthesis/blob/master/Images/ControlPaper/fig3_content.jpg

Step 1:

First you transfer the color from your content image, to your style image like so:

python linear-color-transfer.py --target_image fig3_style1.jpg --source_image fig3_content.jpg --output_image style_colored_pca.png

Then you run that through the lum_transfer.py script like this:

python lum_transfer.py --content_image fig3_content.jpg --style_image style_colored_pca.png --cp_mode lum --output_style_image output_lum_style_pca.png --output_content_image output_lum_style_pca.png

Now you run your content image through lum_transfer.py like this:

python lum_transfer.py --content_image fig3_content.jpg  --style_image fig3_content.jpg --cp_mode lum --output_content_image out_lum_transfer.png

Step 2:

Now you run both newly created images through Neural-Style like this:

th neural_style.lua -original_colors 0 -image_size 1000 -content_weight 1 -style_weight 1e3 -output_image out_lum6_test.png -content_image out_lum_transfer.png -style_image output_lum_style_pca.png -num_iterations 1500 -save_iter 50 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

Once you've finished with Neural-Style, then you use the lum_transfer.py script again like this:

python lum_transfer.py --output_lum2 out_lum6_test_400_r.png --content_image out_lum6_test_400.png --style_image output_style_colored.png --cp_mode lum2 --output_style_image output_style_2.png --output_content_image output_content_2.png --output_image out_combined.png --org_content fig3_content.jpg

Step 3 (Optional):

Now you can leave your image as is, or you can try to change the colors slightly like this:

python linear-color-transfer.py --target_image out_combined.png --source_image fig3_content.jpg --output_image out_combined_lct_pca.png

python linear-color-transfer.py --mode sym --target_image out_combined.png --source_image fig3_content.jpg --output_image out_combined_lct_sym.png

python linear-color-transfer.py --mode chol --target_image out_combined.png --source_image fig3_content.jpg --output_image out_combined_lct_chol.png

And here are my outputs generated by this process:

The final unmodified output:

--mode pca:

--mode sym:

--mode chol:

Album of the full resolution final images: https://imgur.com/a/AW6qU

Screenshot from the research paper:

Control Output using -original_colors 1 and the unmodified style/content images:

The control output with all 3 modes of linear color transfer using linear-color-transfer.py, for comparison: https://imgur.com/a/82xsu

The unmodified content image:

ProGamerGov on 19 Feb 2017

👍1

Looking at the research paper's examples, and my examples, it seems really difficult to actually see the difference between Luminescence Style Transfer, and normal style transfer.

@htoyryla Your modifications to neural_style.lua created this (Though I used -image_size 649 for this output instead of the -image_size 1000 that I used for the above outputs):

And using linear-color-transfer.py has little to no effect on the output with all 3 modes: https://imgur.com/a/wmNyO

My outputs look closer to the research paper's outputs, but yours seems to have more vivid "light spots".

There are more examples from the research paper, and Gatys' Github code here: https://github.com/ProGamerGov/Neural-Tools/wiki/NeuralImageSynthesis-Color-Control-Examples

The -original_colors 1 parameter in Neural-Style uses YUV, where the "Y" part deals with luminance. So this parameter's effect on our results should be noted. Though I am unclear on what part if any, linear-color-transfer.py plays with luminance.

ProGamerGov on 19 Feb 2017

The -original_colors 1 parameter in Neural-Style uses YUV, where the "Y" part deals with luminance. So this parameter's effect on our results should be noted.

-original-colors 1 does not affect the style transfer at all, it only adjusts the output image so that luminance comes from the output image and color from the content image.

One of Gatys' proposals is to make style transfer using luminance only and only afterwards add color just like with original-color 1. Note that this is different from doing the style transfer in color and then copying color from the content image.

The modification is simple. The luminance channels LS and LC are first extracted from the style and content images. Then the Neural Style Transfer algorithm is applied to these images to produce an output luminance image Lˆ. Using the YIQ colour space, the colour information of the content image is represented by the I and Q channels; these are combined with Lˆ to produce the final colour output image (Fig. 3(d)).

In my latter version this is done with -transfer lum -original-colors 1.

Gatys then goes on

If there is a substantial mismatch between the luminance histogram of the style and the content image, it can be help- ful to match the histogram of the style luminance channel LS to that of the content image LC before transferring the style. For that we simply match mean and variance of the content luminance.

In my latter version, this is done (although in YUV space) by -transfer lum -histogram all -original-colors 1

I have not implemented the color matching method described in section 5.2 of the paper. However, when he later writes

In comparison, when using full style transfer and colour matching, the output image really consists of strokes which are blotches of paint, not just variations of light and dark.

I wanted to try the something like this although with simple histogram matching, setting -transfer color -histogram rgb -original_colors 1

Note that when doing histogram matching, the change in the levels of the style image may require different style weight settings.

I hope this clarifies the logic behind my code.

htoyryla on 19 Feb 2017

@htoyryla, I was referring to those examples. Just a few tests with different styles and _models_, my mistake was that I didn't test other models at first and made wrong assumptions.

@ProGamerGov, looks awesome!
Just to be sure: is line 101 supposed to be "content_img = lum_transform(style_img)"?

In original code there is "imgs[cond] = lum_transform(imgs[cond])", which probably means that _content_img_ should be converted from _content_img_, not from _style_img_.

Anyway, results look great!

Second, I think I've found a way to get rid of noise near the borders.
It's very simple - to add 1 pixel gray border around the image at the very beginning, right after "local net = nn.Sequential()" and before "nn.TVLoss" layer.
Something like:

  if params.padding ~= 'default' then
    print('Padding image with zeroes')
    net:add(nn.SpatialZeroPadding(1, 1, 1, 1):type(dtype))
  end

Results (above are regular versions, below are with gray border, variants with "default-reflected-replicated" paddings):

z1-s4-illustration2vec3242-10-1000-0 57-def-refl-repl
z1-b-illustration2vec3242-10-1000-0 57-def-refl-repl

And it doesn't seem to add noise to previously clean images, as far as I can tell:

z1-s2-crowsonkb-5-100-0 0-def-refl-repl

Disadvantage - gray border appears around the edge with some models, so it should be optional:

z1-s5-imagenetlall-10-1000-0 57-def-refl-repl

VaKonS on 19 Feb 2017

It looks like there are many variant procedures possible based on Gatys' paper. Like these lines in @ProGamerGov's code https://gist.github.com/ProGamerGov/ba9a9d54bae53e84ebf0116262df6758#file-lum_transfer-py-L98-L103 correspond to luminance only style transfer (assuming the correction by @VaKonS above) but unlike my implementation based on a paragraph in the paper quoted above, here the mean of style image is additionally adjusted to match the content image.

My -histogram option then does that but also adjusts the variance according to formula 10 in the paper.

htoyryla on 19 Feb 2017

@VaKonS isn't padding with zeroes actually padding with black pixels. It seems straightforward though to modify SpatialZeroPadding to pad with a given value, like 0.5 for gray.

htoyryla on 19 Feb 2017

@htoyryla, nn.SpatialZeroPadding pads with zeroes, but images are converted to mean-centered first, therefore zeroes in processing layers are actually mean values in images, which should be gray pixels in most cases. At least I think so.

_upd_. Images' mean values in "Neural-style" are not "should be gray", but exactly RGB = 123.68, 116.779, 103.939, if I'm not mistaken.

VaKonS on 19 Feb 2017

Oops... my mistake. Of course when the padding is done inside the model the values are as you say. I was thinking in terms of Torch images where the values are between 0 and 1. I've done the same mistake today working on a NoiseMask module which masks part of an image and replaces it by noise (but in that context it is not so critical... actually there is nothing wrong with the module as the mean and std are parameters).

htoyryla on 19 Feb 2017

Experimenting with my code, I have been thinking of what Gatys et al wrote: "If there is a substantial mismatch between the luminance histogram of the style and the content image, it can be help- ful to match the histogram of the style luminance channel LS to that of the content image LC before transferring the style."

I think here is an example. This is a style image which was itself created by neural-style from my own materials.

tytto5e

Using it as a style image without histogram matching can result in something like this

koe2a_240

while with histogram matching one can get something like this (not really like the style but still looks better)

koe2b_400

But it is not so obvious when histogram matching works and when not. From the same two images, with different settings, I get this with histogram matching (far too dark)

hannu9a

and now I get quite close to the original style without histogram matching

hannu9b

In these examples I did not use luminance transfer nor original_colors, just histogram on and off. My feeling based on this is that histogram matching does not necessarily help getting the exact look of the original style, but it can help getting a good balance between style and content in some problematic cases. And it can also be used in creating pictures with a strong interesting style which is noticeably different from the original style.

htoyryla on 19 Feb 2017

@htoyryla, I thought that histogram matching tricks were to make stylized image use _original_ content colors while keeping some elements from style whose colors were not present in content image.

p. s. It's _my_ mistake with image padding – I shouldn't probably have used "image padding", because it's "layer padding", as you have correctly noticed.

VaKonS on 19 Feb 2017

Looking at @ProGamerGov's lum_transfer.py, which if I understand correctly is used to produce an image file for subsequent input into neural_style. In this process it seems to me that the image is stored as an image file (png?) so the question is: is it safe to put, say, LUV image data into a png file and recover the correct data when read into neural-style later. I don't know about png but it seems to me that it might be for greyscale or RGB only.

On the other hand... I cannot say that the whole process using python scripts together with neural-style would be clear to me. Neural-style too expects RGB unless it has been modified. One could, I think, make luminance only transfer by copying L into R, G and B channels before saving image file for neural style (which is what my code does in effect inside neural-style because the model expects RGB channels).

htoyryla on 19 Feb 2017

I thought that histogram matching tricks were to make stylized image use original content colors while keeping some elements from style whose colors were not present in content image.

Reading carefully, yes, that's what the paper says. My first example, I think, works exactly that way. In the second example, histogram matching gave too extreme results but leaving it off gave a very good (in my eyes) result preserving even the sense of depth, even if the original coloring is not preserved.

Personally, I lean towards original artistic application of these techniques, and don't look for a single perfect tool for everything, but rather a versatile toolbox. For an example of an interesting result using histogram matching see my cubistic portrait from yesterday. Funny... that too was made from the same photo as these examples today.

htoyryla on 19 Feb 2017

So I have now made I refined version of my lum_transfer.py script: https://gist.github.com/ProGamerGov/2e7a0fe7a5ef6e117dc0be81df243331

Now the process only takes 4 commands (including neural_style.lua) to complete:

python linear-color-transfer.py --target_image fig3_style1.jpg --source_image fig3_content.jpg --output_image style_colored_pca.png

python lum_transfer.py --content_image fig3_content.jpg --style_image style_colored_pca.png --cp_mode lum --output_style_image output_lum_style_pca.png --output_content_image output_lum_content_pca.png --org_content fig3_content.jpg 

th neural_style.lua commands here...

python lum_transfer.py --output_lum2 out_lum6_test_400_r.png --cp_mode lum2 --output_image out_combined.png --org_content fig3_content.jpg

For lum_transfer.py, the outputs and required inputs are now dependent on the --cp_mode you choose. Though for --cp_mode lum2 and --output_lum2 there is an issue. One must resize either their Neural-Style output to match their original content image's size, or vice version. Though I would image that in order to preserve the quality of your Neural-Style output, you would want to resize your content image.

Does anyone know how I can resize the content image in the script, to match that of the Neural-Style output image, in the Python code?

Edit: I think Gatys solves this issue like this:

hr_init = img_as_float(scipy.misc.imresize(lr_output, imgs['content'].shape))

Or this code has something to do with it:

for cond in conditions:
        imgs[cond] = img_as_float(imread(img_dirs[cond] + img_names[cond]))
        if imgs[cond].ndim == 2:
            imgs[cond] = tile(imgs[cond][:,:,None],(1,1,3))
        elif imgs[cond].shape[2] == 4:
            imgs[cond] = imgs[cond][:,:,:3]
        try:
            imgs[cond] = transform.pyramid_reduce(imgs[cond], sqrt(float(imgs[cond][:,:,0].size) / img_size**2))
        except:
            print('no downsampling: ' + img_names[cond])
        imshow(imgs[cond]);show()

Edit, this does not work:

import scipy

org_content = scipy.misc.imresize(output, org_content.shape)

ProGamerGov on 19 Feb 2017

@ProGamerGov, here, for example, images are transformed like this:

from skimage import io, transform
im = io.imread(input_name)
im = transform.resize(im, (im.shape[0]scale, im.shape[1]scale), order=3)
io.imsave(output_name, im)

upd. This also works for me (note that "transform" must be imported from skimage, it seems, or scipy will not find its "misc" part):

import scipy
from skimage import io, transform
im = io.imread(input_name)
im = scipy.misc.imresize(im, (im.shape[0]*2, im.shape[1]*2))
io.imsave(output_name, im)

VaKonS on 20 Feb 2017

@VaKonS This is the specific part of the script in which the resizing needs to take place:

elif cp_mode == 'lum2':
            output = args.output_lum2
            org_content = args.org_content
        org_content = imread(org_content).astype(float)/256
        output = imread(output).astype(float)/256

            #Resize "org_content" to match the size of "output".

            org_content = rgb2luv(org_content)
            org_content[:,:,0] = output.mean(2)
            output = luv2rgb(org_content)
            output[output<0] = 0
            output[output>1]=1
        imsave(output_a_name, output)

Do I need to read both images before they are converted to the required float value?

ProGamerGov on 20 Feb 2017

This is the control image with no modifications:

This the control image run through the last lum_transfer.py step:

This is the output I made following the steps I have outlined above:

This one is the same one as the above image, except for the fact that I swapped the content and style images for the first step, so linear-color-transfer.py transferred the color from the style image, to the content image:

I am not sure if it's the content and style images that I am using, but the luminance changes seem very subtle to me.

The full images can be viewed here: https://imgur.com/a/1k6nk

Edit:

This is what the output looks like when you skip the first linear-color-transfer.py step:

ProGamerGov on 20 Feb 2017

Maybe it is just the style image's luminance?

Following the luminance transfer steps on this new style image, results in this:

Control Test with the last lum_transfer step and -original_colors 1:

Looking at the images side by side (control on the right):

The difference is much more apparent with this style image. For example, the lighting on the grass is different between the two images

Full versions of these images can be found here: https://imgur.com/a/5daEM

Here's the comparison with linear-color-transfer.py --mode pca:

ProGamerGov on 20 Feb 2017

@ProGamerGov, scipy.misc.imresize seems to automatically convert output to array of [0...255] integers, therefore if you need floats, you'll have to resize image _first_, and then convert if to ".astype(float)/256".

On the contrary, it looks like skimage.transform.resize accepts and returns array of floats [0...1], so with it you'll need to convert to float/256 at first, and after that resize.

p. s. Shouldn't image arrays be divided by 255, by the way?

VaKonS on 20 Feb 2017

Theses are the results of my luminance tests for iteration count, and -image_size:

Full image here (with labels): https://i.imgur.com/xLLukch.png
Full image here (without labels): https://i.imgur.com/DY4dvt9.png
From this album of comparison images: https://imgur.com/a/eoYMf

In this set of comparison images, the effects of luminance transfer/color control, is more visible. I used a variation of this neural_style.lua command (-original colors 1 for the control tests):

th neural_style.lua -original_colors 0 -image_size 1000 -content_weight 1 -style_weight 1e3 -output_image out_lum8_test.png -content_image output_lum_content_pca.png -style_image output_lum_style_pca.png -num_iterations 1500 -save_iter 50 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

ProGamerGov on 20 Feb 2017

@VaKonS Thanks for the help, using skimage.transform.resize did the trick:

org_content = skimage.transform.resize(org_content, output.shape)

https://gist.github.com/ProGamerGov/2e7a0fe7a5ef6e117dc0be81df243331

The script now doesn't require the user to do any manual resizing.

ProGamerGov on 20 Feb 2017

I repeat my question about @ProGamerGov's process, as it seems to me that in this process which is mixing python scripts with neural-style, in order to do luminance only transfer, a python script will produce the luminance-only copies of the images which are then supposed to undergo style transfer in plain neural style. There are some problems in this process as I see it and I am not sure if they have been addressed properly.

How do we get these luminance only images into neural style?

How do we get neural-style to understand that the input is luminance-only and not RGB, and further, how does neural-style process luminance-only data when the model expects three channels (RGB).

If we just save the output of luminance conversion into a png file, then an unmodified neural-style will read it assuming RGB and the result will be something quite different from luminance-only transfer.

The simple solution would be, I think, to save the luminance-only images as RGB so that the L channel is copied, possible scaled, into R,G and B channels. This is what I do in my modified neural-style.

PS. As to my earlier note wondering whether png can be used to contain yuv data instead of rgb (assuming that the receiving end knows that the contents are yuv instead of rgb), I made a test converting to yuv, saving into png, reading back, converting into rgb and saving again. There was an easily noticeable color shift.

orig

final

htoyryla on 20 Feb 2017

@htoyryla

python scripts with neural-style, in order to do luminance only transfer

Looking at the naming of the feature in Gatys' code, he calls it "Color Control", which leads me to believe it's not just about the luminance transfer. His code does not produce the same outputs as that of the research paper, which makes me wonder if the changes are intentional as the code seems to produce the same outputs for the other features in the research paper.

For instance, when comparing the results of your neural_style.lua modifications, to his results, it's like the only difference is that he replaced the color white, with the color yellow:

Results with your code:

From the research paper:

As to my earlier note wondering whether png can be used to contain yuv data instead of rgb (assuming that the receiving end knows that the contents are yuv instead of rgb), I made a test converting to yuv, saving into png, reading back, converting into rgb and saving again. There was an easily noticeable color shift.

So would these results indicate that png can be used to contain yuv data instead of rgb?

ProGamerGov on 20 Feb 2017

So would these results indicate that png can be used to contain yuv data instead of rgb?

My result shows that using png to contain YUV changes the color content. The color nuances in the original image have been replaced by shades of pink when the image has passed from rgb -> yuv -> png -> read as yuv -> rgb.

But using png to carry YUV content is not the main issue, there is also more serious issue that neural-style and VGG are designed to work with RGB, not YUV (to which my solution is converting Y to RGB by setting R,G,B = Y).

You seem to suggest that Gatys' code implements some other scheme than described in the paper. It can well be, but I am not convinced that your approach duplicates his thinking, if you cannot even explain how your process handles the problems I mentioned.

After all, I was just raising some valid questions about the process, and to me nicely looking results are not a proof that the process is correctly implemented. Making correctly working software requires an understanding of both the process and all details of the implementation. If the implementation is divided between separate programs, one needs to understand how to make them working correctly together and passing information correctly between them. For instance, in this particular case one needs to be aware in which format the image is being processed at any point in the process.

It seems to me that I am wasting my time now, so I'll most likely leave this thread.

htoyryla on 20 Feb 2017

@htoyryla, @ProGamerGov, if I'm not mistaken, then the code actually does not save YUV values in PNGs.
line 126: "output = luv2rgb(org_content)" converts YUV colorspace back to RGB before saving.

VaKonS on 21 Feb 2017

I think that at least two functions from "Controlling Perceptual Factors in Neural Style Transfer" can already be used in Neural-style, improving the quality of results and not overcomplicating it:

luminance-matched style transfer with original colors (like if artist drew a picture in his unique manner, but used original-colored paints) – it can be made as extended "-original_colors" option;
padding option, allowing to improve the quality of edges in rendered image.

Though both need to be polished, tested and reimplemented in Torch to use in Neural-style.

VaKonS on 21 Feb 2017

@VaKonS Yes, the code converts the custom gray scale images back to the RGB after working with them in the LUV colorspace. I am currently finalizing my analysis of how the scripts work in relation to Gatys' examples, and Neural-Style's code.

ProGamerGov on 21 Feb 2017

You seem to suggest that Gatys' code implements some other scheme than described in the paper. It can well be, but I am not convinced that your approach duplicates his thinking, if you cannot even explain how your process handles the problems I mentioned.

@htoyryla Sorry, I should have dug into the inner workings of the how, and why of the script far sooner. My bad.

*The following assumes that I have correctly implemented Gatys' code in both scripts. I used ImageMagick to both analyze the inputs before they were run through the scripts, and the outputs after they were run through the scripts: *

The linear-color-transfer.py script makes the RGB channel means, standard deviations, and overall mean/standard deviation, extremely close to that of values of your chosen source image. Min, max, and gamma remain unaffected. With this in mind, it does not seem like this script is crucial to the function of the luminance transfer process. Though this script does seem make the style image's colors work better for style transfer, as seen above in the experiments with Picasso's Seated Nude artwork.

As for the lum-transfer.py script, things are more complicated than it first appears:

There are many different methods and meanings of the word "grayscale", but I think that Gatys is using a custom method the focuses on luminance. Grayscale images only contain shades of grey, and no color. Each pixel has a "luminance" value. Other names for this "luminance" value, are "intensity", and "brightness".

This means that a grayscale in RGB form, represents the luminance of the image. So one can run images representing luminance through Neural-Style. So the scripts always output in the RGB color space, even though they are working with both a grey scale color space and the LUV color space.

I can break the lum-transfer.py and the linear-transfer-color.py code by giving either script a grey scale image with every grey scale algorithm I have tried. This is because Gatys is not using the rec709luma algorithm, or any other algorithm known to ImageMagick. ImageMagick is listed the script's gray scale output in RGB form has it's intensity value listed as, "Undefined" while the intensity value for my control test is listed as "rec709luma".

The lum-transfer.py script's lum mode first converts your input image that's in the RGB color space, to a grey scale format using the grey scale format as way to preserve/focus on the luminance of the input image.

The lum-transfer.py script's lum2 mode takes the RGB output from Neural-Style composed of the two grey scale images you ran through Neural-Style. It also takes your original RGB content image. Your original content image is then converted from the RGB color space to the LUV color space. This RGB to LUV conversion is what the rgb2luv function does. After this, the script the original content image in LUV form, and applies it to the RGB gray scale output image. This is what the luv2rgb function does. The luv2rgb function basically uses the luminance from the LUV color space, to apply the colors onto the grey scale luminance output image.

In summary, the Python scripts I have made, and Gatys' code, are using a gray scale color space to represent luminance, in the RGB color space to bypass the limitations of the pre-trained caffemodel. In order to add the color back to the gray scale luminance output, the LUV color space is used along with the original content image.

Now, the question is are the differences between the Python script aided outputs, and Gatys' outputs because of the neural_style.lua parameters I used, or am I missing a step in the luminance transfer process?

I am still learning how histograms work, so I don't know exactly where, if at all, does histogram matching play a role in what I described above. I also may have missed some things, so please feel free to let me know where I messed up.

ProGamerGov on 21 Feb 2017

Comparing what I have learned/figured out, to what the research paper says, I think the linear-color-transfer.py script is for histogram matching:

If there is a substantial mismatch between the luminance histogram of the style and the content image, it can be helpful to match the histogram of the style luminance channel LS to that of the content image LC before transferring the style. For that we simply match mean and variance of the content luminance.

I think that the lum mode on the lum-transfer.py Python script is responsible for this luminance histogram matching described by the research paper.

For the linear-color-transfer.py Python script, this the relevant part of the research paper:

The one choice to be made is the colour transfer procedure. There are many colour transformation algorithms to choose from; see [5] for a survey. Here we use linear methods, which are simple and effective for colour style transfer. Given the style image, each RGB pixel is transformed as:

where A is a 3 × 3 matrix and b is a 3-vector. This transformation is chosen so that the mean and covariance of the RGB values in the new style image match those of [11] (Appendix B). In general, we find that the colour matching method works reasonably well with Neural Style Transfer (Fig. 3(e)),

Then from the comparison between the two methods:

The colour-matching method is naturally limited by how well the colour transfer from the content image onto the style image works. The colour distribution often cannot be matched perfectly, leading to a mismatch between the colours of the output image and that of the content image.

In contrast, the luminance-only transfer method preserves the colours of the content image perfectly. However, dependencies between the luminance and the colour channels are lost in the output image. While we found that this is usually very difficult to spot, it can be a problem for styles with prominent brushstrokes since a single brushstroke can change colour in an unnatural way. In comparison, when using full style transfer and colour matching, the output image really consists of strokes which are blotches of paint, not just variations of light and dark. For a more detailed discussion of colour preservation in Neural Style Transfer we refer the reader to a technical report in the Supplementary Material, section 2.1.

I am not sure where to find the supplementary material?

Edit: http://bethgelab.org/media/uploads/stylecontrol/supplement/

Second Edit: The supplementary material contains all the raw images that were used in the research paper. This in addition to a lot more examples and details.

There also appears to be a tool that's made to examine what the different layers of a style image look like. This appears to be what the -aesth_input command in his code, is for.

ProGamerGov on 21 Feb 2017

@VaKonS
I was mainly looking at lum_transform when cp_mode = lum which I understood to correspond to luminance-only transfer (with the addition of adjusting the mean). But must say I have not had time to look deep enough. Just raised a concern about an additional complexity when using this kind of mixed approach.

Finally looked more deeply... lum_transform produces a monochrome image so there is no problem, neural-style will internally make a 3xHxW tensor in which all channels are copies of the monochrome image. It is just that neural_style will not be able to restore original_colors because it only gets monochrome images, but of course that can be solved by adding another script into the pipeline.

I apologize for commenting... I was just following my former R&D leader role instincts when this started growing more and more complex. I'll ignore this thread from now on... haven't got the time to follow closely enough and the structure of the solution isn't making it any easier.

My concern was thus limited to program implementation, not to theoretical issues concerning different color spaces. This is because in my view the program needs to be solidly implemented before there is any point starting exploring further improvements on more theoretical level.

I think this all comes down to my preference for neural-style, torch and lua, as well as simplicity and clarity, as a basis for development.

htoyryla on 21 Feb 2017

Pulling images directly from Gatys' code (The iPython notebook example), the content images before the style transfer process are exactly the same as as the ones made by --cp_mode lum:

These are the outputs from from the lum-transfer.py script for comparison (ignore the size difference mistake):

After style transfer, you get pretty much the same output as with Neural-Style:

For adding the color to the output image, lum-transfer.py, and Gatys' code both do the exact same (Script on the left, Gatys' iPython notebook example on the right):

Full image size album here: https://imgur.com/a/HSdcj

Seeing these outputs, I do believe I have replicated Gatys' luminance transfer code in Neural-Style by using the lum-transfer.py script. The differences between his outputs and mine, arise from both differences in the style transfer parameters, and because he uses a two step process.

@htoyryla I think I have with the best of my knowledge, explained how the process works, and now I have proven that there are no missing steps from the process. So I think it is safe to say that I have gotten luminance transfer working in Neural-Style. Though I'll have to play around with your luminance modifications for neural_style.lua some more.

ProGamerGov on 21 Feb 2017

My problem is that with the limited time I have, I cannot follow your process as it is evolving. Like when you now say "after style transfer, you get pretty much the same output as with Neural-Style", I am confused because I believed your process used neural-style for style-transfer proper (and which was the basis for my recent concerns).

Therefore, no point in my continuing to participate. The early part of the thread gave me important impulses but now I think my participation is counterproductive for all of us.

PS.

are using a gray scale color space to represent luminance, in the RGB color space to bypass the limitations of the pre-trained caffemodel. In order to add the color back to the gray scale luminance output, the LUV color space is used

That is pretty much what my neural-version does, too, with the steps inserted into the appropriate places in neural-style. Perhaps it is only that the way you have arranged everything does not work well with my intuition. Like running the same script with cryptic option names to do different things in different parts of the process. That I am not so familiar with numpy does not help either.

The part "limitations of the pre-trained caffemodel" is really not fair. Any trained model has to be based on some constraints on the format of the data, RGB being the most natural choice in my view, and monochrome (greyscale) image can easily be represented in RGB just like we both have done. It was only that my not being fluent in reading numpy made me miss that you too had done so.

htoyryla on 21 Feb 2017

@htoyryla

Like when you now say "after style transfer, you get pretty much the same output as with Neural-Style", I am confused because I believed your process used neural-style for style-transfer proper.

Sorry, my bad. I was referring to running Gatys' code from the iPython notebook example and how it compared with doing the same in Neural-Style + the Python scripts.

Therefore, no point in my continuing to participate. The early part of the thread gave me important impulses but now I think my participation is counterproductive for all of us.

The luminance stuff is pretty much done, but the next feature (and final feature I think?) is called Spatial Control, and it looks as though I need to modify the deeper levels of neural_style.lua, which you know far better than I do, due to your own experimentation with the code. I previously thought I could use some simple Python code for masks, but upon examining the supplementary material for the research paper, it appears I cannot do things in Python.

ProGamerGov on 21 Feb 2017

Using masks in torch based style processing is of interest to me and it would be quite simple to add mask before the gram matrix. But full spatial control, as I understand it, would require the use of multiple gram matrices each with its own mask. I faintly recall that Gatys put the multiple gram matrices in a tensor adding just one more dimension, which sounds simple enough in principle. That also feels interesting for my own purposes but not something to do in one hour, so it'll have to wait.

Using masks for spatial control is also not so easy if you have to make the masks manually. I know because I have used neural-doodle for semantic style transfer. To be really useful, one would need an automatic solution for creating the masks (belonging to the general category of image segmentation). I had a look at one promising solution, but the code was in matlab; the model was caffe but used custom layers not recognized by loadcaffe, using it would have required using a custom build of caffe to run the model for making the masks, so I didn't go further with it.

htoyryla on 21 Feb 2017

@ProGamerGov, @htoyryla, I think I've made "match_color" in Torch.
In case if someone needs it.
https://github.com/VaKonS/neural-style/blob/7682e2e24b650206afeddc576a6ca0778d11260c/match_colors.lua

Results seem to be byte-idenctical to ProGamerGov's script.

Does anybody knows how to remove commits from pull request with browser interface?

VaKonS on 21 Feb 2017

👍2

This might be of interest https://arxiv.org/pdf/1701.08893v1.pdf

htoyryla on 22 Feb 2017

Assuming that match_color is meant to adjust the style image before style transfer (which will then be in color), I added it to my code: https://gist.github.com/htoyryla/9ee49c5ff38dda7d0907b6878c171974

Use parameter -histogram matchcolor (transfer should be left to color and originalcolors I believe shall be off).

This expects to find match_color in a file match_colors.lua in the same directory, use @VaKonS's code but remove everything else than the function.

This should also work with multiple style images but I have not tested it.

Here's a sample output using neural-style default and Gatys' VGG_ILSVRC_19_layers_conv.caffemodel

out

Note to myself:
If both -histogram matchcolor and -transfer lum are set, the images are first modified to greyscale and the color matched, which probably fails because of tensor size mismatch. It might be more interesting to change this order, so that color matching is done first and then -transfer lum would perform a luminance-only style transfer.
Maybe one could use match_color also as an optional method for originalcolor.

htoyryla on 22 Feb 2017

@ProGamerGov , saw in email about your attempt to add loss module that Gatys used. It looks like Gatys' code is based on original neural-style where the loss models get the targets as input. The new neural-style code (updated in December 2016 I think) works differently, the loss module must implement two modes:

'capture' in which it captures the target by its own
'loss' in which it evaluates loss

So what you were trying to do might with good luck work in the pre-12/2017 code.

htoyryla on 23 Feb 2017

👍1

@htoyryla Yea, I removed the comment because I doubted my modifications were even functioning correctly. But thanks for the tip. I'll take a look at older versions of Neural-Style, and then hopefully use that knowledge to port the features to the newer version.

ProGamerGov on 23 Feb 2017

@ProGamerGov, if you want to make your loss modules work in the new neural-style, you need to compare the old and new loss modules. Especially the part in the new modules where if self.mode == 'capture' and if self.mode == 'loss'. The old modules only implement the "loss" part, the targets having been set when the loss modules were created (see PS on this). Setting the target of a loss module basically means forwarding the appropriate image through the model and storing the output of the loss module as the target.

PS. The old way of setting the targets required that as the model was being assembled, after each content or style layer had been added, the appropriate content or style image was forwarded through the model to capture its output as the target for this layer. Using the new way, one only needs to forward each image through the complete model once in capture mode https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L164-L175, and each loss module captures and stores its own targets. But this makes the loss modules more complex.

When I talk about loss modules, I refer to a complete implementation for handling of losses, whereas a loss function for me means the method of calculating loss.

htoyryla on 23 Feb 2017

Changed my code https://gist.github.com/htoyryla/9ee49c5ff38dda7d0907b6878c171974

allows using -histogram matchcolor -transfer lum -original_colors 1 to do color matching first, then a luminance-only style transfer and finally restore original colors
does color matching from content image to output image when -original_colors 2 (don't know if it is useful though)

Sample output using using -histogram matchcolor -transfer lum -original_colors 1
(here match_color.lua modified to swap dimensions 2 and 3, see following comment)

hannu5b-mc-lum-oc1-whswap

htoyryla on 23 Feb 2017

👍3

@VaKonS , one question as I am using your lua implementation of match_color. The comments say that CxWxH is expected, while torch.image uses CxHxW. I don't know if color matching is sensitive to the order of the dimensions. I can of course try swapping them.

htoyryla on 23 Feb 2017

@htoyryla, no need to swap them, it's simply me who didn't notice that dimensions are swapped in Torch. I only checked that intermediate results match arrays in Python.
Probably this is why I had to use "transpose" on "torch.potrf(_Cx_)", while "np.linalg.cholesky(_Cx_)" is not transposed in Python.
Still, resulting images in all 3 modes were exactly the same as Python output during tests.

VaKonS on 23 Feb 2017

@VaKonS, strangely enough I got much better results with my own test material after I swapped the dimensions, but there may be some other reason for that. And thanks for porting the function into lua.

htoyryla on 23 Feb 2017

👍1

@htoyryla, the image is presented as [3 channels * n pixels] array in line 46, and all processing is done on 3 "lines of bytes", until in line 79 they are converted back to "3 * y * x" format.
So, if I'm not mistaken, the function shouldn't depend on image dimensions. But I only tried to repeat the code, maybe changing some internal parameters can improve results.

VaKonS on 23 Feb 2017

👍2

I see... went through it manually in interactive th. It is possible that there was some other difference that I overlooked. I could make a better test., maybe. I don't, however, fully understand the more complex operations (such as the pca specific part, whether it makes a difference if the images are of different size, because in practice they almost always are unless one deliberately deforms either of them (which is not currently done in neural-style). I'll have a look tomorrow.

htoyryla on 23 Feb 2017

An interesting issue when trying to replicate Gatys' color matching match, is "bright spots".

My Neural-Style example is on the left, and Gatys' example is on the right:

Link to the full image: https://i.imgur.com/Cvj2pSm.png

I am not sure what is causing them, and how to limit them. Though Gatys seems to have figured that out.

ProGamerGov on 23 Feb 2017

I have also discovered that the match_color function, does not work with every single style/content image. Though I don't know the exact reason/circumstances behind this apparent malfunction.

linear-color-transfer.py:

linear-color-transfer.py versus lum-transfer.py:

Full images: https://i.imgur.com/w4sTTNa.jpg, https://i.imgur.com/wDjExW2.jpg

You can see that some either really dark, or really bright parts of the style image create issues, with both the match_color function (linear-color-transfer.py), and luminance transfer (lum-transfer.py). Though when used together, the issue becomes better disguised.

Oddly enough, the first style image and content image were part of my attempt to replicate Gatys' images:

Color Matching:

Luminance-Only:

Full images, and the content/style image here: https://imgur.com/a/vjPjj

ProGamerGov on 23 Feb 2017

Areas of black pixels? Maybe when the levels are adjusted (like subtracting the mean) one gets some pixels with negative values which are then clamped to 0 = black?

htoyryla on 23 Feb 2017

@htoyryla That could be the case. I'll try adding some print statements to the code, so that I can see what the levels are, and check for negative values?

ProGamerGov on 23 Feb 2017

In principle yes, in torch I would print min() of image tensors in selected places in code looking for negative values. In python you (at least sometimes) have values in range 0 to 255 but the same should hold there too.

htoyryla on 23 Feb 2017

@htoyryla , It appears there are negative values: https://gist.github.com/ProGamerGov/66cd6e662a5eb9fd9e88aa9810b60361#file-lum-transfer-log-L203-L252

It seems that lum-transfer.py is to blame, at least for this example, and not linear-color-transfer.py, and thus the match_color function.

The line of code responsible for the negative values:

style_img -= style_img.mean(0).mean(0)

The line of code is found here on the unmodified lum-transfer.py script.

I used this command on the modified script:

python lum-transfer.py --content_image newyork_construction.jpg --style_image style_colored_pca.png --cp_mode lum --output_style_image output_lum_style_pca.png --output_content_image output_lum_content_pca.png --org_content newyork_construction.jpg 2>&1 | tee /media/ubuntu/Lexar/python.log

This was the modified script I used to output the values: https://gist.github.com/ProGamerGov/0ad73e5b235487806a35c6ef2fc86f54

The equivalent code in Gatys' iPython Notebook example:

if cp_mode == 'lum':
        org_content = imgs['content'].copy()
        for cond in conditions:
            imgs[cond] = lum_transform(imgs[cond])
        imgs['style'] -= imgs['style'].mean(0).mean(0)
        imgs['style'] += imgs['content'].mean(0).mean(0)
        for cond in conditions:
            imgs[cond][imgs[cond]<0] = 0
            imgs[cond][imgs[cond]>1] = 1

ProGamerGov on 23 Feb 2017

@htoyryla Adding this to my code, seems to have fixed the problem:

style_img [style_img < 0 ] = 0
style_img [style_img > 1 ] = 1

content_img [content_img < 0 ] = 0
content_img [content_img > 1 ] = 1

It was stupid mistake on my part, because I had previously omitted the last step in Gatys code. But that step was what fixed the issue I was having.

This issue does not seem to be related to the color matching bright spots however.

ProGamerGov on 23 Feb 2017

@htoyryla @VaKonS I ran a quick match_colors test in Python with:

source_img = skimage.transform.resize(source_img, target_img.shape)

There appears to be no relation between image dimensions, and the output.

ProGamerGov on 23 Feb 2017

@ProGamerGov it is natural that

style_img -= style_img.mean(0).mean(0)

causes negative values, what matters is after the following line:

style_img += content_img.mean(0).mean(0)

If the mean of the style image > the mean of the content image (style image is lighter) then there may still be negative values (one subtracts more than adds). Setting negative values to zero does not necessarily help either (0 means black in luminance).

htoyryla on 24 Feb 2017

Tested whether swapping h and w makes a difference in the torch version of match_color():

require 'image'
require 'match_colors'

timg = image.load('helsinki000.png',3) ;
simg = image.load('/home/hannu/hannu512.png',3)
oimg = match_color(timg, simg)
oimg2 = match_color(timg:transpose(2,3), simg:transpose(2,3))
oimg2 = oimg2:transpose(2,3)
print(oimg:dist(oimg2))

Yes, there is, but not more than 7.6392223546888e-13 with pca, chol gives 16.130728925385, sym 9.1518068579506. Visually not much difference though.

@ProGamerGov the question about dimensions was related to the torch code where the H and W dimensions have been swapped, nothing to do the python code.

htoyryla on 24 Feb 2017

I have identified the difference between Gatys' Style and content loss functions, versus Neural-Styles

Gatys' MSE code does this:

self.loss = self.loss + self.weights[t] * self.crit:forward(input, self.targets[t])

Which translated to Neural-Style, should be this:

self.loss = self.loss + self.strength * self.crit:forward(self.G, self.target):mul(self.strength)

While default Neural-Style does this:

self.loss = self.strength * self.crit:forward(self.G, self.target)

From this line of code: https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L544

Though when I try to change it, I get this error:

neural_style_gatys.lua:546: attempt to index a number value

Full error: https://gist.github.com/ProGamerGov/5b4644494ea682bbf85250d84da07a09

Edit: I think I translated self.weights into being self.strength, which may be incorrect? Or a different format?

There are a few other very similar differences, so I want to know I am messing up the format or something.

ProGamerGov on 24 Feb 2017

@ProGamerGov, this error probably means that Torch is trying array operations on some part of expression, which is a number.
Try to replace the line with:
self.loss = self.loss + self.strength * self.crit:forward(self.G, self.target) * self.strength

Because :mul() is meant to work "in-place" on a Tensor, if I'm not mistaken.
Maybe result of "self.crit:forward(self.G, self.target)" can not be changed directly or not a tensor at all.

By the way, note that Gatys multiplies by "self.weights[t]" once, and you do it twice.

VaKonS on 25 Feb 2017

@htoyryla, speaking of "Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses" (https://github.com/jcjohnson/neural-style/issues/376#issuecomment-281581370) – I didn't quite understand what those "histogram losses" are, but results look very promising on textures, to say the least.

VaKonS on 25 Feb 2017

It seems to me that Gatys has self.loss = self.loss + ...
because his code is adding up losses using multiple targets within the loss module itself: https://github.com/leongatys/NeuralImageSynthesis/blob/master/LossLayers.lua#L74-L76

Like here https://github.com/leongatys/NeuralImageSynthesis/blob/master/LossLayers.lua#L30-L34 it appears that self.targets is 4-dimensional, each target being 3-dimensional instead of 2 like with neural-style gram matrices. Maybe this is related to using multiple gram matrices with masks.

In other words, Gatys uses multiple targets per loss module, neural-style only one. In addition, the target itself is different.

PS. I see that @ProGamerGov has already modified the loss module to a single target. If one now modifies the line as

self.loss = self.loss +

then self.loss will be the accumulated loss from all iterations so far. I don't think that makes sense, but it is probably not the reason for the error message.

htoyryla on 25 Feb 2017

👍1

@VaKonS I only posted the link as the paper looked relevant, haven't read it carefully yet. But already "histogram loss" sounds interesting. I guess histograms could be used (in addition to gram matrices) to evaluate losses (distance between current histogram and target histogram), although a color image exists only at input, elsewhere we have the feature map activations so the meaning of histogram there is maybe a little vague.

htoyryla on 25 Feb 2017

From the error

/home/ubuntu/torch/install/bin/luajit: /home/ubuntu/torch/install/share/lua/5.1/nn/Container.lua:67:
In 4 module of nn.Sequential:
neural_style_gatys.lua:546: attempt to index a number value

it looks like the problem happens inside the self.crit when forward is called with the given input (pay attention to that the error happens within nn.Sequential). This could mean that there is something wrong with the inputs.

htoyryla on 25 Feb 2017

I have been experimenting with modifying the content and style loss functions in order to gain a better understanding of how to modify them appropriately.

This version of neural_style.lua containts the L2Penalty module from lines 498-539 in Gatys' code: https://gist.github.com/ProGamerGov/ea432324a09822a19af916fe1bfcfc01

To use the L2Penalty module, you specify a weight value with the new -l2_weight parameter. I believe I have implemented the module correctly, by mirroring JcJohnson's total variance code.

Here's an example of the new feature with -l2_weight 0.001 on the right, and -l2_weight 0 on the left:

Full image here: https://i.imgur.com/Sh4U5pD.png

ProGamerGov on 5 Mar 2017

@htoyryla The padding option I had implemented here on lines 121-136 in my pull request version of neural_style.lua did not seem to work correctly. I tried to fix it, but I am unsure if my modifications solved the problem correctly.

ProGamerGov on 5 Mar 2017

@ProGamerGov I can't see any problem immediately, can you tell in which way it is not working.

The code structure is not as clean as one would hope, but L123-136 add the extra padding layer and the convolution layer gets added on L147, with zero padding as set on L134-135. If padding is set to default then L123-136 is not run and the conv layer gets added on L147 with padding intact.

htoyryla on 6 Mar 2017

Looking at the code of L2Penalty, I can't actually see that it is doing much. It says that it is "adding the
[gradOutput] to the gradient of the L2 loss" but I cannot see the gradient of L2 loss being calculated anywhere inside this class. Instead it is adding gradOutput to a weighted copy of the input.

L2 loss itself is calculated and stored in self.loss which is not used anywhere. I get the impression that the class expects something of the code inside which it is to be used. I couldn't find Gatys using L2Penalty anywhere. The same code is found here too https://github.com/aleju/mario-ai/blob/master/layers/L2Penalty.lua .

PS. Looking at how L2 weight decay is usually used, to decay the layer weights towards zero, the code starts to make sense as the gradient of 1/2w^2 is w which kind of explains the "adding gradOutput to a weighted copy of the input", although I wonder why we are adding this to the gradient instead of subtracting if this is meant to decay the values.

However, in neural-style we are not training a model, but creating an image, the pixels of which now take the place of the weights in a layer, and I am not at all sure whether it makes sense to decay the pixel values towards zero.

Just for fun, I changed L2Penalty like this

    self.gradInput:resizeAs(input):copy(input):mul(-m)

and inserted an L2Penalty layer below each StyleLoss layer (instead of between TVLoss and the lowest conv layer) and got an image like this.

l2style1e-5minus_900

and with a slightly higher L2 weight:

l2style5e-5minus_500

htoyryla on 6 Mar 2017

@htoyryla For the padding code issue, the older version with the padding option looked like this: https://github.com/ProGamerGov/neural-style/blob/4bcf625b6e478d35b00901f1fa0f8b80f94a90cd/neural_style.lua#L121-L135

And resulted in a module error when I tried to run it. So I modified it to look like this in order to correct the issue: https://github.com/ProGamerGov/neural-style/blob/master/neural_style.lua#L121-L136

ProGamerGov on 6 Mar 2017

@ProGamerGov I see now... it was a scope problem again. Defining padlayer as a local variable inside the if-then-else block leaves padlayer undefined if the net:add(padlayer) is outside the block. Another solution would have been to declare padlayer before the if-then-else block.

htoyryla on 6 Mar 2017

👍2

Looked at @VaKonS 's new assortment of prepadding options. I have no problem with those... it is only that in my mind padding in convlayers has been a method to get the correct output size. Especially when the model contains one or more FC layers, there is really not any freedom to change padding size, because that will change the output size from a layer and lead to a size mismatch somewhere. Here this is not a problem, as we have only convlayers which automatically adjust to the size of the input. Still, it is good to remember this.

htoyryla on 6 Mar 2017

👍1

@htoyryla, I'm just saving what I'm currently experimenting with, after closing pull request. :)
Opened request didn't allow me to change the file without adding more irrelevant commits to original project.

VaKonS on 6 Mar 2017

@htoyryla This is a bit off topic from the thread, but Say I want use the image package image.scale, image.translate, etc... to modify the current image between or during iterations, so that my changes stick. Like how Lua implementations here, and here work in regards to Deepdream.

Using the image package inside an updateOutput function, or a updateGradInput does not seem to work?

In the feval(x) function, I am not sure what if any variables inside the function to use the image library functions on. Though I did try a variety of values and they did not work.

As I understand things, the feval(x) function is where neural_style.lua computes the gradient with respect to the loss, and thus is run every iteration? And the feval(x) function is the function that runs the loss functions? Is my understand correct, or incorrect?

Where should I be trying to perform these operations?

Edit: This DeepDream.lua code appears to be based off of neural_style.lua's code, including using an early neural_style.lua loadcaffe wrapper code.

ProGamerGov on 7 Mar 2017

It seems to me that combining the approach of these deepdream scripts with neural-style is quite problematic. Neural-style sets up the model including the loss modules and then delegates the task of optimizing the image to the optimizer, l-bfgs or adam. In each iteration, the optimizer calls feval and expects it to return the loss and the correct gradients. Based on these (and the history it stores internally), the optimizer the modifies the image. Feval calculates the loss and the gradients by forwarding the image through the model, backpropagating to get the gradients and reading the losses from the loss modules (which sit inside the model).

It should be technically possible to change the image in feval, but doing so will interfere with the optimizer as the pixels will no longer be in the same place as the gradients calculated from them. I guess this is why the deepdream code examples handle the iterations directly, not through an optimizer.

PS. As to changing img within feval, I tried it and it did not work, because img is a plain tensor, not a Torch image object (img.image was nil). However when I create a plain random tensor exactly like img in th interpreter, it works as a torch image, so there is something going on that I don't immediately grasp.

htoyryla on 7 Mar 2017

@htoyryla, saving x input as an image inside feval(x) function seems to work, at least like this:
if x ~= nil then image.save("feval_x.png", deprocess(x:double())) end

(Note that "deprocess" takes DoubleTensor, and processing within model is done on floats, if I'm not mistaken, therefore :double() is needed).

VaKonS on 7 Mar 2017

I was aware of the possibility of saving as an image and loading back (maybe_save() does save the intermediate images), but it feels terribly inefficient. There must be an easier way. Also don't think de/preprocessing is needed if we only want to scale or transform the tensor which contains the image.

Furthermore, I am not sure if it is wise to modify x directly in the middle of an iteration. But it all depends how the optimizer works internally. Maybe x is simply to pointer to img. Like I said, changing the image outside the optimizer is a bad idea anyway, just wanted to test how bad. Like do we get a size mismatch if we change the size of the image. I think we will.

htoyryla on 7 Mar 2017

Back at my main computer...

This works

    image.save("f.png", img)

this does not

  img = image.scale(img, 511, 511, 'bilinear')

and to me it appears that it is because img, being a plain tensor and not an image created by loading from an image file, lacks .image attribute (which is supposed to contain pointers to functions like scale). The funny thing is that using th interpreter, a plain tensor can be manipulated by image.scale etc and the image attribute does contain those functions.

Anyway, this question is more of general interest to me (as I don't think manipulating the image both directly and by the optimizer makes sense). Moreover, having now seen @ProGamerGov's earlier post about implementing the octaves method inside DeepDreamLoss class, I am not at all sure whether the image which needs to be scaled is the image img which is evolving (which is problematic in neural-style), or some other image which would be used internally by the DeepDreamLoss class (which could make sense... I must admit I have not looked deeper into how this DeepDream through scaling is meant to work).

htoyryla on 7 Mar 2017

@htoyryla, strange that "image.scale" doesn't work – this, for example, works for me (inserted between lines 281 and 282):

    if x ~= nil then
      image.save("feval_x.png", deprocess(x:double()))
      local ti = image.scale(x, 511, 511, 'bilinear')
      image.save("feval_x_scaled.png", deprocess(ti:double()))
    end

Both "feval_x.png" and scaled "feval_x_scaled.png" are saved on every iteration.

VaKonS on 7 Mar 2017

@htoyryla, by the way, maybe that is because I'm running on CPU.
If you are using CUDA, then, I suppose, it should be:
local ti = image.scale(x:double(), 511, 511, 'bilinear')

VaKonS on 7 Mar 2017

@htoyryla Can the image library be substituted for modifying the tensor directly, like jcjohnson does?

On lines L109-L112 on this DeepDream lua code, clipping is performed like this:

if clip then
      bias = Normalization.mean/Normalization.std
      img:clamp(-bias,1/Normalization.std-bias)
   end

On Fast-Neural-Style's Deep DreamLoss:updateGradInput function, clamping appears to be done line this:

self.clipped:resizeAs(input):clamp(input, -self.max_grad, self.max_grad)

I am not sure what operations I should be doing on DeepDreamLoss:__init, DeepDreamLoss:updateOutput, DeepDreamLoss:updateGradInput, but I tried to figure it out below, using this modified neural_style.lua: https://gist.github.com/ProGamerGov/ea432324a09822a19af916fe1bfcfc01

I tried to copy DeepDream's octave related operations from here:

  local octaves = {}
   octaves[octave_n] = torch.add(base_img, -Normalization.mean):div(Normalization.std)
   local _,h,w = unpack(base_img:size():totable())

   for i=octave_n-1,1,-1 do
      octaves[i] = image.scale(octaves[i+1], math.ceil((1/octave_scale)*w), math.ceil((1/octave_scale)*h),'simple')
   end

With this in the DeepDreamLoss:updateGradInput:

     local octaves = {}
     octaves[self.octave_n] = self.gradInput:add(self.clipped, -self.max_grad):div(self.max_grad) 
     for i=self.octave_n-1,1,-1 do
        octaves[i] = octaves[i+1]
        local c,h,w = unpack(self.base_img:size():totable())
        local octave_w = math.ceil((1/self.octave_scale)*w), octaves[i+1] 
        local octave_h = math.ceil((1/self.octave_scale)*h)
        self.gradInput = self.gradInput:resize(1, c, 2, octave_h, 3, octave_w)
     end

Though I have yet figure out the code below for octave, octave_base in pairs(octaves) do: https://github.com/bamos/dream-art/blob/master/deepdream.lua#L135-L156, and the function it uses here: https://github.com/bamos/dream-art/blob/master/deepdream.lua#L87-L114

ProGamerGov on 8 Mar 2017

@VaKonS cuda vs. double was the reason, it also explains why it worked in th. The error message however was misleading:

/home/hannu/torch/install/share/lua/5.1/image/init.lua:716: attempt to index field 'image' (a nil value)

and looking at the source code of image/init.lua I found that it expects to find an attribute named image which was missing.

Using code like this

   x = image.scale(x:double(), x:size(2)-1, x:size(3)-1, 'bilinear'):cuda()
   print("x", x:size())

in different places it looks like

scaling (at the end of feval) x does not make permanent changes to size so x is probably a clone
scaling (at the beginning of feval) x does indeed crash l-bfgs
scaling img makes no permanent changes to size, so l-bfgs probably overwrites img at the end of each iteration

htoyryla on 8 Mar 2017

👍2

@ProGamerGov of course the tensor can be modified directly, but the example you give only modifies the values in the tensor. Scaling an image is quite a different thing, it can be done of course. Anyway, we know now that the tensor can be scaled, but we cannot make permanent changed because l-bfgs doesn't expose the master copy of the image to us. Also changing its size, if we could do it, would crash l-bfgs anyway.

PS. resizeAs is not the same as scale. It only changes the size of the tensor, not the values. If t is

 1   2   3   4
  5   6   7   8
  9  10  11  12
 13  14  15  16

then resizing it to 6x6 gives

  1.0000e+00   2.0000e+00   3.0000e+00   4.0000e+00   5.0000e+00   6.0000e+00
  7.0000e+00   8.0000e+00   9.0000e+00   1.0000e+01   1.1000e+01   1.2000e+01
  1.3000e+01   1.4000e+01   1.5000e+01   1.6000e+01  7.1145e-322  4.9012e+252
 1.8819e+262  1.0258e+200  6.5257e-308  6.7411e+199  1.8698e+262  1.4878e+195
 6.9972e-308  5.7588e+160  4.3529e-114  1.1712e+166  1.4352e+166  1.2633e-118
 1.3581e+243  1.4365e+166  1.2633e-118  3.0195e+169  5.2241e+257  1.3872e-118

so it has simply filled the larger tensor with was in the memory (1 to 16) and the rest is not initialized at all.

Scaling of an image requires recreating the image in a new size while still retaining the visual appearance as much as possible, by approximating the pixel values e.g. through interpolation.

As to your example

self.clipped:resizeAs(input):clamp(input, -self.max_grad, self.max_grad)

What this does is actually not scaling at all, but making self.clipped the same size as input and filling it with the values from input, clamped between -self.max_grad and self.max_grad (i.e. lower and higher values have been replaced by the -max and max values respectively). In other words, clamping is used here to put a limit to absolute values of gradients.

This use of resizeAs is typical in torch.nn modules where the variables need to be resized according to the input size. .

htoyryla on 8 Mar 2017

👍1

@VaKonS this demonstrates that a cuda tensor lacks image capabilities. The first print statement prints out the image capabilities, the second prints nil. I guess this is a sound development choice, but good to know.

require 'image'
require 'cunn'

img = torch.randn(3,64,64):float():mul(0.001)
print(img.image)
img2 = img:cuda()
print(img2.image)

htoyryla on 8 Mar 2017

👍1

@htoyryla So self.clipped:resizeAs(input) in the DeepDreamLoss:updateGradInput function takes the empty tensor created by self.clipped = torch.Tensor() in the DeepDreamLoss:__init function, and resizes it to match the value of input which I assume is the input image in tensor form?

Then :clamp(input, -self.max_grad, self.max_grad) first fills the resized but still empty tensor with the input image, and then -self.max_grad and self.max_grad is used to normalize higher and lower values.

Finally, self.gradInput:add(-self.strength, self.clipped) adds -self.strength to the self.clipped tensor.

Anyway, we know now that the tensor can be scaled, but we cannot make permanent changed because l-bfgs doesn't expose the master copy of the image to us. Also changing its size, if we could do it, would crash l-bfgs anyway.

So performing changes to the image in the feval(x) does not work. Could one use code like you have shown above, to convert the input in either the DeepDreamLoss:updateGradInput or the DeepDreamLoss:updateOutput function, to a non tensor form so that image.scale, and image.translate can be used for the DeepDream operations, and then the resulting output can be reconverted to the tensor form?

Ex: dd_image = image.scale(input:double(), input:size(2)-1, input:size(3)-1, 'bilinear'):cuda()

And then dd_image would be fed into the two DeepDream functions?

ProGamerGov on 8 Mar 2017

On the subject of spaitial control, I have made progress in my understanding of the process.

In the style loss functions:

The spatially limited tensor is defined like this at the start: local input_chan = input:size()[1]

Then it is added the input value.

   if self.guidance then
        input = torch.cat(input, self.guidance, 1)
    end

The spatial targets are given with this in the python code:

                if guide.ndim==2:
                    guide = guide[:,:,None]
                else:
                    guide = guide[:,:,:1]

Where :,:,None means no guides/spatial targets are used, and :,:,:1 means 1 spatial target is used.

I am still don't understand content vs style guides in his code, and how the code knows which part of the guide image to use.

The 3 spatial targets are used here in the style loss function:

    if self.guidance then
        input = input[{{1,input_chan},{},{}}]
    end

ProGamerGov on 8 Mar 2017

@ProGamerGov I'll just answer a few points now.

resizes it to match the value of input which I assume is the input image in tensor form?

The input is the input to the module and depends on where in the model it has been placed. Most nn classes are designed to be general purpose, so that one can stack conv layers, relus etc as long as one obeys certain principles. Also the StyleLoss and ContentLoss modules work this way. For them, the input is the output of the layer below, usually the ReLU of a convlayer.

TVLoss in neural-style is different in that is designed to be placed directly above an image at the input to the whole model. I guess your DeepDream module is intended to be used in the same way, so that the input is really the image. But remember that you cannot change the image directly, only through the gradients you pass to the optimizer, and this can be tricky.

then -self.max_grad and self.max_grad is used to normalize higher and lower values.

This is not really normalization because there is no division involved, one is simply limiting the range.

Finally, self.gradInput:add(-self.strength, self.clipped) adds -self.strength to the self.clipped tensor.

Nope, I think it adds self.clipped (which is a tensor) multiplied by -self.strength (which is a scalar) to self.gradInput.

Could one use code like you have shown above, to convert the input in either the DeepDreamLoss:updateGradInput or the DeepDreamLoss:updateOutput function, to a non tensor form so that image.scale, and image.translate can be used for the DeepDream operations, and then the resulting output can be reconverted to the tensor form?

Ex: dd_image = image.scale(input:double(), input:size(2)-1, input:size(3)-1, 'bilinear'):cuda()

Your question raises many problems, really, but assuming your DeepDream layer is at the bottom of the model, then you can use the input (or a clone of it) as you would an image. There is no conversion into a non-tensor, the input is a tensor, and you can apply image operations to it if you first convert it to float or double. Afterwards, change it back into dtype. My test was using cuda() but you probably want your code to work on CPU as well.

And then dd_image would be fed into the two DeepDream functions?

Which two functions? If you are using input, you are already inside either updateOutput or updateGradInput. If you want to influence the image, your code probably should be in updateGradInput.

OK, now I have commented your post, but I still cannot guarantee that anything will work. For me, these layers that sit directly on top of the image and are influenced through the gradient appear anything but clear. Good luck.

htoyryla on 8 Mar 2017

As to the spatial control, isn't the idea that masks are used to define parts of the image, so that each area can be processed using a different style. To me it looks like the user is responsible for preparing the masks and assigning the styles to them. The idea seems quite simple to me, use multiple gram matrices each behind its own mask. The idea of a mask is basically very simple, like that a b&w image consisting of black and white pixels can be used to block or pass parts of the input into the gram matrix. (Haven't checked Gatys' code, this is simply how I understood the idea when reading the paper... at least the basic principle).

htoyryla on 8 Mar 2017

@htoyryla I have made great progress, but I have run into a few issues that I don't know how to fix. This is what I have so far: https://gist.github.com/ProGamerGov/7b04fe82d44c3e50b5de195200d1bc0f

There are two errors I am currently facing.

I don't know the cause of this one: https://gist.github.com/ProGamerGov/dd5a703603a9211495652e8f14514d72

This is related to Cuda vs cpu: https://gist.github.com/ProGamerGov/35be22fb4ac452ef303e09590dde487a

This line (L709) when un-commented should work, but it does not. When it's un-commented, I get this error. It is meant be this line of code from the DeepDream.lua code.

This line (L695), also does not work. It is meant to do what this line of code does in the DeepDream.lua code.

ProGamerGov on 9 Mar 2017

@ProGamerGov I thought we were discussing about doing some image manipulation inside updateGradInput of a DeepDreamLoss layer, assuming that the layer is placed so it gets the image as an input. Instead I see you have copied a complete DeepDream implementation with its own model, iteration process and everything. Sorry to say, but see no way this could work, makes no sense to me. You are mixing apples with orchards.

Think of it. The DeepDreamLoss module we were discussing is one layer inside a neural network model. The updateGradInput function is run once per each iteration. Inside it you are now loading another full neural network model there and trying to run a full cycle of iterations.

This line (L709) when un-commented should work, but it does not. When it's un-commented, I get this error. It is meant be this line of code from the DeepDream.lua code.

Do you know what a return statement does?

htoyryla on 9 Mar 2017

@htoyryla From my current understanding, I have isolated the code that creates the "DeepDream" hallucinations, and added it into my DeepDream loss functions:

--The DeepDream Magic
      for i=1,iter_n do
        local forward = self.crit:forward(input, self.target)
        -- Set the output gradients at the outermost layer to be equal to the outputs (So they keep getting amplified)
        local forward_grad = forward
        local final = self.crit:updateGradInput(input, forward_grad)
         -- Gradient ascent
        input = input:add(final:mul(step_size/torch.abs(final):mean()))
      end

The code can be found in my modified script here: https://gist.github.com/ProGamerGov/ccc9c41375845f08c4b0f902f251b612#file-neural_style_dd-lua-L642-L649, and source of this variation of the code can be found here in another Lua implementation of DeepDream.

The problem is however I get the following error:

MSECriterion.lua:27: attempt to index local 'target' (a number value)

Full error message: https://gist.github.com/ProGamerGov/aa08b5cdbb7d064f207013014ddebf93

I do not understand the cause of this, as both self.gradInput = self.crit:backward(input, self.target) and self.loss = self.crit:forward(input, self.target) * self.strength work.

ProGamerGov on 10 Mar 2017

Modifying neural_style.lua's feval(x) loss function to feature the above function, results in (Multires starting at -image_size 224):

Full image (And more example outputs): https://imgur.com/a/eKo5H

The hallucinations appears to start randomly in a single spot (unless using a higher step_size value), and then it expands outwards to eventually engulf the entire image over many iterations.

The modified neural_style.lua I used: https://gist.github.com/ProGamerGov/47cbd692e7e6977b6044ef141b0bd81f

ProGamerGov on 10 Mar 2017

This last code seems again something simple and manageable, even if a bit unorthodox. I think the discussion steered off when you asked about how to scale the image permanently inside feval, which led into a series of misunderstandings.

Let's see what is happening here. Line 287 gets the gradient through the network as usual, but you ignore it. Then you run x ten more times through the net, each time modifying it

 x = x:add(grad:mul(step_size/torch.abs(grad):mean()))

Interesting. I think I'll have to try this and see how
step_size/torch.abs(grad):mean()
behaves. Very close to one, sometimes above, sometimes below. So you are running x ten times through the network, taking the gradient, multiplying it with a scalar close to one, adding this to x before the next round. Finally the gradient from the last round gets passed to the optimizer. In addition, changing x might directly affect the image inside the optimizer, although I earlier concluded that x is a clone (that was on l-bfgs, could also be that adam and l-bfgs work differently).

PS. I didn't get anything else but colored noise (using the neural-style defaults). Finally step_size/torch.abs(grad):mean() turned infinite so this is probably not stable.

With Adam it runs more stable but does not evolve beoynd this.

out_600

htoyryla on 10 Mar 2017

@htoyryla After testing the various DeepDream lua implementations by disabling portions of the code and save images after every step, I can confirm the DeepDream function we are experimenting with is responsible for the DeepDream hallucinations.

I think your outputs are due to x = x:add(grad:mul(step_size/torch.abs(grad):mean())) not having an affect on the output. In my testing, that line is crucial for the DeepDream function to work properly.

By placing the DeepDream function into the layer setup process, it appears to work correctly, and creates this output:

Full image: https://i.imgur.com/JcENto9.png

Comparison album (More iterations equals more hallucination details added): https://imgur.com/a/OeMuw

This is the modified neural_style.lua I used to make the above image: https://gist.github.com/ProGamerGov/f666dedde46dc425e8c7714e578300c4

I am not sure at the moment how to go from here, for making the process occur simultaneously with the normal style transfer process. And whether or not the DeepDream function should be located in a different part of the script.

ProGamerGov on 10 Mar 2017

Comparing the above image, with one that uses the octave code in addition to the DeepDream function, it appears as though octaves are for providing control over some aspects of the hallucination features:

Full image: https://i.imgur.com/JuJlnjO.png

A modified version of the above script with octaves: https://gist.github.com/ProGamerGov/0cd618a20cefcbb98dbc912e015fa483

Currently these last two neural_style.lua scripts with the DeepDream code only run the DeepDream code on the content image before the style transfer process occurs. The newly modified DeepDream image is then saved, and does not replace the content image. So basically the DeepDream code uses the default cnn related code and specified layers, but it does not work simultaneously with the code responsible for the style transfer process yet. These two scripts were more so to test whether not the default neural_style.lua code can be used for the DeepDream process, which these experiments proved is possible.

The next step would be to figure out how to fit the code into running simultaneously alongside the style transfer code, and allowing specific DeepDream layers to be set. I am not sure how to do either of those at the moment however.

I should note that the reason I placed the DeepDream function in the layer setup section of neural_style.lua's code, was because I needed the "net" variable.

ProGamerGov on 11 Mar 2017

@ProGamerGov it seems we are talking past each other. You are interested in achieving specific visual results, I was now trying to understand what was happening between your code and the optimizer.

When you write

I think your outputs are due to x = x:add(grad:mul(step_size/torch.abs(grad):mean())) not having an affect on the output. In my testing, that line is crucial for the DeepDream function to work properly.

so I was using exactly your code, unmodified. What interested me was whether the effect is

1) due to gradient, which is the expected way feval influences the optimizer indirectly
2) because of changing x (if passed by reference and not a clone) directly affects the optimizer (which is usually not a good practice and likely to have side effects)

That you say that your line is needed to have an effect proves nothing either way. My concern is whether the effect is through the gradient (note that changing x and recalculating also changes the gradient) or by directly messing with optimizer internals. By the way, an easy way to check whether 2) happens would be to use a clone of x in the loop.

That illustrates my way of working forward, trying the understand what is happening. Understand the processes and programs in order to modify them.

As to your goal of combining the two methods, I am rather skeptical by this time. Neural-style and the deepdream.lua implementations you are trying to import into it are both complete implementations. You have now managed to insert the deepdream code into neural-style so that it is run before the actual neural-style iterations begin. That is fine, you can get the deepdream image and then you might run neural-style to get another image, but it leaves the two processes separate.

I think the discussion of the past few days has been about the possibility of combining them. Currently I don't think it is possible, at least not this way by playing around with code. The problem is not about code, it is how to duplicate the algorithm of deepdream.lua using a different iteration environment. Breaking up and rearranging the algorithm so that it fits the iterative process in neural-style.

PS.

These two scripts were more so to test whether not the default neural_style.lua code can be used for the DeepDream process, which these experiments proved is possible.

If you needed a proof for something trivial, yes. That one can run segments of code sequentially hardly needs to be proven.

htoyryla on 11 Mar 2017

You are interested in achieving specific visual results, I was now trying to understand what was happening between your code and the optimizer.

I think that what was occurring, was that the feval(x) net:forward(x) and local grad = net:updateGradInput(x, dy) lines were being run repeatedly. Which from what I understand, computing the gradient with respect to the loss is what allows style transfer to work. But instead of doing it the normal amount of time, my modifications were computing the gradient for the x value too many times, and then this x value is added to the grad, which then goes on to affect the output image. This is what you described in an earlier comment.

In addition, changing x might directly affect the image inside the optimizer, although I earlier concluded that x is a clone (that was on l-bfgs, could also be that adam and l-bfgs work differently).

My concern is whether the effect is through the gradient (note that changing x and recalculating also changes the gradient) or by directly messing with optimizer internals.

I can't answer this question as I don't know how and what the grad variable is. The output of the feval(x) function seems to come from: return loss, grad:view(grad:nElement()). Though I may be wrong about the grad variable being important for your question. I could also be widely off with my responses here due to my lack of understanding of what you are trying to figure out.

I think the discussion of the past few days has been about the possibility of combining them. Currently I don't think it is possible, at least not this way by playing around with code.

In crowsonkb's style_transfer, simultaneous DeepDream and style transfer seems to work pretty well, so I know that it is at least possible to do.

ProGamerGov on 11 Mar 2017

My current understanding of "DeepDream" hallucinations is that they are created via performing gradient ascent repeatedly. This makes sense considering I've seen DeepDream described as the opposite of the normal gradient descent process. Also from what I understand, normally a layer is selected to be "amplified" for many DeepDream implementations, though this is not required.

With understanding the basics of what makes DeepDream work, JcJohnson's DeepDream code (the original DeepDream code from Fast-Neural-Style, I was messing with), has to also perform gradient ascent repeatedly in order to produce the basic hallucinations that it does. I admit that I don't understand how Fast-Neural-Style's code performs gradient ascent, but I think that it's not doing so enough.

Modifying the DeepDream layer setup process like this (full script here):

if name == deepdream_layers[next_deepdream_idx] then
        print("Setting up Deepdream layer  ", i, ":", layer.name) 
    local loss_module = nn.DeepDreamLoss(params.deepdream_weights):type(dtype)
        for i=1,params.deepdream_iter do
          net:add(loss_module)
        end
        table.insert(deepdream_losses, loss_module)
        next_deepdream_idx = next_deepdream_idx + 1
end

Results in what one would expect from repeated gradient ascent. Neural-Style's normal parameters also interact with the DeepDream parameters as one would expect. Though one of the problems is that the DeepDream hallucinations slowly become brighter and brighter. I am currently testing different ways of correcting this if possible.

ProGamerGov on 11 Mar 2017

If you're interested, I've added some more color transfer functions: "lms", "rgb", "lab", "xyz", "hsl": https://github.com/VaKonS/neural-style/commit/49302ae6f2c7236d8028eee24d55e3dc2a4060ce.
They make very different images, so it's hard to tell which is better.

By the way, @htoyryla, try to use torch.std() instead of torch.var() – in my tests it behaved probably better.

VaKonS on 12 Mar 2017

👍2

Disregard my last comment. Creating a loop in the layer setup process is not needed. I believe I have found a better way to get simultaneous DeepDream and style transfer working, by adding only 1 line of code.

The full output image and variations of the output image can be found here: https://imgur.com/a/WGqgd

The modified script: https://gist.github.com/ProGamerGov/ce36ce2693d2a464c7d6fec0b97cf926

In the script on line 614, I have added:

self.output = self.output:mul(1/self.output:max())

I am currently trying to figure out why/how adding this line works.

For testing, I recommend a -deepdream_weights value of 0.05 for layers relu4_1 and up. For lower relu layers, one must use a lower -deepdream_weights value. Your chosen -style weight and -content_weight also will likely play a role in what -deepdream_weights value works best. Currently only 1 weight value is supported, but multiple -deepdream_layers values are supported.

ProGamerGov on 12 Mar 2017

Now I've definitely seen enough random changes, each morning something different. For me understanding the whole is essential. Trying out things is useful when it adds to this understanding. Neural-style works by inserting loss modules inside the network, then uses gradients in the backward pass, controlled by the optimizer, rippling down to the bottom and there modifying the image. Deepdream.lua implementations use the model as a whole, taking the gradient at the bottom and modifying the image through what they call normalized ascent, without any optimizer. Combining these approaches for me should start from an idea how these two can fruitfully coexist. I've done some attempts of my own to make a deep dream layer inside the model, but these fail to have the effect that direct tampering with the image has. And conversely, direct tampering with the image kills the idea on which neural-style is built, that the optimizer develops the image based on the gradients from the loss modules. Good luck.

PS. I don't personally even like the DeepDream look. Still, a year ago I experimented with using the FC layers in neural-style. The FC layers are classifiers which react to certain content, and them as content layers should result in deepdream-like effects. It did work, but I also found that it was possible to generate more pleasing (to my eyes) images.
http://liipetti.net/erratic/2016/03/28/controlling-image-content-with-fc-layers/
http://liipetti.net/erratic/2016/03/31/i-have-seen-a-neural-mirage/
http://liipetti.net/erratic/2016/04/20/getting-the-space-back/

htoyryla on 12 Mar 2017

@htoyryla Do you happen to still have the modified prototxt file for using the FC layers, and/or the code relating feature names from what the model was seeing, in Neural-Style?

And as for the DeepDream, that was off topic and not within the scope of this thread. I'll try to keep future comments in this thread related to the features in Gatys' code, and the associated research paper.

ProGamerGov on 12 Mar 2017

Any VGG prototxt already contains the FC layers. Like I wrote in my blog, the trick was to avoid size mismatch between the convlayers (which expand according to input) and FC layers which have fixed size. I used adaptive max pooling which worked well enough for 512px image size, but might be inaccurate for much larger images.

Something like this, in the old neural-style code, within the loop which builds net based on cnn.

 for i = 1, #cnn do
    local layer = cnn:get(i)
    if (torch.type(layer) == "nn.View") then 
        addlayer = nn.SpatialAdaptiveMaxPooling(7,7):float()
        if params.gpu >= 0 then
          if params.backend ~= 'clnn' then
            addlayer:cuda()
          else
            addlayer:cl()
          end
        end
    net:add(addlayer)
    end

I used the label names with places205 and places365, I believe that I found the list together with the model(s). At the beginning I read the labels into a table:

 --assuming we are using http://places.csail.mit.edu/downloadCNN.html
  f = io.open("/home/hannu/neural-style/models/places205_categories.txt")  
  for line in f:lines() do
    table.insert(labels, line)
  end

and then later I can print them after doing a net:forward(cimage), where cimage is either the content_image preprocessed for caffe (to show labels for the content image right after net has been built) or the current image (to show labels for the current iteration inside maybe_print).

local p = net:forward(cimage)
  print("----------- seeing the features --------------")
  for i=1, #labels do
    if p[i] > 0.03 then
      print(string.format("%.4f   %s", p[i], labels[i]))
    end
  end

This assumes a SoftMax layer above fc8 so I added it (just after net has been built, before cleaning up cnn)

--vgg16 places lacks softmax at the end, so insert one
  local prob = nn.SoftMax():float()
  if params.gpu >= 0 then
            if params.backend ~= 'clnn' then
              prob:cuda()
            else
              prob:cl()
            end
  end
  net:add(prob)

This is basically what is needed to a) display labels b) use fc layers as content layers. Finding good balance between weights is not easy, especially when using both a conv layer to control spatial arrangement of content and an fc layer to control general content. It helps if one provides separate weight settings for these.

I have been thinking of publishing a complete working solution, maybe when I have put together a version which feels solid enough.

htoyryla on 13 Mar 2017

👍2

@ProGamerGov it is not a problem for me if a thread goes on to do further experiments. There is need for such discussion around neuraI-style, not limited to the existing implementation. It is only that I'd rather not participate if it gets too chaotic (as I perceive it).

htoyryla on 13 Mar 2017

@ProGamerGov this response of yours illustrates the problem in our discussion concerning deepdream

I can't answer this question as I don't know how and what the grad variable is. The output of the feval(x) function seems to come from: return loss, grad:view(grad:nElement()). Though I may be wrong about the grad variable being important for your question. I could also be widely off with my responses here due to my lack of understanding of what you are trying to figure out.

Here you confirm that you don't understand how neural-style works. I wrote

Neural-style works by inserting loss modules inside the network, then uses gradients in the backward pass, controlled by the optimizer, rippling down to the bottom and there modifying the image.

Grad is the gradient obtained from the backward pass, after all layers have contributed to it. If the layers have been implemented properly, it indicates how the image should be modified to minimize the loss. Neural-style gives this to the optimizer which then figures out the how much to change in the direction shown by the gradient.

The standalone deepdream implementions modify the image directly based on the gradient.

To me, solving how these two processes could be combined should be based on an understanding their inherent difference. Not just copying, pasting and modifying code. I was not asking you questions, but trying to point out this. But I have already said this several times. I raise the central issue with what you are trying to do, something essential for this task, and you answer by effectively demonstrating that you don't understand it. This is why I don't enjoy these discussions anymore.

PS. When you write "The output of the feval(x) function seems to come from: return loss, grad:view(grad:nElement())" you miss my point: you also modify x which has been passed by reference (so it appears), in other words, you modify the image directly too.

htoyryla on 13 Mar 2017

@htoyryla May I ask more information about spatial control?
I have seen your saying:

As to the spatial control, isn't the idea that masks are used to define parts of the image, so that each area can be processed using a different style. To me it looks like the user is responsible for preparing the masks and assigning the styles to them. The idea seems quite simple to me, use multiple gram matrices each behind its own mask. The idea of a mask is basically very simple, like that a b&w image consisting of black and white pixels can be used to block or pass parts of the input into the gram matrix. (Haven't checked Gatys' code, this is simply how I understood the idea when reading the paper... at least the basic principle).

After seeing your saying, I finally got where I did wrong. I used a masked input image for style loss computing. And I didn't get expected output image. I should use masks for feature maps to generate new Gram matrixes. So, if I want to used two different styles, one for the ground and another for sky in one picture, Should I define two Gram matrixes, which are generated by two kinds of masked feature maps?

SilvesterHsu on 13 Mar 2017

@SSeanHsu I would proceed by having two Gram matrices, like you say, getting the same input but each through a different mask, but I haven't thought out the implementation in detail. I would place both gram matrices inside the same loss module, to ensure that the output will correctly match the input, and to get the gradInput by adding the gradients from the two Gram matrices filtered by the masks.

I have not attempted this myself yet... maybe later as an exercise. I enjoy using output from style transfer as material that I modify manually, like the background in this selfie.

17155251_10155221805158729_2034571380483984289_n

htoyryla on 13 Mar 2017

👍1

@htoyryla Thank you for sharing the idea. I'll try it. If it succeed, I'll let you know. By the way, your selfie is awesome. I love this one better than the one you are using now.

SilvesterHsu on 13 Mar 2017

@SSeanHsu one more thing. One also needs to find a way to make the style modules to capture two style targets and associate each of them to the right mask/Gram-matrix. And the masks need to be scaled so that they match the size of the output from the underlying convlayer.

htoyryla on 13 Mar 2017

I wrote _"the masks need to be scaled so that they match the size of the output from the underlying convlayer"_ but Gatys' paper says this does not confine the style to the area correctly, because the receptive fields of neurons extend outside the mask boundary (each neuron is affected by a proportionally larger part of the input image when we go to a higher level). Instead they propose using smaller masks for the areas with a specific style and an additional style for the whole image.

The most obvious approach would be to down- sample Tr to the dimensions of each layer’s feature map. However, we often find that doing so fails to keep the de- sired separation of styles by region, e.g., ground texture still appears in the sky. This is because neurons near the boundaries of a guidance region can have large receptive fields that overlap into the other region. Instead we use an eroded version of the spatial guiding channels. We define spatial guidance channels only on the neurons whose receptive field is entirely inside the guidance region and add another global guidance channel that is constant over the entire image. In that way we only enforce spatial guidance for neurons that have a clear assignment to a region and use the unguided style loss for neurons whose receptive field overlaps with the boundary.

Still, I guess, scaling the masks is a good way to start and see what happens.

htoyryla on 14 Mar 2017

Here's now my neural-mirage code (referred to in http://liipetti.net/erratic/2016/04/20/getting-the-space-back/)
https://gist.github.com/htoyryla/806ca4d978f0528114282cd00022ad71

I have been using this with places-205 vgg model from here http://places.csail.mit.edu/downloadCNN.html

Finding a good balance between generated content (fc layers, fc_weight), style and spatial content (relu layers, content_weight) is quite tricky and the weight depend heavily on the layers used.

Set content and fc layers using -content_layers. E.g.

-content_layers relu4_1,fc7,fc8

Set spatial content weight with content_weight, fc generated content weight with -fc_weight.

Looks like it can also produce deepdream-like output. Here I have used

-content_layers relu2_1,relu3_1,relu4_1,fc6
-content_weight 150 -fc_weight 8000 -style_weight 1e4

mirage-out-deepdream1

This is something one gets with mainly fc8 and style layers, only weak spatial control.

tubingen-mirage2_1500

Somewhat more spatial control, but I still can't get close to what I got from the Paris photo a year ago. It could have been a different model, though.

tubingen-mirage3b_700

Like I said, it is difficult to get a good balance between the three main elements: spatial content, fc content and style. I know I have done it before, as shown in the blog, but it takes time to get used to the controls.

Interesting though how drawing-like output one can get without any style image (i.e. using content image as style image).

paris-mirage4_350

htoyryla on 14 Mar 2017

👍1

@htoyryla The first image from the FC layers in have what appears to be the same baseball diamond as the one in this DeepDream image I made many months ago:

Full Image: https://i.imgur.com/dAweBXs.jpg

And the structures in your image, look like the structures from this other DeepDream session of mine: https://imgur.com/a/8vWDB

Full image: https://i.imgur.com/u1yaGKi.jpg

ProGamerGov on 14 Mar 2017

There may be something wrong with my mirage code. Although it does interesting things, I have not been able to achieve some essential effects I have used the same methods earlier. Need to delve deeper into it. Could also be that many of those early results were done on VGG19 and not places-205-vgg. Looks like there is a huge difference between VGG19 and places-205. Totally different animals when it gets to the activation levels from different layers. Makes me thing that one gets stronger hallusinations from VGG, whether deepdream type or natural looking (which I am after).

This came out of VGG19 right away. Spent hours with places-205-vgg and didn't get anywhere near. The only drawback with VGG19 is that I don't have a correct label file for it. With places-205 the labels worked quite good (but did not help much with image creation).

vgg19-paris-mirage5b_1400

Now this is getting somewhere... with VGG19.

vgg19-paris-mirage5c_2500

th neural_mirage5.lua -gpu 0 -content_image /home/hannu/Pictures/IMG_2840.JPG -content_layers relu3_1,fc8 -content_weight 5 -fc_weight 500000 -style_weight 500 -output_image vgg19-paris-mirage6y.png -style_scale 0.6 -num_iterations 4000 -model_file models/VGG_ILSVRC_19_layers.caffemodel -proto_file models/VGG_ILSVRC_19_layers_deploy.prototxt -label_file none -style_image examples/inputs/seated-nude.jpg

After 650 iterations.

vgg19-paris-mirage6y_650

htoyryla on 14 Mar 2017

@htoyryla I didn't succeed on spatial control. This is my method,

I define 2 kinds of Gram Matrix: GramSky and GramGround
The GramSky is generated by feature maps which are multiplied with sky mask. The GramGround uses the same steps, but using ground mask.
I place both gram matrices inside the StyleLoss module. As for the updateGradInput of StyleLoss, I derive the input in the loss function.
Codes are as fellows,
https://github.com/SSeanHsu/neural-style/blob/master/neural_style_spatial_control.lua

SilvesterHsu on 15 Mar 2017

@SSeanHsu it is not so simple at all how to implement the idea in torch/nn/neural-style.

The StyleLoss module gets the input from the underlying convlayer.
The input is branched into two parallel branches.
Each branch has first a mask and then a Gram matrix which is used to calculate losses and the gradient. Losses and gradients from both branches are added together (and with gradOutput) in updateGradInput.
updateOutput should just copy the unmasked input to output.

Then there is the question of setting style target, because now we need two targets. One could try using "captureSky" and "captureGround" for capturing two style targets targetSky and targetGround.

I haven't tried this but if I would, I would proceed like this.

PS. I wrote this before seeing the link to your code.

The first thing I noticed is that you use only a single style, so both Gram matrices are set to the same target (although masked differently). You need two style images, and like I wrote above, two capture modes (or something like that, to capture a different target for each Gram matrix).

A minor thing is that I would not have used two separate classes for the two Gram matrices, instead I would have used two instances of Gram matrix inside StyleLoss each with its own mask. Your approach should work too but I think it is more risky and error-prone.

htoyryla on 15 Mar 2017

@htoyryla I did so, and I also copy the unmasked input to output in updateOutput of both Gram matrixes. Do you mean I should define two modules ( "captureSky" and "captureGround" ) for capturing target? I capture both sky and ground targets in the updateOutput of StyleLoss, one target is GramSky(style image), another is GramGround(style image).
Here is my content image:

And this is mask:

Style image:

So when I pass the style image into StyleLoss, it would pass the GramSky with sky mask and GramGround with ground mask ( I do mask with the feature maps, not the style image ). So I would be able to capture sky target and ground target. The 2 targets I captured are different. Because of the sky mask, GramSky(style image) will be able to capture the style of "the_scream". And because of the ground mask, GramGround(style image) will be able to capture the style of "starry_night".
Here I pick one feature map of style image when passing GramGround, and I copy the unmasked input to output in updateOutput
( before masked / after masked)

I did this way, but I didn't get expected result. Do you have any idea where I did wrong? Or do you have any solutions?

SilvesterHsu on 15 Mar 2017

👍1

Now I see, you are using a single style image, combined of two styles. Fine if that works (and I don't see why it wouldn't).

I understood from your first post today that you couldn't get it to work when you wrote "I didn't succeed on spatial control".

My concept was different, using two (or more) style images, so that here https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L164-L175 we would first set the mode="capture1" and forward the first style image, then set mode=capture2 and forward the second. Correspondingly, in StyleLoss module when mode = "capture1" I would capture target1 and so on. Then there would be no need to create a mixed style image, just use any image as style as before.

Also, like I wrote, I would not modify GramMatrix class at all. I would just create two gram matrix instances here https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L526

  self.gram1 = nn.GramMatrix()
  self.gram2 = nn.GramMatrix()

and then I would apply the masks inside StyleLoss:updateOutput to input just before feeding it to the respective Gram matrices. This makes it easy to add more masked areas and styles. And so on. I may try it out myself when I have time.

htoyryla on 15 Mar 2017

I did that, and it succeeded. I simply scale the masks to generate Tr. As you said,

The most obvious approach would be to down- sample Tr to the dimensions of each layer’s feature map. However, we often find that doing so fails to keep the de- sired separation of styles by region, e.g., ground texture still appears in the sky. This is because neurons near the boundaries of a guidance region can have large receptive fields that overlap into the other region. Instead we use an eroded version of the spatial guiding channels. We define spatial guidance channels only on the neurons whose receptive field is entirely inside the guidance region and add another global guidance channel that is constant over the entire image. In that way we only enforce spatial guidance for neurons that have a clear assignment to a region and use the unguided style loss for neurons whose receptive field overlaps with the boundary.

there is much work to do. And I pretty enjoy this exercise :D

content image:

style image (sky part):

style image (ground part):

generated image (iterations 100 / 400)

SilvesterHsu on 16 Mar 2017

👍2

@htoyryla How would I allow for different DeepDream layer weights for each chosen DeepDream layer in my modified neural_style.lua?

The modified neural_style.lua https://gist.github.com/ProGamerGov/39be3ceb669ffe8369580a0d3ba3d3a7

Currently -deepdream_layers accepts multiple input layers separated by a comma. However -deepdream_weights only accepts a single weight value. I know it requires something like local deepdream_weights = params.deepdream_weights:split(","), but I am not sure what else I need to do for this feature?

ProGamerGov on 19 Mar 2017

@ProGamerGov if you want to apply different weights to layers of the same type, have a look how I implemented style_layers_weights for style layers: https://github.com/htoyryla/neural-style/blob/master/neural_style.lua .

htoyryla on 19 Mar 2017

@htoyryla I was wondering if your solution for supporting FC layers, would also work (with some modifications) could be made to work with inception layers from a model like the Places365 GoogleNet caffemodel? Or if not, do you know where I would start for supporting GoogleNet inception layers in Neural-Style (Preferably without requiring .t7 models)?

ProGamerGov on 20 Mar 2017

@ProGamerGov my guess is that loadcaffe is the showstopper for using networks like GoogleNet, ResNet etc. Loadcaffe is pretty limited as to what layer types it supports. Sometime last summer I spent some time training a fancy model with a special build of Caffe only to find out that loadcaffe would not load the caffemodel.

It is easy to check in th interpreter whether loadcaffe can load this GoogleNet caffemodel. You first require 'loadcaffe' and then cnn = loadcaffe(...) to see if it loads. My bet is negative.

The limitations of loadcaffe is probably why everybody is using t7 files today. With it one can get errors about missing nn classes, but that can be solved by adding the missing classes into to project.

htoyryla on 20 Mar 2017

@htoyryla Loadcaffe appears to be able to load the network, and in fact one can already use the conv2/relu_3x3 layer in the unmodified neural_style.lua:

Using the th interpreter, I get this:

th> require 'loadcaffe'
{
  load : function: 0x4061b9d0
  C : userdata: 0x406c9eb0
}
                                                                      [0.0356s]
th> model = loadcaffe.load('deploy_googlenet_places365.prototxt','googlenet_places365.caffemodel')
Successfully loaded googlenet_places365.caffemodel
warning: module 'data [type Data]' not found
warning: module 'label_data_1_split [type Split]' not found
warning: module 'pool2/3x3_s2_pool2/3x3_s2_0_split [type Split]' not found
warning: module 'inception_3a/output [type Concat]' not found
warning: module 'inception_3a/output_inception_3a/output_0_split [type Split]' not found
warning: module 'inception_3b/output [type Concat]' not found
warning: module 'pool3/3x3_s2_pool3/3x3_s2_0_split [type Split]' not found
warning: module 'inception_4a/output [type Concat]' not found
warning: module 'inception_4a/output_inception_4a/output_0_split [type Split]' not found
conv1/7x7_s2: 64 3 7 7
conv2/3x3_reduce: 64 64 1 1
conv2/3x3: 192 64 3 3
inception_3a/1x1: 64 192 1 1
inception_3a/3x3_reduce: 96 192 1 1
inception_3a/3x3: 128 96 3 3
inception_3a/5x5_reduce: 16 192 1 1
inception_3a/5x5: 32 16 5 5
inception_3a/pool_proj: 32 192 1 1
inception_3b/1x1: 128 256 1 1
inception_3b/3x3_reduce: 128 256 1 1
inception_3b/3x3: 192 128 3 3
inception_3b/5x5_reduce: 32 256 1 1
inception_3b/5x5: 96 32 5 5
inception_3b/pool_proj: 64 256 1 1
inception_4a/1x1: 192 480 1 1
inception_4a/3x3_reduce: 96 480 1 1
inception_4a/3x3: 208 96 3 3
inception_4a/5x5_reduce: 16 480 1 1
inception_4a/5x5: 48 16 5 5
inception_4a/pool_proj: 64 480 1 1
loss1/conv: 128 512 1 1
loss1/fc: 1 1 2048 1024
loss1/classifier: 1 1 1024 365
                                                                      [0.1000s]
th>

There are missing modules for some layers, but the model still seems to work with loadcaffe.

This is the lua prototxt file that neural-style produces: https://gist.github.com/ProGamerGov/8d524a05b8e31edd202d50cd7ca442d6

ProGamerGov on 21 Mar 2017

@ProGamerGov you and I then have a different meaning for "to be able to load the network". For me that implies loading the whole network as it is intended to work.

It looks like loadcaffe had loaded the layers it knows, resulting in what looks like a normal convolutional network, but has not loaded the split and concat layers that would implement the branches that would make the network GoogleNet. At a quick glance it looks like all the convlayers have been inserted in series, while in the original which was trained there were sets of parallel branches in series. Now when the correct structure is lost, you'll probably get size mismatch for all but some lowest layers. It will definitely not help to use adaptive pooling to solve the mismatches, because the layers are now not arranged as they were during training.

I now remember seeing this myself and also that it is possible to use the resulting model in neural-style but it is no longer GoogleNet as it is missing the parallel branches. To me this demonstrates that loadcaffe cannot load a GoogleNet. Yes, you get some kind of a model a part of which may still be used but it is nowhere near a real GoogleNet.

htoyryla on 21 Mar 2017

@htoyryla Taking your advice that loadcaffe was a dead end for supporting other model architectures, I followed this solution here on how to implement Torch model support (including ResNet and Inception models).

The modified neural_style.lua: https://gist.github.com/ProGamerGov/ec7dae4f1a07f5333d820d18a48a18a8

Provided that one folds the BatchNorm layers of a ResNet or Inception model into convolutional layers with a script like this, any Inception or ResNet model should work (I think).

ResNet models seem to be hit or miss in regards to Neural-Style parameters. The inception model I tested created a completely black output when I tried to use -init_image on it.

Edit: It seems that -init image does not work, but -init random does work, and either does -pooling avg (at least on ResNet models). My edits relating to Torch model support can be found on lines 85-93, and line 166.

ProGamerGov on 23 Mar 2017

@ProGamerGov, by the way, it looks like you've ignored the "Preprocessing" part for Torch model.

Sergey saved mean and std values with the model, and used them to preprocess / deprocess images for ResNet model differently than for Caffe models.

VaKonS on 24 Mar 2017

@VaKonS When I was testing the VGG models I converted to Torch models with jcjohnson's /cnn-benchmarks repository, I found that the original preprocessing and deprocessing functions seemed to work correctly. Though it appears they don't work for ResNet models.

This version of the script should correctly support Sergey's mean and std setup for Torch models: https://gist.github.com/ProGamerGov/9f2d9e8089b8569d83b707cc9f36b630

I'm not sure I there's a way to treat some Torch models like Caffe models (as in detecting their architecture), and whether that is necessary for some Caffe models.

ProGamerGov on 24 Mar 2017

👍1

@ProGamerGov, the pre/postprocessing of images probably depends on how a model was trained rather than on a model's architecture.
For VGG19 and VGG16 models, the required preprocessing is given in model's descriptions.

If creator of model used different settings for training, maybe there is no way to detect them automatically, unless they are saved with a model somehow.

On the other hand, "mean-centered" images seem to work fine with other models (NIN ImageNet, Illustration2Vec etc.), so maybe it's not so critical after all.

VaKonS on 25 Mar 2017

@VaKonS is right, as far as I know. In fact, I would differntiate between theee things:

the network architecture, like what layers are used and how they are connected
the value range used during training, raising the need for preprocessing
the format in which the actual trained model is stored

So neither the architecture (like vgg or resnet) nor the save format (caffemodel, t7, pth or whatever) dictate preprocessing, but the need to scale the pixel values to match what the model is trained to see. @ProGamerGov you may remember that for training with caffe, one needs to get the mean of the dataset. So as far as I understand, there is no single mean/std value for vgg either, the values depend on the dataset. On the other hand, differences between datasets may be so small that all models trained with caffe (i.e. handling the value range like caffe does) in practice work with the vgg19 ilsvrc12 preprocessing.

htoyryla on 25 Mar 2017

Using the deep-photo-styletransfer project (line 12 can be commented out), which uses the neural_style.lua code, one can created masked style transfer outputs.

The Content And Style Mask:

The Masked Output (neuralstyle_seg.lua):

th neuralstyle_seg.lua -seed 876 -save_iter 50 -content_image in21_input.png -content_seg in21_mask.png -style_image tar21_style.png -style_seg tar21_mask.png -backend cudnn -cudnn_autotune

The Control Output (From neural_style.lua):

th neural_style.lua -seed 876 -save_iter 50 -content_image in21_input.png -style_image tar21_style.png -image_size 700 -backend cudnn -cudnn_autotune

The masks, content, and style image: https://imgur.com/a/mNIU0

Though when trying to implement this code into the neural_style.lua code, I am running into issues converting the partial style loss function that seems to have been put into the style layer setup process.

ProGamerGov on 28 Mar 2017

@htoyryla How would I go about adding support for multi color masks?

I've got this code I extracted from a heavily modified neural_style.lua project, used for image segmentation, and some attempts to experiment with the code separately from Neural-Style here.

I have also created a few masked versions of Fast-Neural-Style's Chicago content image, here:

And A Style Image, here:

ProGamerGov on 30 Mar 2017

@htoyryla If one downloaded a single image from each of the categories used to train the VGG-16, and VGG-19 models, and then ran a single round of training, would that allow for the two VGG models to both have working label files? My thinking is that this idea might "lock in" the category values, into a known order. Have you tried anything like this before?

ProGamerGov on 7 Apr 2017

@ProGamerGov I understand your question so that you need, for VGG models, a file to convert label numbers to descriptive text (like I have used for the places model).

Running a single image through the VGG model will give you a probability for each of the 1000 outputs, not a category name. You would have to look for the outputs with the highest probabilities, one of which should be the correct one for the category for this image. The problem is that for a given image, it is not guaranteed that the strongest output is the correct one. So this is definitely not an easy way.

On the other hand, as people have been able to participate in the ILSVRC challenge and train their own models for the competition, the list certainly exists somewhere in the internet. Also, if you already know where to get the correct dataset for training, chances are that you will get the label texts from the same source.

In fact, I found two lists which I tried with my code. I guess one of them was the correct one, but still it wasn't useful for me as it was seeing so many irrelevant features in the images (like different kinds of animals where there were none in my images). Seems to me that this indicates that the usefulness of VGG models for style transfer does not rely on their capability to classify objects, but rather on their capability to detect generic visual features.

PS. There are several mentions in the net that in caffe, under data/ilsvrc12 there is a download script get_ilsvrc_aux.sh. One of the files downloaded will be synset_words.txt which should contain 1000 lines, one for each category.

This demo also contains an imagenet synset, in t7 ascii format https://github.com/szagoruyko/torch-opencv-demos/tree/master/imagenet_classification . You can read it in th by

words = torch.load("synset.t7", 'ascii')

htoyryla on 8 Apr 2017

@htoyryla Hi, long time no see. I am still work on the Spatial Control in this paper Controlling Perceptual Factors in Neural Style Transfer. But I have some doubt about details, as followed:

The former contribution of layer l to the total style loss is

And when we do updateGradInput in StyleLoss, the gradInput is

But in Spatial Control, the expression changes into this

So, suppose I only have two regions: Ground & Sky.

When we do updateGradInput in StyleLoss, does the gradInput equal to the expression below?

And does the two expressions below equal to each other?
The value of pixels in selected region on my mask is 1, others are 0. After reading the paper again, I find that I ignored to normalise the mask. But I am confused that why we need to normalise the mask?

SilvesterHsu on 13 Apr 2017

I have gotten Multi-Region Spatial Control working in Neural-Style, in this modified neural_style.lua: https://gist.github.com/ProGamerGov/bcbd27a3d2e431adb73ef158d9990d93

Regions are transferred from the style image(s), to the content image, based on matching colors in mask images. This way of doing Spatial Control means that you can omit regions on style images, by not having all the style mask colors, on your content mask.

ProGamerGov on 18 Apr 2017

@ProGamerGov I have read your code. It is really great. But I did not get why we should count gradient by using masked_features instead of using input line 727 just like this local gradient = self.gram:backward(input, dG). I know masked_features is a set of specially masked input, as well as I know it works. But why? What's more, I also don't understand Gram:updateGradInputline 596. I mean the math expressions. I read a book about matrix Matrix Cookbook. I did not get how Gram:updateGradInput works. I will ask its detail with jcjohnson.

SilvesterHsu on 20 Apr 2017

@htoyryla @SSeanHsu Do have any idea why the masked style transfer code in my modified neural_style.lua here: https://gist.github.com/ProGamerGov/bcbd27a3d2e431adb73ef158d9990d93, gives an error when I attempt to use the NIN model? Every combination of -content_layers and -style_layers that I've tried, does not work.

Successfully loaded models/nin_imagenet_conv.caffemodel
MODULE data UNDEFINED
warning: module 'data [type 5]' not found
conv1: 96 3 11 11
cccp1: 96 96 1 1
cccp2: 96 96 1 1
conv2: 256 96 5 5
cccp3: 256 256 1 1
cccp4: 256 256 1 1
conv3: 384 256 3 3
cccp5: 384 384 1 1
cccp6: 384 384 1 1
conv4-1024: 1024 384 3 3
cccp7-1024: 1024 1024 1 1
cccp8-1024: 1000 1024 1 1
Setting up content layer        15      :       conv3
Setting up style layer          15      :       conv3
Capturing content targets
nn.Sequential {
  [input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> (12) -> (13) -> (14) -> (15) -> (16) -> (17) -> output]
  (1): cudnn.SpatialConvolution(3 -> 96, 11x11, 4,4)
  (2): cudnn.ReLU
  (3): cudnn.SpatialConvolution(96 -> 96, 1x1)
  (4): cudnn.ReLU
  (5): cudnn.SpatialConvolution(96 -> 96, 1x1)
  (6): cudnn.ReLU
  (7): cudnn.SpatialMaxPooling(3x3, 2,2)
  (8): cudnn.SpatialConvolution(96 -> 256, 5x5, 1,1, 2,2)
  (9): cudnn.ReLU
  (10): cudnn.SpatialConvolution(256 -> 256, 1x1)
  (11): cudnn.ReLU
  (12): cudnn.SpatialConvolution(256 -> 256, 1x1)
  (13): cudnn.ReLU
  (14): cudnn.SpatialMaxPooling(3x3, 2,2)
  (15): cudnn.SpatialConvolution(256 -> 384, 3x3, 1,1, 1,1)
  (16): nn.ContentLoss
  (17): nn.MaskedStyleLoss
}
Capturing style target 1
/home/ubuntu/torch/install/bin/luajit: /home/ubuntu/torch/install/share/lua/5.1/nn/Container.lua:67:
In 17 module of nn.Sequential:
/home/ubuntu/torch/install/share/lua/5.1/torch/Tensor.lua:322: incorrect size: only supporting singleton expansion (size=1)
stack traceback:
        [C]: in function 'error'
        /home/ubuntu/torch/install/share/lua/5.1/torch/Tensor.lua:322: in function 'expandAs'
        neural_style_seg.lua:698: in function <neural_style_seg.lua:685>
        [C]: in function 'xpcall'
        /home/ubuntu/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'
        /home/ubuntu/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
        neural_style_seg.lua:265: in function 'main'
        neural_style_seg.lua:838: in main chunk
        [C]: in function 'dofile'
        ...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00405d50

WARNING: If you see a stack trace below, it doesn't point to the place where this error occurred. Please use only the one above.
stack traceback:
        [C]: in function 'error'
        /home/ubuntu/torch/install/share/lua/5.1/nn/Container.lua:67: in function 'rethrowErrors'
        /home/ubuntu/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
        neural_style_seg.lua:265: in function 'main'
        neural_style_seg.lua:838: in main chunk
        [C]: in function 'dofile'
        ...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00405d50
ubuntu@ip-Address:~/neural-style$

There is no size difference between my images, and their associated masks, which I found to be the cause of this error previously.

Line number 265 is referenced in the error message, yet I can't figure why as this code works for VGG models.

Edit:

This line of code:

local l_mask = l_mask_ori:repeatTensor(1,1,1):expandAs(input)

Seems to be the cause.

ProGamerGov on 28 Apr 2017

@ProGamerGov, what the line "local l_mask = l_mask_ori:repeatTensor(1,1,1):expandAs(input)" is supposed to do?
It seems to:

take a mask;
then repeats it _once_ (in that case torch.repeatTensor does nothing, if I'm not mistaken);
then uses torch.expandAs(input) on it.
"expandAs" is intended to repeat 1 element n times, according to shape of reference tensor – 1 pixel to n*m plane, for example.
It can not expand dimensions with more than one element.
And mask is probably not 1-pixel sized.

Did you mean to resize the mask to the size of "input"? Then maybe image.scale could help?
But if masks are already image-sized, then transformations are not needed?

VaKonS on 1 May 2017

Here's something that I tried recently http://liipetti.net/erratic/2017/05/13/randomizing-neural-style-transfer/

htoyryla on 13 May 2017

@htoyryla I think I have been working on something similar recently. I have been messing around with the individual channels of a layer (not sure if every model has something like "channels" for each layer?).

I saw this post on Reddit a long time ago, which detailed using a "iterations/octave curve" to bring out unseen features in a selected layer for DeepDream hallucinations:
https://www.reddit.com/r/deepdream/comments/3d04jy/an_iterationsoctave_curve/

My code here, lets the user select the desired "channel" of a layer, for DeepDream: https://github.com/ProGamerGov/Protobuf-Dreamer, which seems to create outputs that are really similar to that of the Reddit post I linked to.

So I am now wondering if/how "channels" work in Neural-Style, and whether or not it would be possible to control them like how one controls layers. If messing with the DeepDream code by using an octave/iteration curve seems to favor some "channels" more than others, then I wonder if what you are doing is creating a similar effect?

ProGamerGov on 14 May 2017

_"not sure if every model has something like "channels" for each layer?"_

I think in order to work in neural-style there has to be. The fact that there are several channels per layer greatly increase the number of different features that the network can recognize.

_"So I am now wondering if/how "channels" work in Neural-Style, and whether or not it would be possible to control them like how one controls layers."_

An output of a layers is a Tensor C x H x W where the C is the number of channels. Each channel reacts to different features, as one can see using my https://github.com/htoyryla/convis/blob/master/convis.lua.

In the style loss module, neural-style takes the C x H X W layer output and calculates a Gram matrix which is a C x C Tensor. It can be thought of some kind of statistical information.

One could experiment by simply leaving out some channels, which would make the Gram matrix smaller. Or alternatively, zeroing those channels which one wants to exclude. The latter alternative might work even with only a few channels left active. One could also experiment with multiplying each channel with a weight before calculating the Gram matrix. I might try out these now that I am anyway testing how the Gram matrix affects the results (although my interest in not in the hallusinations inherent in the layers but creating and modifying original styles).

_"using an octave/iteration curve seems to favor some "channels" more than others, then I wonder if what you are doing is creating a similar effect?"_

It is quite likely that randomizing the eigenvectors of a Gram matrix really changes the balance between the channels.

htoyryla on 14 May 2017

"One could also experiment with multiplying each channel with a weight before calculating the Gram matrix. "

Something like this. Make a mask Tensor, here I am zeroing the second last channel of the input. Input is the input to the style loss module, that means input is the output of a layer consisting of C channels of H x W each.

```
function inputMask(C, H, W)
local t = torch.Tensor(C,H,W):fill(1)
local m = torch.Tensor(C,H,W):fill(1)
-- make a mask to zero the second last channel
m[C-1] = m[C-1]*0
return t:cmul(m):cuda()
end

Then apply the mask in the Style loss module

function StyleLoss:updateOutput(input)
input = input:cmul(inputMask(input:size(1), input:size(2), input:size(3)))

...

function StyleLoss:updateGradInput(input, gradOutput)
input = input:cmul(inputMask(input:size(1), input:size(2), input:size(3)))
```

This does not yet create hallusinations from the channels; this is simply a way to mask channels and can also be used to apply different weights to channels. Now the same mask is applied to all layers, which is quite awkward. A more useful approach would be to create a mask for each layer when the style loss module is created and store in the module.

Anyhow, here's a picture created using a mask that zeroes half of the channels in each layer; style layers relu3_1 and 4_1 used.

nscw

htoyryla on 14 May 2017

Just noticed that modifying input has side effects, affecting content too, probably need to make a local copy of it inside the style loss module and make sure to pass the unmasked input to output.

function StyleLoss:updateOutput(input)
  local inp = input:clone():cmul(inputMask(input:size(1), input:size(2), input:size(3)))    
  self.G = self.gram:forward(inp)
  self.G:div(inp:nElement())
  if self.mode == 'capture' then
    if self.blend_weight == nil then
      self.target:resizeAs(self.G):copy(self.G)
    elseif self.target:nElement() == 0 then
      self.target:resizeAs(self.G):copy(self.G):mul(self.blend_weight)
    else
      self.target:add(self.blend_weight, self.G)
    end
  elseif self.mode == 'loss' then
    self.loss = self.strength * self.crit:forward(self.G, self.target)
  end
  self.output = input
  return self.output
end

function StyleLoss:updateGradInput(input, gradOutput)
  local inp = input:clone():cmul(inputMask(input:size(1), input:size(2), input:size(3)))    
  if self.mode == 'loss' then
    local dG = self.crit:backward(self.G, self.target)
    dG:div(input:nElement())
    self.gradInput = self.gram:backward(inp, dG)
    if self.normalize then
      self.gradInput:div(torch.norm(self.gradInput, 1) + 1e-8)
    end
    self.gradInput:mul(self.strength)
    self.gradInput:add(gradOutput)
  else
    self.gradInput = gradOutput
  end
  return self.gradInput
end

htoyryla on 14 May 2017

I made a test to use only two channels per layer, zeroing all the others. This did not work well, style losses were very small, style did not have much effect and increasing style weight did not really help.

This worked better. Instead of zeroing channels, multiply them by e.g. 0.2. Then multiply the rest of the channels by a factor greater than 1. In effect we are now applying channel specific weights.

I am now using a modified input for style only; for hallusinations one might try the same method for content, too.

function inputMask(C, H, W)
    local t = torch.Tensor(C,H,W):fill(1)
    local m = torch.Tensor(C,H,W):fill(1)
    for i=1,C do
          if i < 3 then
            m[i] = m[i]*3
          else
        m[i] = m[i]*0.2
          end
    end
    return t:cmul(m):cuda()
end

By using this on relu3_1,relu4_1 with style weight 1e6 I get this.

nscw002-2ch

Having

          if i < 3 then
            m[i] = m[i]*30
          else
        m[i] = m[i]*0.2
          end

and again relu3_1,relu4_1 with style weight 2e4 gives this.

nscw002b-2ch

This means we can achieve different results by selectively emphasize specific channels. My interest is in style, so I have been tampering with how the channels influence the style generation. For hallusinative dreaming, doing the same with content loss might give results?

htoyryla on 14 May 2017

@VaKonS wrote
_"Did you mean to resize the mask to the size of "input"? Then maybe image.scale could help?
But if masks are already image-sized, then transformations are not needed?"_

I have not looked into this new code of @ProGamerGov's . But style loss modules do have to adapt to the input size (meaning the size of the input to the style loss module) because it depends not only on the size of the input image but also on the layer to which the module is attached. Also, as I noticed today when working on the code posted just above, the first time StyleLoss:updateOutput gets called before capture mode has been set, and the input has different dimensions than on all later occasions. Probably as a result of this line https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L218. So the style loss module absolutely has to take care of adapting to the size of input, or otherwise a size mismatch will occur.

Despite this, I don't immediately see how maskedStyleLoss is intended to work either.

htoyryla on 14 May 2017

I put together a script to allow experimenting giving emphasis (5) to specific channels on a style layer. Other channels will be attenuated (0.2). Use param -style_channels (e.g.-style_layers 3_1 -style_channels 17,23).

Note that this is likely to reduce average output from the layers, so style_weight may have to be adjusted.

https://gist.github.com/htoyryla/fe6dd64611638b3db89755d68735d3e3

htoyryla on 14 May 2017

I assume these are the are the correct amount of channels for each layer on the default VGG-19 model?

Layer | Number Of Channels
| ----- |:-----:|
conv1_1 | 64
conv1_2 | 64
conv2_1 | 128
conv2_2 | 128
conv3_1 | 256
conv3_2 | 256
conv3_3 | 256
conv3_4 | 256
conv4_1 | 512
conv4_2 | 512
conv4_3 | 512
conv4_4 | 512
conv5_1 | 512
conv5_2 | 512
conv5_3 | 512
conv5_4 | 512
fc6 | 4096
fc7 | 4096
fc8 | 4096

Neural-Style Console Output:

conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv3_4: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv4_4: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
conv5_4: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8: 1 1 4096 1000

I also have found this site helpful for generating large ranges of sequential numbers, for testing: http://textmechanic.com/text-tools/numeration-tools/generate-list-numbers/

ProGamerGov on 17 May 2017

The conv layers look ok to me.

FC layers do not have channels (in the sense of a HxW map corresponding to an image), only single numeric outputs correlating with features. Yet, masking unwanted outputs may also give interesting results.

htoyryla on 17 May 2017

Speaking of multi-resolution: it looks like transferring style at lower resolution keeps more complex features of style (for example below, not simply shapes of plasticine pieces, but whole flowers).

The second good thing here is that reprocessing at full resolution preserves almost all features of low resolution pass.
It makes possible to process images of any size, I think.

dual_transfer dual_transfer_2

VaKonS on 26 May 2017

_"it looks like transferring style at lower resolution keeps more complex features of style (for example below, not simply shapes of plasticine pieces, but whole flowers)"_

Sounds like the basic scalability problem in neural-style transfer, the result changes as with higher resolution each neuron sees sees only a smaller part of the image. It has been discussed a lot and multi-resolution processing appears to solve it.

Slightly off topic: three of my neurally assisted works are just now on display at Art Fair Suomi, a large contemporary art event in Helsinki: https://twitter.com/htoyryla/status/867460046665523200

htoyryla on 26 May 2017

👍2

Congrats on the Art Fair. Please tell us how they were received, very
curious what the general public think.

On Sun, Jun 4, 2017 at 8:52 PM, ProGamerGov notifications@github.com
wrote:

@htoyryla https://github.com/htoyryla Congratulations on getting your
artwork in an art fair!

Do you know of any neural_style.lua experiments involving a denoiser?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jcjohnson/neural-style/issues/376#issuecomment-306062880,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHAt1TCq0ERB7CB008RgJuvklxunTZBDks5sAwsWgaJpZM4L94hh
.

rodmcfall on 12 Jun 2017

_"Congrats on the Art Fair. Please tell us how they were received, very
curious what the general public think."_

It was a big event, 300 artists, altogether around 1000 widely different works of contemporary art. Only very few artists received individual attention. As far as I know, my works were the only ones made using neural techniques, but this fact was not advertised. For me the main thing was that my works fit well together with the rest (which was really varied). Also, when talking with other artists, they showed a clear interest in the methods I had used. I guess that is because most artists who participated somehow work with alternative or experimental forms of art.

htoyryla on 13 Jun 2017

👍1

Btw, @ProGamerGov, just wanted to thank you for visual examples for different models on Wiki page.
p.s. Updated the link for DeepLab there, they seem to moved the page.

@htoyryla, maybe you'll find it interesting: Network Dissection project has Places model, trained at different iterations – from 1 to 2400818.
Maybe it's possible to discover how training affects stylization.

VaKonS on 22 Jul 2017

👍1

@VaKonS thanks for the link to the Network Dissection site. Also their findings about channels that respond to specific features could be useful, now that I have a version of neural-style which can use speficic channels only.

BTW the places model which is available from multiple iterations is not the VGG16 based places model that I have used, but a smaller one with only 5 conv layers.

htoyryla on 23 Jul 2017

👍2

@VaKonS I was able to influence a model's style transfer ability with finetuning here: https://github.com/jcjohnson/neural-style/issues/292#issuecomment-300685209

I provided a saved model at every 1000 iterations, so that one could see the change over time caused by the fine-tuning.

@htoyryla

I noticed that on some of the channel examples like this, and this, from my Protobuf-Dreamer project, seem to have "dead channels". These dead channels don't cause any visible hallucinations. So I was wondering if we would find similar "dead channels" by creating hallucinations of different channels from Neural-Style models? And if we can, would style transfer be improved by removing these "dead channels", or would it be improved by only using the "dead channels"?

I have also figured out that one can use a model's image classification abilities to identify what each channel hallucination contains (like dogs, plants, cars, etc...). If we used the model to attach the applicable category names to each channel, and then had some kind of search/selection function, could we create better stylized outputs? Like for example, I would select the "dog" related channels, and potentially some other channels (for artistic effects), to stylize a dog content image.

ProGamerGov on 23 Aug 2017

👍1

@htoyryla @VaKonS Have either of you ever come across shapes that resemble animals, or the classic heart shape? And these shapes did not appear to be in any content or style images that you used?

This rooster showed up in one of my style transfer outputs, and I could not find a similar shape in either the content image or the style image.

The heart shape seems to be more common than any other shapes. And there were no hearts in either the content or style images used.

Are these shapes and objects produced by the model being as some kind of error or hallucination? Is it just my brain's pattern/shape recognition abilities being tricked? Or is the geometry of these shapes/objects simple and common enough that they can show up in a combination that tricks the brain?

ProGamerGov on 25 Aug 2017

@htoyryla @jcjohnson A related thing I experienced was that a classification neural network thought this image was a picture of a dog:

I know dogs were obviously part of the training data used to create the model, so I am wondering if there are dog related shapes hidden in the image. If this is the case, then it seems like Neural-Style might be creating Adversarial Example Images, which are meant to trick classification networks into making incorrect predictions, with details that are normally hidden to the naked eye.

ProGamerGov on 25 Aug 2017

@ProGamerGov, regarding the "dog": maybe this "torch-visbox" project could help to identify a part, sensed a "dog" there (together with @htoyryla's "convis"). I didn't try any of them, though.

VaKonS on 27 Aug 2017

Could anyone point me in the right direction for making the -style_image parameter accept a directory as an input?

ProGamerGov on 9 Sep 2017

@ProGamerGov, you can try this.
1) Before the line 64 (params.style_image:split(',')), insert this code:

print( "---" )
print( "-style_image:" )
print( params.style_image or "" )
if params.style_image ~= nil then
print( "---" )
print( "Bytes of 'params.style_image':" )
print( string.byte(params.style_image, 1, -1) )
local style_images_list_with_commas, style_images_list_separators_count = string.gsub(params.style_image, "\10", ",")
print( "Merged", style_images_list_separators_count, "lines, result:", style_images_list_with_commas )
params.style_image = style_images_list_with_commas
end
print( "---" )
print( "New 'style_image':" )
print( params.style_image )
print( "---" )

Print operators are just for check and can be removed, of course):

if params.style_image ~= nil then local style_images_list_with_commas, style_images_list_separators_count = string.gsub(params.style_image, "\10", ",") ; params.style_image = style_images_list_with_commas end

Run neural-style with "find" output as the "-style_image" argument:

th neural_style.lua -style_image "$( find DIRECTORY_NAME/* -type f )"

2nd step will pass the directory listing, and 1st step will convert it to a comma-separated list.

VaKonS on 12 Sep 2017

@VaKonS Thanks, I'll try that out. Experiments involving hundreds, or even thousands of style images at once can become really tedious when they have to be added via the normal way.

ProGamerGov on 13 Sep 2017

@ProGamerGov, speaking of "hundreds, or even thousands" – I didn't try that many images with this method (3 works :) ).
If it will hang or something, here is another way, by passing a directory name:
https://github.com/rtqichen/style-swap

There is a parameter --contentBatch for passing a directory, then files other than images are sorted out in "ImageLoader.lua", and as a result you should have a table "files" with images list (it should be similar to "style_image_list", I guess).

This should work with large number of images.

VaKonS on 13 Sep 2017

@htoyryla Have you done any farther experimentation with neural-channels.lua? And if so, what are your findings?

ProGamerGov on 20 Sep 2017

@ProGamerGov No, I haven't really tested or used it any further. Could be interesting, but there is so much else to do.

htoyryla on 21 Sep 2017

@VaKonS I tried your version of neural_style.lua that's been modified to have built in tiling, but it didn't like me using a -style_scale value of 0.5 for any of the scale related parameters you added. It does look promising though in that it blends the tile edges together better than tiling with an external script.

ProGamerGov on 28 Sep 2017

@VaKonS Thanks for the link to the lua project with batch image collection.

I think that using the code below:

local function styleLoader(dir)
    local style_image_files = paths.dir(dir)
    local i=1
    while i <= #style_image_files do
        if not string.find(style_image_files[i], 'jpg$') 
            and not string.find(style_image_files[i], 'png$')
        and not string.find(style_image_files[i], 'pgm$')
            and not string.find(style_image_files[i], 'ppm$')then
            table.remove(style_image_files, i)
        else
            i = i +1
        end
    end
end

I could simply create a string that style_image_list would accept. Where each time a valid image is found, I append it's name to end to the end of the specified path, and then add a comma for separation (or add the comma first, and then the image and it's path). The style_image_list variable is string based, and thus using a table seems like it could be more complicated than just adding onto the end of a chain of strings.

I'd also have to use the Torch Paths library via: require 'paths', to know how many times to loop for adding a new image to the string. But it would be better if I didn't need to use an extra library.

I also wonder if I can do things in such a way that I only need the existing -style_image parameter, and not a new parameter.

Edit:

This seems to work:

require 'paths'

  local style_image_list
  if is_dir(params.style_image) then     
    style_image_list = styleLoader(params.style_image):split(',')
  else 
    style_image_list = params.style_image:split(',')
  end

These are the functions used:

function is_dir(path)
    f = io.open(path)
    return not f:read(0) and f:seek("end") ~= 0
end


function styleLoader(dir)
    local style_list
    local style_image_files = paths.dir(dir)
    local i=1
    while i <= #style_image_files do
        if not string.find(style_image_files[i], 'jpg$') 
        and not string.find(style_image_files[i], 'JPG$')
            and not string.find(style_image_files[i], 'jpeg$')
        and not string.find(style_image_files[i], 'JPEG$')
        and not string.find(style_image_files[i], 'png$')
        and not string.find(style_image_files[i], 'pgm$')
            and not string.find(style_image_files[i], 'ppm$')then
            table.remove(style_image_files, i)
        else
            style_image_files[i] = dir .. style_image_files[i]
            i = i +1
        end
    end
    style_list = table.concat(style_image_files,",")
    return style_list
end

~~The only issue now is that the image names in the directory lack their associated path.~~ I fixed the issue with: style_image_files[i] = dir .. style_image_files[i]

I found the list of image formats that the Torch Image package accepts, here

The gist file for the above code: https://gist.github.com/ProGamerGov/86c80c74f5e403748659e013447b2499/6de58d6bde6f0fb9ff9b9e20d2ea4294b7d030be

Lines 69-74 and 426-455, implement the new code.

There are some errors when trying to use the -style_image parameter normally:

/home/ubuntu/torch/install/bin/luajit: neural_style_dir.lua:429: attempt to index global 'f' (a nil value)
stack traceback:
        neural_style_dir.lua:429: in function 'is_dir'
        neural_style_dir.lua:70: in function 'main'
        neural_style_dir.lua:644: in main chunk
        [C]: in function 'dofile'
        ...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00405d50
ubuntu@ip-Address:~/neural-style$

It appears that my way of checking for a directory or a specific image, can't handle using multiple specific images with a directory on the front like: examples/inputs/style.png,examples/inputs/style2.png for the -style image parameter.

I'm not sure how to fix this issue while retaining the desired functionality?

Maybe it could be done in such a way that a user could specify multiple different directories and specific images, in the same command? Like for example: -style image examples/inputs/style.png,examples/inputs/style_dir1/,examples/inputs/style2.png,examples/inputs/style_dir2/. It might also be nice to find a way to add a "/" to the end of a specified directory, if the user forgets to do so.

Second Edit:

This should work, while eliminating the unnecessary is_dir function:

local style_image_list, style_input_sorted
local p = 1
local style_input = params.style_image:split(',')

  while p <= #style_input do
    if not paths.dirp(style_input[p])then
      style_input_sorted = (style_input[p]) 
      p = p + 1
    elseif paths.dirp(style_input[p])then
      style_input_sorted = styleLoader(style_input[p]) 
      p = p + 1
    end
    if p == 2 then
      style_image_list = style_input_sorted
    else
      style_image_list = style_image_list .. "," .. style_input_sorted
    end
  end

  style_image_list = style_image_list:split(',')

You can now specify multiple directories, and specific images, all at once, like for example:

-style_image style_dir1/,style_dir2/style_set1/,style1.png,style_dir3/style2.png

And you don't need to follow a specific order:

-style_image style_dir1/,style1.png,style_dir3/style2.png,style_dir2/style_set1/

The current version: https://gist.github.com/ProGamerGov/86c80c74f5e403748659e013447b2499

ProGamerGov on 28 Sep 2017

@ProGamerGov, I tried something like this – first it attempts to open the parameter as directory, then as file, and if everything failed, simply splits it "as is".
p.s. I'll look what's wrong with tiled neural-style version, thanks for checking.

require "paths"

local cmd = torch.CmdLine()
cmd:option('-style_image', 'examples/inputs/seated-nude.jpg',
           'Style target image/directory.')
local params = cmd:parse(arg)

print( "-params.style_image: \"" .. params.style_image .. "\".")
local style_image_list = paths.dir(params.style_image)
if style_image_list ~= nil then -- directory
  print( "Path mode." )
  if string.sub(params.style_image, -1, -1) == "/" then -- remove trailing slash if present
    params.style_image = string.sub(params.style_image, 1, -2)
  end
  local i = 1
  while i <= #style_image_list do
    local fl = string.lower(style_image_list[i])
    if not string.find(fl, 'jpg$')
       and not string.find(fl, 'jpeg$')
       and not string.find(fl, 'png$')
       and not string.find(fl, 'pgm$')
       and not string.find(fl, 'ppm$') then
      table.remove(style_image_list, i)
    else
      style_image_list[i] = params.style_image .. "/" .. style_image_list[i]
      i = i + 1
    end
  end
  assert(#style_image_list > 0, "All files skipped, style images list is empty. Stop.")
else -- file or list
  local is_file = io.open( params.style_image )
  if is_file ~= nil then -- single file, pass name as given
    io.close(is_file)
    print( "File mode." )
    style_image_list = { params.style_image }
  else -- split as comma separated list, don't check items
    print( "List mode." )
    style_image_list = params.style_image:split(',')
  end
end
print( style_image_list )

VaKonS on 1 Oct 2017

@VaKonS Thanks for sharing a more refined version of the directory/image handling code!

You should also consider enabling the issues option on your tiled version of neural_style.lua, so that issues can be reported directly on the project page. Things like saving iterations and initialization images don't work, and it seems odd that the size of the content image dictates the final end size of the tiled output.

Despite the issues, it's really promising! I assume it works similar to crowsonkb's style_transfer?

ProGamerGov on 3 Oct 2017

@ProGamerGov, yes, it's something like @crowsonkb's or maybe @mtyka's style transfer, but with simple fragments overlaying, without modifications of optimizers.

It's strange that style scaling doesn't work for you, it works here. Maybe I forgot to put some type conversions, could you please check it with "-gpu -1 -backend nn" options?
Also, the initialization image seems to work for me – it adopts features from both content and style, yet still noticeable in the result:

cc5sb100ie

Iterations saving _is_ a limitation at the moment, because I can not make stylization work with 1 iteration precision.
You can set "-save_iter" to 1, then it will save every row of processed fragments (instead of every iteration).
p.s. You're welcome to open issues. :)

VaKonS on 7 Oct 2017

Neural-style: Implementing features from the "Controlling Perceptual Factors in Neural Style Transfer" research paper

Most helpful comment

All 228 comments

Combine 2 images with fine and coarse scale:

norm net:

Naive scale combination:

The code in addition to creating the two images for each example, also created 4 stylemix images:

Stylemix image: https://i.imgur.com/m7nRgKP.jpg

Stylemix image: https://i.imgur.com/XPt6N52.jpg

Stylemix image: https://i.imgur.com/Vf3mg2n.jpg

Stylemix image: https://i.imgur.com/c1ZNDoZ.jpg

Step 1:

Step 2:

Step 3 (Optional):

The Content And Style Mask:

The Masked Output (neuralstyle_seg.lua):

The Control Output (From neural_style.lua):

Related issues