Turicreate: Style transfer: Bad results. Greenish, reddish, wavy artifacts.

[banoslo]

I am trying to apply style transfer on images. Unfortunately most results have greenish, reddish, wavy artifacts.

This is my setting:
I use Ubuntu 18 and the Turicreate 5.5 version for Python 3.6 with mxnet CUDA 9.1 on a NVIDIA GTX 950. I use the code lines from the Docs with a 100.000 (on one example 10000) training iterations.

Maybe it is related to https://github.com/apple/turicreate/issues/666 and https://github.com/apple/turicreate/issues/1341

4 examples with models can be downloaded here: https://drive.google.com/open?id=1tTkcpryOW4u5B4MBy4fGHxiBtU2bZU-j

Here is one example:

Output:

Input:

Style:

Here is another example:

Output:

Input:

Style:

As a side note: The original paper seems to have same issues (https://arxiv.org/pdf/1610.07629.pdf). Take a look at page 16 (Roy Lichtenstein, Bicentennial Print (1975)). Zoom-in to the Brad Pitt portrait.

Best regards,

[romanoss]

@banoslo
I had the same results. More Iterations, more content images don't help.
I have quickly looked into the original paper and you are probably right. It seems to be a feature of the model and this is underwhelming to say the least. Especially if one compares the results to the ones from a vgg19 model.
@TobyRoseman should I provide some images for comparison?

[TobyRoseman]

@romanoss - Yes, please share your images for comparison. Having more examples of this problem would be helpful. Thank you.

[romanoss]

@TobyRoseman
orig vs vgg19 vs tc 5.4 -with stripes is tc version - imho a big difference

[romanoss]

@TobyRoseman
this should be resolved now - as I tried it with the 5.5 changes

[banoslo]

Wait. Why did you close the ticket? This is not solved for version 5.5 actually. Did you reproduce the artifacts with my image files? You can easily reproduce them with files at https://drive.google.com/open?id=1tTkcpryOW4u5B4MBy4fGHxiBtU2bZU-j as I written before.

[TobyRoseman]

@banoslo - apologizes for closing the issues. We improved style transfer with the 5.5 release and @romanoss seemed to indicate this issue had been fixed. Can you please summarize what is still outstanding with this issue?

[romanoss]

hmm- I am not sure about your problem @banoslo .
Generally, it seems that this model is prone to produce some stripes, as one can see in the original paper too. The worst artifacts went away for me with the 5.5 patch

Nevertheless, there are some points I don't understand right now.
Actual changes and some testing in the past indicate that one should calculate the model with iterations = styles * 10000 (which is ridiculous on a Cpu :) )
I did not try this, but it seems that the model converges fast (~3000 iters) and stays that way for a long time and then improves again at a high iteration value.

Did you try high iteration values?

regards
Roman

[banoslo]

@romanoss @TobyRoseman

As written in my description, I use 5.5 with 10000 - 100000 iterations and have the issues as shown in my examples (also download the files from Google Drive).

Best regards,
Daniel

[romanoss]

@TobyRoseman @banoslo
Ok - I managed to test the 5.6 version of style transfer in tc
I trained 1 style for about 10000 iterations in (512,512) and as far as one can see the problems of version 5.4 are resolved here(stripes version is 5.4)

btw default iterations changed from 5*numStyles+2000 to numStyles*10000 which is quite a difference
fun fact: if you train 20 styles this is 200000 iterations which are 5 days on a Tesla T4 and probably over a year for a mac :)

[banoslo]

Thanks for your reply.

But if you zoom in, you will see a dramatic increase in many light greenish violett waves in the first image!

[romanoss]

@banoslo
first image is tc 5.4 version, that really had severe problems - there was an update in 5.5, which is much nicer

[banoslo]

Thanks for you fast reply.

Agree, the first one is problematic.

But the second image still has got the greenish red wave issue. Take a look at the right side of the nose. Some light red/green issues at the beard. The beanie has the typical green all over the place and a profound red/green issue at the upper right.

Best regards

[romanoss]

@banoslo
I can see what you mean.
hmm-I guessed the red/green values at the beanie were a style feature and I did not recognize the small wave at the right side of the nose -hmm - hard to say. @TobyRoseman what do u think?

[romanoss]

@banoslo I can try this one again with 10000 it with the coco image set instead of my 200 private content images

[romanoss]

@banoslo you are definitely right - the issue still exists - to a lesser degree but it exists.
Here are two more pictures:
5.4

5.6

@TobyRoseman can you see it?

The strips have a lower frequency and intensity
The effect is seen better on low res images
the waves in red sections have a different direction than in greenish sections

[banoslo]

Thanks Roman for verifying this. Best regards!

[romanoss]

@abhishekpratapa can you pls take a look?

[TobyRoseman]

@romanoss and @banoslo - Thanks for the detailed discussion. I do see what you mean now; looks like 5.6 is a big improvement but we still have a way to go. Do either of you have any ideas about how this could be fixed?

@abhishekpratapa - what do you think?

[romanoss]

@TobyRoseman
@abhishekpratapa added some padding layers - maybe they can be tuned?

[banoslo]

@TobyRoseman Maybe you could ask Leon Gatys, he works at Apple now as well. Maybe, he saw similar issues in his implementations.

[romanoss]

@TobyRoseman @banoslo @abhishekpratapa

I checked the original paper and found:

We used the same network architecture as in Johnson et al. (2016),
except for two key details:
zero-padding is replaced with mirror-padding, and transposed convolutions (also sometimes called
deconvolutions) are replaced with nearest-neighbor upsampling followed by a convolution. The
use of mirror-padding avoids border patterns sometimes caused by zero-padding in SAME-padded
convolutions, while the replacement for transposed convolutions avoids checkerboard patterning, as
discussed in Odena et al. (2016).

This model seems to be built like the tensorflow/magenta equivalent
Adding the padding was done in 5.5
Avoiding the checkerboard patterning seems to be the way to go.
Can you put it on 5.8?

btw fine tuning all params doesn't help

[TobyRoseman]

Is this still an issue with the latest version of TuriCreate? Since 5.8 we have done a large rewrite of the Style Transfer Toolkit.

[TobyRoseman]

I have not heard back here. I'm going to close this issue. Please reopen, if it is still a problem in the latest version of TuriCreate.