Turicreate: Style Transfer issue, Model fine tunig and general questions

[josefgrunig]

First of all I want to say that I really appreciate the nice and hard work on Turi Create: its incredible how this tool simplifies Style Transfer. I am opening this issue because experimenting with it I found a visual issue and need some parameter clarification that I was not able to find in the docs.

The issue: I found that the resulting images have some strange color banding. They are always diagonal and present strange green/purple color. Maybe related to this bug: #666
Please see image below:

Apart this, I am writing to show my results on my first style transfer test. Actually I was expecting something different thus I am asking here for more fine tuning on the parameters.

------
Model                                    : resnet-16
Number of unique styles                  : 1
Input image shape                        : (256, 256)

Training summary
----------------
Training time                            : 3h 54m
Training epochs                          : 2051
Training iterations                      : 10000
Number of style images                   : 1
Number of content images                 : 39
Final loss                               : 1.6845

Loss plots:

Results:

What I am trying to reach out is to bring more pencil stroke on the final image. I would like it to look like a hand drawn painting. Is that possible or did I misunderstood how the algorithm works?

I have seen that I can manipulate the Style Loss Multiplier (style_loss_mult), but not sure how to change those numbers (sorry I am quite new to this subject)

I have trained on 40 portrait images, is that good or should I increase the number of images?
Should the content have a more general subject in it?

The Style image is 400x400px is that enough or should be higher in resolution? Or should it match the network size (256x256px)?

Using more similar styles in input can increase the quality of the network?
Is it possible to blend between styles, like explained in the paper this algorithm comes from?

Any suggestion is welcome and much appreciated. I think answer to my questions could help a lot other people in the community looking for fine tunig their models

Thank you

[josefgrunig]

I have given another try with the more used StarryNight style:

again here I get something different from what I am expecting.
I am looking to achieve a result more like this one:

Any hint or ideas?

[josefgrunig]

using params={'style_loss_mult': [1e-2, 1e-2, 1e-2, 1e-2]} I got a definitely more interesting result, will play around those values a little bit more.
Is it possible to get an explanation/documentation of all parameters of the style_transfer.create function?

[tankfly2014]

HELLO.
TURI Style Transfer build image is green/purple color.
This is a very very very very very very very very very very very bug.
haha.
I wait next version fix it,but I think bug (is here )long time and never .

[srikris]

@tankfly2014 Sorry for the trouble and the delay in fixing this issue. We are looking at this bug in #666. It's a tricky bug to fix and we haven't yet gotten to the bottom of it. We will keep you posted.

[Umity]

any update on this? how could you created that demo images?

[srikris]

@Umity We are looking into this. Can you share some of your images and results for us to better diagnose the issue.

[Umity]

@Umity We are looking into this. Can you share some of your images and results for us to better diagnose the issue.

Hi Krishna, thank you for asking. And I am really glad you guys are really looking into it.

The results are awful. Do you think we are doing something wrong? I had around 5000 photos in my content folder, took more than a day to bake the model file and the result was bad. Here is the result image: https://imgur.com/a/PMVIBVY

There is a tutorial on web and you can see its result image in there: https://appcoda.com/coreml-turi-create/

I have a very simple question;

Are we doing something wrong? Or do you think you guys edited these photos https://apple.github.io/turicreate/docs/userguide/style_transfer/ with a different version of Turicreate???

[josefgrunig]

Hi @Umity,

I am a complete newby in the Style Transfer matter and came across the same issues.
The main two problems in my case where the number of iterations and the style loss multipliers.
I wrote a short article on my test results here; maybe it helps.

Regarding the banding issue #666 it gets less visible by raising number of iterations.
The reason I opened this thread was exactly for asking a more detailed explanations of those low level parameters that can definitely help in getting better result.

I suggest also reading the paper this algorithm comes from A Learned Representation For Artistic Style, where those parameters are used.

[Umity]

@josefgrunig

Wow! Great article! Thank you. How can i contact you via email?

[josefgrunig]

@Umity You can contact me from LinkedIn or: name dot surname at gmail dot com

[srikris]

@abhishekpratapa Can you share the new results based on your updated PR.