Neural-style: Tiling Renders

[wmco]

Hi ! I've had a lot of fun with this algorithm and I'm hoping to contribute. I have a fair amount of experience programming, but I'm new to torch, tensors, and neural nets.

Here's my question :

1. I take my content image and divide it tiles 2x3 ( vertical image )
2. Each tile is padded 250 pixels around the edges ( I extended the image canvas for the extra pixels )
3. I send each tile to neural-style
4. After render time, I chop off the offsets and recombine the tiles.
5. I'm using the -init image flag and a reasonable content weight
6. The tiles contain the geometry info ( which reflects their offsets ) i.e. bottom right, knows it's offset relative to the initial image it was cropped from.

Using neural-delta2 I'm able to get very "close" results. ( due to the better propagation )

I just don't feel we should have to use panorama software to stitch.

What am I missing ? Is there a way that we can use the offset info in the tile ?

It will be a while before I can get anymore hardware. I'd love to get tiling working.

For now, I'd like to understand why I don't get better results along those edges. I tried to ensure that the size of the offset ( padding on the tiles ) would be as least as large as my biggest feature.

Any clues would be greatly appreciated.

cheers

[htoyryla]

Tiling hasn't really been my priority as I am more inclined to producing elements (not tiles but independent parts like human figures) of an image using different neural transforms and manually assembling a picture from these parts together with hand drawn elements. But for me, it would appear surprising if the separately produced tiles would fit seamlessly, if there is nothing in the process to guarantee that they would.

The problem of making the tile edges match, however, reminds me of the patches or MRF approach to style transfer. While neural-style uses a statistical approach to control style (measuring style with Gram matrices), the patches approach disassembles the style image and then builds a version of the content image out of these patches trying to match them as well as possible. My favorite solution is neural-doodle, in particular the forward branch version (see more in http://liipetti.net/erratic/2016/08/16/neural-style-transfer-after-the-first-year/). Neural-doodle even has a slices parameter which would appear to do something like produce the image in parts in order to save memory.

[crowsonkb]

I have a Caffe implementation using tiling. There are several things that go into it. The main 'trick' I use is to translate the output buffer, feature maps, and optimizer (Adam) internal state by a different random offset every iteration, wrapping as necessary. (Gram matrices are translation-invariant and can be left alone.) The result is effective tile seam suppression as each new iteration's changes obscure the previous iteration's subtle seam and the seam does not accumulate in the same place over time. The initial computation of the Gram matrices and feature maps has to be done in a way that fits into GPU memory, so I average together ten feature maps computed in tiles with different translations to form the final feature maps. Finally, my implementation automates stylization at multiple scales, using each scale's resampled last iterate as the initial iterate for each larger scale. This lets it start at a high-quality initial state from a non-tiled resolution.

[wmco]

Hannu, Thank you for the excellent explanation. It's easy to see why my method was not working and how to move forward ! Very exciting !

[wmco]

Katherine, Thank you too ! I can't wait to try it out ! Exciting times.