Dlib: CIFAR10 DCGAN ISSUE

[intellizd]

I am trying to implement dcgan-cifar.
However, a problem occurred while learning CIFAR10 MODELS.
Maybe,I think there is a problem with the function . Binary Cross Entropy (BCE)? that I developed.
I made several loss functions.And it's testing.
I don't think I know exactly what mechanism Dlib's loss function is yet.

Cifar-DCGAN Layer Architectures

    using generator_type =
        loss_binary_cross_entropy < fc_no_bias <1,
        htan<contp<1, 4, 2, 1,
        relu<bn_con<contp<64, 4, 2, 1,
        relu<bn_con<contp<128, 4, 2, 1,
        relu<bn_con<contp<256, 4, 2, 1, //16
        relu<bn_con<contp<512, 4, 1, 0, //16
        input<noise_t>
        >>>>>>>>>>>>>>>>;
    using discriminator_type =
        loss_binary_cross_entropy <fc_no_bias<1,
        conp<1, 4, 1, 0, //1
        leakyrelu<bn_con<conp<512, 4, 2, 1, //2
        leakyrelu<bn_con<conp<256, 4, 2, 1, //4
        leakyrelu<bn_con<conp<128, 4, 2, 1, //8
        leakyrelu<conp<64, 4, 2, 1, //16
        dlib::input<gray_pixel>
        >>>>>>>>>>>>>>;

BCE Source

    tt::sigmoid(grad, output_tensor);
            // The loss we output is the average loss over the mini-batch.
            const double scale = 1.0 / output_tensor.num_samples();
            double loss = 0;
            float* g = grad.host();
            float eps = 1e-12;
            // gradient wrt grad_output
            const float* io = input_tensor.host();
            const float* out_data = output_tensor.host();




            for (long i = 0; i < output_tensor.num_samples(); ++i)
            {
                float y = *truth++;
                float y_ = g[i];

                float BCEL = (y * safe_log(y_) - (1 - y) * safe_log(1 - y_)); //this is binary cross entorpy loss function 

                if (y > 0)
                {
                    loss += scale * BCEL;
                    g[i] = scale * (g[i] - 1);
                }
                else
                {
                    loss +=  scale * BCEL;
                    g[i] =  scale * g[i];

                }


            }

[intellizd]

I had designed DCGAN Layer with reference to pythorch's DCGAN.
but there was a missing part in that part.

So I made some corrections by referring to the orignal DCGAN paper.
I change the LeakyRelu slope parameter from 0.001x to 0.2x and
generator noise connecting with layers of fc layer that include 1024 paramaeter
When I added this part, the image began to be created like the paper.

It's almost done.
I'm now working on fine tuning and finishing.

I will share the source as soon as it is completed.

thanks.

[intellizd]

After the final tune-up,
Here's the result:

It's almost the same, but there's a little noise.
I try to remove this noise and improve it with color images.

I'm trying to modify mean & standard deviation parameters in batch_normalization(bn_) layer,
but I'm not sure how to modify them.

@davisking Please advise to me.

[Cydral]

That's brilliant! Thank you for sharing these results. I look forward to seeing results also in color...
The noises noted in the fake image compared to the original one perhaps come from the training data's distribution? Is the input noise updated, for example, at each Epoch? Couldn't this affect the quality of the projection and therefore the output image, for example? This network type is really exciting.

[davisking]

I haven't played around with GANs much, so I'm not the best person to ask about the tricks necessary to get them working.

[intellizd]

@Cydral Thanks for your compliment.
First, it is necessary to fix the z= noise value to verify that it is being generated correctly.
I think you want to projection image that generated image from each input noise.

Next, I will test GAN by random noise if GAN by fixed noise that is stable.
Color image is next. because color image is 3 times slow.
I think we need some time to develop it now.
I ask for your continued attention. thanks

@davisking
I am using Dlib Frameworks to develop GAN.
It wasn't about the GAN, it was about the bn_layer.
I want to manually set the Mean value and standard deviation in bn_ layer.
I saw a bn_ layer source and it was very complicated!

[Cydral]

@intellizd, that's correct. I verified the code you shared and yes, I had the impression that the noises vector could change between several iterations, which in my opinion should penalize convergence on the one hand, and disturb the stability of the parametric model on the other. I'm doing tests myself (and this for some time now) but I'm still stuck on the back-propagation part. I try with convoluted and deconvoluted images to avoid Gaussian noise as an input... but for the moment, it doesn't want to work. So yes, you're right, I'm trying a pix2pix transfer instead. Your progress is very helpful to me so I stay in touch!

[intellizd]

We need more understanding of noise vectors in generator.
the noise vector shall be assigned to the image to be generated.
and we can generate distributed images such like face image category that male & female.
most GANs Example, noise input Z is generated indiscriminately without this.
we must understand the Original DC-GANs thesis before implementing GANs.