Deeplearning4j: Results problem of keras CNN autoencoder with dl4j

Created on 16 Oct 2019 · 24Comments · Source: eclipse/deeplearning4j

Issue Description

I am trying to deploy my keras CNN model implemented on Java (CNN denoiser) , putting noisy image in input and cleaned image. (Mnist denoising autoencoder)

The problem in when i test in java the model, i don't have the same output as when i tested it on python.
Input ------ ------ ------- output (in java)
Capture

Version Information

1.0.0-beta5
Platform information (OS, etc)
Cuda10.0

Contributing

package org.test.cnn;

import java.awt.image.BufferedImage;
//import java.awt.image.BufferedImage;
import java.awt.image.WritableRaster;
import java.io.File;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardOpenOption;

import javax.imageio.ImageIO;

import org.datavec.image.loader.ImageLoader;
import org.datavec.image.loader.NativeImageLoader;
import org.deeplearning4j.nn.graph.ComputationGraph;
import org.deeplearning4j.nn.modelimport.keras.KerasModelImport;
import org.deeplearning4j.nn.modelimport.keras.exceptions.InvalidKerasConfigurationException;
import org.deeplearning4j.nn.modelimport.keras.exceptions.UnsupportedKerasConfigurationException;
import org.nd4j.linalg.api.buffer.DataType;
import org.nd4j.linalg.api.ndarray.INDArray;
import org.nd4j.linalg.factory.Nd4j;

import ij.ImagePlus;
import ij.io.FileSaver;
import ij.plugin.filter.Writer;
import ij.process.ByteProcessor;
import ij.process.ImageProcessor;
import loci.formats.FormatException;
import loci.formats.ImageWriter;
import loci.formats.gui.BufferedImageWriter;
import loci.plugins.BF;

public class TestMnist {
public static void main(String[] args) {
ComputationGraph model = null;
String name = "m_dnoise";

    try {
        model = KerasModelImport.importKerasModelAndWeights("model/" + name + ".h5");
        System.out.println(model.summary());
        System.out.println("Model " + name + " loaded successfully.");
        String name_image = "input/0.btf";
        ImagePlus[] imgs = BF.openImagePlus(name_image);
        ImagePlus img = imgs[0];
        img.show();
        final ImageProcessor imp = img.getProcessor();
        INDArray data = Nd4j.zeros(1, 1, 28, 28);

        for (int i = 0; i < imp.getHeight(); i++) {
            for (int j = 0; j < imp.getWidth(); j++) {

                data.putScalar(0, 0, i, j, imp.getPixelValue(i, j));

            }
        }

        INDArray[] output = model.output(data);

        int dim = 28;
        BufferedImage imgout = new BufferedImage(dim, dim, BufferedImage.TYPE_USHORT_GRAY);
        BufferedImage imgin = new BufferedImage(dim, dim, BufferedImage.TYPE_USHORT_GRAY);
        WritableRaster rasterout = imgout.getRaster();
        WritableRaster rasterin = imgin.getRaster();
        for (int i = 0; i < dim; i++) {
            for (int j = 0; j < dim; j++) {

                int value_in = Math.round(imp.getPixelValue(i, j) * 255);
                int value_out = (int) Math.round(output[0].getDouble(0, 0, i, j) * 255);// .getPixelValue(i, j)*255;

                int pixelsin[] = new int[1];
                int pixelsout[] = new int[1];

                pixelsin[0] = value_in;
                pixelsout[0] = value_out;

                rasterin.setPixel(i, j, pixelsin);
                rasterout.setPixel(i, j, pixelsout);

            }
        }
        Path pathout = Paths.get("D:\\dl4j_mnist_denoise\\deeplearning4j-examples\\dl4j-examples\\output\\out.tif");
        Path pathin = Paths.get("D:\\dl4j_mnist_denoise\\deeplearning4j-examples\\dl4j-examples\\output\\in.tif");
        OutputStream out = Files.newOutputStream(pathout, StandardOpenOption.CREATE);
        OutputStream in = Files.newOutputStream(pathin, StandardOpenOption.CREATE);
        ImageIO.write(imgout, "TIF", out);
        ImageIO.write(imgin, "TIF", in);
        ImagePlus[] outimgs = BF.openImagePlus("Output/out.tif");
        ImagePlus outvisu = outimgs[0];
        outvisu.show();

        ImagePlus[] inimgs = BF.openImagePlus("Output/in.tif");
        ImagePlus invisu = inimgs[0];
        invisu.show();
    } catch (IOException | UnsupportedKerasConfigurationException | InvalidKerasConfigurationException
            | FormatException e) {

        e.printStackTrace();
    }

}

}

Bug DL4J Keras

Source

LyesSp

All 24 comments

Thank you for filing this issue. Can you add to the issue everything I will need to reproduce what you are seeing? Also do you see the problem with CPU as well as GPU?

keras model
keras code
input images, output etc
OS version
Keras version

eraly on 16 Oct 2019

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import keras
from keras.layers import Activation, Dense, Input
from keras.layers import Conv2D, Flatten
from keras.layers import Reshape, Conv2DTranspose
from keras.models import Model
from keras import backend as K
from keras.datasets import mnist
import numpy as np
import matplotlib.pyplot as plt
from PIL import Image

np.random.seed(1337)

MNIST dataset

(x_train, _), (x_test, _) = mnist.load_data()

image_size = x_train.shape[1]
x_train = np.reshape(x_train, [-1, image_size, image_size, 1])
x_test = np.reshape(x_test, [-1, image_size, image_size, 1])
x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255

Generate corrupted MNIST images by adding noise with normal dist

centered at 0.5 and std=0.5

noise = np.random.normal(loc=0.5, scale=0.5, size=x_train.shape)
x_train_noisy = x_train + noise
noise = np.random.normal(loc=0.5, scale=0.5, size=x_test.shape)
x_test_noisy = x_test + noise

x_train_noisy = np.clip(x_train_noisy, 0., 1.)
x_test_noisy = np.clip(x_test_noisy, 0., 1.)

Network parameters

input_shape = (image_size, image_size, 1)
batch_size = 128
kernel_size = 3
latent_dim = 16

Encoder/Decoder number of CNN layers and filters per layer

layer_filters = [32, 64]

Build the Autoencoder Model

First build the Encoder Model

inputs = Input(shape=input_shape, name='encoder_input')
x = inputs

Stack of Conv2D blocks

Notes:

1) Use Batch Normalization before ReLU on deep networks

2) Use MaxPooling2D as alternative to strides>1

- faster but not as good as strides>1

for filters in layer_filters:
x = Conv2D(filters=filters,
kernel_size=kernel_size,
strides=2,
activation='relu',
padding='same')(x)

Shape info needed to build Decoder Model

shape = K.int_shape(x)

Generate the latent vector

x = Flatten()(x)
latent = Dense(latent_dim, name='latent_vector')(x)
x = Dense(shape[1] * shape[2] * shape[3])(latent)
x = Reshape((shape[1], shape[2], shape[3]))(x)
for filters in layer_filters[::-1]:
x = Conv2DTranspose(filters=filters,
kernel_size=kernel_size,
strides=2,
activation='relu',
padding='same')(x)

x = Conv2DTranspose(filters=1,
kernel_size=kernel_size,
padding='same')(x)

outputs = Activation('sigmoid', name='decoder_output')(x)

Instantiate Decoder Model

model = Model(inputs, outputs, name='decoder')
model.summary()
model.compile(loss='mse', optimizer='adam')

Train the autoencoder

model.fit(x_train_noisy,
x_train,
validation_data=(x_test_noisy, x_test),
epochs=30,
batch_size=batch_size)
#

Predict the Autoencoder output from corrupted test images

x_decoded = model.predict(x_test_noisy)
#

Display the 1st 8 corrupted and denoised images

rows, cols = 10, 30
num = rows * cols
imgs = np.concatenate([x_test[:num], x_test_noisy[:num], x_decoded[:num]])
imgs = imgs.reshape((rows * 3, cols, image_size, image_size))
imgs = np.vstack(np.split(imgs, rows, axis=1))
imgs = imgs.reshape((rows * 3, -1, image_size, image_size))
imgs = np.vstack([np.hstack(i) for i in imgs])
imgs = (imgs * 255).astype(np.uint8)
plt.figure()
plt.axis('off')
plt.title('Original images: top rows, '
'Corrupted Input: middle rows, '
'Denoised Input: third rows')
plt.imshow(imgs, interpolation='none', cmap='gray')
Image.fromarray(imgs).save('corrupted_and_denoised.png')
plt.show()
import tifffile as tif

tif.imsave("out.btf",x_test_noisy[0,:,:,0])
model.save("m_dnoise.h5")

LyesSp on 17 Oct 2019

The input is the saved image "tif.imsave("out.btf",x_test_noisy[0,:,:,0])"
The model is "m_dnoise.h5"
I'am using python with keras 2.2.4 on windows
The problem is the same using GPU or CPU.

LyesSp on 17 Oct 2019

Great, thanks. I'll take a look.

eraly on 17 Oct 2019

Looks like an issue with the Deconvolution2DLayer. Will post more details as I have them. Thanks

eraly on 22 Oct 2019

Ok, thanks.I think it's the same problem on the issue #8298

LyesSp on 23 Oct 2019

@LyesSp Yes it is. We've been updating that one more if you want to keep track of progress.

eraly on 24 Oct 2019

This is almost certainly fixed now - see most recent comments here.
https://github.com/eclipse/deeplearning4j/issues/8298

Once it's available on snapshots, let's test and confirm it's fixed before closing this issue.

AlexDBlack on 26 Oct 2019

ok thanks

LyesSp on 31 Oct 2019

@LyesSp can you confirm this issue resolved now?

raver119 on 10 Nov 2019

@raver119 I have problems on the compilation of https://github.com/KonduitAI/deeplearning4j/

[ERROR] Failed to execute goal org.bytedeco:javacpp:1.5.2:build (javacpp-compiler) on project nd4j-native: Failed to execute JavaCPP Builder: Cannot run program "g++" (in directory "D:\deeplearning4j\nd4j\nd4j-backends\nd4j-backend-impls\nd4j-native\target\classes\org\nd4j\nativeblas\windows-x86_64"): CreateProcess error=2, Le fichier spécifié est introuvable -> [Help 1]

LyesSp on 18 Nov 2019

but why you're trying to build from source? just use snapshots

raver119 on 18 Nov 2019

ok i will try, following these steps. https://deeplearning4j.org/docs/latest/deeplearning4j-config-snapshots , it's good ?

LyesSp on 18 Nov 2019

👍1

yes, that'll work

raver119 on 18 Nov 2019

@raver119 I am using snapshots but i have the same results

o.n.l.f.Nd4jBackend - Loaded [JCublasBackend] backend o.n.v.VersionCheck - *** ND4J VERSION CHECK FAILED - INCOMPATIBLE VERSIONS FOUND *** o.n.v.VersionCheck - Incompatible versions (different version number) of DL4J, ND4J, RL4J, DataVec, Arbiter are unlikely to function correctly o.n.v.VersionCheck - Versions of artifacts found on classpath: o.n.v.VersionCheck - org.datavec : datavec-api : 1.0.0-beta5 o.n.v.VersionCheck - org.datavec : datavec-data-codec : 1.0.0-beta5 o.n.v.VersionCheck - org.datavec : datavec-data-image : 1.0.0-beta5 o.n.v.VersionCheck - org.datavec : datavec-hadoop : 1.0.0-beta5 o.n.v.VersionCheck - org.deeplearning4j : arbiter-core : 1.0.0-beta5 o.n.v.VersionCheck - org.deeplearning4j : arbiter-deeplearning4j : 1.0.0-beta5 o.n.v.VersionCheck - org.deeplearning4j : arbiter-ui_2.11 : 1.0.0-beta5 o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-common : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-core : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-datasets : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-datavec-iterators : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-modelimport : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-nlp : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-nn : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-parallel-wrapper : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-play_2.11 : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-tsne : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-ui-components : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-ui-model : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-ui_2.11 : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-util : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-utility-iterators : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : deeplearning4j-zoo : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.deeplearning4j : nearestneighbor-core : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.nd4j : guava : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.nd4j : jackson : 1.0.0-beta5 o.n.v.VersionCheck - org.nd4j : nd4j-aeron : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.nd4j : nd4j-api : 1.0.0-beta5 o.n.v.VersionCheck - org.nd4j : nd4j-buffer : 1.0.0-beta5 o.n.v.VersionCheck - org.nd4j : nd4j-common : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.nd4j : nd4j-context : 1.0.0-beta5 o.n.v.VersionCheck - org.nd4j : nd4j-cuda-10.1 : 1.0.0-beta5 o.n.v.VersionCheck - org.nd4j : nd4j-jackson : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.nd4j : nd4j-native-api : 1.0.0-beta5 o.n.v.VersionCheck - org.nd4j : nd4j-native-platform : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.nd4j : nd4j-native : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.nd4j : nd4j-parameter-server-client : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.nd4j : nd4j-parameter-server-model : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.nd4j : nd4j-parameter-server : 1.0.0-SNAPSHOT o.n.v.VersionCheck - org.nd4j : protobuf : 1.0.0-beta5 ND4J CUDA build version: 10.1.243

LyesSp on 18 Nov 2019

Ok, we’ll keep looking into it. Thank you for feedback

raver119 on 18 Nov 2019

@LyesSp that warning - as is says - means you're mixing 1.0.0-beta5 and 1.0.0-SNAPSHOT versions.
If you're using snapshots, use only snapshots, for everything - DL4J, ND4J, DataVec, etc.

There's really only two ways that can happen.

You have a mix of both versions in your pom.xml (or equivalent for other build tools)
You are using Eclipse IDE and changed the version after creating the project (if so, recreate your project, Eclipse is notorious for incorrect/undefined behaviour with maven + changing dependencies)

AlexDBlack on 19 Nov 2019

@AlexDBlack @raver119.Yes i fixed thanks.
Now it's working on my mnist project i will try to test this for multi channels output

LyesSp on 19 Nov 2019

👍1

@AlexDBlack it's globally better for the multi channels output but, in the details the keras output is different from dl4j, i don't know if it's comming from (weights rescaling or biais or .....) used on dl4j . As you can see in the background of grayscale image here.
dl4jvskeras

LyesSp on 20 Nov 2019

Are you running with CPU or GPU?

raver119 on 20 Nov 2019

@raver119 GPU

LyesSp on 20 Nov 2019

Please try CPU then, and tell us if there's the same issue.

raver119 on 20 Nov 2019

@raver119, with CPU it's look working good yes. i have images with good quality without bars.

LyesSp on 20 Nov 2019

I see. In this case I’d propose to way day or two. Theres a nasty bug we’re working right now, that affects CUDA in certain ops.

With best regards,
raver119
On Nov 20, 2019, 18:15 +0300, LyesSp notifications@github.com, wrote:

@raver119, with CPU it's look working good yes. i have images with good quality without bars.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

raver119 on 20 Nov 2019

Was this page helpful?

0 / 5 - 0 ratings

Related issues

libnd4j: space_to_batch op validation fails for CNN1d input case

AlexDBlack · 4Comments

Type inconsistency

hanbaoan123 · 4Comments

DL4J: UI functions on Java 8 only?

Storm-cev · 5Comments

libnd4j: add broadcast_to op

AlexDBlack · 4Comments

[Question] Does DL4J works fine on NVidia Jetson ?

Paranormaly · 5Comments