Deeplearning4j: Unsupported space_to_batch operation in Lambda layer for YOLO import

[saudet]

Issue Description

While trying to import a Keras model for YOLO created via YAD2K, I get an "Unsupported keras layer type Lambda" error for this layer:

space_to_depth_x2 (Lambda)       (None, 19, 19, 256)   0           leaky_re_lu_21[0][0]     

It corresponds to https://www.tensorflow.org/api_docs/python/tf/space_to_depth in TensorFlow. It does not look like a complicated operation, but do you know if we have a layer like that somewhere in DL4J? What do you think @AlexDBlack?

Version Information

• Deeplearning4j 0.9.2-SNAPSHOT

[maxpumperla]

@saudet lambda layers in keras allow you to essentially provide a little lambda function of your choice, a simple way to add a new layer. Supporting Lambda in general in DL4J would need Samediff in place, as we need to compute the backward pass. So, I think it will take a while for us to fully support this.

On the other hand, the specific operation you mention, sure, we could implement that as a one-off. How urgent or important is this?

[saudet]

We're going to need it for accuracy with object detection at some point. I think @wangfeng-skymind might need it soon (?) so it may be semi-urgent... Do we have a layer like space_to_depth already though?

[maxpumperla]

@raver119 is currently implementing a ton of things for tf import. if we have an equivalent of space_to_depth in libnd4j/nd4j, I could relatively quickly build a dl4j layer around it.

[raver119]

space_to_depth is implemented.

[maxpumperla]

@saudet we even have this in SameDiff by now. Supporting lambda layers in general is still a long way to go, but we could have this a utility layer. Is this still important or urgent or both? :)

[nlebreton]

@maxpumperla thx for this impl. I will test it by trying to import a (full) yolo v2 asap and update this issue accordingly.

[maxpumperla]

@nlebreton wait, it's in ND4J right now, I still need to wrap this into DL4J _and_ support keras import.

@saudet can you please (again, I guess) point me to the model specification that you try to import?

[saudet]

It's basically the steps here at https://github.com/allanzelener/YAD2K#quick-start:

git clone https://github.com/allanzelener/YAD2K
cd YAD2K
wget http://pjreddie.com/media/files/yolo.weights
wget https://raw.githubusercontent.com/pjreddie/darknet/master/cfg/yolo.cfg
python3 yad2k.py yolo.cfg yolo.weights yolo.h5

[saudet]

IMO, I think it's important to be able to import full YOLOv2 models for decent accuracy with object detection. I'll leave it up to @jpatanooga and @agibsonccc to set priorities about this and other things though.

[nlebreton]

I can confirm that from a business point of view, Tiny Yolo is not accurate enough for many use-cases. I work on a proof of concept trying to convince my partners that java and dl4j are viable compared to the classical python approach. One strong argument is "Keras can be imported" but today state of the art objects detection model Yolo cannot be imported. I really hope that this capability will be added soon.

[maxpumperla]

@nlebreton @saudet we're slowly but surely getting there, see: https://github.com/deeplearning4j/deeplearning4j/pull/4691

the full yolo h5 exceeds the github file size limit (it's well over 200mb), so I can't just put it into our resources repo. Is there a smaller version that reproduces the basic pattern (i.e. concat a conv layer with the output of space_to_depth)?

[nlebreton]

@maxpumperla I was able to reproduce the issue with that one ? Its size is 36kb.
my_model.zip

[maxpumperla]

@nlebreton @saudet importing full yolo-voc works on the branch now, will try to test some more, then wrap this up.

[maxpumperla]

perfect, yolo seems to work as well. however, I see that your my_model.zip fails for some reason @nlebreton.

[maxpumperla]

ok, I also added a keras configuration test (loading json only), which you can see here:

https://github.com/deeplearning4j/deeplearning4j/blob/4a25300f1452d656b15686cbb224f28c28aa70c1/deeplearning4j-modelimport/src/test/java/org/deeplearning4j/nn/modelimport/keras/configurations/Keras2ModelConfigurationTest.java#L44-L46

[maxpumperla]

I'm renaming this issue to this more specific case and open a new "Lambda layer support" issue. thanks!

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.