Keras: NumPy (or JAX) backend for Keras.

[danFromTelAviv]

Most of keras is compatible with numpy / scipy.
It would be awesome ( and probably competitively fast at inference on cpu) if keras was available with a numpy/scipy backend.

This would help simplify inference significantly on a variety of machines ( in my case a non-raspbery pi arm computer). It would also be very fast.

Alternatively a simplified keras inference engine that does not require tensorflow/cntk would great as well.

Here is a link to something similar that unfortunately lost maintenance: https://github.com/riga/tfdeploy

[kgrm]

All existing backends provide several "standard" deep learning operations such as convolution/deconvolution, pooling, activation functions, etc, as well as their gradients, which would have to be implemented from scratch for a numpy backend. This seems to me to be outside the scope of keras.

[jlherren]

Most of keras is compatible with numpy / scipy.

That's maybe a bit of a stretch. I think that all of Keras' backends can do symbolic math for various transformations and optimizations of the function graph, which is quite crucial to the execution speed. Neither numpy nor scipy can do any of that. So unless that part is covered by something, I don't really see the point. I'm not familiar with sympy and its tensor module, but perhaps there is some potential there.

Can you explain the motivation behind this a bit more? Does your machine really implement full numpy and scipy support but none of Keras' backends? Tensorflow runs on raspberries, so maybe it does on your arm machine? At least some version of theano runs on arm too.

That said, I don't think writing a custom pure numpy (maybe not even scipy) backend is very hard, especially if you restrict yourself to simple models. It just won't be as performant as you think it would be.

[gabrieldemarmiesse]

There is a "numpy backend" in the test suite. See this file:
https://github.com/keras-team/keras/blob/master/tests/keras/backend/reference_operations.py

It's being worked on right now (PR welcome). It's mainly for testing purposes but nobody prevents you from copy pasting the code and load it as an external backend. Loading external backends is not a documented feature, but it's possible, you just have to write:

{
    "floatx": "float32",
    "epsilon": 1e-07,
    "backend": "numpy_backend",
    "image_data_format": "channels_last"
}

in your ~/.keras/keras.json.

Then copy the code from reference_operations.py and put it in a file called numpy_backend.py. I cannot garantee that everything will work as expected, but it should behave kind of like tensorflow eager or even a hypothetical pytorch backend.

You should use the model subclassing and then call the method call on your input numpy arrays.

There might be a few things to modify here and there, but it should work with a bit of tweaking. Don't forget to write how you made everything work somewhere.

I get the point of a numpy backend to remove dependencies for inference. It's possible that one day, this numpy backend gets out of the test suite and goes to the backends directory. We never know. For now, you should be able to use it as an external backend.

[gabrieldemarmiesse]

Let me know if you encounter any issues with this setup. I'm happy to help. I'd be interested in seeing a numpy backend for inference too :)

[farizrahman4u]

Is numpy faster than available backends on certain platforms? Any numbers available?

[gabrieldemarmiesse]

I don't expect it to be faster. But it'd be nice to have numbers indeed. What I see is mostly a benefit in terms of dependencies for the inference part.

[danFromTelAviv]

Thanks for the feed back to everyone.
What i would love to have ( and help create) is keras simplicity and speed at inference ( on arm ).

I spent about 2 weeks ( 9 hours a day ) with a linix expert trying to install tensorflow on arm ( tinker board ). It is very non trivial unless it's a raspberry pi which google recently started officially supporting. cntk is explicitly not supported. Theano is very easy to install but not maintained :/.
I have found that these libraries are also far from optimized for arm. While regular algorithmic code runs about 1x-2x slower as my i7 a convnet I built runs about 7x slower.

I obviously can not beat the speed of the most optimized frameworks on their intended x86 + gpu targets - but as for arm computers ( which is where a lot of models actually end up running for IoT devices/routers/some servers...etc ) I can likely get at least comparable speed with much less hustle of installation.

Again - I am only interested in inference - sort of like tf lite. Maybe this can be like a keras lite prototype ( if ya'll are on board for that ).

@kgrm Inference is also much simpler to implement...

@gabrieldemarmiesse - that is super awesome - I'll try it out .

I also started working on a cython + numpy implementation that is showing a lot of promise to be super fast for non gpu enviroment.
I tested a conv2d + seperable conv2d and it seems to perform with similar or even faster speeds compared to tf for my use cases ( very small images + kernels). This may be since there is almost zero overhead.

You can check out my ( very early ) work:
https://github.com/danFromTelAviv/cynfrence

I tried my best to follow tf standards.

@farizrahman4u - I will run torough tests and let you know. ( It is a great honor to have you comment on my post btw )

[farizrahman4u]

@danFromTelAviv Thanks for the detailed response. Note that i have an existing architecture for running Keras with imperative backends (like pytorch) and currently i use it for running keras models on Nd4j (www.nd4j.org). See . Nd4j api is quite similar to numpy. If you can show me some solid numbers, we can definitely get this going.

[gabrieldemarmiesse]

Having support for imperative backends in keras would be super nice. Maybe that would help keras and tf.keras be in sync since tf.keras already supports eager execution (just a guess)? And maybe reduce @fchollet 's workload, since keeping keras and tf.keras in sync must be very time-consuming.

[danFromTelAviv]

@gabrieldemarmiesse I just looked at your code. It is beautifully simple. My only concern is runtime for conv / seperable conv and maybe rnn. I think I'll ditch my version that I was working on in favor of yours but i'll merge in the convolutions that I wrote in cython ( and maybe a couple more functions that can gain significant speedup over good old numpy ). It looks far more maintainable but I'll have to compare runtime just to make sure...

@farizrahman4u That is awesome. I'm at ECCV right now. When I get back to work I'll start testing and update you as I go.

[danFromTelAviv]

Short update.
I ran tests on my i-3 with a windows linux subsystem :
1000 runs (conv2d with (2,2) stride) = 2.062500 (s) - cython
1000 runs (conv2d with (2,2) stride) = 1.140625 (s) - tf

* edit *
after some minor tweaks
1000 runs (conv2d with (2,2) stride) = 1.468750 (s) - cython

So not as fast as I was precieving. But not a bad starting point at all. I think it maybe they are using fft based convolutions rather than direct. I'm not really sure how to aply strides/dilation in fourier space but it seems doable. Otherwise I was also thinking about looking deeper into how np/tf do their convolutions and just add in the striding / dilation options as needed. 2x is also with in range of just pure code optimization so maybe i'll attack it that way although I'm more experienced with highlevel.

[gabrieldemarmiesse]

I found a way to speedup the convolution in the numpy backend while staying in pure python:

@normalize_conv
def conv(x, w, padding, data_format):
    _y = []
    for j in range(w.shape[1]):
        __y = []
        for k in range(w.shape[0]):
            __y.append(signal.convolve(x[:, k], w[None, k, j], mode=padding))
        _y.append(np.sum(__y, axis=0))
    return np.stack(_y, axis=1)

But it's still nowhere as fast as tensorflow for example.

See this PR: #11156

[danFromTelAviv]

nice. What is the run time compared to tf roughly? Maybe we can request scipy to add dilated strided nd convolutions ?

Update with regards to the cython implementation ->
1) The above run times did not include parallelization which should give > 1.5x speedup with 2 cores.
2) I found out time.clock() can not time parallelized cython properly ( I get 2x slow down ).
I think even 30% slower than tf is good enough for now. Especially with the assumption that when I get around to it I can recover that speed with parallelization.

I think I will start to integrate with @gabrieldemarmiesse 's code and add tests and test on arm...etc
What do you guys think?

[gabrieldemarmiesse]

• If numpy / scipy could have a convolution operation for 1d/2d/3d equivalent to what deep learning frameworks have, it would be awesome.
• Currently, not only our convolution is slow (I think something like 10-20x slower than TF) but our pooling operations in numpy in reference_operations.py is very slow too.
• We cannot use Cython in the keras codebase. That would complicate the build too much.
• We cannot go with a numpy backend in the keras codebase unless we have a fast pooling and convolution operation. Those operations should only use scipy/numpy. For pooling, I think numpy tricks can help us, but I'm not so sure for the convolution.

[gabrieldemarmiesse]

I ran a VGG with the numpy backend from reference_operations.py and it was orders of magnitude slower than TF. Mainly because of pooling and convolutions. I don't have precise numbers though.

[danFromTelAviv]

Yeah... orders of magnitude is an issue.
I opened an issue for scipy - hopefully, they think it's a significant feature and select it for dev. I tried to follow their code and had a hard time so I don't think I'll be able to contribute there.

As for pooling - average pooling is just a strided convolution with a np.ones. There is also a max filter similarly. So it's two bird with one stone if we can figure out the convolutions. There are also rnns to watch out for.

In any case, for truly competative speed, none of the for loops can be in python.
Cython code precompiled or at least translated to c seems not that bad for distribution but it's not as easy as python obviously.
Official cython advice:
1) Turn the cython code into c automatically with cythonize ( no cython needed on user's computer ).
2) Add it to setup.py like the following.

from distutils.core import setup
from distutils.extension import Extension

setup(
    ext_modules = [Extension("example", ["example.c"])]
)

Maybe a simpler custom backend interface ( imperative backends as you suggested ) would be the best way to go - this way it's not part of official keras. ( If the scipy/ numpy avenue doesn't work out )

[farizrahman4u]

Hmm. Let's not do the cython stuff. I know where it goes.. you will end up writing your own TF. Let's have a Keras fork with a basic template for imperative backends. I will add a bunch ops for numpy, and you guys can help out with the rest?

[danFromTelAviv]

Fair enough. @gabrieldemarmiesse - it's not going to be fast enough for practical use but will open a really big door for innovation and maybe speed eventually will come too. You in?

Let's get a list of things that need to be done somewhere so we know who's on what. Where is the most convenient ( maybe trello or something)? maybe we should even do TDD ( seems classic here ).
** note that between gabriel's code and mine there's already a lot of functionality in pure numpy/scipy. ( mine is still not well tested ).

[gabrieldemarmiesse]

Let's have a Keras fork with a basic template for imperative backends.

@farizrahman4u From what I know, this keras-team/keras repo will have to support TF eager. Why not work directly here to support TF eager? We'll load the numpy backend as an external one when we're done. What do you think?

[farizrahman4u]

@gabrieldemarmiesse We don't have to make much changes to this repo to support eager. Keras simply calls tf functions. Whether to execute ops eagerly or not is upto TF. Current codebase should be almost eager compatible. Maybe there are some api changes (additions) to be made to support eager workflow more explicitly, and there are upcoming breaking api changes in TF2, but I think fchollet and google will handle it :)

Here i am talking about truly imperative backends (which doesn't have the concept of placeholders). These include: pytorch, nd4j, numpy. I already have partially working prototypes for pytorch and nd4j. Lets make it a single project, with switchable backends. I can do this over the weekend (next weekend), but willl need help with adding ops.

[gabrieldemarmiesse]

Sure. I'd be interested in helping out adding ops. I mean, especially with numpy, it's not very difficult (pytorch too. There is just their weird to_device()to handle).

One request though. I would like to add missing numpy backend functions to references_operations.py in this repo. This allows a healthier testing of the backends in keras-team/keras. (because I don't like the fact that we test all backends against each other right now. If one test fail, we don't know which backend caused the tests to fail, it's contrary to the spirit of unit testing). You can always port them to your fork of course. I'd just ask some help since I need someone to read and review my PRs in this repo, so I'll count on you for that.

In short, I'll give priority to references_operations.py in this repo for the moment.

[farizrahman4u]

Sure. You can keep adding ops to reference_operations.py, I will just copy from there for my numpy backend?

[danFromTelAviv]

I am relatively free this week ( we are on holidays over here ). I am not sure though exactly what functionality is needed and how the architecture of this project is set.

I think I will be most helpful if you provide me with a list of small functionalities that you want me to help out with and I can bring back a set of functions + tests.

I haven't had a chance to try out pytorch yet so I'm probably more useful with numpy as well.

[gabrieldemarmiesse]

If you want to help, you can do PRs to add ops to
reference_operations.py. I think it's the simplest since you don't need
much knowledge of keras' internals to do that.

On Sat, 22 Sep 2018, 15:58 danFromTelAviv, [e-mail] wrote:

I am relatively free this week ( we are on holidays over here ). I am not
sure though exactly what functionality is needed and how the architecture
of this project is set.

I think I will be most helpful if you provide me with a list of small
functionalities that you want me to help out with and I can bring back a
set of functions + tests.

I haven't had a chance to try out pytorch yet so I'm probably more useful
with numpy as well.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/keras/issues/11068#issuecomment-423746004,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMS2Kz0DlVWXc6wFpwFf1ANHF1oBrSjoks5udkIIgaJpZM4WXh8V
.

[danFromTelAviv]

just to clarify - I should look at tensorflow_backend.py as a reference and write the functions that are missing? I assume that tf specific things like clear_session() and such are not needed?

[gabrieldemarmiesse]

Indeed, but you should look at https://github.com/keras-team/keras/blob/master/tests/keras/backend/backend_test.py

Specifically, we want to remove the use of the variable BACKENDS and replace it with the variable WITH_NP. We can only do that for operations which are implemented in numpy.

I can guide you for a first PR if you want:

• Look at this line: https://github.com/keras-team/keras/blob/e00be3d95dc7d88706a5b76d4b11856c8a7b5f6f/tests/keras/backend/backend_test.py#L208

• We use BACKENDS. Replace it with WITH_NP (like for the 'dot' operation just a few lines above).

• Run the test. It won't work. This is because the numpy backend doesn't have the 'batch_dot' op yet.

• Add batch_dot in the numpy backend (reference_operations.py).

• Run the same test again.

• Success!

• PR.

I'll review your PR for simplicity when you do it.

[danFromTelAviv]

ok, excellent. I'll let you know if/when I run into an issue.

[gabrieldemarmiesse]

If you want to know why we're doing that in the test suite, this is because we compare backends against each other to check that they all produce the same results. When using BACKENDS we compare the three backends together. And it's not great, because the error won't tell us easily which backend has the op which is failing. By using WITH_NP we just compare the current backend to the numpy backend to check that this is the same result. Then we know exactly which backend is failing if there is an error.

[danFromTelAviv]

oh ok. that makes sense.

[danFromTelAviv]

It took me a bit of time to get the diff and i figure it should be useful .
There may be a couple of functions missing/ not needed but this is the bulk of it.

_has_nchw_support
_to_tensor
is_sparse
to_dense
is_keras_tensor
is_tensor
placeholder
is_placeholder
shape
int_shape
ndim
zeros
ones
zeros_like
ones_like
identity
count_params
cast
update
update_add
update_sub
moving_average_update
batch_dot
gather
sin
cos
_regular_normalize_batch_in_training
_broadcast_normalize_batch_in_training
_fused_normalize_batch_in_training
normalize_batch_in_training
batch_normalization
tile
squeeze
temporal_padding
spatial_2d_padding
spatial_3d_padding
stack
one_hot
slice
get_value
batch_get_value
set_value
batch_set_value
get_variable_shape
class Function(object)
function
gradients
stop_gradient
softsign
sparse_categorical_crossentropy
dropout
in_top_k
_preprocess_conv1d_input
_preprocess_conv2d_input
_preprocess_conv3d_input
_preprocess_padding
conv2d_transpose
separable_conv3d
conv3d_transpose
random_normal
random_uniform
random_binomial
truncated_normal
ctc_label_dense_to_sparse
ctc_batch_cost
ctc_decode
map_fn
foldl
foldr
local_conv1d
local_conv2d
name_scope
depthwise_conv2d
squeeze

[gabrieldemarmiesse]

See #11205 for an example of PR which would help populate the numpy backend.

[danFromTelAviv]

I created one pull request. Please review it and give me feed back. I assume I did at least one thing not as you expected so don't be shy with comments :)

[danFromTelAviv]

I just refreshed the page and saw ur example. I changed the formatting - I just copied what you said in your request.
Do you want me to do a pull request per function or can I group a few functions together?

[gabrieldemarmiesse]

I suggest that you push new commits to the branch which you used for the pull request to modify it. No need to close it and open another one. Try to follow the style of the one I made.

Small PRs are the best types of PRs. One PR per op would be nice.

[danFromTelAviv]

I get fails in the CI system -
"The command "if [[ "$TEST_MODE" == "INTEGRATION_TESTS" ]]; then PYTHONPATH=$PWD:$PYTHONPATH py.test tests/integration_tests; elif [[ "$TEST_MODE" == "PEP8" ]]; then PYTHONPATH=$PWD:$PYTHONPATH py.test --pep8 -m pep8 -n0; elif [[ "$TEST_MODE" == "DOC" ]]; then PYTHONPATH=$PWD:$PYTHONPATH py.test tests/test_documentation.py; else PYTHONPATH=$PWD:$PYTHONPATH py.test tests/ --ignore=tests/integration_tests --ignore=tests/test_documentation.py --ignore=tests/keras/legacy/layers_test.py --cov-config .coveragerc --cov=keras tests/; fi" exited with 1."

Is there a better error message? When I run the file "backend_tests.py" tests locally I don't have these errors?
The failed test is : tests/keras/backend/reference_operations.py which is not even a test?

[gabrieldemarmiesse]

There are the tracebacks in the travis logs. You should be able to find out what went wrong. Worst case scenario, you can run travis locally with docker to reproduce the error.

[danFromTelAviv]

ok. It took a few tries but i finally passed all of the ci tests. enjoy the review . for the future PRs ill do one function at a time.

[danFromTelAviv]

I wanted to create batch_normalization() but I'm not sure about two things :
1) how is axis taken into account - should it be applied in some way in the function?
2) what is the shape of mean, var, gamma, beta?

If I understand correctly the above tensors should be the same dimensions as x except for in "axis" where their dimension should be 1?

[gabrieldemarmiesse]

@danFromTelAviv can you make a pull request with a starting implementation of batchnorm (even if it doesn't work). Then we'll be able to look into it easily and discuss the implementation.

[danFromTelAviv]

sure. ill be on that this evening.

[gabrieldemarmiesse]

Thanks!

[danFromTelAviv]

is scipy ok to use? What do you think? I was thinking for sparse / dense matrix.

[gabrieldemarmiesse]

Scipy is already used in the numpy backend, so I would say it's ok.

[alexbw]

Curious if there is a status-update on how complete a NumPy backend is for Keras?

[danFromTelAviv]

It's coming along nicely ( mostly thanks to gabriel ). I think for most architectures you could already use it.
If there is something in particular that you need and is missing I'll try to prioritize it.

[alexbw]

Awesome! I'd like to do something _really_ simple, like:

from keras.models import Sequential
from keras.datasets import mnist
from keras.layers import Dense

(x_train, y_train), _ = mnist.load_data()
x_train = x_train.reshape(len(x_train), -1)
model = Sequential()
model.add(Dense(units=64, activation='relu', input_dim=x_train.shape[1]))
model.add(Dense(units=10, activation='softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer='sgd',
              metrics=['accuracy'])

# x_train and y_train are Numpy arrays --just like in the Scikit-Learn API.
model.fit(x_train, y_train, epochs=5, batch_size=32)

using NumPy. There's a few helper functions missing, which I patched locally. I ran into some Keras internals w/ placeholders that I'm not sure how to resolve. Any help is appreciated!

FYI I'm doing this to test compatibility of JAX with Keras. If the NumPy backend works, it should be automatically compatible with JAX (I hope, and if not I will fix it), which means that the NumPy backend will run on CPU, GPU and TPU, and be quite fast.

[gabrieldemarmiesse]

Currently the numpy backend can only be used for inference. Not training.

To support training, we would have to implement the back-propagation, which I believe will never been done since the purpose of keras is to use backends, not create backends. We made the numpy backend for testing, used it to display simple code in the docs, and once keras supports tensorflow eager, it should be fairly easy to use the numpy backend for inferencing without having to make a fork of keras. You'll likely never be able to use the numpy backend with model.fit unless you implement the backpropagation yourself. Wait a bit and the inference will come, I believe after TF eager is supported.

[gabrieldemarmiesse]

By the way, thanks to @andhus , we have a RNN numpy implementation. See #11622 .

[alexbw]

The backpropagation I can handle (perhaps will need some help from a Keras
expert to find out where to plug in), because both autograd and JAX support
easy automatic differentiation.
I'll be waiting eagerly for NumPy inference! Should I just monitor this
issue?
Also, will supporting 'inference' include K.placeholder, and other internal
structures like that?

On Thu, Nov 29, 2018 at 3:46 PM Gabriel de Marmiesse <
[email protected]> wrote:

Currently the numpy backend only supports inference. Not training.

To support training, we would have to implement the back-propagation,
which I believe will never been done since the purpose of keras is to use
backends, not create backends. We made the numpy backend for testing, used
it to display simple code in the docs, and once keras supports tensorflow
eager, it should be fairly easy to use the numpy backend for inferencing
without having to make a fork of keras. You'll likely never be able to use
the numpy backend with model.fit unless you implement the backpropagation
yourself. Wait a bit and the inference will come, I believe after TF eager
is supported.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/keras/issues/11068#issuecomment-442986798,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJ4j6JCHvCY9TN304mENcMY7uj12XVKks5u0Ee8gaJpZM4WXh8V
.

[gabrieldemarmiesse]

I was able to get the inference working with numpy by patching keras here and there and use model subclassing. By calling the function call(), I was able to run the graph without ever using placeholders. I believe the issue of placeholders is going to go away once keras supports TF eager.

See https://keras.io/models/about-keras-models/#model-subclassing

[alexbw]

Very cool. Is there an open issue/PR for the changes coming for TF eager support?

[gabrieldemarmiesse]

tf.keras supports tf eager. I believe @fchollet will re-implement it for keras-team/keras once TF 2.0 is out and that he'll have less work.

[alexbw]

@gabrieldemarmiesse would you mind sharing your patches?

EDIT: looks like a patch to base_layer is required for this, to conditionally call compute_output_shape instead of call if a deferred object is passed in in base_layer:Layer:__call__.

[alexbw]

FYI I am currently specializing the backend for github.com/google/jax, but most of the changes could easily be shared between a pure-NumPy backend and a JAX backend.

https://github.com/alexbw/keras/tree/jax-backend

The numpy/jax_backend.py file is pretty straight-forward. Where things get hairy is specializing code in training.py and elsewhere that assume a deferred computation model. If you take a peek in TensorFlow's Keras implementation, you'll find a lot of if in_eager_mode(): do one thing else: do another thing type code.

[gabrieldemarmiesse]

@alexbw Sorry for the delay in answering. My week has been faily busy and I'm currently trying to revive keras-contrib.

Currently, the only way to use the numpy backend is for inference. The way to use it with a minimal amount of hacks is to use the model subclassing API (you'll still need to make a fork of keras).

When using the model subclassing API, instead of using model.predict, you should use model.call(). This is a trick to prevent keras from creating a backend Function (a graph) and use placeholders.

If you want to use keras with a numpy backend for training (with JAX), it's very likely that you'll have to wait until keras-team/keras implements tensorflow eager. Since tensorflow eager is also imperative (like JAX) it should be easy to then implement the JAX backend for training in a fork.

[alexbw]

@gabrieldemarmiesse No worries, just reporting my progress. In my fork, making predictions (inference) now works. Just as a proof-of-concept, I'm patching in the changes from TensorFlow's version of Keras which supports tensorflow eager. I don't know when that will be officially moved to OSS Keras, so I'm just hacking something together as a prototype.

[gabrieldemarmiesse]

Currently the pain point of our numpy backend (performance-wise) is the pooling operation and the convolution. If you have any insights, this would be much appreciated. We do way too many for loops. I don't think JAX would like it either.

[alexbw]

There is a lot of hackery currently, but on my branch linked above you can "fake train" a simple MLP with JAX (you'll need the unmerged keras branch, and in-place updates are busted). I expect that my branch will never be mergeable, because of the incoming changes for TF Eager support.

The only part that is interesting is where we calculate gradients, and make them available for optimizers. Pretty simple! The rest is gross plumbing.

Things I'll work on as I get time:

• I'm calculating the forward pass twice, unnecessarily. Remove that.
• JAX has a jit decorator that should dramatically speed things up
• Test on GPU. We might be disadvantaged on CPU, because JAX currently doesn't use MKL.
• In-place updates are a no-no in JAX, so I'll need to port the apply_gradients functionality from TF optimizers in Keras.

This test script below works with the keras branch of JAX.

import os
os.environ['KERAS_BACKEND'] = 'jax'
from keras.models import Sequential
from keras.layers import Dense
from keras.datasets import mnist
from keras.utils import np_utils
from jax import device_put
import jax.numpy as np
import numpy as onp
## MNIST:
onp.random.seed(0)
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_train = X_train.reshape(-1, 28*28)
X_test /= 255
X_test = X_test.reshape(-1, 28*28)
Y_train = np_utils.to_categorical(y_train, 10)
Y_test = np_utils.to_categorical(y_test, 10)

model = Sequential()
# --------------------------------------------------
d = Dense(units=64, activation='relu', input_dim=28*28)
model.add(d)
d = Dense(units=10, activation='softmax')
model.add(d)
# --------------------------------------------------
out = model.predict(X_train[:1])
# --------------------------------------------------
out = model.predict(X_train[:10])
# --------------------------------------------------
model.compile(loss='categorical_crossentropy',
              optimizer='sgd',
              metrics=['accuracy'])
model.train_on_batch(X_train, Y_train)
# --------------------------------------------------
model.fit(X_train, Y_train, epochs=5, batch_size=32)

[gabrieldemarmiesse]

Sir, I tip my hat to you. Great proof of concept. I would not have thought for a minute that the numpy backend (which was only there for testing at first) would have allowed something like this :)

[alexbw]

Thanks! My bet is that with a little work, the JAX backend can be as fast or faster than the others.
FYI we have convolution and pooling operations that would be easy to port. I think it'd be best to wait until eager support gets merged in, to avoid doing too much duplicate work.

[danFromTelAviv]

@alexbw That is awesome!
I know cython was pushed down but it may be a simple add-on for faster performance as a separate package that just wraps the numpy back end and replaces the slow functions.

Last week I finally got an ssd variant to run at 30fps on a mid-range arm cpu with cython. Numba is also really coming up it looks like - It may soon be robust and simple enough as a required package for keras. It also has simple gpu support which is really useful in this case.

Anyways - not worrying about that just yet.

[Mofef]

Wow this is great news! :)

If I conclude correctly:

• Basically everything is done and we are now waiting for TF eager support
• For now I can't make a saved model work with the numpy backend if it wasn't built using the subclassing api

Please correct me if I got sth. wrong

And if someone would give me a hint on how to modify keras to test this, that would be awesome!

[kvrooman]

If you're looking for speed and want to avoid loops and list creation in the convolution function, here's a batched single channel 2d convolve using einsum and .as_strided. Alternatively, one can use skimage.util.view_as_windows in place of .as_strided. Adapting it to multi-channel images and multi-channel filters would need some amendments and the conv pre-process would need to setup the einsum strings. As a plus, you avoid using scipy ( technically not a numpy backend... )

def conv2d(x, f):
    """Image of dimension b, m, n and filter of dimension p, q """
    shape = f.shape + (x.shape[0],) + tuple(np.subtract(x.shape[1:], f.shape) + 1)
    strides = (x.strides * 2)[1:]
    M = np.lib.stride_tricks.as_strided(x, shape=shape, strides=strides)
    y = np.einsum('pq,pqbmn->bmn', f, M)
    return y

Source:
https://stackoverflow.com/questions/45540000/batch-convolution-2d-in-numpy-without-scipy