Deepspeech: Intel Movidius NCS Support

[madhavajay]

• **Have I written custom code Not yet
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
• TensorFlow installed from (our builds, or upstream TensorFlow): tags/v0.1.1
• TensorFlow version (use command below): TensorFlow 1.4
• Python version: Which ever
• Bazel version (if compiling from source): N/A
• GCC/Compiler version (if compiling from source): N/A
• CUDA/cuDNN version: N/A
• GPU model and memory: Movidius NCS
• Exact command to reproduce:

I am interested in compiling the TensorFlow Protobuff to a NCS compatible Model file and making the necessary changes to allow the NCS to accelerate DeepSpeech.

It says here that the NCSDK supports TensorFlow 1.4
https://github.com/movidius/ncsdk/releases

The 0.1.1 release was built using TensorFlow 1.4 right, so that shouldn't be a problem?
I was under the impression that the Mozilla DeepSpeech project was using their own fork of TensorFlow at some point though, is that likely to pose a problem?

Can anyone else see for any reason why this might not work or have any other suggestions or feedback?

Also can anyone comment on how the final Python Wheel is being compiled?
I see taskcluster scripts and mention of the native client and wheel, does the wheel actually use the native client in it bridged to C? Or is the python release actually all in python?

[lissyx]

@madhavajay Thanks, but most of your questions should be asked in Discourse: https://discourse.mozilla.org/c/deep-speech

I don't know anything about NCSDK, so we cannot help you on that. Any contribution is welcome, though. The v0.1.1 was TensorFlow r1.4, yes. We have a fork but it's mostly for some light changes, nothing big. We don't build anymore the TensorFlow python wheel :), and native client is actually code to perform inference of the network.

[lissyx]

Taking a closer look to https://github.com/movidius/ncsdk/releases, it seems not far away from what tfcompile does. However, it seems to have huge limitation in term of ops, according to the "Firmware feature". I fear you are going to not be able to use it :(

[madhavajay]

Thanks @lissyx yes I wanted to post in Discourse but I cant log in or sign up:
https://twitter.com/madhavajay/status/974727853483986944

Yes I was afraid there might be some issues in terms of supported Operations, I assume attempting to convert the protobuff using their tool should pass or fail based on those issues though right?
My guess is that some of the RNN stuff could be a problem. I know most of the focus is on CNN optimisations and RNN ops are often the last ones to be supported.

If the protobuff converts successfully and the ops are supported, then I guess its just a matter of forking and patching the 0.1.1 release with code to use the NCSDK?

Are you saying that even in 0.1.1 the native client is where all the action is and I will need to edit the C code to figure out where to use the NCSDK rather than just changing some python code which loads tensorflow?

[lissyx]

@madhavajay Sorry for the Discourse issue, but it's good you pinged people on Twitter. Can you be more descriptive of why / how Github and Google logins don't work? Regarding the ops, yes, I share your concerns about RNNs.

If the conversion works, then you can try and start hacking, but we will have to figure out something, since right now it's all within libdeepspeech.so, and this more or less relies on tensorflow. It's all about C++ yes, there is no Python involved. The inference relies on TensorFlow's C++ API.

Besides, please start hacking from master, not 0.1.1 :)

[lissyx]

Frankly, I'd say get it to work, whatever the most hackish way it might be. If it runs then it's always time to make it, a PR, then we can make it nicer and land it :-).

[madhavajay]

Sorry regarding login, it now says that my GitHub needs 2FA... Also I cant change it to a normal signup because of the automated redirection. I tried logging out of github but i think i need a full cookie wipe because the redirect happens so fast I cant clear the cookies on sso. When I tried the other day I managed to get the login box but when I added an email instead the link I got in the email didnt work either...

Gonna wipe my cookies and start again see what happens this time.

Regarding the TF and C++ stuff. Okay so it would depend on how NCS is able to accelerate TF C++ code and models, at least I know where to begin looking, but it sounds fairly hard.

I would hope this stuff will become less of a problem with XLA and TF Lite type work where the NNAPI on Android will provide a layer of abstraction that hardware drivers can provide custom optimisation support. I hope that something like this becomes part of the main TF project so people dont need 3rd party hardware optimisation code paths and model conversion formats.

Also I assume working off master is a bad idea since its all setup for newer TF versions right, so surely the pb files are going to be incompatible? Or do you use the exact same network setup as earlier versions?

[madhavajay]

Sorry about this, something is foobared, I have completely wiped my cookies.

I entered my email to signup to discourse. Got the email which reads:
Mozilla

Click and confirm that you want to sign in to discourse.mozilla.org. Please sign in using the same web browser that you used to generate this link. The link will expire in five minutes.

SIGN IN TO DISCOURSE.MOZILLA.ORG

The page then says this:
Sorry, you may not login using passwordless email. We require login to be performed using the most secure method available for your account, which is GitHub.

Hahahahaha. Now the google button is gone.
When i reload the page from the login/ signup i see the google button again.
Clicking it gives me:

Sorry, you may not login using Google. We require login to be performed using the most secure method available for your account, which is GitHub.

Okay so I have to use GitHub, but im not allowed to use GitHub unless i have 2FA....
I assume something is wrong here.

[lissyx]

@madhavajay DeepSpeech master tracks some stable TensorFlow version, not master. We currently switched to r1.6. If you do the work againt v0.1.1 it will be useless, since we cannot integrate it. If you want to play with TensorFlow master, then use DeepSpeech tf-master branch.

[madhavajay]

Yes, I understand that but if the NCSDK doesnt work in anything newer than TF 1.4 currently then it wont work anyway.

It could be that if it works in an older version it could be parked until the newer support is released. Anyway for now this is looking like its going to be too difficult right now, but I really appreciate the time you have taken to explain to me where the integration surface is.

BTW can anyone suggest any good resources on TensorFlow and C++. Iv done a bit of Objective-C so I do know a bit of C, but TensorFlow in C++ kinda intimidates me.

[lissyx]

@madhavajay I don't know the details of their "1.4" limitation. My understanding is that they work on a protobuf file, not rely on the API. So as long as the model is produced with TensorFlow r1.4, it would be fine? Or did I misunderstood anything?

[madhavajay]

Are you using the same protobuff file from 0.1.1 release with master? I just assumed it would change slightly since the versions of protobuff are changing constantly in the tensorflow repo.

Also I don't know I havent really gotten in depth with the NCSDK, but the examples seem to be including their code into your project and converting the model file to their format, so I don't know it might not be so difficult.

[lissyx]

Well, the protobuf for release model has nothing to do with the release versions we use: it's being trained on a different, dedicated infra. We have tests to ensure it works correctly with our inference code release though, but you can safely take it and try to hack.

[madhavajay]

Yes, that was my intention to go back to 0.1.1 with TF 1.4 and try it there. But for now, I would say, close this ticket, thanks for all the time and explanation! 👍

[lissyx]

@madhavajay So you give up ? I would have been interested in what we can expect from this device :-)

[madhavajay]

I haven't given up, I just haven't started yet. I am also interested in what can be done with the device, unfortunately I need to prioritise my time with other tasks. If I get a chance to revisit this I will. If anyone else wants to take a stab that would be great.

[madhavajay]

Maybe the Intel NCS team would be interested in adding it to their "supported models" as a good demo of their hardware on RNNs?

[lissyx]

@madhavajay Maybe, but we don't have contact with them, so, it's not in our hands. If you ever come up with anything that looks promising, feel free to chime here :)

[madhavajay]

@nealvis at Intel NCSDK, care to comment?

[nealvis]

@madhavajay Thanks for asking. After glancing through this thread it looks like you are looking for RNN support which unfortunately isn't present in the current release of the NCSDK. Let me know if I misinterpreted that though.

[madhavajay]

@nealvis Yeah that would probably be necessary. Is this on the roadmap, and if so would Intel consider using the Mozilla DeepSpeech project to test their Movidius support / integration? That would be a pretty awesome demo and perfect real world use case of cutting edge RNN usage. 👍 Plus Open Source... 🧐

[hordiales]

Hi! Any news about this? Someone with the movidius neural stick working in the inference process?

[herdtie]

I think about buying me a movidus, specifically for speech processing. If this worked I would not hesitate

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.