Spleeter: [Feature] TFLite (Pretrained Models)

[jinay1991]

Description

As the repository provides, Tensorflow v1.15.2 Checkpoints which can be converted to SavedModel for all the pretrained models.

This issue is to request support for TFLite and support for TensorFlow v2.1.

Additional information

• Current version of trained model (i.e. SavedModel) can not be converted to TFLite due to issues. This feature support is also indented to fix that issue and possibly retrain the model with TF v2.1

[mmoussallam]

Hi @jinay1991

Point taken, we'll investigate porting the model to TF2 compat in the coming weeks.

[waqasakram117]

Hi @mmoussallam
Please make compatible with TF lite as well so it can target a vast variety.

[Vnicius]

This will be amazing

[abhi-rawat1]

Point taken, we'll investigate porting the model to TF2 compat in the coming weeks.

@mmoussallam Eagerly waiting of this to get out. Do we know when TFLite version will be available?

[searchff]

Description

As the repository provides, Tensorflow v1.15.2 Checkpoints which can be converted to SavedModel for all the pretrained models.

This issue is to request support for TFLite and support for TensorFlow v2.1.

Additional information

• Current version of trained model (i.e. SavedModel) can not be converted to TFLite due to issues. This feature support is also indented to fix that issue and possibly retrain the model with TF v2.1

Would you please share the tflite file?

[VVasanth]

@mmoussallam - Would be really great if you could throw some light on when TF2.0 would be available.

[guomingjin]

Hi everyone, I have already converted the models into c++ implementation with android jni project, which doesn't depend on any deep learning framework.
Those models are really great and pretty cool.
I am arranging to make a real-time mobile app.
Thank you for your great work.
Regards.

[waqasakram117]

@guomingjin It is great so, can you please share your amazing work with us.

[actuy]

Hi everyone, I have already converted the models into c++ implementation with android jni project, which doesn't depend on any deep learning framework.
Those models are really great and pretty cool.
I am arranging to make a real-time mobile app.
Thank you for your great work.
Regards.

@guomingjin That's amazing! Could you share this with us?

[jinay1991]

Hi @jinay1991

Point taken, we'll investigate porting the model to TF2 compat in the coming weeks.

@mmoussallam Is there any update on this? Is anyone working on this?

[mmoussallam]

Closing this since v2 is now TF2 compatible

[jinay1991]

@mmoussallam Thanks for taking care of support for TF2. Will you be kind to update pre-trained models as well with recent Spleeter Version?

[agur]

@mmoussallam Thanks for taking care of support for TF2. Will you be kind to update pre-trained models as well with recent Spleeter Version?

Is all needed for TFLite support is to re-train after the TF2 support was coded?

Is the progress here https://github.com/jinay1991/spleeter/issues/1 needed? Ready?

Can we please figure the needed steps together? Would be happy to help make it easy to use on Mobile.

[jinay1991]

Hello @agur Thanks for your msg. We have a open feature request for the same https://github.com/deezer/spleeter/issues/477.

With the help of tinoucas/spleeter-tflite-convert, I was manage to convert all the existing models (TF v1.13) in a hacky way (i.e. Input/Output Tensors did not have their shapes defined in TFLite model hence ResizeTensors was necessary for TFLite Models afterwards and in addition to that binary for TFLite is comparatively too big due to the usage of TF_SELECT_OPS). Hence I am assuming that with retraining of the Model with TFv2 support would atleast avoid this hacky way and one could use Quantisation and Optimisation techniques to make binary smaller yet have same results.

1. Retrain and Release Models with TFv2 versions
2. Once TFv2 models are available in fact it shall be easy to release TFLite Models as well.
3. If possible, remove TF OPS and use compatible OPS between TF/TFLite OPS. ops_compatibility
4. Provide post-training quantisation for both types of models post_training_quantization, this requires training dataset hence it is best is someone who is training the model provides this thing as well.

And I think these are all we need for now in order to integrate this with any platform. Once done one can check/deploy with https://github.com/jinay1991/spleeter/tree/master/spleeter/test

[agur]

Hi @jinay1991, appreciate the prompt response!

Should this discussion be on #477 then? (And should #477 title be "Spleeter Models for Tensorflow 2 and tflite (mobile)" rather than the existing: ~Spleeter Models for Tensorflow tflite~?)

If the other routes are hacky and we're clear that re-train models for TF2 which is easy to convert to tflite is the right way than maybe we want to only focus on that now.

And for 1.: Do we have the original data/music the original was trained with? So re-train can happen? And even if it is heavy/complex/won't-happen-too-often - we may want to ensure the process is as easy to repeat as possible for the future while running the current pass of re-train.

[jinay1991]

Hi @jinay1991, appreciate the prompt response!

Should this discussion be on #477 then? (And should #477 title be "Spleeter Models for Tensorflow 2 and tflite (mobile)" rather than the existing: ~Spleeter Models for Tensorflow tflite~?)

If the other routes are hacky and we're clear that re-train models for TF2 which is easy to convert to tflite is the right way than maybe we want to only focus on that now.

And for 1.: Do we have the original data/music the original was trained with? So re-train can happen? And even if it is heavy/complex/won't-happen-too-often - we may want to ensure the process is as easy to repeat as possible for the future while running the current pass of re-train.

Yes I agree with you on all points. You can find dataset on musDB dataset Please check and add LICENSE file as well with Trained Model for consumers to know about.

Lets carry out all further discussion can be carried out on #477 .

[VVasanth]

HI @agur, @jinay1991 - I am currently involved in porting Spleeter code into TF2.0 with the latest features...Following are the summary of the activities completed till date:

--> Have used musdb dataset for training purpose...as per Spleeter...
--> Have reproduced the data processing pipeline, as similar to Spleeter
--> Have recreated the model generation process with identical network but using TF2.0 features of GradientTape...
--> I am abe to create the model and export it to TFLite format as well...
--> Have built the android app that processes the data and feeds it to the TFLite model...there are still some work to be done and will complete them prob by today...
--> You can find the github repository link below...
--> I have trained the model only with less data and for only 2000 runs on my CPU (which ran for ~5 days)...Spleeter has been trained for 200K runs on the complete dataset on GPUs over a period of a week...so, accuracy of my model is pretty less...I will focus on training my model on Google Collab and improving the accuracy in coming week...

So, right now - I have the Spleeter version ported to TF2.0 with all its features and the Android app that leverages the generated TFLite model available...But the accuracy is less :-(

If you are interested, you can join with me and work on improving. the accuracy..thereby, we will be able to build an audio separation model deployed on edge...

Spleeter TF2.0 Model (Unofficial): https://github.com/VVasanth/SpleeterTF2.0_Unofficial
Spleeter TF2.0 Mobile App Github link (Unofficial) : https://github.com/VVasanth/Spleeter_Unofficial_TF20_MobileApp

Pls let me know if you need any additional info and like to participate on this...

[agur]

Answered at: https://github.com/deezer/spleeter/issues/477#issuecomment-740527178, we can continue there.