Tfjs: tf.loadModel not working in react native

[tangtai]

TensorFlow.js version

0.10.3

Browser version

react-native 0.55.3

Problem:

tf.loadModel not working for local folder and remo url in react native, load local model not working for React

Feature request:

tf.loadModel react-native support

Code to reproduce the bug / link to feature request

const path = 'https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_0.25_224/model.json';
const localPath = './tfjsmodel/model.json';

  async loadmodel(p) {
    const model = await this.loadHostedPretrainedModel(p);
    console.log(model);
  }

  async loadHostedPretrainedModel(p) {
    try {
      const model = await tf.loadModel(p);
      return model;
    } catch (err) {
      console.error(err);
    }
  }

this is the demo app i created for running tfjs in react-native, it can train and predict model locally,
if you are interested this is the repo: https://github.com/tangtai/tfjsdemo

error i get when try to load model from url, i tried the code on react as well it returns the model successfully:

error i get when try to load model from local folder, for React it returns pending Promise :

the model i got from tensorflowjs file convertor:

[timotheebernard]

Hi @tangtai, I'm not really familiar with React but I think you need to find a way to serve your model with a http server, moreover, it must allow CORS requests.

Is there a way to do this within the React ecosystem? Otherwise, I described a solution in a similar issue: #257.

[tangtai]

@timotheebernard thanks for your response, i tried to host the model on AWS s3 bucket and turn on CORS request it worked for React(web app), but didn't work for react native(mobile app), apparently according to https://facebook.github.io/react-native/docs/network.html native apps have not concept of CORS.

how ever i can only fetch the json file from AWS and it returns the model configuration for me:

I'm wondering if we can somehow bypass the CORS issue and turn the model.json file into a tf.Sequential model we can use in our native app that would be great!

as i was trying to figure out what tf.loadModel does, i think here is what it uses the model.json file to configure the model https://github.com/tensorflow/tfjs-layers/blob/72d82c90619c449548c5e881c64fcc5a299c5cfc/src/models.ts#L200 but im not 100% sure.

and i also tried tfjs examples on chrome on my phone, the model still works offline, once the model is loaded.

[pyu10055]

@tangtai does React native support loading file locally? If so, this would be similar to node.js env, we can provide API to load the model and weights from local file system instead of urls.

[tangtai]

hi @pyu10055 , react native can definitely load local json files, as for other files that created by the tensorflowjs converter i am not so sure, since they don't have a file format. I'm wondering how exactly does the loadModel function turns the json file into a tf.Sequential(). Even if the model is loaded from url other than CORS, if tf.Sequential() is a javascript object, i think react native will be able to store the object locally in async Storage, and use the model offline.

[pyu10055]

Adding @caisq to this conversation. That could be a better route, we should move to a more generic loading API, sitting on top of the new serialization mechanism, which allows you to create io handler to store and load models for different storages.

[caisq]

tf.loadModel() works by

1. making a request to the specified model.json path. In the JSON response, there is a field called weightsManifest, containing a number of relative paths.
2. requesting binary weight values from the relative paths.

So he server that serves model.json must be able to work with relative path. This is the case for Google Cloud Storage, as many of our example and demo models are hosted there. Isn't this also the case for AWS S3 buckets work with relative paths too? It has been a while since I last used S3.

[tangtai]

@caisq for AWS s3 buckets you can enable CORS sharing, but it is a web thing, react native can't do it, while React can because it is a web framework. i think i understand the other files are storing the weights of the neural net, while the json file is the configuration of the neural net. it is just strange to me that you have to host your json model to access it, even-though the model is stored locally, is there a way around without hosting it for CORS sharing?

[caisq]

@tangtai Looking at your original error message, it seems to fail at the step where FileReader.readAsArrayBuffer is called. This is the 2nd step, i.e., the step where the binary weights are loaded. It says the method is not implemented. So it seems to be not directly related to CORS, but a limitation of your browser. What browser and platform is this?

[tangtai]

@caisq I’m using react-native which is a JavaScript platform for mobile development, it says on its documents it does not support CORS, can android use tfjs?

[caisq]

@tangtai Actually, tf.loadModel in version 0.10.3 of tfjs doesn't actually call FileReader.readAsArrayBuffer directory. It should call only fetch. Is the call from your own code?

[tangtai]

the version of tfjs Im using is @ version 0.10.3.

[Ivikassingh]

you can try webpack to import the cdn of tensorflowjs in Reactnative

[pyankoff]

Managed to make it work by using a custom loader:

const loader = {
  load: async () => {
    return {
      modelTopology: topology,
      weightSpecs: specs
      weightData: data,
    };
  }
}

const model = await tf.loadLayersModel(loader)

[prasadkawthekar]

@tangtai could you please share the code you used to load a tf.loadLayersModel from AWS s3 bucket on React webapp? Thanks!

[tafsiri]

We just released a package for this! https://www.npmjs.com/package/@tensorflow/tfjs-react-native try it out and feel free to re-open this issues if you have trouble with it.