Tfjs: Local fetch not working in android tfjs-react-native API

[AbdulRashidReshamwala]

TensorFlow.js version

@tensorflow/tfjs" : 1.7.4
@tensorflow/tfjs-react-native : 0.2.3

Browser version

expo : 37.0.3

Describe the problem or feature request

Local fetch does not work on android but works fine on ios

Code to reproduce the bug / link to feature request

import React, { useEffect, useState } from "react";
import { StyleSheet, Text, View, Button, Image } from "react-native";
import { ImageBrowser } from "expo-multiple-media-imagepicker";
import * as tf from "@tensorflow/tfjs";
import * as tf_rn from "@tensorflow/tfjs-react-native";
import * as Permissions from "expo-permissions";
import * as jpeg from "jpeg-js";

const askPerm = async () => {
  return await Permissions.askAsync(Permissions.CAMERA_ROLL);
};


export default function App() {
  const [imagePickerOpen, setImagePickerOpen] = useState(false);
  const [TFReady, setTFReady] = useState(false);

  useEffect(() => {
    askPerm();
    tf.ready().then(() => setTFReady(true));
  }, []);

  const imageBrowserCallback = async (s) => {
    let files = await s;
    console.log(files);
    const r = await tf_rn.fetch(files[0].uri, {}, { isBinary: true });
    const rawImageData = await r.arrayBuffer();
    const imageTensor = imageToTensor(rawImageData);
    setImagePickerOpen(false);
  };
  return (
    <View style={styles.container}>
      <Text>TF Loaded {TFReady.toString()}</Text>
      {imagePickerOpen ? (
        <ImageBrowser
          max={20} // Maximum number of pickable image. default is None
          headerButtonColor={"#E31676"} // Button color on header.
          badgeColor={"#E31676"} // Badge color when picking.
          callback={imageBrowserCallback}
        />
      ) : (
        <Button
          title="open"
          onPress={() => setImagePickerOpen(!imagePickerOpen)}
        ></Button>
      )}
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    backgroundColor: "#fff",
    alignItems: "center",
    justifyContent: "center",
  },
});

Example of a files[] for android

Array [
Object {
    "albumId": "-1034915852",
    "creationTime": 1491896243000,
    "duration": 0,
    "exif": Object {
      "DateTime": "2017:04:11 13:07:23",
      "DateTimeOriginal": "2017:04:11 13:07:23",
      "ImageLength": 637,
      "ImageWidth": 622,
      "LightSource": 0,
      "Orientation": 0,
      "UserComment": "{\"sha1\":\"4e7b7ebc00464c46dfc8b03c0b8ad968770bfc6b\",\"ext\":\"jpg\"}",
    },
    "filename": "Share-your-encounters-with-disguised-devils.jpg",
    "height": 637,
    "id": "2638",
    "localUri": "file:///storage/emulated/0/pictures/9gag/Share-your-encounters-with-disguised-devils.jpg",
    "location": null,
    "mediaType": "photo",
    "modificationTime": 1491896243000,
    "uri": "file:///storage/emulated/0/pictures/9gag/Share-your-encounters-with-disguised-devils.jpg",
    "width": 622,
  }
]

ios:

Array [
  Object {
    "creationTime": 1588249831000,
    "duration": 0,
    "exif": Object {
      "ColorModel": "RGB",
      "Depth": 16,
      "HasAlpha": true,
      "Orientation": 1,
      "PixelHeight": 2688,
      "PixelWidth": 1242,
      "ProfileName": "Display P3",
      "{Exif}": Object {
        "DateTimeOriginal": "2020:04:30 18:00:31",
        "PixelXDimension": 1242,
        "PixelYDimension": 2688,
        "UserComment": "Screenshot",
      },
      "{PNG}": Object {
        "InterlaceType": 0,
      },
      "{TIFF}": Object {
        "Orientation": 1,
      },
    },
    "filename": "IMG_0546.PNG",
    "height": 2688,
    "id": "30493338-1075-414A-A9D0-8C2FC25F52C4/L0/001",
    "isFavorite": false,
    "isHidden": false,
    "localUri": "file:///var/mobile/Media/DCIM/100APPLE/IMG_0546.PNG",
    "location": null,
    "mediaSubtypes": Array [
      "screenshot",
    ],
    "mediaType": "photo",
    "modificationTime": 1588249832257,
    "orientation": 1,
    "uri": "assets-library://asset/asset.PNG?id=30493338-1075-414A-A9D0-8C2FC25F52C4&ext=PNG",
    "width": 1242,
  },
]

Error faced in android:

[Unhandled promise rejection: TypeError: Network request failed]
- node_modules/@tensorflow/tfjs-react-native/dist/platform_react_native.js:97:22 in xhr.onerror
- node_modules/event-target-shim/dist/event-target-shim.js:818:39 in EventTarget.prototype.dispatchEvent
- node_modules/react-native/Libraries/Network/XMLHttpRequest.js:574:29 in setReadyState
- node_modules/react-native/Libraries/Network/XMLHttpRequest.js:388:25 in __didCompleteResponse
- node_modules/react-native/Libraries/vendor/emitter/EventEmitter.js:190:12 in emit
- node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:436:47 in __callFunction
- node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:111:26 in __guard$argument_0
- node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:384:10 in __guard
- node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:110:17 in __guard$argument_0
* [native code]:null in callFunctionReturnFlushedQueue

[zalkier]

I am facing the exact same problem in Android, with the package.json:

{
  "main": "node_modules/expo/AppEntry.js",
  "scripts": {
    "start": "expo start",
    "android": "expo start --android",
    "ios": "expo start --ios",
    "web": "expo start --web",
    "eject": "expo eject"
  },
  "dependencies": {
    "@react-native-community/async-storage": "^1.9.0",
    "@tensorflow-models/mobilenet": "^2.0.4",
    "@tensorflow/tfjs": "^1.7.3",
    "@tensorflow/tfjs-react-native": "^0.2.3",
    "expo": "~37.0.3",
    "expo-camera": "^8.2.0",
    "expo-gl": "^8.1.0",
    "jpeg-js": "^0.4.0",
    "react": "~16.9.0",
    "react-dom": "~16.9.0",
    "react-native": "https://github.com/expo/react-native/archive/sdk-37.0.1.tar.gz",
    "react-native-fs": "^2.16.6",
    "react-native-web": "~0.11.7"
  },
  "devDependencies": {
    "@babel/core": "^7.8.6",
    "babel-preset-expo": "~8.1.0"
  },
  "private": true
}

and with the App.js:

 import React from 'react'
import { StyleSheet, Text, View, StatusBar, ActivityIndicator, TouchableOpacity, Image,  } from 'react-native'
import * as tf from '@tensorflow/tfjs'
import { fetch, bundleResourceIO, decodeJpeg } from '@tensorflow/tfjs-react-native'

//import * as mobilenet from '@tensorflow-models/mobilenet'

import Constants from 'expo-constants'
import * as Permissions from 'expo-permissions'

import * as jpeg from 'jpeg-js'

import * as ImagePicker from 'expo-image-picker'

import * as FileSystem from 'expo-file-system'

class App extends React.Component {
  state = {
    isTfReady: false,
    isModelReady: false,
    predictions: null,
    image: null
  }

  async componentDidMount() {
    await tf.ready()
    this.setState({
      isTfReady: true
    })

    const modelJSON = require('./assets/model/model.json');
    const modelWeights = require('./assets/model/modelweights.bin');

    this.model = await tf.loadLayersModel(bundleResourceIO(modelJSON, modelWeights));

    this.setState({ isModelReady: true })

    // add this
    this.getPermissionAsync()
  }

  getPermissionAsync = async () => {
    if (Constants.platform.ios) {
      const { status } = await Permissions.askAsync(Permissions.CAMERA_ROLL)
      if (status !== 'granted') {
        alert('Sorry, we need camera roll permissions to make this work!')
      }
    }
  }

  imageToTensor(rawImageData) {
    const TO_UINT8ARRAY = true
    const { width, height, data } = jpeg.decode(rawImageData, TO_UINT8ARRAY)
    // Drop the alpha channel info for mobilenet
    const buffer = new Uint8Array(width * height * 3)
    let offset = 0 // offset into original data
    for (let i = 0; i < buffer.length; i += 3) {
      buffer[i] = data[offset]
      buffer[i + 1] = data[offset + 1]
      buffer[i + 2] = data[offset + 2]

      offset += 4
    }

    return tf.tensor3d(buffer, [height, width, 3])
  }

  classifyImage = async () => {
    try {
      const imageAssetPath = Image.resolveAssetSource(this.state.image)
      const response = await fetch(imageAssetPath.uri, {},  { isBinary: true })
      const rawImageData = await response.arrayBuffer()

      const imageTensor = decodeJpeg(rawImageData)

      const predictions = await this.model.predict(imageTensor)

      this.setState({ predictions })
      console.log(predictions)
    } catch (error) {
      console.log(error)
    }
  }

  selectImage = async () => {
    try {
      let response = await ImagePicker.launchImageLibraryAsync({
        mediaTypes: ImagePicker.MediaTypeOptions.All,
        allowsEditing: true,
        aspect: [4, 3]
      })

      if (!response.cancelled) {
        const source = { uri: response.uri }
        this.setState({ image: source })
        this.classifyImage()
      }
    } catch (error) {
      console.log(error)
    }
  }

  render() {
    const { isTfReady, isModelReady, predictions, image } = this.state

    return (
      <View style={styles.container}>
        <StatusBar barStyle='light-content' />
        <View style={styles.loadingContainer}>
          <Text style={styles.commonTextStyles}>
            TFJS ready? {isTfReady ? <Text>✅</Text> : ''}
          </Text>

          <View style={styles.loadingModelContainer}>
            <Text style={styles.text}>Model ready? </Text>
            {isModelReady ? (
              <Text style={styles.text}>✅</Text>
            ) : (
                <ActivityIndicator size='small' />
              )}
          </View>
        </View>
        <TouchableOpacity
          style={styles.imageWrapper}
          onPress={isModelReady ? this.selectImage : undefined}>
          {image && <Image source={image} style={styles.imageContainer} />}

          {isModelReady && !image && (
            <Text style={styles.transparentText}>Tap to choose image</Text>
          )}
        </TouchableOpacity>
        <View style={styles.predictionWrapper}>
          {isModelReady && image && (
            <Text style={styles.text}>
              Predictions: {predictions ? '' : 'Predicting...'}
            </Text>
          )}
          {isModelReady &&
            predictions &&
            predictions.map(p => this.renderPrediction(p))}
        </View>
        <View style={styles.footer}>
          <Text style={styles.poweredBy}>Powered by:</Text>
          <Image source={require('./assets/tfjs.jpg')} style={styles.tfLogo} />
        </View>
      </View>
    )
  }
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    backgroundColor: '#171f24',
    alignItems: 'center'
  },
  loadingContainer: {
    marginTop: 80,
    justifyContent: 'center'
  },
  text: {
    color: '#ffffff',
    fontSize: 16
  },
  loadingModelContainer: {
    flexDirection: 'row',
    marginTop: 10
  },
  imageWrapper: {
    width: 280,
    height: 280,
    padding: 10,
    borderColor: '#cf667f',
    borderWidth: 5,
    borderStyle: 'dashed',
    marginTop: 40,
    marginBottom: 10,
    position: 'relative',
    justifyContent: 'center',
    alignItems: 'center'
  },
  imageContainer: {
    width: 250,
    height: 250,
    position: 'absolute',
    top: 10,
    left: 10,
    bottom: 10,
    right: 10
  },
  predictionWrapper: {
    height: 100,
    width: '100%',
    flexDirection: 'column',
    alignItems: 'center'
  },
  transparentText: {
    color: '#ffffff',
    opacity: 0.7
  },
  footer: {
    marginTop: 40
  },
  poweredBy: {
    fontSize: 20,
    color: '#e69e34',
    marginBottom: 6
  },
  tfLogo: {
    width: 125,
    height: 70
  }
})

export default App

Results in:

Network request failed
- node_modules\@tensorflow\tfjs-react-native\dist\platform_react_native.js:97:22 in xhr.onerror
- node_modules\event-target-shim\dist\event-target-shim.js:818:39 in EventTarget.prototype.dispatchEvent
- node_modules\react-native\Libraries\Network\XMLHttpRequest.js:574:29 in setReadyState
- node_modules\react-native\Libraries\Network\XMLHttpRequest.js:388:25 in __didCompleteResponse
- node_modules\react-native\Libraries\vendor\emitter\EventEmitter.js:190:12 in emit
- node_modules\react-native\Libraries\BatchedBridge\MessageQueue.js:436:47 in __callFunction
- node_modules\react-native\Libraries\BatchedBridge\MessageQueue.js:111:26 in __guard$argument_0
- node_modules\react-native\Libraries\BatchedBridge\MessageQueue.js:384:10 in __guard
- node_modules\react-native\Libraries\BatchedBridge\MessageQueue.js:110:17 in __guard$argument_0
* [native code]:null in callFunctionReturnFlushedQueue

[pathakcodes]

Any updates on the issue. I am facing exactly the same issue as @zalkier
Please see, Thanks

[daoshengmu]

It doesn't work for me too. I am developing an Android app. I have tried giving a file path from external storage or a HTTPS URL from S3. Both of them didn't work. Btw, my package version is tfjs-react-native:0.1.0-alpha.2

[hayzem]

I'm facing same problem here with trying to fetch local image uri starting with file://
version: @tensorflow/[email protected]

[tafsiri]

Digging into this it looks like fetch is not a good fit for loading these types of files. The api our custom fetch is trying to implement leaves behavior for file:// scheme urls undefined. I probably wouldn't rely on on it for this even where it seems to work. Instead I would use an existing file reading mechanism in react native to get the data. You want to be able to read the file data into one of two formats:

• A UInt8Array
• A base64 encoded string (which will then be converted to a UInt8Array)

Here is a skeleton of some code using expo-filesystem that you can adapt to your code. We do something similar in our bundleResourceIO model loader (though we use a different library for reading the file).

const fileUri = 'NON-HTTP-URI-GOES-HERE';      
const imgB64 = await FileSystem.readAsStringAsync(fileUri, {
    encoding: FileSystem.EncodingType.Base64,
});
const imgBuffer = tf.util.encodeString(imgB64, 'base64').buffer;
const raw = new Uint8Array(imgBuffer)  
const imageTensor = decodeJpeg(raw);

Give that a try and let me know if that works.

I'd also recommend that any one on 0.1.0-alpha.2 upgrade to 0.2.3.

[hayzem]

After I realized the error was caused by fetch I decided to try another method to load the image and as you suggested I ended up using expo-filesystem which loaded the local image successfully on my android device.

Then I faced another error SOI not found which is not related with this issue but related to decoding JPEG. Your example helped me to solve it by just using tf.util.encodeString.

It works as expected. Thanks.

I'd also recommend that any one on 0.1.0-alpha.2 upgrade to 0.2.3.

For the record I have not upgraded.

[daoshengmu]

const fileUri = 'NON-HTTP-URI-GOES-HERE';      
const imgB64 = await FileSystem.readAsStringAsync(fileUri, {
  encoding: FileSystem.EncodingType.Base64,
});
const imgBuffer = tf.util.encodeString(imgB64, 'base64').buffer;
const raw = new Uint8Array(imgBuffer)  
const imageTensor = decodeJpeg(raw);

It works for my Android devices too. Thanks for your response. Trying to use 0.2.3 instead although there are some dependency issues are happening.

[ryanvolum]

@tafsiri @daoshengmu, any idea why your proposed code would only work for images of specific orientations? It seems bizarre, but when running this with expo I can't classify landscape images. If I instead crop a portrait image, it works fine.

[tafsiri]

@ryanvolum when you say can't classify, do you get an error or just not the expected results. If the latter it could be that once you resize the image to the dimensions expected by the model the landscape images are more distorted than your portrait ones.

[ryanvolum]

@tafsiri thanks for the quick answer! I should have been more specific. I'm building a React Native app that classifies images from my camera roll. The app is currently using two models - mobilenet (trained on Imagenet data) and a COCOSSD object detection model. I'm testing my app using expo on my android device, and it keeps crashing outright when I run the image processing code above on _certain_ images.

My app is unfortunately crashing silently, so I'm left without clues to what's happening. I speculate, though, that the app is running out of memory. I can't imagine why turning a JPEG image to a tensor (or any of the preprocessing code to turn the image into a Uint8 array) would cause this kind of memory consumption, but I have no other explanation. What's really throwing me for a loop is that it works fine (albeit slow) with some images, but crashes with others.

I also realize this is a weird place to report this issue - I'm happy to open a new one, but am honestly not sure it's even a tfjs-react-native issue.

[ryanvolum]

@tafsiri Incidentally, I also can't seem to run the app in the Android emulator on my Windows machine. Here I do explicitly get OOM errors. I'm guessing this is because my PC can't emulate the GPU on my Android device, but would also be surprised if a GPU were really necessary for turning an image into a tensor.

[tafsiri]

@ryanvolum the GPU emulation is required if using the default backend in tfjs-react-native. If you want to use CPU, try calling tf.setBackend('cpu') near the top of the program, this should make it more likely to work in an emulator.

Try looking at https://reactnative.dev/docs/debugging#accessing-console-logs for tips on getting lower level error messages from react native to try and explain the cause of the crash.

[tafsiri]

Another idea. is there any chance that some of the images are not jpeg encoded?

[ryanvolum]

Another idea. is there any chance that some of the images are not jpeg encoded?

I'll check to be sure, but can't imagine why this wouldn't be the case as I'm using images straight from my camera roll, all taken by my phone's camera.

[ryanvolum]

@tafsiri thanks for the suggestions. I'm now running successfully in the emulator 🎊 I was actually able to accelerate my emulator by enabling it to use my PC's GPU, which made a world of difference.

I believe the OOM issues I'm continuing to run into are a product of image size. In the tfjs-react native sample it doesn't appear that you're doing any image resizing. Given that mobile phones take very high resolution photos, should I consider downscaling + resizing the photos I'm using before running inference on a mobilenet model? I see the tf.image.resizeBilinear helper, but am not super clear on whether or how to use it for this scenario.

[tafsiri]

@ryanvolum you should definitely downscale your images before running inference, typically the models are trained on relatively small images compared to typical camera resolutions (for example our mobilenet wrapper will internally resize inputs to 224x224). So finding a smaller size that will give you adequate performance will help performance. On the memory usage front, if you can resize the image before you even get it into react native, you will save having to allocate those large base64 encoded strings.

Another perforamance tip. If you are processing many images, it will be faster if they are all the same size (or use a small number of fixed dimensions), every time you pass a different sized image through your pipeline a bunch of texture initialization (and memory usage) will happen. If you use a small set of fixed sizes you only pay the overhead for each different size once.

[ryanvolum]

Thanks @tafsiri!

I think it would be hugely beneficial to include code (in samples/docs) that does this. I can't think of a mobile scenario where images are already downscaled, so anybody hoping to use tfjs-react-native will have to figure this out.

I think one of the great thing about tfjs is that it democratizes data science to a group of developers (web) who wouldn't otherwise have the skills to take advantage of ML models. That said, that group still needs some extra hand-holding to push them over the finish line. To make matters more difficult, the JS/Node community doesn't have the tooling/docs to determine how best to do things like properly downscaling an image. Docs and tooling for doing this in Python, on the other hand, are plentiful.

Just my two cents - thanks for all the prompt responses!

[tafsiri]

Thanks for your perspective! In this case we do have tools for downscaling (for example the tensor camera component does this on the GPU) and we have the various resize* functions for downscaling as well (https://js.tensorflow.org/api/latest/#image.resizeBilinear, we also have a bunch of examples that use this). Though this latter case requires a tensor to already be created.

In this case though my other suggestions are more about react native specific issues around loading images without consuming a bunch of memory, if there was a way to do this without forcing serialization to strings that would be great (but RN seems to limit this). Or if it were done natively before tfjs tries to load it that would also be greta. However these issues are pretty much react native specific so from my perspective it would be great to have solutions developed/proposed by folks who well integrated into the React Native ecosystem. We are hoping to get more of this perspective via the newly launched tfjs SIG (check it out in case this is something that interests you!).

[joaorenatojr87]

Digging into this it looks like fetch is not a good fit for loading these types of files. The api our custom fetch is trying to implement leaves behavior for file:// scheme urls undefined. I probably wouldn't rely on on it for this even where it seems to work. Instead I would use an existing file reading mechanism in react native to get the data. You want to be able to read the file data into one of two formats:

• A UInt8Array
• A base64 encoded string (which will then be converted to a UInt8Array)

Here is a skeleton of some code using expo-filesystem that you can adapt to your code. We do something similar in our bundleResourceIO model loader (though we use a different library for reading the file).

const fileUri = 'NON-HTTP-URI-GOES-HERE';      
const imgB64 = await FileSystem.readAsStringAsync(fileUri, {
  encoding: FileSystem.EncodingType.Base64,
});
const imgBuffer = tf.util.encodeString(imgB64, 'base64').buffer;
const raw = new Uint8Array(imgBuffer)  
const imageTensor = decodeJpeg(raw);

Give that a try and let me know if that works.

I'd also recommend that any one on 0.1.0-alpha.2 upgrade to 0.2.3.

Thanks so much for showing this approach. Worked well for me!