React-native: Blobs

[philikon]

Here at Silk Labs we've had a Blob implementation for React Native for a while. We want to contribute it back upstream. This issue discusses the API and implementation and tracks progress.

Motivation

Like in the browser, Blobs are opaque lumps of (usually binary) data that can be read into TypedArrays or strings, but don't have to be. Concretely in React Native that means that the binary data lives in native and not in JS, unless you explicitly request it. Why is this useful? Say you're receiving some data via XHR, WebSocket, or WebRTC, or some custom method that you've implemented and you want to display it in an <Image> view. Or you want to send it along through some other means without looking paying the cost of transferring it to the JS thread and back.

We already have something somewhat similar to this for images on iOS (RCTImageStoreManager). This would be a generalization of that.

API

The web has an API for Blobs. It's not particularly pretty or anything, but it's kind of a standard, so I think we should use that, or at least a pretty good subset.

Creating

Blobs typically get created when JS is receiving data, e.g. via XHR, WebSocket, etc. I am planning on making

xhr.responseType = 'blob';

and

websocket.binaryType = 'blob';

work so that when you specify those settings, the XHR and WebSocket responses will return Blob objects and not transfer the data to the JS thread. This works exactly like on the web.

Mutating

A subset of mutation operations will be supported:

let combinedBlob = new Blob([existingBlob, anotherBlob]);
let halfBlob = blob.slice(0, blob.size / 2);

I'm not planning on supporting creating a blob in JS from JS data (e.g. a string or TypedArray), though it should be easily implementable.

Consuming

Like in the browser, you'll be able to map Blobs to URIs that you can then use in things that eat URIs, e.g. the view:

let url = URL.createObjectURL(blob);

Also like in the browser, you'll be able to read the blob into a TypedArray or string. I am planning on simply implementing the web's FileReader API.

Freeing

The following method is part of the Blob standard, but is currently not implemented by browsers. However, because we can't automatically garbage collect our blobs like browsers can, it will be necessary to provide a manual way of freeing blobs, so that consumers can make use of it when appropriate:

blob.close();

Here's a complete example of how consuming blobs would work: https://gist.github.com/philikon/4592efd153673f3e64dec353046af7a8

Implementation

Proposal

React Native already has a loose concept of a "blob", in other words, a lump of data that's not directly accessible by JS. The iOS (RCTImageStoreManager) mentioned above is an example.

XHR and most notably the <Image> tag already have a way to express a reference to a blob:

{uri: "some://uri/that/gets/resolved"}

How does some://uri/that/gets/resolved get resolved? On iOS, it gets resolved through a RCTURLRequestHandler. On Android, it gets resolved through a ContentProvider.

For instance, you can upload binary data provided by a system resource in the following way:

const xhr = new XMLHttpRequest();
xhr.open(...);
xhr.send({uri: "some://big/blob/of/binary/data"})

As long as some://big/blob/of/binary/data is resolvable by a RCTURLRequestHandler or ContentProvider, this will Just Work(tm) already.

On this basis, I'm proposing to implement Blobs as simply another way of URI-addressable binary data. Our implementation does exactly that.

Progress

Prep work:

• [x] Make XMLHttpRequest and XMLHttpRequest.upload proper EventTargets so that we can receive progress events even when we're not receiving data to JS, because we're receiving it into a blob (#7017)
• [x] Add responseType as a concept to RCTNetworking, so that we can add 'blob' later (#8324)

Implementation work, most likely by porting silk-react-native-blobs:

• [ ] Provide native blob storage with NativeModule API, and JS implementation of Blob class
• [ ] Provide URI integration (RCTURLRequestHandler on iOS, ContentProvider on Android)

Future implementation work:

• [ ] XHR integration: allow xhr.responseType = 'blob' and xhr.send(blob)
• [ ] WebSocket integration: allow websocket.binaryType = 'blob' and websocket.send(blob)
• [ ] Implement FileReader API for reading blobs into JS e.g. as TypedArrays. This can just be a thin wrapper around XHR, yay!

[ide]

Can you coordinate with @satya164, he has a Blob implementation almost ready I believe.

[philikon]

Yep, that's the reason for this issue :), so that we can coordinate. I believe he's uplifting our Blob implementation.

[satya164]

Hey! Yeah, so I've been working to port changes from @philikon's implementation to RN master.

I've only worked on Android part so far.

My focus was to first get it working with firebase so people could upload files to firebase. This has slightly different requirements since we need to create a blob locally from a file before uploading.

So I think in addition to the proposed APIs, we need two more non-standard methods,

Blob.fromURI(uri: string, options: { type: string }): Promise<Blob>
File.fromURI(uri: string, options: { type: string }): Promise<File>

I just copied @philikon's implementation for the blob module and with minor changes, and writing some code to support XHR I've been able to get the following working,

• [x] Create blobs when data is received via XHR
• [x] Create blobs from a local file URI
• [x] Upload blobs to a server (e.g. - firebase) via XHR

I couldn't get URL.createObjectURL working from the current implementation, looking into it right now.

I've not yet tested the WebSocket implementation yet, but since that was already implemented I guess shouldn't take much time to get it working.

So things I need to do are,

• [x] Get content provider working
• [x] Get the WebSocket implementation working

What I'd like to explore is, right now when you create a blob from URI, I read it to memory as bytes. But we could probably just store the uri and read the bytes when needed (we'd need get the bytes length though, so we'd need to read it twice!)

I also want to know if there's a full test suite available to test my implementation against.

[philikon]

So I think in addition to the proposed APIs, we need two more non-standard methods,

Blob.fromURI(uri: string, options: { type: string }): Promise
File.fromURI(uri: string, options: { type: string }): Promise

• I don't understand the need for File. What use cases does File solve that aren't covered by Blob?
• What specific use cases does Blob.fromURI(...) solve? If I have a URI, why wouldn't I make an informal {uri: ...} blob object? If there's an RCTURLRequestHandler / ContentProvider for the URI in question, it'll already work with XHR! The only thing that we gain with Blob support is concatenation and slicing, but as you say, that's really hard to do if we don't at least know the size of each part. I'm not familiar with Firebase, but I'd like to hear more about why a generic URI-based Blob is a good idea. For now it seems pretty hard to implement Blob semantics around any lump of data that's not already stored in memory. Maybe a different API that also happens to expose uri for XHR-compatibility would be better suited for Firebase integration?
• It also sounds like you might be thinking about this backwards. If people want to create blobs from some sort of datasource, the datasource should return Blob objects, e.g. someFirebaseAPI.get(options).then((blob) => ...). Or would Blob.fromURI() be an API that's meant only for React Native modules, much like the non-standard Blob.create() I introduced?

[satya164]

I don't understand the need for File. What use cases does File solve that aren't covered by Blob

File implements the File API from the spec. It's just a thin wrapper around blob.

What specific use cases does Blob.fromURI(...) solve? If I have a URI, why wouldn't I make an informal {uri: ...} blob object? If there's an RCTURLRequestHandler / ContentProvider for the URI in question, it'll already work with XHR! The only thing that we gain with Blob support is concatenation and slicing, but as you say, that's really hard to do if we don't at least know the size of each part. I'm not familiar with Firebase, but I'd like to hear more about why a generic URI-based Blob is a good idea. For now it seems pretty hard to implement Blob semantics around any lump of data that's not already stored in memory. Maybe a different API that also happens to expose uri for XHR-compatibility would be better suited for Firebase integration?

Libraries like firebase expect a blob or file object for uploading. We cannot pass a { uri: string } object to firebase. So allowing to create a file/blob object from an URI is the only way people can use firebase for now. I don't like it much, but I don't think there are any options.

Firebase API looks like this,

const ref = firebase.storage().ref().child('image.jpg');
const snapshot = await ref.put(blob);

It also sounds like you might be thinking about this backwards. If people want to create blobs from some sort of datasource, the datasource should return Blob objects, e.g. someFirebaseAPI.get(options).then((blob) => ...). Or would Blob.fromURI() be an API that's meant only for React Native modules, much like the non-standard Blob.create() I introduced?

This is so that we'll be able to pass blob objects to libraries which don't accept uris. I think in general this should be discouraged though, and libraries can internally use this API if they need.

[philikon]

File implements the File API from the spec. It's just a thin wrapper around blob.

Meh ok. Seems unnecessary, unless there's demand from 3rd party libraries out there. (Perhaps Firebase?)

Libraries like firebase expect a blob or file object for uploading.

Ok, so then just do

const response = await fetch(uri);
const blob = await response.blob();
someFirebaseAPI.consume(blob);

Or in XHR spelling:

const xhr = new XMLHttpRequest();
xhr.open('GET', uri);
xhr.responseType = 'blob';
xhr.onload = () => {
  const blob = xhr.response;
  someFirebaseAPI.consume(blob);
};
xhr.send();

Either variant will load the data from the URI into a Blob that can then be passed to Firebase. Though, I still don't quite understand what Firebase is going to do with it? Upload it to the server?

[satya164]

Meh ok. Seems unnecessary

It's very tiny amount code to polyfill a web API, which makes sure libs which need File constructor will work.

Either variant will load the data from the URI into a Blob that can then be passed to Firebase.

I didn't know I can do fetch from local URI. Or are you proposing to implement this API?

Though, I still don't quite understand what Firebase is going to do with it? Upload it to the server?

Yes

[philikon]

I didn't know I can do fetch from local URI.

Yep, should already work!

(* Definitely on iOS. No idea about Android. If XHR can't currently download from content:// URIs, then we should add that support.)

[philikon]

It's very tiny amount code to polyfill a web API, which makes sure libs which need File constructor will work.

Sure, ok.

[brentvatne]

It also sounds like you might be thinking about this backwards. If people want to create blobs from some sort of datasource, the datasource should return Blob objects, e.g. someFirebaseAPI.get(options).then((blob) => ...). Or would Blob.fromURI() be an API that's meant only for React Native modules, much like the non-standard Blob.create() I introduced?

I had the same thought, the issue that @satya164 pointed out to me is that if the datasource API (let's say ImagePicker which lets you pick an image from your camera roll or take a photo) returns a Blob every time then people who are not interested in the Blob but rather in other other values like the URI might just forget to close the Blob. We could make it an optional parameter on those APIs:

let imageBlob = ImagePicker.launchImageLibraryAsync({
  allowsEditing: false,
  aspect: [1, 2],
  responseType: 'blob',
})

Thoughts @satya164 @philikon?

(the ImagePicker I refer to here is the one in Exponent - see docs if you're curious)

[ide]

Another approach would be to write a separate module that creates a Blob from a URI -- BlobLoader.loadFromURIAsync(uri). This would keep the function out of the Blob/FileSystem namespace but still let you work with any API that gives you back a local URI.

That said, one thing I like about libraries that return Blobs (like Brent's ImagePicker example) is that you either get back a valid Blob (assuming it's eagerly loaded) or the call fails. There's no uncertain state halfway where you get back a valid URI but then the file on disk gets deleted before you load the Blob from the URI.

So we might want both -- Blob-aware APIs that give you back Blobs in a more atomic way, and a generic URI-to-Blob loader that works with any URI-aware API.

[satya164]

Another approach would be to write a separate module that creates a Blob from a URI

Separate module for creating blobs from URI sounds good to me

We could make it an optional parameter on those APIs

That could work, it only saves one line of code of code though (not really since you've to specify responseType: 'blob' anyways).

While the blob-aware APIs seem nice to me in a general sense, in case of RN I'm not super happy about them since the user always has to manually free the blob. . One thing I wanted to explore is to lazy-load URI based blobs. But this negates the case @ide mentioned about not having to deal with uncertain state.

[hramos]

Closing this issue because it has been inactive for a while. If you think it should still be opened let us know why.

[austinksmith]

@hramos Please reopen , blob support is very critical to my current issue. https://github.com/facebook/react-native/issues/16034#issuecomment-333385849

[satya164]

Theres no point in re-opening. There's already a WIP PR implementing it.

[austinksmith]

Can you post that here and on the ticket I've opened? ETA on completion etc.?

[satya164]

There's no ETA. https://github.com/facebook/react-native/pull/11573

[austinksmith]

So how can we get this issue prioritized, it appears that PR has been in limbo for almost a year

[satya164]

Only way is to help me with the iOS code and then wait for review.

[austinksmith]

@satya164 What is needed on IOS side?

[pradeep250677]

@satya164 could you share me with Andriod code? Binary data communication API between JSC and Native using RN bridge is very important. I am the only concern for Andriod.

[satya164]

@pradeep250677 you can check the pr linked above.