React-native-fs: Enable read and write streams

[petuomin]

It would be useful for some cases to be able to stream the data into a file / from a file.

[mhollweck]

+1

[johanneslumpe]

@cjdell do you have any thoughts on this?

[cjdell]

@johanneslumpe I believe this is an important feature to have, and maybe useful to mitigate the performance penalty of traversing over the native bridge. I think taking inspiration from Node's fs.createReadStream and fs.createWriteStream would be a good idea, however we would likely have to settle for a subset of Node's API.

It will take me quite a bit of time to put this together and I have a busy schedule so I can't make any promises of an ETA just yet.

[timsuchanek]

@cjdell any updates on this? If you have some beginning thoughts / approaches I really would like to work on such a feature for android.

[lagondo]

+1

[faceyspacey]

+1

@cjdell have you had any time to look at this?

I could really use this to upgrade my youtube video uploader to a resumable one. Youtube has a resumable upload API:

https://developers.google.com/youtube/v3/guides/using_resumable_upload_protocol

And there's a node package that does this, but requires createReadStream to operate:

https://github.com/grayleonard/node-youtube-resumable-upload/issues/11

I've been hoping to rewrite that package to make use of react-native-fs that supports createReadStream.

To solve my particularly problem, I generally need to be able to specify a starting point in bytes (though won't be as efficient for large files), whether that relies on createReadStream or not. Perhaps someone has another idea of how to accomplish this?

[adamkdean]

I too require createReadStream, for what it's worth. :(

[rngadam]

Another example where this would be useful:

Decrypting a file where I'm relying on react-native-crypto to read an encrypted file (encrypted with a Java utility) and decrypt as per this function:

function decrypt(key, input, output) {
  var decrypt = crypto.createDecipher('aes-128-ecb', key)

  var r = fs.createReadStream(input);
  var w = fs.createWriteStream(output);

  r.pipe(decrypt).pipe(w)
}

having this would be great:

const RNFS = require('react-native-fs');
[...]
  var r = RNFS.createReadStream(input);
  var w = RNFS.createWriteStream(output);
[...]

[kevinresol]

I have written a pretty low-level read write library, on which a streaming API can be built.
https://github.com/kevinresol/react-native-fs-stream

It is pretty much proof-of-concept only. Feel free to take to code and integrate to this library.

[MarkMurphy]

I'm looking to have support for this so that I can implement the TUS resumable upload protocol. We're building a client that will upload large video files so being able to upload them in chunks with the ability to start/restart at any offset would be ideal.

The api that node offeres for createReadStream has all the features I'm looking for:

fs.createReadStream(path[, options])

options can include start and end values to read a range of bytes from the file instead of the entire file. Both start and end are inclusive and start counting at 0. If fd is specified and start is omitted or undefined, fs.createReadStream() reads sequentially from the current file position. The encoding can be any one of those accepted by Buffer.

[itinance]

We have a similar scenario in the current project where large videos should be downloaded from S3/Cloudfront and uploaded to AWS in chunks and with resume mode depending on network connectivity.
I think here about to make this complete in native without to handle these chunks in javascript-land. I think about a background task (thread) that is doing the whole work and informs the RN-land (i mean the javascript-layer) with a progress-event (progress in generell, event when stopping because of lost connectivity, event when resuming the download/upload and so on...) So then no base64 encoding/decoding would be necessary any more, particularly as the files are stored on device as binary and the video-player will play them as binary as they are ;)

Is there a special reason why you would do the handling of chunks on the javascript-layer, @MarkMurphy?

[MarkMurphy]

Is there a special reason why you would do the handling of chunks on the javascript-layer, @MarkMurphy?

I doesn't have to but then your native layer goes from just being a filesystem api to being a file system + network api which is what we're seeing a lot of these libraries becoming. There's an intermixing of concerns here.

and in general, I just want to write javascript 😄 and reuse stuff I've already written for browser projects.

[itinance]

In order to fully understand, i will draw a picture of the process to see, if we both discuss the same scenario: :)

1. user is recording a video with his smartphone
2. the video-recording-app (wether it is the default system-application or a specific one) will save the file on the device's drive
3. RN-developer wants to upload the file to a server, even if it is very large, even if the connectivity is slow or sometimes broken. The maingoal is to get this video up to the backend server or some cloudstorage.

I think you have another point that i am not aware of yet, when you are saying "reuse stuff i've already written for browser projects". What exactly is this what you want to reuse from browser-projects?

[MarkMurphy]

@itinance I'm see my initial comment about implementing the TUS protocol.

[faceyspacey]

@MarkMurphy does the TUS protocol support youtube uploads?

[JamesMGreene]

You could also get this functionality from the "react-native-fetch-blob" module.

[eykrehbein]

Any update on this?

[Leeway213]

have you tried to use 'ascii' to encode you binary data. and write the ascii string to file.

Here is a sample in node runtime:

const buffer = fs.readFileSync('./test.png');

for (let i = 0; i < buffer.length; i++) {
  const num = buffer[i];
  const str = String.fromCharCode(num);
  fs.appendFileSync('./test1.png', str, { encoding: 'ascii' });
}

[AppleMango23]

guys i found this but i have no idea how to put this into my code which is :

RNFetchBlob.fs.writeStream(
PATH_TO_FILE,
// encoding, should be one of base64, utf8, ascii
'utf8',
// should data append to existing content ?
true
)
.then(ofstream => ofstream.write('foo'))
.then(ofstream => ofstream.write('bar'))
.then(ofstream => ofstream.write('foobar'))
.then(ofstream => ofstream.close())
.catch(console.error)
>

which is this one.. do u guys know how?
wfs .createReadStream("/dir/somefile.dat") .pipe(fs.createWriteStream("./somefile.dat"));

[AppleMango23]

You could also get this functionality from the "react-native-fetch-blob" module.

I am using that but i do not know how to use it into my code, can u help me out?

[toddmitchell]

@AppleMango23

I'm not sure if you've solved this issue but here's what I did. (based on info from this: https://stackoverflow.com/questions/37104199/how-to-await-for-a-callback-to-return)

async encryptStream(sourcePath: string, destinationPath: string): Promise {

  const buffer_size: number = 4095;

  let outputStream: any;
  await RNFetchBlob.fs.writeStream(destinationPath, 'base64', true).then(os => {
      outputStream = os;
      console.log('output stream created')
  });

  return new Promise<IMediaItem>((resolve, reject) => {

      RNFetchBlob.fs.readStream(sourcePath, 'base64', buffer_size).then((streamReader) => {

          streamReader.open();

          streamReader.onData((chunk) => {
              //console.log('wrote chunk', chunk);

              //TODO: perform encryption here
              outputStream.write(chunk.toString());
          })

          streamReader.onError((err) => {
              reject(err);
          })

          streamReader.onEnd(() => {
              outputStream.close();

              resolve(result);
          });
      });
  })

}

Usage:

await this.encryptStream(sourcePath, destinationPath);

[CozmaCatalin]

+1

[ArsalImam]

I need an alternative to read/write streams

[AngeloLanch]

+1