Uppy: Question: Is there a way to pre-process uploads? Encryption

[ThisIsMissEm]

Hi!

I've a use case where before upload I want to encrypt the file on the client; Is there a way I can hook into uppy to intercept the file between it being added and it being uploaded?

I can see onBeforeFileAdded, but I'm not sure how I'd replace the added file?

[arturi]

Hi! Yeah, you could use that, but if you set autoProceed: false in Uppy options, then it won’t start the upload automatically, so you should be able to do this:

uppy.on('core:file-added', (newFile) => {
  const updatedFiles = Object.assign({}, this.state.files)
  const file = Object.assign({}, newFile)
  const encryptedFile = encrypt(file)
  updatedFiles[encryptedFile.id] = encryptedFile
  uppy.setState(files: updatedFiles)
})

Something like that, so that the file replacement is immutable. Same would work with onBeforeFileAdded if you want to start upload immediately after file is added, and it’s probably a better way. Then you can do all of the above at first, then return Promise.resolve().

We could maybe change it so that onBeforeFileAdded returns an updated file in Promise.resolve(), what do you think @goto-bus-stop?

[ThisIsMissEm]

@arturi thanks for the quick reply!

The whole returning an empty resolved promise does look a little strange. Generally I see code which is more input/output driven, where promises are just a future, rather than a signal to proceed.

Something like:

onBeforeFileAdded(currentFile, files) {
  const encryptedFileData = encrypt(file.blob) // not sure if this is right?
  Promise.resolve({ ...file, blob: encryptedFileData })
}

Would make a lot more sense. I also note that the documentation is missing information about the schema of arguments; e.g., what shape is newFile or currentFile, what properties or methods does it have?

[arturi]

The whole returning an empty resolved promise does look a little strange

The original purpose of this method was to implement your own restrictions, so you can run a check and if a file doesn’t pass, you reject, and if does, you resolve. That was the thinking behind it. I like you suggestion, we’ll discuss and likely implement it.

what shape is newFile

Agreed, we should add that to docs. Shape of file is something like this: https://github.com/transloadit/uppy/blob/78c9be52f864d1908335355720613d64843cddba/src/core/Core.js#L321-L343. You can also check in dev tools by doing uppy.state.files if you’ve set debug: true.

Thanks for your suggestions! PRs are always welcome too ;)

[ThisIsMissEm]

Okay, cool. So, shortly I'll be building a uppy plugin for doing public-key encryption of files before they're uploaded using libsodium's Sealed Boxes; In my implementation, I need to receive a File object, and then my plugin encrypts it, before handing it back to uppy — there is some asynchronicity here, which hopefully won't be a problem?

You can see the rough code for this over at: https://www.dropbox.com/s/x5a2txuqymb076b/secure-transfer.html?dl=0

I'll be open-sourcing this plugin, too. End-to-end encryption for everyone!

[goto-bus-stop]

I need to receive a File object, and then my plugin encrypts it, before handing it back to uppy — there is some asynchronicity here, which hopefully won't be a problem?

This is going to be a problem with onBeforeFileAdded, because it runs before Uppy does _anything_ else with the file (eg. before showing it in the UI). If the file data changes inside onBeforeFileAdded, uppy can't generate previews etc, and if onBeforeFileAdded does a lot of work, it will take a while for the file to show up in the UI.

A good place for something like encryption would be a custom plugin that adds a preprocessor hook. Internally when uploading a file, Uppy passes it through three steps that can be hooked into by plugins: preprocessing, uploading, and postprocessing. This isn't documented right now (neither is anything related to custom plugins), but we'll hopefully get to that sooner rather than later :eyes:

A custom plugin defines an install method, which can use the this.core.addPreProcessor method to hook into the preprocessing pipeline. A preprocessor is a function that receives the file IDs that are being uploaded, and returns a Promise that resolves when the preprocessing work is finished. (Like in onBeforeFileAdded, the Promise resolution value is ignored, it's only used for sequencing.)

For example:

function encrypt (blob) { /* etc */ }

class Encryption extends Plugin {
  constructor (uppy, options = {}) {
    // some boilerplate
    super(uppy, options)
    this.id = options.id || 'encryption'
    this.type = 'modifier' // value doesn't really matter, we may make this optional later

    this.prepareUpload = this.prepareUpload.bind(this)
  }

  prepareUpload (fileIDs) {
    const promises = fileIDs.map((fileID) => {
      const data = this.core.getFile(fileID).data
      return encrypt(data).then((encrypted) => {
        const files = this.core.getState().files
        this.core.setState({
          files: Object.assign({}, files, {
            [fileID]: Object.assign({}, files[fileID], { data: encrypted })
          })
        })
      })
    })
    return Promise.all(promises)
  }

  install () {
    this.core.addPreProcessor(this.prepareUpload)
  }
}

(We should add a core.setFileState(fileID, data) method to remove some of the setState boilerplate there!)

That would be used as

uppy.use(Encryption, { /* options */ })

Hope that makes sense! An E2E encryption plugin would be super sweet :D

[kvz]

Quite excited about the idea of an E2E encryption plugin. How would everybody in this thread feel about submitting that as a PR against Uppy itself so that it would be more visible and scrutinised by others, as well as maintained under our shared roof?

[ThisIsMissEm]

Hi @kvz, I’ll actually be releasing this as an open-source plugin, but under @unobvious-technology, as there’s additional parts around it. (Working name is libconfidant)

It’ll be three parts: encryption/decryption library, uppy plugin, react.ja viewer.

[ThisIsMissEm]

Oh, there’s also the bin script to generate a keypair for encryption (you can’t just use an ssh keypair)

[ThisIsMissEm]

@goto-bus-stop thanks so much for that example code! Definitely looks like the better integration point.

[stoopman]

Hi @ThisIsMissEm, I'm curious if you ended up with a working solution for the client side encryption.

[ThisIsMissEm]

Hiya Rick,

I came up with something at a conceptual level, and can explain it all to you, but I never wrote all the code for it. (I became distracted by other things, and my motivations and ideas shifted)

More than happy to do up a document explaining how this would work. Perhaps you'd like to commission it?

Yours,
Emelia

On 4. Jan 2019, at 13:25, Rick Stoopman [e-mail] wrote:

Hi @ThisIsMissEm, I'm curious if you ended up with a working solution for the client side encryption.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[stoopman]

Hi Emelia,
That would be great! I'm currently working on a project which uses the AWS
S3 MultiPart upload integration in Uppy, but it also requires encryption of
the upload chunks in the browser.

I'll be happy to implement your design and share the results once it's
finished :)

Cheers,
Rick

[kvz]

Hi there, closing this due to inactivity. If you'd still like to share your work we'd be more than happy to look/link at it! (even if it's just a WIP!)

[JStumpp]

@stoopman any progress on this? We also need AWS S3 Multipart upload with encryption.

Thanks,

Jürgen

[stoopman]

@JStumpp no not yet. @ThisIsMissEm are you still up for documenting your approach? Thanks!

[ThisIsMissEm]

@JStumpp @stoopman It's very far removed from my current work — but you can find the general implementation of using libsodium here: https://www.dropbox.com/s/x5a2txuqymb076b/secure-transfer.html?dl=0