Uppy: Create a working Transloadit Uppy integration

[kvz]

Renee will take the lead here until Artur has time to help out.

For reference https://github.com/transloadit/uppy/pull/177

We should:

• [x] First, have an Uppy call with @ifedapoolarewaju @goto-bus-stop @arturi @tim-kos @Acconut @kvz to kick things off
• [x] First step: create a Transloadit plugin that makes a request to api2 and received an assembly_id, see https://github.com/tus/tusd/wiki/Uploading-to-Transloadit-using-tus#creating-a-resumable-assembly for details (@goto-bus-stop)
• [x] Add ability to handle / bubble up errors to the user (maybe behind a "more details" button) (maybe not part of the 177 PR)
• [x] Tim/Kev add links to sites that use the jquery-sdk
• [ ] A better "all done" screen (<-- can happen before encoding if waitOnEncoding: false + waitOnMetaData: true or requireUploadMetaData)
• [x] Encoding progress updates in the Dashboard status bar
• [x] Can we add params
• [x] ^ can we use the https://github.com/transloadit/uppy/blob/master/src/plugins/MetaData.js plugin for this?
• [x] Can we add signature
• [x] Can we add form fields
• [x] Add one (or three?) Uppy+Transloadit demos to https://transloadit.com/
• [x] Add links to the encoding results(?)

[goto-bus-stop]

• Add ability to handle / bubble up errors to the user (maybe behind a "more details" button) (maybe not part of the 177 PR)

I think we need to do this throughout Uppy at some point, also thinking about how to deal with some files, but not all, failing to upload, and things like retrying failed uploads.

• Can we add params <-- can we use the https://github.com/transloadit/uppy/blob/master/src/plugins/MetaData.js plugin for this?

Should metadata be related to params, or to fields?

I also added another point:

• Encoding progress updates in the Dashboard status bar

(Since right now it just says 🔄 Encoding, but we receive events for every assembly result from the socket so we can be a bit more specific there.)

[kvz]

I think we need to do this throughout Uppy at some point, also thinking about how to deal with some files, but not all, failing to upload, and things like retrying failed uploads.

yes, likely a @todo suffices for the PR in question, but since this issue tracks what would be required to have a transloadit+uppy integration, let's keep the sub-issue here open, as we'll want the end users to be able to communicate the actual errors to us should they happen, so we have a better chance at 0 errors, later : )

Should metadata be related to params, or to fields?

I guess both 😱

so we can be a bit more specific there

Yes, very good!

[kvz]

Gonna strip barely from this issue since we got that now 😄

[oyeanuj]

@kvz @goto-bus-stop Great to see this happen! Wondering a couple of things -

1. When is this being planned to release? I noticed that the Changelog says the next release was March 31st but I don't see a release then.

2. And at the time of the next release, what parts or features with the integration would be working and what would not be possible to do?

Just want to get a sense to plan my own work.

Thank you!

[kvz]

Thanks for taking an interest @oyeanuj!

When is this being planned to release? I noticed that the Changelog says the next release was March 31st but I don't see a release then.

We're currently doing a big internal push with the team, this has hurt our Uppy release schedule, we're going to pick things up in May again. But luckily, we aim to have a working Transloadit integration sooner than that - be it subject to change & (quite some planned) improvements still. For starters we want to demo uppy on the transloadit.com homepage, so templates, good ui feedback, etc will have to be working

And at the time of the next release, what parts or features with the integration would be working and what would not be possible to do?

For this i refer to the subtodos in this issue. If we complete those, i'd say 90% of use cases of people currently using the jquery-sdk should be covered.

[oyeanuj]

@kvz Thanks for the detailed update, just a couple of quick follow ups -

(quite some planned) improvements still... 90% of use cases of people currently using the jquery-sdk should be covered.

1. Could you give me a sense of what those 10% or planned improvements might cover? The list in subtodos above are fairly technical and it is hard for me to translate those to functionality (since I am not as familiar with the library).

2. If I have to use Transloadit on a new project (React), is there some Uppy branch that you'd recommend I use to get started? Or wait for a few days, since it is only a few more days to May (or even use the existing Transloadit API or react-transloadit)?

Any tips are much appreciated 😄

[kvz]

Thanks @oyeanuj, I suggest waiting a few more days until we can get some pending work merged and have an Uppy demo on the transloadit.com website. We'll have a React project of our own that we'll want to add Uppy to next, so it seems we will be laying down a lot of groundwork right in front of you. We'd love to have you as a beta-tester and keep you informed of the latest&greatest if you'd like that? I think everything that uppy & transloadit can do will work, with the exception of:

• (maybe) form fields
• pulling files from external sources like dropbox/gdrive/etc. this requires us to run uppy-server in production, and quite some more work is involved with that still, so we'll soft-launch without support for those sources initially

Does this help?

[oyeanuj]

@kvz That was perfect, thank you! And I'd be happy to be a beta-tester, please keep me posted (or open an issue) with the React project integration. I'll try to follow alongside and integrate it in my project as well!

[kvz]

Hey @oyeanuj just wanted to let you know that we shipped a first version a few minutes ago!
https://transloadit.com/#uppy-demo

This was roughly the code that was used: https://gist.github.com/kvz/8ae07aa8c063c8e55abbc1580b50c8a4 (we still need to brush this up, but i had the feeling you'd prefer _early_ over _perfect_)

Please note that everything might still change, even in minor/patch version bumps. So you probably want to pin versions tightly, to ensure things stay working as they do now for you.

[oyeanuj]

@kvz Thank you so much for the update, I'm going to dig in to this more this weekend!

_PS. I just noticed that from your link, in the text next to the demo, there are links to the 'Demo pages'. All of those links still show NodeJS code or JQuery but not Uppy (which is what I expected as a user). Not sure if that was intentional but thought I'd bring it up in case!_

[kvz]

PS. I just noticed that from your link, in the text next to the demo, there are links to the 'Demo pages'. All of those links still show NodeJS code or JQuery but not Uppy (which is what I expected as a user). Not sure if that was intentional but thought I'd bring it up in case!

Yes thanks! We have about 50 demos online that still use our jquery-sdk indeed, we'll be gradually porting them over, but the truth is, we wanted to first launch 1, get some feedback, hammer out issues, before going big-bang 💥 🌍 on this :)

[oyeanuj]

@kvz Thank you for all your helpfulness so far.

I had a look, and I am looking forward to playing with uppy and transloadit together. I had a few quick questions if you have any thoughts on them (happy to open another ticket if this is too off-topic)

1. Upload to Transloadit: If one needs processing for images, video, etc, I imagine the recommended way would be to upload to Transloadit which exports to S3? I imagine there is no need for uppy-server when uploading directly to Transloadit?

2. Post-processed URL: Since Uppy has a socket connection, will Transloadit inform when the encoding and processing assembly is done, and what the new url is? And if yes, how does one tell Uppy where or how to send that new url to an endpoint?

3. Tus vs Multipart: I notice in the API that you've to choose between one of them. Since tus is the main reason behind this, I imagine one would always choose this. Is there a good reason/usecase to opt for Multipart?

4. Upload to S3 directly: In the Readme, it suggests that an S3 adapter is coming soon. Any updates on that (since for the files not need processing, I was planning to upload directly to S3)?Also, would S3's Multipart be compatible with Tus?

5. Use without UI: I posted my question in this thread.

Thank you again!

[kvz]

Upload to Transloadit: If one needs processing for images, video, etc, I imagine the recommended way would be to upload to Transloadit which exports to S3? I imagine there is no need for uppy-server when uploading directly to Transloadit?

Yes, Transloadit will be running uppy-servers for our customers, to enable googledrive/dropbox etc sources. We already run the tusd servers required to handle uploads.

Post-processed URL: Since Uppy has a socket connection, will Transloadit inform when the encoding and processing assembly is done, and what the new url is? And if yes, how does one tell Uppy where or how to send that new url to an endpoint?

Yes Uppy already is aware of results. In addition, you could also enable notify_urls that Transloadit will ping with the JSON payload when an Assembly is done.

Tus vs Multipart: I notice in the API that you've to choose between one of them. Since tus is the main reason behind this, I imagine one would always choose this. Is there a good reason/usecase to opt for Multipart?

If you don't want to use Transloadit, and you don't have any tusd servers running, you may want to re-use your legacy apache2 webserver that is not tus-capable. For these circumstances, you can use the Multipart plugin.

Upload to S3 directly: In the Readme, it suggests that an S3 adapter is coming soon. Any updates on that (since for the files not need processing, I was planning to upload directly to S3)?Also, would S3's Multipart be compatible with Tus?

Direct S3 uploads would be instead of tus, 'soon' is likely not very soon though, as we have some other priorities first.

Use without UI: I posted my question in this thread.

I'll leave this to the team there (and team, also if i made a mistake here feel free to correct me)

[hpvd]

very nice :-)
smoooth experience...
was trying demo "cropping images"

Only thing still missing from users pov is some kind of file restriction (type, size, number):
I was able to upload a 94MB .exe
-> with no message at start or even at the end that cropping of this file type was not possible...

it's already in the Backlog section in https://github.com/transloadit/uppy/blob/master/CHANGELOG.md
and discussed in some issues....

[oyeanuj]

@kvz Thanks Kevin for the answer. I'll wait on @goto-bus-stop response to see how much can be used without the UI. Quick follow-up though on S3:

• Once you do have the wrapper, do you expect it to be compatible with chunking and resumability that S3 provides, similar to tus? Or are they incompatible protocols?

[kvz]

If/when we are going to make direct s3 uploads available, it will have as much resumability properties as we can offer. iirc that means only in 5mb chunks, so for small files the resumability would not work.

[arturi]

@hpvd

Only thing still missing from users pov is some kind of file restriction

Yes, good catch, we’ll look into adding that soon. You are persistent ;)

[Acconut]

Here is another feature which would be needed to replace the jQuery SDK: https://github.com/transloadit/uppy/issues/200

[kvz]

🎉 🎉 ❤️ 🎉 🎉