Meteor-files: Sugessions for S3 / Upload Failed File / re-upload process

Created on 29 Aug 2020 · 5Comments · Source: veliovgroup/Meteor-Files

I'm having an issue:

S3 uplaod failed file just left on the saved directory (caused by network or size... whatever)
The file served well because it is treated as a normal saved file before it was uploaded successfully
But it seems that not re-try to be uploaded to S3 automatically.

I had an issue and I've solved it on my own:

I couldn't find and not sure this is intentional or there's an option to re-try s3 upload again.

I have a suggestion:

Meteor-Files upload file to S3 onAfterUpload event, so the whole procedule to save it to the local server system was already be done.
The failure action should be not interrupt the other process so I believe that Left the failed the file as a serviced by server would be good to go.
But there's two cases to use Meteor-Files with S3, 1. saved huge/many files / 2. consider multiple file server for scale-out the system
So, I think that would be great to have two options to be used on right situation.

Option 1. Left the failed file and servce as normal file system service.
Option 2. Re-upload it seamlessly when it's available during it is served as a normal file.
(re-try limits / re-try interval / notify this as a event ... options would be great as well for this)

Thanks

question

Source

kakadais

All 5 comments

Hello @kakadais ,

I believe case you've described is way beyond this library. Everything written in hooks is under your control, you can push it to the task queue and manage file upload to S3 with task manager, including failed uploads.

This library wiki and documentation are complimentary suggestions and examples of how Meteor-Files can be used to solve day-to-day tasks, the implementation itself — is under your control.

On other hand — I'd be happy to make our docs better with your suggestions, so feel free to send a PR

dr-dimitru on 16 Sep 2020

👍1

Cool.

Then I'll make some example or suggestion to handle failed S3 upload files.

I think the best thing that we used this library is the single instance of the file management.
Which means that when we use GridFS or S3, we can scale-up and out the servers as many as we want, without any concern of the file service syncs.

I used to use GridFS as well and I couldn't find the upload failure on noremal case,
so if we could handle S3 failure case that would be good enough to get this single instance of the file management.

Thanks a lot always for great library-

kakadais on 24 Sep 2020

@dr-dimitru Can I get some advice?

I think this could be started from bound() function in onAfterUpload event,
and when I make an error on purpose I could see some retryable and retryDelay option on the log.

If this spec is some option to retry for failed action,
could you let me know some documents or link where could I refer to see this spec?

following is the log.

W20200925-00:20:17.790(9)? (STDERR) InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
W20200925-00:20:17.791(9)? (STDERR) at Request.extractError (/Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/services/s3.js:837:35)
W20200925-00:20:17.791(9)? (STDERR) at Request.callListeners (/Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
W20200925-00:20:17.791(9)? (STDERR) at Request.emit (/Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
W20200925-00:20:17.791(9)? (STDERR) at Request.emit (/Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/request.js:688:14)
W20200925-00:20:17.791(9)? (STDERR) at Request.transition (/Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/request.js:22:10)
W20200925-00:20:17.791(9)? (STDERR) at AcceptorStateMachine.runTo (/Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/state_machine.js:14:12)
W20200925-00:20:17.791(9)? (STDERR) at /Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/state_machine.js:26:10
W20200925-00:20:17.791(9)? (STDERR) at Request. (/Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/request.js:38:9)
W20200925-00:20:17.791(9)? (STDERR) at Request. (/Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/request.js:690:12)
W20200925-00:20:17.791(9)? (STDERR) at Request.callListeners (/Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
W20200925-00:20:17.792(9)? (STDERR) at Request.emit (/Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
W20200925-00:20:17.792(9)? (STDERR) at Request.emit (/Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/request.js:688:14)
W20200925-00:20:17.792(9)? (STDERR) at Request.transition (/Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/request.js:22:10)
W20200925-00:20:17.792(9)? (STDERR) at AcceptorStateMachine.runTo (/Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/state_machine.js:14:12)
W20200925-00:20:17.792(9)? (STDERR) at /Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/state_machine.js:26:10
W20200925-00:20:17.792(9)? (STDERR) at Request. (/Users/kakadais/WebstormProjects/zmapbo/node_modules/aws-sdk/lib/request.js:38:9) {
W20200925-00:20:17.792(9)? (STDERR) message: 'The AWS Access Key Id you provided does not exist in our records.',
W20200925-00:20:17.792(9)? (STDERR) code: 'InvalidAccessKeyId',
W20200925-00:20:17.792(9)? (STDERR) region: null,
W20200925-00:20:17.792(9)? (STDERR) time: 2020-09-24T15:20:17.784Z,
W20200925-00:20:17.792(9)? (STDERR) requestId: '1FDFC0B677A4E275',
W20200925-00:20:17.792(9)? (STDERR) extendedRequestId: '0aQEOKgKtEPdpQSY3jKIdCBgMcDcI4OKqnjKFp1O/BIHLWMIH8Fx6XZw2rWyxku9gVAtW5qnYmY=',
W20200925-00:20:17.792(9)? (STDERR) cfId: undefined,
W20200925-00:20:17.793(9)? (STDERR) statusCode: 403,
W20200925-00:20:17.793(9)? (STDERR) retryable: false,
W20200925-00:20:17.793(9)? (STDERR) retryDelay: 46.5444581545641
W20200925-00:20:17.793(9)? (STDERR) }