Parse-server: Status of legacy files on Parse S3

Created on 29 Jan 2016  ·  88Comments  ·  Source: parse-community/parse-server

What will happen to file migration? I can't really find instructions on how to transfer/migrate over files to your own bucket.

Most helpful comment

Any ETA for the migration tool?

All 88 comments

We'll be working on a plan for this. Right now when you migrate data, the files stay on Parse's S3 bucket, but parse-server knows how to continue serving them..

It would be so nice to have a config setting with our own S3 settings which end up getting used for file uploads.

So we are assured that file migration will be happening? So that we can continue develop using PFFile before migration?

@hashmapped Yes

Without PFFile migration my apps are dead in the water, please prioritise this as it would allow me to migrate alot sooner

So if I do a database migration right now, when a user creates a new PFFile - will the MongoDB entry just point to a new file hosted by Parse or will something else happen ?

New files uploaded to parse-server are hosted in GridStore(Mongo), only files uploaded through api.parse.com are still hosted by Parse. We are definitely going to solve this, and an awesome engineer here is working on an great update.

+1 we really need this. Do we at least have a timeframe for this? like when it likely will be available?

Also would be good if the client SDK can directly upload to S3 without hoping on first to parse server, abstracted in the client so we can still use the parse file api.

I would love to see a script that helps the user (idempotently) download every file from Parse's S3 http://files.parsetfss.com/ endpoint and from GridFS, and uploads them to a third location of the user's choice (authentication might be an issue here of course — parse-server will need a way to handle this): whether this be an S3 bucket, or Dropbox, or Cloudinary, or whatever.

The script should also update the URL held in the relevant Mongo file field for each class.

It's scary to think that after a year, all files held on Parse will disappear unless action is taken!

@fatuhoku that is very easy, I have already written one to copy all my files off Parse to an alternative server for backup purposes. Just use the javascript or php api's and you will get it done in minutes.

The real issue is that until I move to my own server files uploaded to api.parse.com are hosted by Parse.
What is needed is two fold.
1) When I tell api.parse.com that I want to move to MongoDB - to move all my existing files over as well as the database and store them with MongoDB Gridstore.
2) When (1) above is complete and I have moved to my own MongoDB for api.parse.com to store all new files to MongoDB gridstore and STOP hosting the files yourself.

This way I can follow Parse's guide of step 1) point at own database , 2) build own server, 3) issue app update.

Currently with files this is not possible.

An alternative (or additional) process could be telling both api.parse.com and the new parse server to host files on a S3 instance instead of MongoDB gridstore.
Either way this is a glaring omission that needs resolving asap.

@chrisfoulds do you have that script to hand by any chance? I'd be grateful if you stuck it on a gist so that we all have something to work off. Re. new files hitting api.parse.com being stored there, I imagine you could write a Cloud Code function (for every class that has files...!) that saves them to GridDB explicitly.

Im not sure if its finished but in the FilesAdaptor.js you should abe able to set this to S3 soon. https://github.com/ParsePlatform/parse-server/blob/master/FilesAdapter.js see this bit of code for the s3 implementation https://github.com/ParsePlatform/parse-server/blob/master/S3Adapter.js .. I wish someone would document how to use this over gridstore. I personally haven't had time to test it out.

@gateway The S3Adapter seems to be working. I have not seen official documentation but here is a sample of my index.js. Note the additional require at the top to bring in the S3Adapter. It is noted in #113 that this may eventually be exposed by default so the require() won't be needed.

What happens when you use this adapter is that any new files added will go to your S3 bucket instead of GridFS in your MongoDB database.

var express = require('express');
var ParseServer = require('parse-server').ParseServer;
var S3Adapter = require('parse-server/S3Adapter');
...
var api = new ParseServer({
  databaseURI: databaseUri || 'mongodb://.....',
  cloud: process.env.CLOUD_CODE_MAIN || __dirname + '/cloud/main.js',
  appId: 'MY_APP_ID',
  clientKey: 'MY_CLIENT_KEY', 
  masterKey: 'MY_MASTER_KEY',
  filesAdapter: new S3Adapter(
    "KEY",
    "SECRET_KEY",
    {bucket: "my-bucket-name", bucketPrefix: "some-folder/", directAccess: true}
  )
});

@gfosco When can we expect an update on migrating existing files to a self-hosted S3 bucket? I have tested migration (and am using my fileKey) but all existing files result in a 404. The PFFile.url ends up pointing to a self hosted file instead of the parse hosted file.
Example URL:
http://foo.herokuapp.com/parse/files/dWOKPEJ9ihbPYARGV7lwBUz3vQOASuhfWrNl7PCtxrau/78a0a3aa-2ef3-46c8-be1b-14059308cc4a-my_image.jpg
I should note I am using the S3Adapter so that new files get stored to a self-hosted S3 bucket instead of GridFS. Is this resulting in the parse-server trying to also Retrieve files from my S3 instead of the existing parse hosted file?

The file migration solution will need to account for both existing files and new files. Existing files will ideally be able to be migrated to the same S3 bucket/folder as new files. If they cannot be migrated to the same self-hosted S3 bucket/folder as new files, the parse-server should work seamlessly for those of us opting to use the S3Adapter fileAdapter by delegating to some other S3 bucket for existing files and to the self-hosted S3 bucket for new files.

Thanks for helping us. So far things are working pretty smooth in general, but the PFFile question is a roadblock to finalizing migration.

@fatuhoku I am sure we'll get an awesome solution for migrating files from the Parse team.
In the meantime (and for paranoid backup) I whipped a quick Python script to manually download files. To use it you'll need to export your collections to json using the "Export app data" option in your apps App Settings.
Then you'll need to run the script for each file that contains pointers to files. It is a quick and dirty script that met my needs but it covers the basic requirements of a brute force file downloader. You'll very likely need to make some modifications to it to process your exported json files.
https://gist.github.com/barnaclejive/bac711c85e5e9a94a8b1

@drew-gross @nlutsenko
Thank you for sharing this.

Do we have plan to add adapter for other file storage services like Dropbox or Azure file storage ?

Immediate plans - no, but since it's fairly easy to create your own file adapter we would love to see the community build the ones that they think are the most helpful and we'll happily merge them.

Can we save the images (or files) on local server (E.g: http://domain.com/uploads/photo.png) instead of using S3 bucket?

You can do that in mongo using GridStoreAdapter (the default), or implement a new FileAdapter to store files on the local filesystem.

Ive migrated my app to Parse Server and mongoDB. However my development client cannot access the files since the mongodb cant seem to find whee the files are. Am I missing something here ?

@poojanjhaveri Make sure you add the fileKey in your new server. I do not think this is in the example program.
var api = new ParseServer({
...
fileKey: process.env.PARSE_FILE_KEY || 'My file key from parse.com'

});

@Controlhaus : Thanks for your help. It works now

Thanks for all the input and answers it has helped me alot with my migration from Parse to AWS BS. However, I still have a couple of issues I can't seem to find answers for...or at least that I can understand to act upon. So any help would be appreciated.

Using the below changes to the index.js I can access access the AWS ParseDB/MongoDB and I can upload and retrieve images and display pre and post migration images (have not finalized the migration).
This is just base changes with the fileKey added which I assume is why I can see my pre migration images. But when I change the index.js to include the S3 items as the instructions detail I get an error about data not being formatted correctly just by building the app. Next section below

Working Index.js without S3:
var express = require('express');
var ParseServer = require('parse-server').ParseServer;

var databaseUri = process.env.DATABASE_URI || process.env.MONGOLAB_URI

if (!databaseUri) {
console.log('DATABASE_URI not specified, falling back to localhost.');
}

var api = new ParseServer({
databaseURI: databaseUri || 'mongodb://accessID:[email protected]:xxxx/parse-aws',
cloud: process.env.CLOUD_CODE_MAIN || __dirname + '/cloud/main.js',
appId: process.env.APP_ID || 'AppID',
fileKey: process.env.FILE_KEY || 'fileKey',
masterKey: process.env.MASTER_KEY || 'masterkey'
});

This is the changes to the index.js I made for using S3 bucket to store my images.

Revised Index.js to use S3:
var express = require('express');
var ParseServer = require('parse-server').ParseServer;
var S3Adapter = require('parse-server').S3Adapter;
var databaseUri = process.env.DATABASE_URI || process.env.MONGOLAB_URI

if (!databaseUri) {
console.log('DATABASE_URI not specified, falling back to localhost.');
}

var api = new ParseServer({
databaseURI: databaseUri || 'mongodb://accessID:[email protected]:xxxxx/parse-aws',
cloud: process.env.CLOUD_CODE_MAIN || __dirname + '/cloud/main.js',
appId: process.env.APP_ID || 'AppID',
fileKey: process.env.FILE_KEY || 'FileKey',
masterKey: process.env.MASTER_KEY || 'MasterKey'
filesAdapter: new S3Adapter(
'myaccess key', 'secret key', 'bucket' {directAccess: true}),

});

After deploying this configuration and then trying to access my app which has no issues using the first version of the index.js above I get the following errors/messages.

Build and Run App in xcode I get this error
If I try to login to the app, I get the error message (Data could not be read because it is in the wrong format)

Nothing is different in my AppDelegate between the two...and if I redeploy the first Index.js I have no issues and no S3.

The last question (if I get the above to work) is How Do I Get All the Pre-Migration Images from Parse into my bucket so that all the images are in the same location and I am not using the MongoDB
GridStore.

Thanks in advance for any help

You're missing a closing single quote on 'FileKey in your revised code. Not sure if that is causing issue or not....
As of now, you have to manually migrate image from Parse to your S3 container to complete a full migration.

Thanks...that was my fault editing the actual key values...in the index.js the closing quote is there. editing above...

You'll get that error if you server fails to startup. If you look at your logs you'll see the syntax error mentioned about the missing quote character.

Thanks but I don't understand your comment. The quote issue was only on my post when I edited out my ids per my reply to onosol's comment. I corrected the post and both of the index.js do not have a missing quote. I also tried to look at the logs and don't see anything that l can translate into why it does not work.

Below is the log that references the index.js...

[email protected] start /var/app/current
node index.js

/var/app/current/index.js:19
filesAdapter: new S3Adapter(
^^^^^^^^^^^^

SyntaxError: Unexpected identifier
at exports.runInThisContext (vm.js:53:16)
at Module._compile (module.js:374:25)
at Object.Module._extensions..js (module.js:417:10)
at Module.load (module.js:344:32)
at Function.Module._load (module.js:301:12)
at Function.Module.runMain (module.js:442:10)
at startup (node.js:136:18)
at node.js:966:3

npm ERR! Linux 4.1.17-22.30.amzn1.x86_64
npm ERR! argv "/opt/elasticbeanstalk/node-install/node-v4.3.0-linux-x64/bin/node" "/opt/elasticbeanstalk/node-install/node-v4.3.0-linux-x64/bin/npm" "start"
npm ERR! node v4.3.0
npm ERR! npm v2.14.12
npm ERR! code ELIFECYCLE
npm ERR! [email protected] start: node index.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script 'node index.js'.
npm ERR! This is most likely a problem with the parse-server-example package,
npm ERR! not with npm itself.

That's pretty clearly a syntax error in your code. filesAdapter should be being defined inside an object passed to new ParseServer({ .... in here .... })

Do we have any update on migration from parse hosted s3 files to our own buckets? Are we days/weeks away from getting this functionality?

Until the migration tool
1) Moves my files over to a store of my choice
2) Once I put the "redirect" in place to use my own server and until a user upgrades to the latest version of my app (I can't force them to do it immediately! ) that points them straight at my server then files sent to api.parse.com go to my new location instead of staying on parse.

I like many others are completely stuck

Seems to be a problem in FilesController.js when Parse Server checks wether to use parsetfss.com or adapter for a file:

if (filename.indexOf('tfss-') === 0) {..

I have old files on Parse that do not begin with 'tfss-', and they are wrongly sent to adapter.

Looks like Parse Server does:

filename = randomHexString(32) + '_' + filename;

on any new files, so I guess a fix would be to check for that underscore, since it would never match old files.

I like to keep text data in my database. Blob stores exist for storing assets. It would be nice to have assets moved to S3 rather than dumping them in Mongo using GridFS.

Any ETA for the migration tool?

+1

@trondwernerhansen hi, so it looks like you have the right idea, do we need parse team to do this?

I guess they will fix it as part of rolling out a proper file migration solution

I'm currently finishing the migration of my app to my own Parse Server & mLab. Unfortunately I still have a lot of files stored on Parse. Do you think it is a good approach to finish the migration and wait for the file migration tool to migrate my files on Parse to my own S3 bucket?

After finishing the migration using your tool, will I need to push a new update of my app to the users? Or it will be only a server side change?

Thank you for your work on Parse Server. I'm facing no big problems to migrate (except the file migration issue) and I can notice that you're constantly updating the documentation. I hope you'll have news about file migration tool for us soon!

I'm in the same situation as @rdgborges . I have been doing a lot of testing and am finally ready to complete migration... however, I have no idea what the migration strategy is going to be for legacy files. I'm assuming it will be as pain-free as the migration process so far, but I think we'd all like at least some rough description of what the legacy file migration will look like - even if won't be ready to use for a while.

I can second @rdgborges comment about the great work that has been done so far. The support has been great, just waiting to feel more confident that this last (but very critical) piece will be as smooth.

The recommended date for full migration is one month from today, but there have been no updates as to the status of the file migration tool in the last 3 months. It's the only part of our overall migration that is still missing, and the only thing preventing us from moving away from parse.com.

Could we have some idea please of when to expect an update on this tool - is it still being actively developed or should we be seeking an alternative course of action at this point?

It looks like @flovilmart wrote a handy little nodejs utility that gets the urls for all files across all classes: parse-server-modules/parse-files-utils

I'm not sure if Parse is planning to extend this utility, but I think it could be updated to do the following:

  1. Detect if file is a Parse hosted file.
  2. Transfer file to S3, GCS, or filesystem with a new filename that doesn't start with "tfss-" or match legacyFileRegex.
  3. Update filename saved in the database.

Anyone see any problems with this approach?
Anyone have any better ideas?

Hey guys, we've just released a File Migration Tool - https://mv.sashido.io, which can transfer all your files from Parse to Your AWS S3 bucket :wink: We hope you will like it!

@pivanov You have a typo in your URL but looking forward to taking a look.

Remove the www. :)

sorry and thanks guys 👍

@pivanov Is this tool open source? Naturally worried about security with secure keys etc 👍

It's normal (you must!!!) to think about the security. We use the same tool in our Dashboard :) It's not open source yet ... that's why we made this as service ;) and we will open source it soon :) we just need to do few other cool things before that ... but anyway you can use it if you want :)

@pivanov Thanks for sharing!

Can you please describe what kind of database writes you are doing with your tool? Are you just removing "tfss-" from file field values?

Any ETA on open sourcing? If not, then I'll likely release something in the next week or so.

We simple remove "tfss-" from file fields as you mention :) We will open source it till end of August.

1582 is closed, but it doesn't appear that there is a solution where we can release clients that use parse-server without breaking existing clients that use parse.com with regard to files created on parse-server?? @JeremyPlease you seem to be in the same boat. Have you decided on a course of action?

@acinader, we're very close to have a generic file migrator, that leverage the same files adapters that we use on parse-server. Hang on tight!

@flovilmart thanks for the speedy response. i'm hoping that your solution will allow us to release a new client that points at our parse-server and old clients that are still using parse.com will be able to see files uploaded by the new clients that are using parse-server. Not clear if a generic file migrator would cover that use case, but i'll be optimistic ;).

We are in the process of migrating our files over from parse to s3, but we would like our app users to still be able to access their files before the update their app to use our own server.
Any suggestions to allow us to write to both our s3 bucket and the parse file server, so our app users who don't upgraded immediately can don't run into problems.

@leapingcoyote yes - take a look at the discussion here and that might help you. Basically, you will need to keep the file names the same, migrate the files, and ignore the fileKey in your Parse Server (check this PR and this condition).

For others peoples reference I came up with a simple solution if it's a mobile app that is using parse.

In my case each user has just two files associated with them.
When they update the app to the version of the app that goes direct to my server instead of api.parse.com I do an 'Upgrade Process'.
I just basically load those two files which Parse server will pull from parse.com buckets. I then immediately do a save on both which then goes via my server and drops them in my S3. Users data is transferred, so instead of making it mass migration I made the users do it.
Not perfect and does not get 100% coverage but it gets all active users which is all I care about.

Does anyone have tried this tool: https://mv.sashido.io ?

Any update on this one ?

@poojanjhaveri
you have the following options available: https://mv.sashido.io or this command line tool

The parse-server-modules/parse-files-utils tool suggests two strategies for migrating parse files.

README has been recently updated to describe the two options:
https://github.com/parse-server-modules/parse-files-utils#parse-file-migrations

@JeremyPlease Thanks. @natanrolnik @vladiGtr Have you guys used mv.sashido ?

Finally I didn't want to use any alternative since they didn't seem what I was looking for.

I'm showing my code here just in case you need it. It's working on me (migrating really slow, but it's going on!)

Good luck!

Is someone already working on this ? I would like to work on if not started already

@RafaRuiz I think your code is broken with the last parse php sdk..

$parseFile->getName();

2016/09/13 18:56:27 [error] 25386#0: *329857 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Call to a member function getName() on a non-object in /var/www/aaa.com/tests/data/migrate.php on line 34" while reading response header from upstream, client: , server: aaa.com, request: "GET /tests/data/migrate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "aaa.com"

Also I dont think this takes into account if you have more than one image in the row.. like a profile pic, small, med, large etc.. I might be able to fix it just letting you know.

@gfosco Any updates on this ? Things are moving to danger zone if this is further delayed. Is there any active development on this ?

@flovilmart so is parse-files-utils the official answer or not? This ticket appears to be closed on the basis of that utility existing, but it comes with a giant warning claiming that it's "unofficial" and might break things.

Yeah, it's as good as it gets, not sure if anything else will be released.

Thanks for all of these comments. They have helped me thus far. However I have added the File Key to my index.js as mentioned above :

var api = new ParseServer({
databaseURI: databaseUri || 'mongodb://xxxxxxx',
cloud: process.env.CLOUD_CODE_MAIN || __dirname + '/cloud/main.js',
appId: process.env.APP_ID || 'myAppID',
masterKey: process.env.MASTER_KEY || 'MyMasterKey', //Add your master key here. Keep it secret!
serverURL: process.env.SERVER_URL || 'http://myServerURL', // Don't forget to change to https if needed
fileKey: process.env.PARSE_FILE_KEY || 'myFileKey',
liveQuery: {
classNames: ["Posts", "Comments"] // List of classes to support for query subscriptions
}
});
// Client-keys like the javascript key or the .NET key are not necessary with parse-server
// If you wish you require them, you can set them as options in the initialization above:
// javascriptKey, restAPIKey, dotNetKey, clientKey

and I am still not able to access my imageFiles that were stored to Parse prior to migration. When I run my app, I get the following error :

Has anyone else ran into this? What have I missed? Thanks in advance.

Can someone please answer this:
http://stackoverflow.com/questions/41737158/migrating-parse-files-from-gridstore-to-s3

Description:
We migrated the data(without files) to mLab and Heroku. So the old files are still on Parse.
Since then, any new file added goes into Gridstore, which is the default file storage for mLab.
Now I migrated old parse files from Parse to an S3 Bucket using sashido
The files are migrated and are accessible using S3Adapter in Heroku.
But the files on Gridstore are not accessible now. How can I migrate them to the same S3 bucket and change references in mLab?

Open a ticket in SashiDo an mention Pavel :)

@vinaykaitha I just answered it

This might be useful for some of you:
https://github.com/modernistik/parse-file-migration

aaaaaannddd it's gone.

Hello everyone! We have just realized that lots of our files did not migrate completely to our servers and, as expected, we cannot access files on http://files.parsetfss.com/. This is core to our business and we need to get those files back even if we have to pay. Please, please, please, is there a way we can access them? We'll do anything that is necessary.

@ortimanu
Today I received a message from back4app. Maybe it will help you.

How come they can still access those files?

I have no idea

Hi, guys. We are doing like this:
https://github.com/parse-server-modules/parse-files-utils/pull/67

It is already merged to the master so you can still use parse-file-utils.

Parse.com announced it will stop anytime though.

I migrated using sashido, just the files with mv.sashido.io.
Everything went fine and I can access the files in s3. but I can't access the files using parse dashboard, or my apps. what am I missing?

@ortimanu ping me on [email protected]

It might be permission issue sometimes.

On 31 January 2017 at 17:56, Manuel Ortiz notifications@github.com wrote:

Hello everyone! We have just realized that lots of our files did not
migrate completely to our servers and, as expected, we cannot access files
on http://files.parsetfss.com/. This is core to our business and we need
to get those files back even if we have to pay. Please, please, please, is
there a way we can access them? We'll do anything that is necessary.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ParsePlatform/parse-server/issues/8#issuecomment-276350392,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADRXACci3cXsU_RcEQ2s5NKKG5eZht_qks5rXyhvgaJpZM4HO2Bk
.

--
alvajindia.....
"nthng spzl..nthng mattrzz..."

@Mars If you are seeing access denied message, its because files were
uploaded with limited permission. You can make the files public and then it
will be available

On 1 February 2017 at 00:14, Mars notifications@github.com wrote:

I migrated using sashido, just the files with mv.sashido.io.
Everything went fine and I can access the files in s3. but I can't access
the files using parse dashboard, or my apps. what am I missing?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ParsePlatform/parse-server/issues/8#issuecomment-276453080,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADRXAI6dy2aNUzcFMLPh0lad4rwLu6ocks5rX4DygaJpZM4HO2Bk
.

--
alvajindia.....
"nthng spzl..nthng mattrzz..."

@alvajindia no, its giving me a "file not found" the urls have been updated with the mv-prefix do I need to set up a file adapter to make this work ?

Did you set the Parse Public URL in your Parse-Server? I forgot to add it
and all my file URLs started with localhost instead of the public URL.
Can you check the URL that is giving your Parse-Server when accessing to
files?

El mié., 1 feb. 2017 7:45, Mars notifications@github.com escribió:

@alvajindia https://github.com/alvajindia no, its giving me a "file not
found" the urls have been updated with the mv-prefix do I need to set up a
file adapter to make this work ?


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ParsePlatform/parse-server/issues/8#issuecomment-276585071,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAsPn5473tJ-lTrjhB1xLY0-wEt_Vgd2ks5rYCoXgaJpZM4HO2Bk
.

@arturogutierrez the file urls are pointing to the right domain as far as I can see in the parse-dashboard

a migrated file looks like this
https://[mydomain]/apps/[myapp]/files/[filekey]/mv-b0acb1ae-d6f7-44c9-921d-0f332366692a-file
newly uploaded and working files look like this, these were uploaded to my local parse
https://[mydomain]/apps/[myapp]/files/[filekey]/7213555b291d0c4ca7f99942c460a9ee_file

Could the file adapter cause this?

Guys, thanks to parse-server-modules/parse-files-utils#67 I could download all my files and transfer them to my S3 bucket. Thanks guys for everything!

Was this page helpful?
0 / 5 - 0 ratings