Parse-server: Add 10k+ rows to a Class in Parse Server

[AmbroiseCollon]

I've got 10k+ new questions objects to my Question class (therefore they don't have objectIds for now). I want to add them to my dashboard. On Parse.com, I would have uploaded a CSV. But Parse Server and open source dashboard don't allow this functionality.

Here is what I tried so far :

• Use mongoimport to upload a JSON. All went well. But the created objectIds are really long (Ex:577b7488e9d6261a308ed262) and curiously resembling to each other . They all have at least 80% of the characters in common. And any object that points to a question fetch nothing when trying to fetch the question. So I guessed that bypassing parse server to create objects was a bad idea.
• Write a node script to upload the question using Parse save method. But 10k+ objects is way too big and the script always end up failing.
• Use this gist that combines curl and cloud code. But my curl request is too

I'm out of ideas.

[AmbroiseCollon]

If it can be useful to anyone:

I created a simple script for generating unique parse like object ids. Added them to my CSV file with column title "_id". Passed it to JSON and used mongoimport. In the JSON I also added createdAt and updatedAt field. Finally my JSON looked like:

[
  {
    "_id": "Q88swkLOVL",
    "category": "TV SERIES",
    "difficulty": 1,
    "question": "Whose final \"Late Show\" episode aired on May 20, 2015?",
    "incorrectAnswer1": "Jay Leno",
    "incorrectAnswer2": "Stephen Colbert",
    "incorrectAnswer3": "Conan O'Brien",
    "correctAnswer": "David Letterman",
    "createdAt": {
        "$date": "2016-07-05T11:27:15.784Z"
    },
    "updatedAt": {
        "$date": "2016-07-05T11:27:15.784Z"
    }
  },
...
]

I leave this open as this solution doesn't seem very handy to me. And the upload csv or json feature would be a lot greater.

[gulo-gulo]

I happened to see your issue while having my second cup of coffee, so it's entirely possible (though unlikely) that I am way off, so I'll let you decide:

parse-server/src/cryptoUtils.js
contains the following:

// Returns a new random alphanumeric string of the given size.
//
// Note: to simplify implementation, the result has slight modulo bias,
// because chars length of 62 doesn't divide the number of all bytes
// (256) evenly. Such bias is acceptable for most cases when the output
// length is long enough and doesn't need to be uniform.
export function randomString(size: number): string {
  if (size === 0) {
    throw new Error('Zero-length randomString is useless.');
  }
  let chars = ('ABCDEFGHIJKLMNOPQRSTUVWXYZ' +
               'abcdefghijklmnopqrstuvwxyz' +
               '0123456789');
  let objectId = '';
  let bytes = randomBytes(size);
  for (let i = 0; i < bytes.length; ++i) {
    objectId += chars[bytes.readUInt8(i) % chars.length];
  }
  return objectId;
}

// Returns a new random alphanumeric string suitable for object ID.
export function newObjectId(): string {
  //TODO: increase length to better protect against collisions.
  return randomString(10);
}

Additionally, when migrating to your own mongodb database, you have to set the failIndexKeyTooLong to false since it defaults to true for standard mongodb configurations

If you look here:
parse-server/src/RestWrite.js
you'll find:
var cryptoUtils = require('./cryptoUtils')
and

RestWrite.prototype.setRequiredFieldsIfNeeded = function() {
  if (this.data) {
    // Add default fields
    this.data.updatedAt = this.updatedAt;
    if (!this.query) {
      this.data.createdAt = this.updatedAt;

      // Only assign new objectId if we are creating new object
      if (!this.data.objectId) {
        this.data.objectId = cryptoUtils.newObjectId();
      }
    }
  }
  return Promise.resolve();
};

It appears that RestWrite.prototype.setRequiredFieldsIfNeeded is responsible for setting all the values that include a randomly generated objectId, a createdAt time and an updatedAt time. In my experience when using Parse, these are the 3 fields that are always defined for any given row (or object, if you prefer) in a table.

I may be wrong and what I've described may not actually cover everything, but it appears that you need to create your own logic to replicate the creation of the following fields:

1. createdAt
2. updatedAt
3. objectId (see parse-server/src/cryptoUtils.js and the function newObjectId())

Keep in mind that the Parse Server handles everything in one place for a reason, so I would actually advise against directly talking to your mongodb instance. I need another cup of coffee. Give that a shot in an isolated test environment and see if it works the way you want it to.

Another option would be something that uses the REST API by sending batches (of 50 maximum per api call) to the parse server or to simply use a helper cloud code function that you can iterate through:

For example, in python:

import requests,json
def send_data_to_parse(all_batch_requests):
    headers = {
        "X-Parse-Application-Id": "<appId>",
        "X-Parse-REST-API-Key": "<rest api key, if necessary>",
        "Content-Type": "application/json"}
    batch_requests = {
        "requests": all_batch_requests
    }
    response = requests.post('https://api.yourserver.com/parse/batch', headers=headers, data=json.dumps(batch_requests))
    print response.json()


def create_two_objects_with_parse_api():
    first_request = {
        "method": "POST",
        "path": "/parse/classes/Questions",
        "body": {
            "title": "first question for whatever",
            "text": "Is this a question? really?"
        }
    }
    second_request = {
        "method": "POST",
        "path": "/parse/classes/Questions",
        "body": {
            "title": "second question for whatever",
            "text": "Is this another question? come on dude"
        }
    }
    all_requests = list()
    all_requests.append(first_request)
    all_requests.append(second_request)
    create_two_objects_with_parse_api(all_requests)

I just wrote that up though, and I didn't test it because I've written code like that probably hundreds of times using python to talk with the REST API. Hopefully that's helpful for you.

It may even be easier to do it in the cloud but javascript is too verbose for me personally, especially while I'm enjoying my coffee.

[AmbroiseCollon]

Thank you for your answer ! And thank you for pointing these relevants files to me. It would have save me the pain to write my own script. Your answer convince me that it should be made available in Parse-Dashboard for instance.

Thanks !

[flovilmart]

Seems that we got a solid answer here, let's close that for now