Jsforce: Is it possible to build a batch job using JSForce to update around 1M Contacts?

[sureshshamanth]

Hello,
I'm looking to build a batch job that will update 1M contact's email, that is to mask them to prevent users from getting sandbox emails (Which is a common problem), however, I've hit a dead end where I built an apex batch job which works fine in selective sandboxes, so was looking to use JSForce to build a batch job for this simple task, is it possible using JSForce, if so please tell me which method of the bulk api should I be looking into?

Thanks in advance!

[TonyWhiteSMS]

Yes it should be possible using
https://jsforce.github.io/document/#update-delete-queried-records

[sureshshamanth]

I tried doing that for 1M records and I get the error "Failed to load resource: the server responded with a status of 400 (Unable to forward request due to: Read timed out)"

When I put a limit of 10,000 or less it works, so want to know how I could make it work for 1M records

[Zambonilli]

You need to break up the 1M records into batches of 10,000 and submit 10,000 at a time. With just updating the email you should not hit a byte limit and only add 100 API calls to your daily allowance.

[sureshshamanth]

Can someone please tell me how I need to go about building this? with little more detail, because from the issues I read looks like bulk API doesn't really work as expected, and if I were to create chunks of 10k records, how would I keep submitting them? should I wait for each job to finish? or does the job created by jsforce take care of that?

Any ideas suggestions by people have already built something similar is appreciated

[Zambonilli]

You'll need to create chunks of 10k records to submit to the bulk API. How you submit them is more of a question of what you can handle on the client. I personally submit 10 chunks of 10k at a time using async.js eachLimit function.

[sureshshamanth]

I'm new to jsforce, so can you please give more detail like code snippet or how you go about doing that? because I'm getting stuck at using the bulk API itself, not able to update 10k records at a time, leave alone 1M.