Screwdriver: Store API crashing frequently

[Zhongtang]

What happened:

We are observing curl: (56) Unexpected EOF in multiple build logs in the artifact bookend step, when it attempts to upload large artifacts to the store. Running kubectl describe indicates store API pods have been crashing quite frequently (every several hours). When we look at the store API pod logs, we are seeing FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory and the pods crashed and restarted because of this.

What you expected to happen:

Store API should be able to handle artifacts upload when the size of the artifacts are within the range in our policy.

How to reproduce it:

We don't have a consistent way to reproduce it yet. But it seems it happens quite frequently with large artifacts upload.

[minz1027]

I think we found the cause:

Inside store, we convert payload to an object and then send it to S3: https://github.com/screwdriver-cd/store/blob/master/plugins/builds.js#L84

But in catbox-s3's set function, it will try to do JSON.stringify if the data is an object: https://github.com/fhemberger/catbox-s3/blob/master/lib/index.js#L236

When the payload is too big, JSON.stringify failed.