Prisma-client-js: Large Payloads

[aarshaw]

Hi, I'm attempting to post quite a lot of data at the one time to a Postgres db via prisma2.

Trying to deduce the source of the following error:

{
  "statusCode": 413,
  "statusMessage": "Payload Too Large",
}

Is this limit configurable anywhere? - and where is it coming from - the payload itself is GraphQL - so I'm assuming somewhere in the query engine - but why is an HTTP request being made with a GraphQL body - I was under the impression this was all done locally?

[janpio]

What commands or code exactly are you using?

Can you post the complete error you are getting - or is it really just this JSON?

and where is it coming from - the payload itself is GraphQL - so I'm assuming somewhere in the query engine - but why is an HTTP request being made with a GraphQL body - I was under the impression this was all done locally?

The query engine is a sidecar process that runs locally and that offers a HTTP API that uses GraphQL as its "transmission format".

[aarshaw]

The command that is calling it is an update - it is a fairly large write as it basically writes a bank account and all it's transactions into a users account - hence the large payload.

Here is the full error... the GraphQL content goes on for a long time

Error: 
Invalid `photon.()` invocation in internal/process/next_tick.js:68:7



Reason: {
  "name": "HTTPError",
  "host": "localhost:65106",
  "hostname": "localhost",
  "method": "POST",
  "path": "/",
  "protocol": "http:",
  "url": "http://localhost:65106/",
  "gotOptions": {
    "path": "/",
    "protocol": "http:",
    "slashes": true,
    "auth": null,
    "host": "localhost:65106",
    "port": "65106",
    "hostname": "localhost",
    "hash": null,
    "search": null,
    "query": null,
    "pathname": "/",
    "href": "http://localhost:65106/",
    "retry": {
      "methods": {},
      "statusCodes": {},
      "errorCodes": {}
    },
    "headers": {
      "user-agent": "got/9.6.0 (https://github.com/sindresorhus/got)",
      "content-type": "application/json",
      "accept": "application/json",
      "accept-encoding": "gzip, deflate",
      "content-length": 3030526
    },
    "hooks": {
      "beforeRequest": [],
      "beforeRedirect": [],
      "beforeRetry": [],
      "afterResponse": [],
      "beforeError": [],
      "init": []
    },
    "decompress": true,
    "throwHttpErrors": true,
    "followRedirect": true,
    "stream": false,
    "form": false,
    "json": true,
    "cache": false,
    "useElectronNet": false,
    "body": "{\"query\":\"mutation {\\n  updateOneAccount(\\n    where: {\\n      id: \\\"ck19j3d0l0001vxupivvjb8h8\\\"\\n    }\\n    data: {\\n      bankConnections: {\\n        create: [\\n          {\\n            provider: \\\"mock\\\"\\n            sourceId: \\\"c0UTLh6/

...
  companyStructure\\n    companyNumber\\n    hasSupport\\n    termsAccepted\\n    waitlistType\\n    chAuthCode\\n    utr\\n    hmrcUsername\\n    hmrcPassword\\n    isVATRegistered\\n    vatScheme\\n    vatNumber\\n    isRegisteredEmployer\\n    payrollId\\n    simplicity\\n    appetite\\n    industry\\n    domain\\n    range\\n  }\\n}\",\"variables\":{}}",
    "method": "POST"
  },
  "statusCode": 413,
  "statusMessage": "Payload Too Large",
  "headers": {
    "connection": "close",
    "content-length": "0",
    "date": "Mon, 07 Oct 2019 12:09:03 GMT"
  },
  "body": ""
}

[janpio]

I transferred this into the photonjs repo as this is about using Photon.

[aarshaw]

@janpio I think the source of this issue is definitely in the query-engine - the max payload size tends to be an issue that stems from the server.

Have done a bit of digging and looks like the max payload size ~> from what I can tell the server (I don't speak rust) can take 32kb by default... this looks to be configurable using JsonConfig and increasing the max body size

I really do think this is worth doing as the GraphQL requests can get quite large given the amount of whitespace they contain and mine is probably going to be a fairly common use case and should be a pretty quick win if this is all that needs doing

[xero0001]

I'm also having this large payload issue. The error message is exactly same with my case.

https://github.com/prisma/prisma2/issues/771

I'm building online text editor and it uploads image which is converted into base64, so text size is little bit large.

But in my case, I didn't get the error when payload size was less than around 200kb(or much larger. I don't know how much it gets larger when PNG is converted into base64. Error started to happen when I contained PNG image larger than 200kb, and it was fine when it was under 200kb), which is definitely larger than 32kb. So I'm wondering if the error is really caused by default 32kb setting.

Also, querying over 200kb doesn't generate error at all.
But updating large data produces "Payload too large" error.

[xero0001]

Here's my git repository which can reproduce the error, according to pantharshit00's request at https://github.com/prisma/prisma2/issues/771.

https://github.com/xero0001/prisma2-payloaderror-reproducer

The code is based on "react-graphql-fullstack" example(https://github.com/prisma/prisma-examples/tree/prisma2/misc/react-graphql-fullstack).

I just added "bodyParserOptions" at server side which sets the limitation of body size to 50mb, and there's nothing else I added or modified.(using body-parser middleware directly or using apollo-server-express didn't resolve the error, as I mentioned https://github.com/prisma/prisma2/issues/771)

The error is reproducable by,

1. Running client and server
2. Copying the texts of "base64.txt"
3. Press create draft button(on client side)
4. And pasting it multiple times and click the create button.

The size of "base64.txt" is about 50kb and if you paste it 5 or more times in one post you will see the 413 payload too large error.

Thanks to pantharshit00 and I hope this could help!

[pantharshit00]

I am unable to reproduce this.

Photon handled the base64 text very easily. I used photon version [email protected], binary version: 20b6dc13949cccccfef5be07c0be7a3d7c858abe

[xero0001]

Thanks for replying.

Error is produced when you "pasted it 5 or more times in one post".

As I see from your comment, I think you pasted it just once.

If you paste it just one time, that's just 50kb. I think error is produced around 200kb.

As English is not my native language, my words could have been confusing. 👏

[xero0001]

I updated my "base64.txt" on github which is now five-times-duplicated version of old "base64.txt"

you can just download it from my github repository
https://github.com/xero0001/prisma2-payloaderror-reproducer/blob/master/base64.txt
it looks like this - which has 5 paragraph elements.

You can try with this!

And this is the error screenshot captured from safari.

[pantharshit00]

Ok, I can reproduce this now.

Thanks for the detailed reproduction. I think we can have an option to increase the max load.

[xero0001]

It's amazing! Thank you.

[pimeys]

This has been fixed in my experimental branch already, we tested it with the example given here with @pantharshit00 and the error is gone.

Merging this to master needs first the core refactoring from @dpetrick, which happens today or tomorrow. After that we'll start porting the experimental async/await branch to master, introducing a new web server with higher limits (and in general waaaay faster query engine).

If you want to try it now, you need to compile a bit of rust:

• Clone the bench branch from https://github.com/prisma/prisma-engine/tree/bench
• Install rustup and with it install rustc beta:

> rustup install beta
> rustup default beta

• Compile query-engine in release mode:

> cargo build --release --bin prisma

• The release binary will be in target/release/prisma

The binary needs to be copied somewhere in node_modules, replacing the existing query-engine-linux-glibc-libssl1.1.0 or query-engine-darwin or whatever binary fits to the current distro.

Or wait until next Monday! :)

[pimeys]

Fixed on master.