Prisma1: Import fails with exceeding size reason.

[donedgardo]

Bug Report

Current behavior
Running prisma import -d export-2018-03-06T04:48:12.466Z.zip -e .env.prod doesnt complete import instead debugs witha exceeding size exeption:

Unzipping... 225ms

Validating data... 540ms

Uploading nodes...
  client Uploading to endpoint https://us1.prisma.sh/edgardo-carreras/beaconsofhope-prisma/prod/import +0ms
Error: The request content was malformed:
EntityStreamSizeException: actual entity size (Some(10049921)) exceeded content length limit (8388608 bytes)! You can configure this by setting `akka.http.[server|client].parsing.max-content-length` or calling `HttpEntity.withSizeLimit` before materializing the dataBytes stream.
    at Client.<anonymous> (/Users/donedgardo/.nvm/versions/node/v8.9.4/lib/node_modules/prisma/node_modules/prisma-cli-engine/src/Client/Client.ts:321:13)
    at step (/Users/donedgardo/.nvm/versions/node/v8.9.4/lib/node_modules/prisma/node_modules/prisma-cli-engine/dist/Client/Client.js:32:23)
    at Object.next (/Users/donedgardo/.nvm/versions/node/v8.9.4/lib/node_modules/prisma/node_modules/prisma-cli-engine/dist/Client/Client.js:13:53)
    at fulfilled (/Users/donedgardo/.nvm/versions/node/v8.9.4/lib/node_modules/prisma/node_modules/prisma-cli-engine/dist/Client/Client.js:4:58)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:228:7)
Exiting with code: 1

Expected behavior?
Successful import or better description of how to configure service to skip size limit.

[marktani]

Thanks for the report @donedgardo, we're looking into this soon 🙂

[donedgardo]

https://github.com/graphcool/prisma/blob/c77e582b5fd6edc62d18bb03d060f5f2d3d2172a/server/servers/api/src/main/resources/application.conf#L18 @do4gr

[horsejockey]

@donedgardo were you able to find any kind of workaround for this?

[donedgardo]

I haven’t. Dont know how to build scala with different configurations

On Tue, Mar 13, 2018 at 12:09 PM Matthew McArthur [e-mail]
wrote:

@donedgardo https://github.com/donedgardo were you able to find any
kind of workaround for this?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/graphcool/prisma/issues/2024#issuecomment-372721677,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACXlULWURyM6cB3sjy7NYi7c9V48V6IDks5td-8_gaJpZM4SeG2h
.

[horsejockey]

@donedgardo I wrote this python script that uses the api to do the migration. Takes a long time cause the python requests library was giving me trouble when I sent to much data so I had to sub-batch it during import.
https://gist.github.com/horsejockey/54748992ac98ae92d0a6f05480750e0c

[johhansantana]

Tried the fix in unstable channel beta locally
v1.5.0-beta.1

Got the following error:

Uploading nodes...
  client Uploading to endpoint http://localhost:4466/<my-project-name>/dev/import +0ms
FetchError: request to http://localhost:4466/<my-project-name>/dev/import failed, reason: socket hang up

@do4gr

[do4gr]

Hey, @johhansantana,

thanks for quickly testing it!

I am still working on this issue even though the PR closed it. Enabling the import of larger file sizes exposed some performance problems that I am cleaning up at the moment. Your error is a symptom of this. I'll post again once I feel the issue is solved.

[giapnhtadi]

@do4gr This issue was fixed or not? I got same issue, so please don't close the task when it is not fixed yet.

[marktani]

Hey @giapnhtadi,

this fix is available already on the unstable channel.

It will be rolled out to the development clusters with the next point release, and will then also be available on private clusters.

If you're running this locally, can you confirm that you run against a cluster with a version that contains 1.5.0-beta? You can check using prisma cluster list.

[johhansantana]

I think what @giapnhtadi meant is that the issue is not really fixed because you can't import a backup yet successfully, even in the unstable channel.

[marktani]

Hey @donedgardo @johhansantana @giapnhtadi, this PR includes a fix for the problems you observes, as well as performance improvements. It is available on the latest beta. I would love to hear your feedback about it :slightly_smiling_face:

[marktani]

I'm so sorry, I mixed it up :see_no_evil: https://github.com/graphcool/prisma/pull/2115 is the PR in question, and it's still open. We'll merge it soon, and I will notify you then.

[johhansantana]

@marktani Weird thing is that I tried today again to do the import locally and it worked this time. It took a few minutes to complete but the manual import worked.

Not sure if there's anything that has been changed since my last attempt though. Can anyone else confirm?

[do4gr]

@johhansantana The import was sometimes failing due to using too much of the JVM's memory since the local Docker has only very little memory available So it was not entirely predictable when or if it would fail. If you had enough free memory before and were not doing anything else with your server it might have gone through even before the changes. But if the import spiked memory usage too high the Garbage Collection sometimes failed for the local servers. The last PR includes an almost complete rework of import to use less memory. It also reduces the chunk size and speeds up the import noticeably so you do not get a timeout response from the CLI that you might've gotten before if the server takes more than 300 seconds to import one chunk. We will not enforce the new 1mb chunk size as a limit during import, but if you still experience timeouts using 10mb chunks you should re-export again using the new 1mb chunk size that we set or split old chunks.

[do4gr]

The fix is now rolled out.