Prisma1: [Import/Export] Data import fails when trying to import nodes with two scalar list fields

[Jaszkowic]

Describe the bug

Scenario: Export data with prisma export, reset prisma with prisma reset, import data with prisma import -- data ...: Data import fails when trying to import nodes with scalar list fields. Import works fine, when scalar list fields are empty.

Error:

Unzipping 12ms
Uncaught exception, cleaning up: Error: Object {"_typeName":"Course","id":"cjj71n9fd5x3k0807wpv0o3jg","colors":["#ff0000","#abcdef"]} lacks the following properties: tools
Validating data ✔

To Reproduce

1. Setup Prisma Server with Postgresql Connector

1. Schema:

type User {
    id: ID! @unique,
    createdAt: DateTime!,
    courses: [Course!]! @relation(name: "CoursesForUser", onDelete: CASCADE)
}

type Course {
    id: ID! @unique,
    createdAt: DateTime!,
    colors: [String!]!,
    tools: [String!]!,
    user: [User!]! @relation(name: "CoursesForUser")
}

1. Create data with an instance of Course having colors and tools field set to a non-empty list.

2. Export the data using prisma export ...

3. Import the data using prisma import --data ...

Expected behavior

Data ist imported succesfully.

Versions (please complete the following information):

• OS: macOS High Sierra (10.13.5)
• prisma prisma/1.11.0 (darwin-x64) node-v9.11.1

Additional context

NDF export looks fine according to https://www.prisma.io/docs/reference/data-import-and-export/normalized-data-format-teroo5uxih/ , content of file <export.zip>/lists/000001.json:

{
  "valueType": "lists",
  "values": [
    {
      "_typeName": "Course",
      "id": "cjj71n9fd5x3k0807wpv0o3jg",
      "colors": [
        "#ff0000",
        "#abcdef"
      ]
    },
    {
      "_typeName": "Course",
      "id": "cjj71n9fd5x3k0807wpv0o3jg",
      "tools": [
        "Zeichnung exportieren",
        "Zoom",
        "Template ein- und ausblenden",
        "Replay",
        "Undo/Redo"
      ]
    }
  ]
}

(Might be related to https://github.com/prismagraphql/prisma/issues/2621 ?)

[marktani]

I can confirm this bug, thanks for the great reproduction! 🙂

[divyenduz]

I can validate a similar issue with an even smaller reproduction, however, the error encountered is different.

1. datamodel.graphql

type Course {
  id: ID! @unique
  colors: [String!]!
}

1. seed.graphql

mutation {
  createCourse(data: {colors: {set: ["red"]}}) {
    id
  }
}

1. prisma.yml

endpoint: http://localhost:4466/scalar-import/dev
datamodel: datamodel.graphql
seed:
  import: seed.graphql

1. prisma export

2. prisma import --data <zip file name>

This yields the following error

divyendusingh [prisma-2731]$ prisma import --data export-2018-07-12T09:53:58.300Z.zip
Unzipping 11ms
Validating data 7ms

Uploading nodes...
[
  "Failure inserting Course with Id: cjjidmf6j00an0788mhw3yhl0. Cause: java.sql.SQLIntegrityConstraintViolationException:
Duplicate entry 'cjjidmf6j00an0788mhw3yhl0' for key 'PRIMARY'",
  "Failure inserting RelayRow with Id: cjjidmf6j00an0788mhw3yhl0. Cause: java.sql.SQLIntegrityConstraintViolationException
: Duplicate entry 'cjjidmf6j00an0788mhw3yhl0' for key 'PRIMARY'"
]

Validating if this and the original error are same/related. Will create a separate issue if needed.

[divyenduz]

I can confirm that these are separate issues indeed. For the originally reported replication, the generated export is

{
  "valueType": "lists",
  "values": [
    {
      "_typeName": "Course",
      "id": "cjjiek1xs00cg07882vzpzxq1",
      "colors": ["red"]
    },
    {
      "_typeName": "Course",
      "id": "cjjiek1xs00cg07882vzpzxq1",
      "tools": ["tool 1"]
    }
  ]
}

For the following seed.graphql

mutation {
  createUser(
    data: {
      courses: {create: {tools: {set: ["tool 1"]}, colors: {set: ["red"]}}}
    }
  ) {
    id
  }
}

Adding the missing required scalar lists to the NDF removes the validation error and yields the constraint error.

[marktani]

@divyenduz, a question for your first comment: have you run prisma reset between 4. and 5.? If not, then this behaviour is the expected behaviour. On 1.12-beta I get a different error message:

prisma import --data export.zip
Unzipping 15ms
Validating data 8ms

Uploading nodes...
[
  "Failure inserting Course with Id: cjjihlkvn006w0914ag58kejq. Cause: duplicate key value violates unique constraint \"Course_pkey\"\n  Detail: Key (id)=(cjjihlkvn006w0914ag58kejq) already exists.",
  "Failure inserting RelayRow with Id: cjjihlkvn006w0914ag58kejq. Cause: duplicate key value violates unique constraint \"_RelayId_pkey\"\n  Detail: Key (id)=(cjjihlkvn006w0914ag58kejq) already exists."
]

This is expected, as you try to import nodes with the same id which is not handled in any special way as per the specification and leads to undefined/faulty behaviour (quoting the documentation):

Note that import operations are not idempotent. This means running an import always adds data to your service. It never updates existing nodes. This means importing the same dataset multiple times will lead to undefined behaviour.
For example, importing a node with the same id more than once will lead to undefined behaviour and likely break your service!

[marktani]

@divyenduz, regarding your second comment:

The documentation is not clear how nodes with two scalar lists are formatted in NDF. This is what this issue boils down to.

I see two options for the expected format in NDF:

1. One entry per node and scalar list

The originally exported data has one entry per node and per scalar list. If there are 5 nodes that all have the scalar lists tools and colors set, we have 10 entries in this export. In your example, we have two entries for cjjiek1xs00cg07882vzpzxq1 in total, one for tools and colors each:

{
  "valueType": "lists",
  "values": [
    {
      "_typeName": "Course",
      "id": "cjjiek1xs00cg07882vzpzxq1",
      "colors": ["red"]
    },
    {
      "_typeName": "Course",
      "id": "cjjiek1xs00cg07882vzpzxq1",
      "tools": ["tool 1"]
    }
  ]
}

I believe this to be the correct format for NDF.

1. One entry per node, all scalar lists combined

Another possibility would be to combine all scalar lists for a given node into one entry. For 5 nodes, we have 5 entries with an entry for tools and colors each. On your example dataset, this is the result for the only node cjjiek1xs00cg07882vzpzxq1:

{
  "valueType": "lists",
  "values": [
    {
      "_typeName": "Course",
      "id": "cjjiek1xs00cg07882vzpzxq1",
      "colors": ["red"],
      "tools": ["tool 1"]
    }
  ]
}

---

In any case, I am not sure what the expected NDF is. We should clarify this and update the documentation accordingly.

[divyenduz]

@marktani : Good catch, I had missed running prisma reset indeed.

Next point is to find the correct NDF format. Currently generated NDF resembles 1.

[timsuchanek]

Thanks for the good overview everyone!
The correct NDF format is indeed 1.
The CLI is incorrectly checking for all fields to be existing in the lists type. Here one scalar field should be sufficient.
We chose format 1 because it's easier to split big lists.

[marktani]

This is now available at 1.13-beta.

[marktani]

Fixed in 1.13.