Amplify-cli: Ability to specify existing dynamodb source when adding Graphql api.

[tarunguddeti]

When adding graphql api , the amplify cli is creating a new dynamodb table and using it as a Data Source for Graphql. It would be really useful if there could be a feature where we can add an existing Dynamodb Table as the data source.

[kaustavghosh06]

@tarunguddeti That's a great question. Could you please open this as in issue/feature request in this repo -> https://github.com/aws-amplify/amplify-cli

[jordanranz]

I am moving this issue to the repo mentioned above.

[mikeparisstuff]

@tarunguddeti Thanks for the question. This have been discussed multiple times and is a feature that we would like to support. We can use this ticket to kick off a discussion around the design of this feature.

Updates to @model

As I see it, the most obvious hook point would be to allow @model to specify an existing table by name & region. It is technically possible to call tables in different accounts but since this will require extra config in the other account, I suggest we do not support cross account access at first. I propose an updated @model directive.

directive @model(
    queries: ModelQueryMap,
    mutations: ModelMutationMap,
    subscriptions: ModelSubscriptionMap,
    table: TableConfig,
) on OBJECT
# You must provide a name. The region will default to the same region as the stack.
input TableConfig { name: String!, region: String }

# These are the same as before
input ModelMutationMap { create: String, update: String, delete: String }
input ModelQueryMap { get: String, list: String }
input ModelSubscriptionMap {
    onCreate: [String]
    onUpdate: [String]
    onDelete: [String]
}

If you already had a table named "Tasks" and you wanted to front it by an AppSync API, then you could use the following markup:

type Task @model(table: { name: "Tasks" }) {
  id: ID!
  title: String!
  content: String
}

This would result in the creation of the data source, data source iam role, and the CRUDL+sub resolvers but would target the existing table in your account by name instead of creating a new one. This solves part of the problem. Since the transform will no longer be administering the table, we need a way to inform the transform of the key schema and indexes and may also want to consider adding new directives for different access patterns.

The @index directive

We have plans to introduce an @index directive (see the design #314) that will allow for arbitrary key & index structures on all @model types. I view this as a pre-req for this feature. In terms of building support for new access patterns that allows querying indexes at the top level as well as using them to connect related data.

Updated @connection and/or new directives for plain DynamoDB operations.

Currently the @connection directive is the only way to configure DynamoDB query operations (without using custom resolvers). The @connection directive does a few things:
1) It creates a GSI (in one-to-many situations)
2) It configures a DynamoDB query operation against that index
3) It passes along the parent object's $ctx.source.id as the hash key in the DynamoDB query which allows for the efficient look up of related info

A @query directive

This is a design to allow users to bring their own (pre-configured) tables, so it seems like it would be useful to have a directive that is capable of doing steps 2 & 3 without doing step 1. For example, I propose a new @query directive.

directive @query(index: String, mapToKey: KeyMapping) on FIELD_DEFINITION
input KeyMapping { hash: String!, sort: String }

Compounding on our example above, let's say the Task table used a primary key schema of (projectId [HASH], taskId [SORT]). Let's say it also provided a global secondary index on the (ownerId [HASH], taskId[SORT]) fields to allow users to easily lookup their owned items. You should be able to model this in its entirety using the new design. For example,

# If using Amplify CLI storage tables with multienv, you could pass  "Tasks-${env}". 
type Task @model(table: { name: "Tasks" }) {
  projectId: ID! @index(type: HASH)
  taskId: ID! @index(type: SORT) @index(name: "OwnerIndex", type: SORT)
  ownerId: ID! @index(name: "OwnerIndex", type: HASH)
  title: String!
  content: String
}

Let's say you had another table, Comments, that had a column that related it to a task by taskId.

type Task @model(table: { name: "Tasks" }) {
  # ... fields from above

  # Add a query operation. Omitting the "index" defaults to using the table's primary key.
  # The field mappings allow us to map a source attribute to an attribute in the other index.
  # I.E. the resultant query would allow you supply a sort key expression but the hash key would
  # be auto-set to $ctx.source.taskId.
  comments: [Comment] @query(mapToKey: { hash: "$ctx.source.taskId" }])
}
type Comment @model(table: { name: "Comments" }) {
  taskId: ID! @index(type: HASH)
  createdAt: ID! @index(type: SORT)
}

An updated @connection directive

As an alternative approach, we could change the @connection directive to support similar behavior as above.

# Currently we have
directive @connection(name: String, keyField: String, sortField: String) on FIELD_DEFINITION

# We could change to
directive @connection(
  # You could use name, keyField, & sortField to create GSIs as usual.
  name: String, keyField: String, sortField: String,

  # OR you can use index/mapToKey to use already existing indexes. You cannot mix and match.
  # For backwards compat, you need to pass "primary" (or similar) to hit a table's primary key 
  # because @connection without args already has behavior.
  index: String, mapToKey: KeyMapping
) on FIELD_DEFINITION

The same example as above would look like this with @connection:

type Task @model(table: { name: "Tasks" }) {
  # ... fields from above

  # Add a query operation. Omitting the "index" defaults to using the table's primary key.
  # The field mappings allow us to map a source attribute to an attribute in the other index.
  # I.E. the resultant query would allow you supply a sort key expression but the hash key would
  # be auto-set to $ctx.source.taskId.
  comments: [Comment] @connection(index: "primary", mapToKey: { hash: "$ctx.source.taskId" }])
  # mapToKey is optional and if not provided, the user would have to provide the taskId
  # explicitly in the query. e.g. `query { task { comments(taskId: "123") { ... } }` which
  # is not ideal in this scenario.
}
type Comment @model(table: { name: "Comments" }) {
  taskId: ID! @index(type: HASH)
  createdAt: ID! @index(type: SORT)
}

Requesting feedback

Let me know what you think. I am particularly interested in what you think of augmenting @connection vs adding new directives @query/@getitem etc. The KeyMapping concept is very fresh, and I imagine there are many ways to accomplish a similar goal so would be interested in hearing more options.

[kevinwolfdev]

Would be nice if we can add tables generated by amplify add storage on some similar way that we do with the current @function directive.

type User @model(table: "UsersTable-${env}") {
  id
}

[clinicalinkgithub]

@tarunguddeti Thanks for the question. This have been discussed multiple times and is a feature that we would like to support. We can use this ticket to kick off a discussion around the design of this feature.

Updates to @model

As I see it, the most obvious hook point would be to allow @model to specify an existing table by name & region. It is technically possible to call tables in different accounts but since this will require extra config in the other account, I suggest we do not support cross account access at first. I propose an updated @model directive.

directive @model(
    queries: ModelQueryMap,
    mutations: ModelMutationMap,
    subscriptions: ModelSubscriptionMap,
    table: TableConfig,
) on OBJECT
# You must provide a name. The region will default to the same region as the stack.
input TableConfig { name: String!, region: String }

# These are the same as before
input ModelMutationMap { create: String, update: String, delete: String }
input ModelQueryMap { get: String, list: String }
input ModelSubscriptionMap {
    onCreate: [String]
    onUpdate: [String]
    onDelete: [String]
}

If you already had a table named "Tasks" and you wanted to front it by an AppSync API, then you could use the following markup:

type Task @model(table: { name: "Tasks" }) {
  id: ID!
  title: String!
  content: String
}

This would result in the creation of the data source, data source iam role, and the CRUDL+sub resolvers but would target the existing table in your account by name instead of creating a new one. This solves part of the problem. Since the transform will no longer be administering the table, we need a way to inform the transform of the key schema and indexes and may also want to consider adding new directives for different access patterns.

The @index directive

We have plans to introduce an @index directive (see the design #314) that will allow for arbitrary key & index structures on all @model types. I view this as a pre-req for this feature. In terms of building support for new access patterns that allows querying indexes at the top level as well as using them to connect related data.

Updated @connection and/or new directives for plain DynamoDB operations.

Currently the @connection directive is the only way to configure DynamoDB query operations (without using custom resolvers). The @connection directive does a few things:

1. It creates a GSI (in one-to-many situations)
2. It configures a DynamoDB query operation against that index
3. It passes along the parent object's $ctx.source.id as the hash key in the DynamoDB query which allows for the efficient look up of related info

A @query directive

This is a design to allow users to bring their own (pre-configured) tables, so it seems like it would be useful to have a directive that is capable of doing steps 2 & 3 without doing step 1. For example, I propose a new @query directive.

directive @query(index: String, mapToKey: KeyMapping) on FIELD_DEFINITION
input KeyMapping { hash: String!, sort: String }

Compounding on our example above, let's say the Task table used a primary key schema of (projectId [HASH], taskId [SORT]). Let's say it also provided a global secondary index on the (ownerId [HASH], taskId[SORT]) fields to allow users to easily lookup their owned items. You should be able to model this in its entirety using the new design. For example,

# If using Amplify CLI storage tables with multienv, you could pass  "Tasks-${env}". 
type Task @model(table: { name: "Tasks" }) {
  projectId: ID! @index(type: HASH)
  taskId: ID! @index(type: SORT) @index(name: "OwnerIndex", type: SORT)
  ownerId: ID! @index(name: "OwnerIndex", type: HASH)
  title: String!
  content: String
}

Let's say you had another table, Comments, that had a column that related it to a task by taskId.

type Task @model(table: { name: "Tasks" }) {
  # ... fields from above

  # Add a query operation. Omitting the "index" defaults to using the table's primary key.
  # The field mappings allow us to map a source attribute to an attribute in the other index.
  # I.E. the resultant query would allow you supply a sort key expression but the hash key would
  # be auto-set to $ctx.source.taskId.
  comments: [Comment] @query(mapToKey: { hash: "$ctx.source.taskId" }])
}
type Comment @model(table: { name: "Comments" }) {
  taskId: ID! @index(type: HASH)
  createdAt: ID! @index(type: SORT)
}

An updated @connection directive

As an alternative approach, we could change the @connection directive to support similar behavior as above.

# Currently we have
directive @connection(name: String, keyField: String, sortField: String) on FIELD_DEFINITION

# We could change to
directive @connection(
  # You could use name, keyField, & sortField to create GSIs as usual.
  name: String, keyField: String, sortField: String,

  # OR you can use index/mapToKey to use already existing indexes. You cannot mix and match.
  # For backwards compat, you need to pass "primary" (or similar) to hit a table's primary key 
  # because @connection without args already has behavior.
  index: String, mapToKey: KeyMapping
) on FIELD_DEFINITION

The same example as above would look like this with @connection:

type Task @model(table: { name: "Tasks" }) {
  # ... fields from above

  # Add a query operation. Omitting the "index" defaults to using the table's primary key.
  # The field mappings allow us to map a source attribute to an attribute in the other index.
  # I.E. the resultant query would allow you supply a sort key expression but the hash key would
  # be auto-set to $ctx.source.taskId.
  comments: [Comment] @connection(index: "primary", mapToKey: { hash: "$ctx.source.taskId" }])
  # mapToKey is optional and if not provided, the user would have to provide the taskId
  # explicitly in the query. e.g. `query { task { comments(taskId: "123") { ... } }` which
  # is not ideal in this scenario.
}
type Comment @model(table: { name: "Comments" }) {
  taskId: ID! @index(type: HASH)
  createdAt: ID! @index(type: SORT)
}

Requesting feedback

Let me know what you think. I am particularly interested in what you think of augmenting @connection vs adding new directives @query/@getitem etc. The KeyMapping concept is very fresh, and I imagine there are many ways to accomplish a similar goal so would be interested in hearing more options.

@mikeparisstuff is there any movement on this with @model(table: { name: "Tasks" }) option???

[hisham]

One variation of this feature that would be handy to me: specify dynamodb table name to connect to in one env (prod), but have default tables that would be created by Amplify used in other envs.

[khaschuluu]

+1

[Zaib10]

@clinicalinkgithub
i'm facing issue while connecting to exiting tables in dynamoDB is there any other way to connect to the existing tables? "amplify push" creates new tables
i'm also tested with your given code i'm facing some issue
Here is my code I want to connect my existing table Planet.

directive @model(
queries: QueryInput,
mutations: MutationInput,
subscriptions: SuscriptionInput,
table: TableConfig,
) on OBJECT
input TableConfig { name: String!, region: String }

input MutationInput { guid: String, name: String }
input QueryInput { guid: String, name: String }
input SuscriptionInput {
onCreate: [String]
onUpdate: [String]
onDelete: [String]
}
type Planet @model(table: { name: "Planet" }) @key(fields: ["guid"]) {
guid: String!
name: String!
}

And when i run "amplify push " it gives the following error

Error: Unknown argument "table" on directive "@model".

There can be only one directive named "model".

[Zaib10]

@clinicalinkgithub @hisham @khaschuluu @yuth @mikeparisstuff @kevinwolfdev
We are looking forward for your help this is urgent for me please participate with your experience with an mentioned issue thanks in advance.