Amplify-cli: Schema Migration

[rohanmuplara]

I have an existing aws amplify schema with data deployed to dynamodb tables. I want to change the aws amplify schema. When I change the schema, how do I include the data in my old tables and migrate them to the new tables created by aws amplify.

[attilah]

@rohanmuplara data migration is that is not supported and out of scope for the CLI. You have to use a DynamoDB supported data migration method to make it happen.

I'm closing the issue but if you have a new related question feel free to reopen it.

[hisham]

@attilah - what are the dynamodb support data migration methods? I have not seen any besides adhoc scripts. Links to these supported methods would be appreciated.

The reality is the Amplify CLI's is many people's first entry on building full stack apps over purely AWS tech. The GraphQL transformer makes it easy for people to create tables and relational models. We can't then say, once a person has a real app with real data, oh you need data migration? Sorry that's out of scope.

Sure if it's out of scope for CLI fine, but at least point us to some best practices.

Currently in my app for data migration, I've done it through ad hoc scripts, and also used Apollo Links to migrate data on the fly in the frontend.

I'm still looking for better ways, and there doesn't seem to be standard ways to do it like what people are used to from a Rails or Django type backend.

[rohanmuplara]

@attilah
Is it on the roadmap to support migrations? How do you expect to support production applications?

[ambientlight]

@hisham:

1. you might consider going with lambda called from CI on deployments that will run sequential scan with modelVersion filter (you need to store modelVersion in your records) and then call batchWriteItem for old modelVersion items to update. Though this method can be expensive when your dynamoDB table is large.
2. you can write custom resolvers for your queries that will do what's referred to as lazy migration. On user queries, fetch data from dynamoDB, check the modelVersion and then update back only when item is actually requested.

The GraphQL transformer makes it easy for people to create tables and relational models. We can't then say, once a person has a real app with real data, oh you need data migration? Sorry that's out of scope.

It is not yet there to be used as a sole tool if you just rely on amplify-cli out-of-the-box functionality. To me it's great use in provisioning and deploying cloud formation, this is quite handy with appsync. (off topic but) I personally use it together with Django + Zappa + PynamoDB. django is great with relational with its ORM and with zappa you can pack it into lambda and it has build-in support for more sugar like remote commands, CloudWatch event rules. PynamoDB gives ORM-like abstraction over dynamo, though a tricky part is a need to maintain a pynamoDB model that matches amplify graphql @model definition. for the example full-scan migration for me looked like:

def full_scan_migrate():
    full_fixtures_scan = Fixture.scan(
        filter_condition=(Fixture.modelVersion != FIXTURE_MODEL_VERSION))
    fixtures = list(full_fixtures_scan)
    for fixture in fixtures:
        migrate(fixture)

    with Fixture.batch_write() as batch:
        for fixture in fixtures:
            batch.save(fixture)


def migrate(fixture: Fixture):
    # added start time and changed status
    if fixture.modelVersion == 0 and FIXTURE_MODEL_VERSION == 1:
        fixture.startTime = datetime.utcfromtimestamp(fixture.startTs)
        fixture.status = GameStatus.from_status(fixture.status).value
        fixture.gsi1SortKey = '#'.join([
            fixture.status,
            fixture.competitionId
        ])
        fixture.modelVersion = 1

The reality is the Amplify CLI's is many people's first entry on building full stack apps over purely AWS tech.

You make a good point, and this is an issue, especially when people jump into it after reading posts like using-aws-amplify-feels-like-cheating, since for non out-of-the-box thing, background in CloudFormation is needed to begin with. CLI really abstracts away a set of AWS tech which won't make Amplify feel as a smooth sale for newcomers straight away.

[ambientlight]

@hisham, @rohanmuplara, @attilah: Maybe we can start thinking about RFC for build-in migration support. Build-in lazy migrations can be a viable option given there are pipeline resolvers in place, a resolver in pipeline let's say can call a function category lambda for a custom migration on queries, this can integrate with existing amplify generate @model, @key when that part will utilize pipeline resolvers. Does it make sense to you?

[hisham]

Good points @ambientlight and thanks for the tip on using Lambda upon CI and using modelVersion. I've been delaying rolling out a full fledged solution of mine in hopes that Amplify CLI team will offer some recommended way. After all I'd rather focus my time and energy on the business domain of my app rather than internal tooling. And thanks for the tip on using Django with Amplify, I personally have been eyeing NestJs but I really like Amplify's GraphQL transformer and I feel I already invested into the learning curve involved with the Amplify CLI and AWS in general.

I would love an RFC. cc'ing @undefobj and @attilah for consideration.