Amplify-js: DataStore/Conflict Detection/AppSync - ALLOW UNDELETE OF DELETED ITEMS (RECYCLE BIN)

[sacrampton]

Describe the bug
With conflict detection enabled, the schema.graphql that is generated automatically adds the following fields to types. It also needs to add the _ttl value to each type so that you can see when items are going to be removed from DynamoDB

_version: Int!
_deleted: Boolean
_lastChangedAt: AWSTimestamp!

Only the _version is added to input lists (ie: input create* / input update* / input delete*) where you can provide a _version. It is not added to any of the lists that allow you to list/search fields - so for example you can't list all records that are not deleted. If I have 20 records in my database and 5 have _deleted=true then all my queries will always get all 20 records until the _ttl expires and those records are removed. We need to be able to filter for _deleted = false or not exists, but at the moment these fields are not exposed.

Also, if I want to un-delete a record I need the resolver to allow me to be able to set _deleted to false and _ttl = null and at this time I can't access anything other than _version as an input.

Amplify CLI Version

4.19.0

To Reproduce
Try to query for _deleted = true

Expected behavior
Query for all records (list and searchable) where _deleted ne true. Update _deleted to false and delete the _ttl field for undelete

Additional context

I can implement this by modifying the schema, but that is a lot of work to build/maintain on my part and this is a pretty basic function that I think is required by everyone so I'm listing this as something that should be implemented by the Amplify team.

[sacrampton]

@UnleashedMind & @SwaySway - we don't seem to be making much progress with this issue the way I wrote it so I'm going to have another go at doing this.

For the purposes of this example, lets say I have a table/type called Asset. So when I call the deleteAsset resolver it creates a field called _delete = true and _ttl = int (30 days from time of deletion). Of course it also increments the _version and _lastChangedAt values.

With this record "flagged" as deleted, and held there for 30 days before the _ttl fires and deletes the record from DynamoDB, I want to be able to UNDELETE the record. In DynamoDB all I need to do is to delete the _ttl field and it won't be deleted from DynamoDB. But I want to do this through AppSync.

So the first thing I need to be able to do is to see the contents of the Recycle Bin (ie. listAssets(filter:{_deleted:{eq: true}}) } - however the listAssets takes its input from ModelAssetFilterInput and the _deleted field is not part of the ModelAssetFilterInput definition so its impossible to filter for _deleted=true.

So lets say we do know the id of the record in the "recycle bin" that we want to undelete then we should be able to set

updateAsset(input:{
   id: "8ce079c6-1db2-48c1-a5a6-e60c9d22daab"
  _version: 11
  _deleted: null
  _ttl: null
})

To be able to do this the definition of UpdateAssetInput must include _deleted and _ttl - but it doesn't. Even if I manually do this editing of the definitions I can't get it to work where I can null the _deleted & _ttl fields.

[sacrampton]

Hi @SwaySway & @UnleashedMind - understand you guys are busy, but I created this issue a month ago today and haven't had any feedback at all on it, so was wondering if you could take a look. Thanks

[Mentioum]

Also keen to know how we are supposed to filter out _deleted items. I also feel it should be added to the ModelAssetFilterInput definition.

[sacrampton]

If you are using @searchable it is deleting the record from ElasticSearch so any search there will return all live (non-deleted) records. But being able to find in DynamoDB the _deleted records and then undelete them (_deleted: null, _ttl: null) remains something that is a challenge

[undefobj]

Hi @sacrampton I took a read of this issue and I understand you're looking for capabilities to undelete items. The soft delete implemented in AppSync is for consistency guarantees and not to be controlled by the client, which is also why _lastChangedAt is not in the current input. The only reason that _version is in the update input is for the client to provide its current version, but that is still controlled in the service. When we developed this feature in AppSync it was to allow administrators to recover from erroneous deletions in collaborative applications.

To open this up for a "Recycle Bin" like feature, we'll have to investigate this further and look at the consistency implications throughout the system. For now I'm going to mark this as a feature request and measure the interest from other community members as we have a large backlog of DataStore requests. In the meantime if you need this I believe you should be able to address with a custom resolver.

[undefobj]

Transferring to JS repo for tracking.

[sacrampton]

Hi @undefobj - thanks for getting back to me on this - appreciate it.

I've been trying to do this in custom resolvers, but it fails when I try to null the _deleted/_ttl fields through AppSync. With _ttl is says it can't be modified through the client.

[undefobj]

Hi @undefobj - thanks for getting back to me on this - appreciate it.

I've been trying to do this in custom resolvers, but it fails when I try to null the _deleted/_ttl fields through AppSync. With _ttl is says it can't be modified through the client.

You're most likely going to need to do a regular DynamoDB resolver with a custom data source. The AppSync resolvers have constraints on what can be done from a client for good reason - mistakes can lead to data loss and consistency issues.

My .02 I'd think a little more through your business use case for such a feature. It's you're looking to recover from mistakes as an admin, I'd probably recommend a different API to interact with the records as an admin.

[jocarrillo]

How should we filter deleted records from list queries? I'm using appsync and the issue is ModeXXXFilterInput do not accepts the _deleted field as one of the filters...

[renegoretzka]

Just my 2 Cents would be that it would be just enough that Amplify or lets say the API does not spit out those deleted database entries by default. It really makes no sense to still receive them with the API call and handle them with filter (what is actually is, as @jocarrillo has said, not supported by the FilterInput, which makes handling this even worse). I would really appreciate the use of AutoMerge and still can handle filtering or even get rid of those _deleted: true entries in the response.

[ramakrishnan]

@undefobj
Just to add an usecase which I am going through.
Through DataSource I delete an document.

const [original] = await DataStore.query(this.MODEL, conditions)
return DataStore.delete(original)

This action immediately removed the document from IndexedDB, but unfortunately the backend VTL logic prohibits this delete, now how to get the document inserted back to IndexedDB.

One more usecase during create

const newDoc = DataStore.save(new this.MODEL(input))
DataStore.save(newDocument)

• Assume a document is created when system was offline.
• By then a similar document also gets created by other user. (Causing the above created by DataStore as Duplicate Error)
• When DataStore tries to sync it receives a syncError
• As a conflict resolution strategy I try to delete the localModel DataStore.delete(error.localModel)
• A delete mutation happens, which tries to delete a document which is not present in DynamoDB, which again returns an error.

Ideal behaviour would be a mark the document as soft delete and stop calling mutation.