Pouchdb: Deleted documents not syncing when using filter and put()

[ddolheguy]

Issue

I am using the below setup with SYNC in my React Native app with PouchDB (pouchdb-core 6.0.7) and I found that when I delete a document (using put with _delete:true) on the client, it wasn't replicating back to the server at all. From the server to the client works fine, but not from the client back.

BUT, if I include "|| doc._deleted === true;" in the _design/expense document below at the end of the current javascript line, it works fine.

Can i ask? When PouchDB performs a sync on a _deleted document, does it only send the _ID, _REV and _DELETED fields? I'm starting to think that my CouchDB isn't getting the field "userKey" even though I'm including it when I perform a PUT and hence the filter is ignoring it back to the server?

this.db.sync(remoteDb, {
   live: true,
   retry: true,
   filter: 'expense/byuserkey',
   query_params: {userKey: key}
}).on('change', change => { 
     console.log('CHANGE OCCURRED');
});

CouchDB Filter looks like this:

{
  "_id": "_design/expense",
  "_rev": "1-edfa576ff61f1ca52c3c0b80bc698555",
  "language": "javascript",
  "filters": {
    "byuserkey": "function (doc, req) { return doc._id === '_design/expense' || doc.userKey === req.query.userKey }"
  }
}

Info

• Environment: React-Native (AsyncStorage)
• Platform: iOS (React native)
• Adapter: Async Storage
• Server: CouchDB

[nolanlawson]

Your filter function will return false for your deleted document because it doesn't contain a userKey field. The fix is to merely add _deleted: true to the document and then do a put() instead of using remove(). Please see https://pouchdb.com/api.html#filtered-replication for details.

If that doesn't fix your problem, please reopen the issue. :)

[ddolheguy]

Unfortunately that's exactly what I did and the local PouchDb does not replicate the updated document (with the _deleted: true property) to the server.

I performed these steps exactly:

1. Created a new document on the client as follows:

{
   "_id": "244af5e7-5054-4fe9-9476-ad74ececdd50",
   "data": "this is some data",
   "userKey": "91b768b4-b0f7-4850-83c1-c4a30fa87fc6"
}

1. This saves correctly and replicates to the server as expected with the filter settings I listed in the original part of the this issue.

2. I then on the client get the document using PouchDb using db.get('244af5e7-5054-4fe9-9476-ad74ececdd50') which returns the document in step 4 below.

3. I then add _deleted: true to the document and then put the document to PouchDb as follows:

// NOTE: The below document comes from the db.get request, I've put it in a variable for display on this issue.
const document = {
   "_deleted": true,
   "_id": "244af5e7-5054-4fe9-9476-ad74ececdd50",
   "_rev": "1-a8b5bba736b01bf2924ebf3440284038",
   "data": "this is some data",
   "userKey": "91b768b4-b0f7-4850-83c1-c4a30fa87fc6"
}

db.put(document)
   .then(result => {});

After performing the above, I can confirm the 2 things.

1. The client never replicates this document to the server even though the "sync" process is running and I can see PAUSE / ACTIVE requests going through (but no CHANGES requests). Server's version of the document never changes.

1. The document is updated on the client and I see the update come through the change feed of the local db with the flag _deleted: true.

Following the above test which I've performed several times, it looks like filtering might be broken for documents updated with _deleted: true??

[ddolheguy]

Just to add, I debugged this much further and can confirm PouchDB is not sending the entire document to the server which is being saved with the _deleted: true property.

As per below (Although a different document to the above), I did exactly the same thing and updated the document with the _deleted:true and with the "userKey": "91b768b4-b0f7-4850-83c1-c4a30fa87fc6" which I confirmed upon save.

As you can see below, this is the changes HTTP request which is sent to the server when I remove the filtering from the client (so just have fully replication of all documents).

There is no other fields sent to the server other than the ID and REV properties when a updated occurs with the _deleted: true property

Document with _deleted: true set on client

{
    "ok": true,
    "start_time": "2016-11-21T21:37:31.011Z",
    "docs_read": 2,
    "docs_written": 2,
    "doc_write_failures": 0,
    "errors": [],
    "last_seq": 14,
    "docs": [{
        "_deleted": true,
        "_id": "d9aa3644-4e30-4a32-8468-d2645e1a0fb1",
        "_rev": "2-7d75efa183aff0601be68c0a736ba127",
        "_revisions": {
            "start": 2,
            "ids": ["7d75efa183aff0601be68c0a736ba127"]
        }
    }]
}

Finally I can confirm that if i only UPDATE the document and do not set the property _deleted:true then the document sent to the server contains the full document fields including the userKey property.

Document updated without _deleted: true set on client (i.e. normal update)

{
    "ok": true,
    "start_time": "2016-11-21T21:44:16.756Z",
    "docs_read": 2,
    "docs_written": 2,
    "doc_write_failures": 0,
    "errors": [],
    "last_seq": 18,
    "docs": [{
        "data": "this is some data",
        "userKey": "91b768b4-b0f7-4850-83c1-c4a30fa87fc6",
        "_id": "acef0d9e-70c1-4bd2-984b-58bd80ffc61a",
        "_rev": "2-45b7f54b50dfa878a2388a2ae5b14093",
        "_revisions": {
            "start": 2,
            "ids": ["45b7f54b50dfa878a2388a2ae5b14093"]
        }
    }]
}

[nolanlawson]

Can you please provide a live test case to reproduce? It would save us a lot of time in attempting to resolve this issue. Thank you!

[ddolheguy]

Hi Guys,

I've finally found the issue and I'd like to document it here in case anyone else finds this. Firstly PouchDB is fine, the problem is in the AsyncStorage adapter for React Native.

Specifically the issue is with the Bulk_Docs process:
https://github.com/stockulus/pouchdb-react-native/blob/master/packages/pouchdb-adapter-asyncstorage/src/bulk_docs.js#L132-L135

After extensive debugging, I found that when you mark a document as _deleted: true, the AsyncStorage adapter deletes all the properties on the document as part of the Bulk_Docs update except for the deleted, _id and _rev fields.

This breaks design and breaks filtered replication hence why it wasn't working for me.

Next step is to fix it :) I'm going to spend the next couple of days testing and see if I can work out how this needs to be resolved.

[ddolheguy]

Please close this issue as it has now been resolved with a PR i raised today which has been merged. The issue was with the AsyncStorage adapter and not with PouchDB itself.