We include the contact hierarchy in all data records and contacts. This duplicates the data which means updates don't propagate. It also means each record and contact is approximately double as big as it should be requiring more data to be replicated and more disk space on the device.
Investigate using joins with views to get all the required data in one query without duplicating it.
garethbowen
All 24 comments
Given that nested contacts are not updated when the original contact is updated, I think this is more than technical debt because users expect a change to a contact's info to be reflected in all related contacts. We should aim to get to this as early as feasible.
abbyad
on 2 Dec 2016
sglangevin
on 2 Feb 2017
sglangevin
on 2 Feb 2017
After talking with @sglangevin we agreed that this isn't the top priority in terms of technical debt or the top priority bug to be fixed, so I've bumped it out of this iteration. It's still important and will still be done.
garethbowen
on 13 Feb 2017
This may have some impact on analytics. Curious what level of nesting we will be removing.
In the example below where type = 'person', would we be...
(a) Removing all contents in the curly brace (just leaving the _id); or
(b) Would we just be removing the element pointed at by the arrow ({parent,parent})
If (b), would we remove that element altogether? Or would we replace it with the _id of the parent's parent? Sorry, not sure how best to represent / explain this.
michaelkohn
on 11 Apr 2017
Pretty sure any parent or contact properties would just be a string with a UUID in it.
mandric
on 11 Apr 2017
@michaelkohn Gareth can give the most correct answer as he's doing the work, but as I understand it we'd be effectively doing option a, except that the entire value of parent would be the id (ie not parent: { _id: 'foo' } but instead parent: 'foo').
SCdF
on 11 Apr 2017
@michaelkohn @SCdF @mandric I'm still prototyping but I think we'll have to have a list of all ancestors, so parents: [ "a", "b", "c", ... "x" ]. The structure may change from this but IDs of all ancestors will be available in the doc.
garethbowen
on 11 Apr 2017
@garethbowen I recognize this is still WIP... but I'm curious what the current thinking is in terms of updates that happen higher up in the hierarchy. So if you move a health_center from one district_hospital to another or one clinic to a different health_center, those updates would cascade down and update the parents: [ "a", "b", "c", ... "x" ] for all person docs, correct?
michaelkohn
on 18 Apr 2017
@michaelkohn No, unfortunately each doc will have to hold a reference to their entire hierarchy, so a report will store the IDs for its contact, its contacts parent, grandparent, etc. This is because couchdb views cannot recursively fetch other docs. Any other field will be updated automatically however.
My recommendation is that changing a hierarchy is part of a separate api that searches for and updates all affected contacts and reports. This is potentially an expensive operation, but I can't see any technical solution that's still efficient for querying.
garethbowen
on 18 Apr 2017
Thanks @garethbowen ... so without that separate API, the parents array on a data_record would always just be a snapshot of the hierarchy at the time the data_record was submitted.
From what I can tell, when you transfer a CHW to a different place, the hierarchy stored on the CHW's person record is already getting updated properly. This is actually more important to me than the data_records hierarchy because my queries always join the contact on the data_record with config data (hierarchies derived from contacts) to determine which places the CHW belongs to.
michaelkohn
on 18 Apr 2017
@mandric @estellecomment Would you do me the honour of review this enormous PR?
Now when creating reports instead of inlining the entire contact we just store the IDs required to hydrate the contacts later.
Primarily this means that contact information will always be up to date (with the one exception of contact hierarchies), so if a contacts phone number or name is changed then this will be reflected throughout the UI without having to update all docs.
It also means we store and transfer much less data which in turn improves database performance.
The expected contact information is stored as contact: { _id: 'a', parent: { _id: 'b' } }. This is a slightly clunky style but means the feature is backwards compatible so any existing documents that have inlined contacts will work, and the obsolete inlined data will be ignored.
I recommend reviewing in this order...
I plan to write up a doc explaining the technical side of how this works tomorrow to share the knowledge which may help with this review.
Leave the actual merging to me...
garethbowen
on 17 May 2017
Google Doc explaining the concept
garethbowen
on 18 May 2017
Merged! Thanks @estellecomment and @mandric for fantastic and detailed reviews. If I missed some feedback in that epic review I'm happy to change in master but I needed to merge for my sanity...
garethbowen
on 24 May 2017
@garethbowen does this change affect how will be accessing nested elements from within an xform? If yes, what is the suggested way?
derickl
on 14 Aug 2017
Yes, this will affect that. You'll be able to access any fields about the selected contact the normal way. For any fields you're using from the parent, you'll need to pull those field via the contact-summary and make them available to the form, the same way you'd be making any other data available to the form via contact-summary. I'm planning to write up some info about this for projects that are moving from v2.12 (or earlier) to v2.13.
sglangevin
on 15 Aug 2017
The lineage in the contact-summary for places and people does not work on an instance upgraded to 2.13.0-beta.4. Normally the parent places are listed as links in the Belongs to field.
abbyad
on 15 Aug 2017
@abbyad I tested that and it was working in beta.3, perhaps this regressed?
sglangevin
on 15 Aug 2017
You have to change the configuration of your contact-summary for it to work.
sglangevin
on 15 Aug 2017
Is the only change to have return result; instead of just result; at the end?
abbyad
on 15 Aug 2017
No. You also have to update the "Belongs to" field.
label: 'contact.parent', value: lineage, filter: 'lineage'
sglangevin
on 15 Aug 2017
Aha, the value of value changes from contact.parent to lineage . Works now.
Is everything else sufficient acceptance tested to move back to Ready?
abbyad
on 15 Aug 2017
Yeah, I think so. I've done a fair amount of testing already and filed several bugs which have been fixed that are related to this change.
sglangevin
on 15 Aug 2017
Ok, closing and returning to Ready.
abbyad
on 15 Aug 2017
Related issues
SCdF
路
3Comments
ngaruko
路
3Comments
alxndrsn
路
6Comments
diannakane
路
6Comments
alxndrsn
路
4Comments
Most helpful comment
Merged! Thanks @estellecomment and @mandric for fantastic and detailed reviews. If I missed some feedback in that epic review I'm happy to change in master but I needed to merge for my sanity...