Platform: NGRX Offline patterns

[MattiJarvinen]

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[ ] Feature request
[X] Documentation issue or request

What is the current behavior?

We currently have no documented patterns for handling offline Effects calling remote services.

Expected behavior:

We should start discussion on ideas on how to handle offline state changes. Maybe that way we as a developer community get past the we're working on it and more immediate matters taking most of our time.

Other information:

I tried to device most used problem scope, without entity relations to keep things simple at first.

NGRX offline pattern optimistic effects with outbox:

Here I'm trying to describe possible logic for an retry outbox based logic. This example is not complete and I have currently no real idea on how to handle following cases:

• action outbox -- failed item retries as a retry queue handling as a FIFO
• multiple offline updates to same entity with first update failing properly 400
• entity backup -- maybe sneak peak to https://github.com/johnpapa/angular-ngrx-data tracking

Some actions that are perhaps the most needed for offline use are:

• CREATE
• UPDATE
• DELETE

Offline execution of Actions & Reducers & Effects

State description:

export interface TodoState extends EntityState<TodoModel> {
 backupEntities: { 
  [id: string|number]: TodoModel
 }
}

// Outbox should be it's own feature state since it's a FIFO queue
export interface OutboxState {
 fifoHandleIds: string[],
 outbox: {
  [outboxHandle: string]: ADD_TODO|UPDATE_TODO|DELETE_TODO //...
 },
}

CREATE

• ADD_TODO:
  
  
  
  • payload:
    
    
    
    
    
    • action generator / class adds an operation handle if not given
    
    
    
    • action generator / class generates id as UUID if not given
    
    
    
  
  
  • reducer:
  
  
  • handles the optimistic creation
  
  
  • effect:
    
    
    
    
    
    • calls REST service without added UUID
    
    
    
  
  
• ADD_TODO_SUCCESS:
  
  
  
  • payload:
    
    
    
    
    
    • generated UUID from ADD_TODO payload
    
    
    
    • operation handle from ADD_TODO payload
    
    
    
    • server response as the payload
    
    
    
  
  
  • reducer:
    
    
    
    
    
    • updates entity in store to new UUID with payload
    
    
    
    • checks if operation handle is in outbox
    
    
    
    • if it is remove related entry ( this was a retry )
    
    
    
    • if it isn't do nothing
    
    
    
  
  
• ADD_TODO_ERROR:
  
  
  
  • payload:
    
    
    
    
    
    • original ADD_TODO action
    
    
    
    • server response
    
    
    
  
  
  • reducer:
    
    
    
    
    
    • if server response 500 || timeout
    
    
    
    • adds the whole action into outbox state
    
    
    
    • if server response 400 remove original added UUID entity
    
    
    
  
  

UPDATE

• UPDATE_TODO:
  
  
  
  • payload:
    
    
    
    
    
    • action generator / class adds an operation handle if not given
    
    
    
  
  
  • reducer:
    
    
    
    
    
    • takes backup of the original entity
    
    
    
    • handles the optimistic update
    
    
    
  
  
  • effect:
    
    
    
    
    
    • calls the REST service
    
    
    
  
  
• UPDATE_TODO_SUCCESS:
  
  
  
  • payload:
    
    
    
    
    
    • operation handle
    
    
    
    • server response as the payload
    
    
    
  
  
  • reducer:
    
    
    
    
    
    • checks if operation handle is in outbox
    
    
    
    • if it is remove related entry ( this was a retry )
    
    
    
    • if it isn't do nothing
    
    
    
  
  
• UPDATE_TODO_ERROR:
  
  
  
  • payload:
    
    
    
    
    
    • original UPDATE_TODO action
    
    
    
    • server response
    
    
    
  
  
  • reducer:
    
    
    
    
    
    • if server response 500 || timeout
    
    
    
    • adds the whole action into outbox state
    
    
    
    • if server response 400
    
    
    
    • remove updates made by using backup
    
    
    
    • remove backup
    
    
    
  
  

DELETE

• DELETE_TODO:
  
  
  
  • payload:
    
    
    
    
    
    • action generator / class adds an operation handle if not given
    
    
    
  
  
  • reducer:
    
    
    
    
    
    • takes backup of the original entity
    
    
    
    • handles the optimistic store delete
    
    
    
  
  
  • effect:
    
    
    
    
    
    • calls the REST service
    
    
    
  
  
• DELETE_TODO_SUCCESS:
  
  
  
  • payload:
    
    
    
    
    
    • operation handle
    
    
    
    • server response as the payload
    
    
    
  
  
  • reducer:
    
    
    
    
    
    • checks if operation handle is in the outbox
    
    
    
    • if it is remove related entry ( this was a retry )
    
    
    
    • if it isn't do nothing
    
    
    
  
  
• DELETE_TODO_ERROR:
  
  
  
  • payload:
    
    
    
    
    
    • original DELETE_TODO action
    
    
    
    • server response
    
    
    
  
  
  • reducer:
    
    
    
    
    
    • if server response 500 || timeout
    
    
    
    • adds the whole action into outbox state
    
    
    
    • if server response 400
    
    
    
    • restore deleted entity using backup
    
    
    
    • remove backup
    
    
    
  
  

If outbox != empty and we're "online" retry FIFO outbox action with decaying retry interval ( up to a point like max 30min interval or so).

On logout clear outbox... everything in the outbox is lost.

[Valindo]

Hey is there any update on this? Can we expect this soon?

[michaeljota]

Do you think this can be doable with the new pipe in @ngrx/effects, the act pipe? Or maybe a similar pipe, that allows to dispatch an optimistic action, and then a revert action if anything went wrong after retrying.

What do you think?

[leonardochaia]

@MattiJarvinen interesting thooughts. Appreciate it.

I'm about to dig into this topic and any updated guidance is appreciated.
I've also seen a comment from @MikeRyanDev, was there any advancements on this topic?

Thank you guys. 🙏

[wundo]

Hi @leonardochaia
were you able to dig into this? what were your findings?

Cheers

[leonardochaia]

Hi @wundo, unfortunately I was delayed with other projects and haven't been able to dig into this. It's still on the backlog so I'll definitively take a look soon.

[AdditionAddict]

@leonardochaia any update on progress or findings?

Personally I think if we're going to look at offline, let's do it properly with an offline storage like indexeddb. At least that's my opinion. And thinking more fully about offline (imagine 8 hours offline rather than intermittent) forces us to cover more bases.

page 45 of Online and Offline Operation ofJ2EE Enterprise Applications details the possible out-of sync states for a local and replica of a data object (or entity in our framework thinking) and outcomes/conflicts which I've put below.

Local down, replica across. Replica is most likely the server but could be any two data stores of objects you want to sync. Most of bracketed terms can occur if there is more than one data store (think sever iPad, desktop etc)
| | Unchanged | Added | Updated | Deleted | Not Existing
-- | -- | -- | -- | -- | --
Unchanged | (ignore) | (may not yet exist in local) | UPDATE_LOCAL | DELETE_LOCAL | (may not be Unchanged in local)
Added | (may not exist in replica) | (same object may not be added in both local / replica) | (may not yet exist in replica) | (may not exist in local) | CREATE_REPLICA
Updated | UPDATE_REPLICA | (may not exist in local) | (conflict) UPDATE_LOCAL / UPDATE_REPLICA | (conflict) DELETE_LOCAL / UPDATE_REPLICA | CREATE_REPLICA
Deleted | DELETE_REPLICA | (may not exist in local) | (conflict) UPDATE_LOCAL / DELETE_REPLICA | (ignore) | DELETE_LOCAL
Not Existing | (may not be Unchanged in replica) | CREATE_LOCAL | CREATE_LOCAL | DELETE_REPLICA | (impossible)

I've replaced the clear state for unchanged and modified for updated to match @ngrx/data.

Synchronization Process

Store A sends list of updates (Added / Updated / Deleted) to Store B
Store B merges updates and reconciles conflicts
Store B creates list of updated objects (Added / Updated / Deleted)
Apply updates to Store A (Added / Updated / Deleted)

"The resolution of a conflict is called reconciliation and is always performed on the data store receiving updates from another data store."

Conflict Strategy

Client Wins
Server Wins
Last Change Wins (if entities have last updated timestamps)
First Change Wins (if entities have last updated timestamps)

Upshot is that if @ngrx/data could provide some basics syncing could be made possible and extended out of this with its changeState though I do worry how much support @ngrx/data has.

[samratarmas]

Hello @AdditionAddict , I have been following this thread of offline and sync design and recently looked at the issue number 2359 that you closed on the @ngrx/data side, after the team showing no interest in adding the sync capabilities. Just out of curiosity have you been still working on it? Would love to hear about it, regards!

[AdditionAddict]

Hello @AdditionAddict , I have been following this thread of offline and sync design and recently looked at the issue number 2359 that you closed on the @ngrx/data side, after the team showing no interest in adding the sync capabilities. Just out of curiosity have you been still working on it? Would love to hear about it, regards!

@samratarmas I'm currently exploring ideas. To be fair to the team they can't work with wishy washy proposals that are already covered by change tracking (didn't know this in depth at the time). They need concrete ideas that won't affect current users negatively and falls within reactive scope. I still think @ngrx/data is the basis on which to proceed rather than start from scratch, but it may be a case of breaking into it at key points by extending some current classes and providing the suite of classes to deal with offline behaviour.

The main crux with the current codebase is that the main EntityEffect assumes the app is in an 'Online' mode which could occasionally fail rather than allowing a switch between 'Online' or 'Offline' modes. This is the natural place to decide if we are in 'Online' or 'Offline' mode is here in the effect.

Therefore to support offline first means extending EntityEffects with EntityEffectsNetworkAware and replacing the persist$

  persist$: Observable<Action> = createEffect(() =>
    this.actions.pipe(
      ofEntityOp(persistOps),
      mergeMap(action => {
        const online$ = this.onlineCheckService.online$();
        return of(action).pipe(
          /** Lazy check online stream */
          withLatestFrom(online$),
          switchMap(([action, isOnline]) => {
            if (isOnline) {
              return this.persist(action);
            } else {
              return this.offlinePersist(action);
            }
          })
        );
      })
    )
  );

My current implementation is to mimick persist() with offlinePersist() so that say a QUERY_ALL maps to QUERY_ALL_OFFLINE_SUCCESS or QUERY_ALL_OFFLINE_ERROR for example.

Current Model

Potential Model

There are natural patterns for switching between 'Online' or 'Offline' modes combining naviator.online (converted to stream of course), a user toggle if they wish to have more direct control and an API circuit breaker pattern based of _Angular Design Patterns by Mathiey Nayrolles page 119_

Saturating store

For each store / entityName:

• Get from indexeddb:- store.getAll()
• create { unchanged, added, deleted, updated } via reduce
• populate store via EntityServices getEntityCollectionService
• use cache methods to populate ngrx store upsertManyInCache / addManyToCache / updateManyInCache / removeManyFromCache

Offline Behaviour

Add / Delete / Update → app uses NgRx, persist via IndexedDB / localStorage (main thing will be API that could be implemented separately by user)

Online Behaviour

I need to think more carefully about this but roughly speaking (for my use case anyway)

• Sync per store - Developer to provide entity order Parent <--> Child
• Queries require preservechanges strategy
• Effects to keep indexeddb in line with ngrx
• Need to solve the 'Out of collection' problem (websockets should normally be used but I don't have this option)
• Need to think really really really hard about updates...

I'm currently struggling with is indexeddb and understanding the current @ngrx/data codebase to the level I need to (not looked at architecture in this manner before) and need to layout the models more concretely. Next step is to lay everything out in typescript models...