Firebase-js-sdk: ServerTimestampValue locally evaluates to null in Firestore
Describe your environment
- Operating System version: OSX 10.13
- Firebase SDK version: 4.5.0
- Firebase Product: firestore
Describe the problem
In the documentation of ServerTimestampValue it is written:
When evaluated locally (e.g. for snapshot.data()), they evaluate to null.
Ok, but imagine that I have a list of notes created by the user and until data will be synchronized with the firestore server, we will get null, which means that I cannot show when the note was created. In the documentation there is also:
..They can only exist in the local view of a document. Therefore they do not need to be parsed or serialized.
If the field containing ServerTimestampValue is not serialized (that is, value is not send to the server, only field type), then there is no reason why locally it cannot return local time until it will be replaced by the server time.
The fix for that is very simple - just one line to be changed in FieldValue.ts (https://github.com/mmlleevvyy/firebase-js-sdk/commit/c5887dd497ffc436b377d21c27388d1d6ba71901). Tested the change and it works as expected - when saved, the timestamp field returns local time, but after synchronization the returned value is a timestamp set by the server.
Steps to reproduce:
- Create new document, with timestamp field
- Save it
doc.set({timestampField: firestore.FieldValue.serverTimestamp()}) - Until data are synchronized, the data of saved document returns null for timestampField
MarkChrisLevy
All 7 comments
Hmmm this issue does not seem to follow the issue template. Make sure you provide all the required information.
google-oss-bot
on 8 Oct 2017
Thanks for writing in!
Server timestamps are unfortunately extremely tricky. I'll try to explain why we have the behavior we have but the short version is that local clocks keep terrible time and your proposal breaks expectations when timestamps are used to order results.
Consider a list of documents ordered by update time, where update time is a server timestamp. If two documents are updated near each other in time and the local clock is slow their relative position in the list can jump as the first document's server timestamp is fixated then jump again as the second timestamp is fixated.
This problem is something we call "flicker". Initial versions of the RTDB just used the local time as a stand-in for a server timestamp, but this caused a bunch of support load due to exactly this problem.
Later versions of the RTDB included server time in the initial handshake, which allowed the client to compensate for a bad local clock, but this still caused problems in cases where writes preceded the initial connection (because the client was offline altogether or the network was just slow).
Flicker caused by this essentially remains an unsolved problem in the RTDB SDKs. Expectations burned in by existing usage make this unfixable there.
For Firestore we decided that flicker-free ordering was more important than providing a local estimate and one of our top design goals for locally estimated values is to avoid this kind of behavior because it's very tricky for users to defend against.
To avoid reordering flicker, internally server timestamps order themselves always after any actual timestamp (as server timestamps are fixated by the server they become regular timestamps). We use the local write time to order among pending server timestamps.
This gives the intended effect because writes are queued locally and server time is monotonically increasing (it's TrueTime). As each write commits the server timestamp converts to a regular timestamp and the relative position of the server timestamp in the list does not change.
So the trouble that your proposal introduces is that the value of the local estimate of the server timestamp can cause the document to now look out of place in the ordering if the local clock is running slow. That is: your last updated document might appear to have an update time before the last fixated server timestamp and would look out of order even though the ordering is actually correct.
If we supplied the local estimate anyway, we'd have to give users that cared about this case some way to defend against it, by indicating somehow that the timestamp was an estimate. However we decided against this because nobody would read the documentation and the app would be subtly wrong until you got your first bug report.
An alternative would be to return the estimate as some non-Date value but this causes problems when serializing to POJOs on Android and we'd like the contract to be consistent across platforms. If we made our non-Date extend Date and throw if it's used as a date this causes tricky runtime failures that are hard for users to test for since latency compensated views are fundamentally racing with the server.
Normally we try to do the thing that most reasonable people would expect to be the default by default but this is a rare case where there's no good default behavior. A local timestamp might actually work for your case while others would prefer to show a sync-pending icon instead. Unfortunately there's no good way for us to know which is the right behavior so we have to force you to choose for yourself.
I hope this helps you understand why we can't make the change you're proposing.
Thanks!
wilhuff
on 13 Oct 2017
@wilhuff Wow!! Didn't expect so detailed answer. Thank you for your time to write it! Now I understand your point of view and have to say that is reasonable, but maybe you could provide ability to configure, how it should work locally? In my case it is better to show locale date (and have flickering) instead of "null"/unknown... Sure, I could have another field "localUpdateTimestamp" for every document, but it doesn't look good ;-)
MarkChrisLevy
on 13 Oct 2017
We've considered how to do this and it's actually something we'd like to do because our current implementation makes the local write time inaccessible.
For the moment we've punted on this because we couldn't quickly decide if it was sufficient to configure this instance-wide or if we needed to make it per query or even per snapshot. We need to balance simplicity of use across common cases. "Just" making something configurable ends up committing us to continue to do so even if it turns out there was a better way to do it. Undoing this later becomes disruptive, so given the existence of a simple work-around we prefer not to address it until we have time to do it right.
This is something we'll revisit but it won't happen soon just because of the sheer volume of other higher priority things to do.
So yes, for now the best thing to do in your case is to keep a localUpdateTimestamp and display with serverUptimeTimestamp ? serverUptimeTimestamp : localUpdateTimestamp.
Cheers!
wilhuff
on 13 Oct 2017
This might be related to an issue I've come across, thought I'd post here before starting a new thread.
I have a collection that is queried... ref.orderBy('createdAt', 'desc').endBefore(docSnapshot). If the document's createdAt hasn't been updated with the server timestamp, you get errors below. I can see now why querying by date when there's nothing there might kick up a fuss, and am working around - it's just the errors are pretty cryptic.
Firestore (4.5.0) 2017-10-19T17:12:21.933Z: FIRESTORE (4.5.0) INTERNAL ASSERTION FAILED: Unknown FieldValue {"localWriteTime":{"seconds":1508433141,"nanos":144000000},"typeOrder":3}
error @ log.js:67
fail @ assert.js:43
webpackJsonp.../../../../firebase/firestore/remote/serializer.js.JsonProtoSerializer.toValue @ serializer.js:296
(anonymous) @ serializer.js:866
webpackJsonp.../../../../firebase/firestore/remote/serializer.js.JsonProtoSerializer.toCursor @ serializer.js:865
webpackJsonp.../../../../firebase/firestore/remote/serializer.js.JsonProtoSerializer.toQueryTarget @ serializer.js:742
webpackJsonp.../../../../firebase/firestore/local/local_serializer.js.LocalSerializer.toDbTarget @ local_serializer.js:111
webpackJsonp.../../../../firebase/firestore/local/indexeddb_query_cache.js.IndexedDbQueryCache.addQueryData @ indexeddb_query_cache.js:93
(anonymous) @ local_store.js:494
(anonymous) @ persistence_promise.js:121
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.wrapUserFunction @ persistence_promise.js:108
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.wrapSuccess @ persistence_promise.js:120
_this.nextCallback @ persistence_promise.js:92
PersistencePromise._this.isDone @ persistence_promise.js:64
(anonymous) @ persistence_promise.js:121
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.wrapUserFunction @ persistence_promise.js:108
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.wrapSuccess @ persistence_promise.js:120
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.next @ persistence_promise.js:85
_this.nextCallback @ persistence_promise.js:92
PersistencePromise._this.isDone @ persistence_promise.js:64
(anonymous) @ persistence_promise.js:121
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.wrapUserFunction @ persistence_promise.js:108
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.wrapSuccess @ persistence_promise.js:120
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.next @ persistence_promise.js:85
_this.nextCallback @ persistence_promise.js:92
PersistencePromise._this.isDone @ persistence_promise.js:64
cursorRequest.onsuccess @ simple_db.js:326
wrapFn @ zone.js:1166
webpackJsonp.../../../../zone.js/dist/zone.js.ZoneDelegate.invokeTask @ zone.js:425
onInvokeTask @ core.es5.js:3881
webpackJsonp.../../../../zone.js/dist/zone.js.ZoneDelegate.invokeTask @ zone.js:424
webpackJsonp.../../../../zone.js/dist/zone.js.Zone.runTask @ zone.js:192
webpackJsonp.../../../../zone.js/dist/zone.js.ZoneTask.invokeTask @ zone.js:499
invokeTask @ zone.js:1540
globalZoneAwareCallback @ zone.js:1566
IndexedDB (async)
webpackJsonp.../../../../firebase/firestore/local/simple_db.js.SimpleDbStore.cursor @ simple_db.js:368
webpackJsonp.../../../../firebase/firestore/local/simple_db.js.SimpleDbStore.iterate @ simple_db.js:314
webpackJsonp.../../../../firebase/firestore/local/indexeddb_query_cache.js.IndexedDbQueryCache.getQueryData @ indexeddb_query_cache.js:116
(anonymous) @ local_store.js:484
(anonymous) @ indexeddb_persistence.js:148
(anonymous) @ persistence_promise.js:121
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.wrapUserFunction @ persistence_promise.js:108
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.wrapSuccess @ persistence_promise.js:120
_this.nextCallback @ persistence_promise.js:92
PersistencePromise._this.isDone @ persistence_promise.js:64
(anonymous) @ persistence_promise.js:121
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.wrapUserFunction @ persistence_promise.js:108
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.wrapSuccess @ persistence_promise.js:120
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.next @ persistence_promise.js:85
_this.nextCallback @ persistence_promise.js:92
PersistencePromise._this.isDone @ persistence_promise.js:64
(anonymous) @ persistence_promise.js:121
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.wrapUserFunction @ persistence_promise.js:108
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.wrapSuccess @ persistence_promise.js:120
webpackJsonp.../../../../firebase/firestore/local/persistence_promise.js.PersistencePromise.next @ persistence_promise.js:85
_this.nextCallback @ persistence_promise.js:92
PersistencePromise._this.isDone @ persistence_promise.js:64
request.onsuccess @ simple_db.js:386
wrapFn @ zone.js:1166
webpackJsonp.../../../../zone.js/dist/zone.js.ZoneDelegate.invokeTask @ zone.js:425
onInvokeTask @ core.es5.js:3881
webpackJsonp.../../../../zone.js/dist/zone.js.ZoneDelegate.invokeTask @ zone.js:424
webpackJsonp.../../../../zone.js/dist/zone.js.Zone.runTask @ zone.js:192
webpackJsonp.../../../../zone.js/dist/zone.js.ZoneTask.invokeTask @ zone.js:499
invokeTask @ zone.js:1540
globalZoneAwareCallback @ zone.js:1566
IndexedDB (async)
webpackJsonp.../../../../firebase/firestore/local/simple_db.js.SimpleDbStore.get @ simple_db.js:266
webpackJsonp.../../../../firebase/firestore/local/indexeddb_persistence.js.IndexedDbPersistence.ensureOwnerLease @ indexeddb_persistence.js:221
(anonymous) @ indexeddb_persistence.js:147
webpackJsonp.../../../../firebase/firestore/local/simple_db.js.SimpleDb.runTransaction @ simple_db.js:110
webpackJsonp.../../../../firebase/firestore/local/indexeddb_persistence.js.IndexedDbPersistence.runTransaction @ indexeddb_persistence.js:145
webpackJsonp.../../../../firebase/firestore/local/local_store.js.LocalStore.allocateQuery @ local_store.js:482
webpackJsonp.../../../../firebase/firestore/core/sync_engine.js.SyncEngine.listen @ sync_engine.js:157
webpackJsonp.../../../../firebase/firestore/core/event_manager.js.EventManager.listen @ event_manager.js:76
(anonymous) @ firestore_client.js:266
(anonymous) @ async_queue.js:81
webpackJsonp.../../../../zone.js/dist/zone.js.ZoneDelegate.invoke @ zone.js:392
onInvoke @ core.es5.js:3890
webpackJsonp.../../../../zone.js/dist/zone.js.ZoneDelegate.invoke @ zone.js:391
webpackJsonp.../../../../zone.js/dist/zone.js.Zone.run @ zone.js:142
(anonymous) @ zone.js:873
webpackJsonp.../../../../zone.js/dist/zone.js.ZoneDelegate.invokeTask @ zone.js:425
onInvokeTask @ core.es5.js:3881
webpackJsonp.../../../../zone.js/dist/zone.js.ZoneDelegate.invokeTask @ zone.js:424
webpackJsonp.../../../../zone.js/dist/zone.js.Zone.runTask @ zone.js:192
drainMicroTaskQueue @ zone.js:602
webpackJsonp.../../../../zone.js/dist/zone.js.ZoneTask.invokeTask @ zone.js:503
invokeTask @ zone.js:1540
globalZoneAwareCallback @ zone.js:1566
sionfletcher
on 19 Oct 2017
Internal assertion failures are never supposed to happen. Could you start a new issue for this? Please also include your browser/os info. Thanks!
wilhuff
on 19 Oct 2017
I think it's an issue with angularfire2, couldn't replicate with just firebase.js. I've submitted an issue here
sionfletcher
on 20 Oct 2017
Related issues
stamler
路
3Comments
hvonralvert
路
3Comments
Kosai106
路
3Comments
camden-kid
路
4Comments
dsgriffin
路
4Comments
Most helpful comment
Thanks for writing in!
Server timestamps are unfortunately extremely tricky. I'll try to explain why we have the behavior we have but the short version is that local clocks keep terrible time and your proposal breaks expectations when timestamps are used to order results.
Consider a list of documents ordered by update time, where update time is a server timestamp. If two documents are updated near each other in time and the local clock is slow their relative position in the list can jump as the first document's server timestamp is fixated then jump again as the second timestamp is fixated.
This problem is something we call "flicker". Initial versions of the RTDB just used the local time as a stand-in for a server timestamp, but this caused a bunch of support load due to exactly this problem.
Later versions of the RTDB included server time in the initial handshake, which allowed the client to compensate for a bad local clock, but this still caused problems in cases where writes preceded the initial connection (because the client was offline altogether or the network was just slow).
Flicker caused by this essentially remains an unsolved problem in the RTDB SDKs. Expectations burned in by existing usage make this unfixable there.
For Firestore we decided that flicker-free ordering was more important than providing a local estimate and one of our top design goals for locally estimated values is to avoid this kind of behavior because it's very tricky for users to defend against.
To avoid reordering flicker, internally server timestamps order themselves always after any actual timestamp (as server timestamps are fixated by the server they become regular timestamps). We use the local write time to order among pending server timestamps.
This gives the intended effect because writes are queued locally and server time is monotonically increasing (it's TrueTime). As each write commits the server timestamp converts to a regular timestamp and the relative position of the server timestamp in the list does not change.
So the trouble that your proposal introduces is that the value of the local estimate of the server timestamp can cause the document to now look out of place in the ordering if the local clock is running slow. That is: your last updated document might appear to have an update time before the last fixated server timestamp and would look out of order even though the ordering is actually correct.
If we supplied the local estimate anyway, we'd have to give users that cared about this case some way to defend against it, by indicating somehow that the timestamp was an estimate. However we decided against this because nobody would read the documentation and the app would be subtly wrong until you got your first bug report.
An alternative would be to return the estimate as some non-Date value but this causes problems when serializing to POJOs on Android and we'd like the contract to be consistent across platforms. If we made our non-Date extend Date and throw if it's used as a date this causes tricky runtime failures that are hard for users to test for since latency compensated views are fundamentally racing with the server.
Normally we try to do the thing that most reasonable people would expect to be the default by default but this is a rare case where there's no good default behavior. A local timestamp might actually work for your case while others would prefer to show a sync-pending icon instead. Unfortunately there's no good way for us to know which is the right behavior so we have to force you to choose for yourself.
I hope this helps you understand why we can't make the change you're proposing.
Thanks!