Openbci_gui: LSL manual timestamping interferes with LSL clock_offset correction

[cboulay]

Problem

Hi. I'm one of the LSL maintainers. I'm trying to help a user with a problem. They are unable to synchronize OpenBCI streams with those from different sources when loading xdf files saved in LabRecorder.

The xdf importer adjusts the timestamps in each stream to a common clock: lsl's local_clock on the computer that was running LabRecorder. This adjustment relies on a series of clock_offsets stored alongside the stream. The clock_offsets are calculated online and stored every ~5 seconds using the LabRecorder computer's and the remote computer's lsl::local_clock.
When the remote stream's timestamps also uses lsl::local_clock, then synchronizing the streams on data import 'just works'.

When the remote streams timestamps are set manually, as you might want to do if you had information from the hardware about sample_acq_time which you know to be superior to the default sample_push_time, then it's important to convert sample_acq_time into the same base clock as lsl_local_clock.
If a different base clock is used then post-hoc synchronization of different LSL streams is impossible.

FYI, lsl's clock is std::chrono::high_resolution_clock::now().time_since_epoch().

Another important details is that the timestamps should be in seconds. When using push_chunk, timestamps for samples after the first sample in a chunk are auto-incremented based on the stream's nominal_rate as +1/rate seconds. If millisecond timestamps are used then the first timestamp in a chunk will be in milliseconds but subsequent timestamps in that chunk will be incremented in seconds. This would have the effect of all timestamps within a chunk being very close in time, and then relatively large gaps between chunks. This will interfere with xdf importer's ability to dejitter timestamps.

Expected

As it appears the manual timestamps are simply the time at sample push and do not make use of any hardware timing info, there's no benefit to supplying the timestamps manually. It would be simpler to not provide timestamps at all and let LSL timestamp them for you. This will enable post-hoc synchronization of different streams and fix the chunk timestamping error above.

The downside to this is that it might break compatibility with other software you might have that expects the LSL timestamps to be System.currentTimeMillis(). For example, if any of your example Matlab or Python scripts for reading LSL streams expects timestamps in msec then these will have to be adjusted.

Edit: I took a look at the examples and none that I saw made explicit use of timestamps so it shouldn't be an issue.

[retiutut]

Thank you for a thorough breakdown of what is happening. We will review this carefully.

[LJGz]

I am the user, and happy to help where I can, I can provide example data or test code with our setup.

[retiutut]

So we should revert

 outlet_data.push_chunk(dataToSend, System.currentTimeMillis());`

TO

outlet_data.push_chunk(dataToSend);`

?

[cboulay]

That should be all that is required, though I'm not sure how this interacts with #530.

[retiutut]

I am getting LSL streams from openbci gui to pylsl and trying to get an absolute timestamp with the correct epoch. But since the timestamp sent by Gui is relative, matching it with any other external timeseries becomes hard. Anyway to avoid this simply by adding to the timestamp the current time from the gui perhaps?

If a different base clock is used then post-hoc synchronization of different LSL streams is impossible.

I'd like to reconcile both issues.

So the goal in both cases is to sync multiple LSL streams coming from different sources?
Example: 1 stream from GUI, another stream from a 3rd party Python script.

If so, I'm not sure why #530 was necessary,.

[cboulay]

Right. #530 was not necessary (and in fact was incorrect).

If all streams are on the same PC (here I'm using PC for all types of computers, even Mac and mobile), then synchronization is trivial.

If streams are on different PCs, then the raw timestamps by default returned by inlet.pull_chunk are those generated by the local_clock on the source PC. What local_clock uses, or more precisely what high_resolution_clock uses, for epoch is unix time (1970-01-01 midnight) on most platforms, but on MacOS (at least older versions) its epoch is when the PC was last rebooted.
In #530 @compmonks was using MacOS so this is likely the problem. Even if all computers used Unix time as t=0, it's still bad to rely on this clock because computer's are easily off by a few seconds relative to each other and there's also timezone issues that come into play. The solution is to use the aforementioned clock_offsets that are calculated by an inlet.

While the default during loading xdf files is to use those clock_offsets to correct timestamps, that is not the default for online pulling from an outlet e.g. using pylsl. The solution to #530 was to set the postprocessing flags:
https://github.com/labstreaminglayer/liblsl-Python/blob/master/pylsl/pylsl.py#L80-L86

@compmonks should instead use something like the following:
my_inlet = StreamInlet(stream_info, processing_flags=pylsl.proc_ALL)

So why don't we synchronize timestamps by default? Probably because processing_flags was a relatively recent addition to pylsl and we wanted to maintain backward compatibility. Maybe we should raise a warning whenever post processing is not used and clock_offsets are found to be > eps.

[retiutut]

From @Andrey1994 :

timestamps collected on the SDK are much more precise and dont introduce additional delays like here...

He is referring to the timestamps managed by the BrainFlow SDK (data acquisition). @cboulay Would this be helpful? Or should we send data using push_chunk(data) as discussed on this thread?

[cboulay]

I'm not familiar with the BrainFlow SDK and I don't know where to begin looking to see how they manage timestamps. But I believe it's possible to get more precise timestamps. (I also expect they don't fully understand what LSL does under the hood so the difference might not be as great as you expect).

You can certainly use more precise timestamps, and I encourage you to do so if you have them, but they have to be converted to the LSL clock first. This is usually as simple as: lsl_ts = precise_ts * scale + offset. Note that if precise_ts is coming from a source different to what the LSL clock is coming from, then scale and offset might drift over time.