Godot: Network time update causing Godot physics simulation to hang / speed up

[wombatwingdings]

Godot version: 3.1 build from source

OS/device including version:

Android 7.1.2 - Moto G 2nd gen GPU: Adreno 305

Issue description:

I've been having issues with _physics_process() running normally, then suddenly start running at more than twice the speed as if it's in fast forward. It looks very amusing and breaks lots of things. It keeps running like this until the process exits. I have similar problem using _process() where it just freezes and hangs instead. After some investigation with adb logcat, it appears that it all starts going wrong after this happens:

AlarmClock: AlarmInitReceiver android.intent.action.TIME_SET

This completely screws up the physics processing. These are sent if you have "Settings -> Date & time -> Use network-provided date and time" set. Switching this off, stops the problem from happening. On my home WiFi, this seems to be generated every few minutes, meaning, it breaks quite often. My home setup is just a typical setup. Nothing unusual.

Not sure if there's anything that can be done here. It doesn't break other games made with (for example Unity). Maybe there physics processing needs some sort of sanity check between frames?

Steps to reproduce:

Minimal reproduction project:

[Faless]

A place to look:
OS_Unix::get_ticks_usec uses gettimeofday and calculate it based on ticks_start.
When the time changes, the result of OS_Unix::get_ticks_usec() will no longer be truthful (and you might end up going back in time, or fast forward a second).
Might this be the problem in physics? @AndreaCatania
For sure, at least one iteration will be messed up, but it should be able to recover.
And if this is the problem, should we try to find another way to find the running time without relying on system timestamp (which might change as result of NTP sync)?

[Faless]

Update:
From http://man7.org/linux/man-pages/man2/gettimeofday.2.html :

  The time returned by gettimeofday() is affected by discontinuous
  jumps in the system time (e.g., if the system administrator manually
  changes the system time).  If you need a monotonically increasing
  clock, see clock_gettime(2).

~I guess we should use clock_gettime for get_ticks_usec()
http://man7.org/linux/man-pages/man2/timer_create.2.html
http://man7.org/linux/man-pages/man2/clock_gettime.2.html~

Nope, that is still affected by NTP changes...

[jabcross]

This is also happening under X11. _process(delta) gets a negative delta, and that seems to be the cause of the weirdness.

A fix that comes to mind is having Godot keep a reference time that gets defines on load, and updated whenever a negative delta is detected.

I'll try to implement that.

[Faless]

@jabcross that already happens here:
https://github.com/godotengine/godot/blob/master/drivers/unix/os_unix.cpp#L130

I think the problem is that we should check (and eliminate) negative numbers when they occurs due to NTP sync here:
https://github.com/godotengine/godot/blob/master/drivers/unix/os_unix.cpp#L253

The question is: can we do better? Because we can detect it, but we will still have at least one integration step with a fake time (can it be 0.0 @AndreaCatania ? I guess as long it's not negative it's fine)

[jabcross]

Update:
From http://man7.org/linux/man-pages/man2/gettimeofday.2.html :

  The time returned by gettimeofday() is affected by discontinuous
  jumps in the system time (e.g., if the system administrator manually
  changes the system time).  If you need a monotonically increasing
  clock, see clock_gettime(2).

I guess we should use clock_gettime for get_ticks_usec()
http://man7.org/linux/man-pages/man2/timer_create.2.html
http://man7.org/linux/man-pages/man2/clock_gettime.2.html

Nope, that is still affected by NTP changes...

https://www.softwariness.com/articles/monotonic-clocks-windows-and-posix/

This article mentions that there are two monotonic clocks, one of which doesn't get affected by NTP changes. Did you test it with CLOCK_MONOTONIC_RAW?

[jabcross]

@jabcross that already happens here:
https://github.com/godotengine/godot/blob/master/drivers/unix/os_unix.cpp#L130

I think the problem is that we should check (and eliminate) negative numbers when they occurs due to NTP sync here:
https://github.com/godotengine/godot/blob/master/drivers/unix/os_unix.cpp#L253

The question is: can we do better? Because we can detect it, but we will still have at least one integration step with a fake time (can it be 0.0 @AndreaCatania ? I guess as long it's not negative it's fine)

Here, the problem doesn't happen in a single frame. If you set you clock back in time for, say, a hour, ever subsequent frame gets a negative delta.

The MRP is pretty easy:

func _process(delta):
    if (delta < 0):
        print("Woop")
    $Sprite.rotate( delta)

Once I set my system time back one hour, I get "Woop"s until I kill the process.

[Faless]

Did you test it with CLOCK_MONOTONIC_RAW?

No, I didn't test it, but I assume that would be a solution.
Sadly, it's a Linux specific clock (i.e. it won't work on Mac nor on *BSDs, although it might be available on Android, I didn't check).

[jabcross]

Maybe this C++11 steady_clock will fix it.

[Faless]

@jabcross yes, I did expect that. The problem happens when the system clock goes back in time.
In the last function I linked, we can detect that, and somehow recover (e.g. with an extra variable updated every iteration, by offsetting ticks_start so that the last counted ticks is always <= than the current one). I'm just not sure it's the best fix

Maybe this C++ steady_clock will fix it.

The problem I see with steady_clocks is that the resolution is implementation dependent, and might not be high precision (which is quite important for physics).

I wonder what other engines does.

[Faless]

So, SDL uses monotonic clocks when they are available:

At least Andorid/OSX/Linux support is there. I guess iOS too.

https://hg.libsdl.org/SDL/file/b504b29c3831/src/timer/unix/SDL_systimer.c

On Windows (and UWP) we already use QueryPerformanceCounter which is monotonic.

Browsers might be supported via emscripten_get_now which uses performance.now() which used to be both monotonic and high precision (but now it is no longer "that high precision" due to spectre mitigation -.-)

[jabcross]

Cocos2d seems to use steady_clock. https://github.com/cocos2d/cocos2d-x/blob/f8ed20e4b3bf74f0812e019d58d823d72ef8d4a9/cocos/base/CCDirector.cpp#L385

[jabcross]

Using steady_clock on Linux does seem to work, however I had to compile for C++11. For now, I'm ignoring the precision problem.

Any idea if we're planning to officially upgrade any time soon?

uint64_t OS_Unix::get_ticks_usec() const {

    uint64_t longtime = std::chrono::duration_cast<std::chrono::microseconds>(
        std::chrono::steady_clock::now().time_since_epoch()).count();

    longtime -= ticks_start;

    return longtime;
}

Now to see if it just works on Android.

[EDIT]

It does seem to work on Android with no obvious physics issue. I haven't done any thorough testing, though.

[AndreaCatania]

It doesn't seems related to physics, I'm removing myself as assignment, but let me know if you need help from me