Minikube: hyperkit clock falls behind: failed to write or validate certificate: the certificate is not valid yet

[vincent-dm]

Minikube version: v0.18.0

Environment:

• OS: MacOS Sierra (10.12.4 (16E195))
• VM Driver : xhyve
• ISO version : v0.18.0

What happened:
When I wake my laptop after sleep, the clock of the Minikube VM lags behind. This causes problems with the registry credentials plugin, since AWS rejects credentials requests with an invalid timestamp.

What you expected to happen:
The clock should sync after computer wakeup.

How to reproduce it (as minimally and precisely as possible):
See above.

[aaron-prindle]

Thanks for reporting this. This seems to be a know issue for users w/ xhyve + docker:

https://github.com/docker/for-mac/issues/17
https://forums.docker.com/t/time-in-container-is-out-of-sync/16566/8
For now you should be able to minikube ssh into the VM and then run:
docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)
to sync the time back after you sleep.

[aaron-prindle]

It seems that you can automate this by using sleepwatcher:

brew install sleepwatcher
brew services start sleepwatcher
echo minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y) > ~/.wakeup
chmod +x ~/.wakeup

[r2d4]

ref https://github.com/moby/hyperkit/issues/110

[vincent-dm]

@aaron-prindle thanks for the workaround, but unfortunately I can't get it working.
When I minikube ssh into the VM and run docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y) it pulls an alpine image and then nsenter throws an error:

$ minikube ssh
$ docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine

627beaf3eaaf: Already exists
Digest: sha256:58e1a1bb75db1b5a24a462dd5e2915277ea06438c3f105138f97eb53149673c4
Status: Downloaded newer image for alpine:latest
BusyBox v1.25.1 (2016-10-26 16:15:20 GMT) multi-call binary.

Usage: nsenter [OPTIONS] [PROG [ARGS]]

    -t, --target=PID        Target process to get namespaces from
    -m, --mount[=FILE]      Enter mount namespace
    -u, --uts[=FILE]        Enter UTS namespace (hostname etc)
    -i, --ipc[=FILE]        Enter System V IPC namespace
    -n, --net[=FILE]        Enter network namespace
    -p, --pid[=FILE]        Enter pid namespace
    -U, --user[=FILE]       Enter user namespace
    -S, --setuid=UID        Set uid in entered namespace
    -G, --setgid=GID        Set gid in entered namespace
    --preserve-credentials      Don't touch uids or gids
    -r, --root[=DIR]        Set root directory
    -w, --wd[=DIR]          Set working directory
    -F, --no-fork           Don't fork before exec'ing PROG

I'm also a bit in the dark about how this is supposed to work: as I understand it, the Minikube VM has a skewed clock, so how is launching a pod going to solve this?

Wouldn't it be more logical to run some kind of command directly in the Minikube VM to fix its clock? (like fetching the time from a time server)

I tried to come up with a working command myself, but all the usual linux commands seem to be missing from the flavour of Linux that the Minikube VM is running...

[aaron-prindle]

I actually tested this command w/ debian and incorrectly assumed it would be the same on alpine. Also I realize now that this must be a single command as the host terminal needs to complete the date to update in the vm. The following command should work properly:
minikube ssh -- docker run -i --rm --privileged --pid=host debian nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)

This is just a workaround for now. We are considering making time-syncing daemon for the VM to solve this but this is not really directly related to minikube.

[vincent-dm]

Thanks, this workaround does the trick!

[sebgoa]

Ran into this on a VirtualBox minikube 0.19.0 as well.

The command did the trick

[tristanpemble]

I'm playing with using a privileged pod running ntpd to keep my Minikube VM's clock in sync. If it works well, I will try to submit a PR to bundle it as an addon (if that aligns with how you envisioned a fix)

[gavrie]

If you use Homebrew to install minikube and kubectl, you may have to add /usr/local/bin to your path in the ~/.wakeup script:

PATH=$PATH:/usr/local/bin
minikube ssh -- docker run -i --rm --privileged --pid=host debian nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)

I also added the following line to set the timezone of the VM to the one of my Mac:

minikube ssh -- sudo ln -sf /usr/share/zoneinfo/Asia/Jerusalem /etc/localtime

(Instead of Asia/Jerusalem, you may want to determine your own timezone by running sudo systemsetup -gettimezone).

[bboreham]

out of interest, why run docker run --privileged ... date instead of sudo date ?

[hasan-abdul-rahman]

@aaron-prindle
Please note, your echo command will hardcode a specific date value into the ~/.wakeup script.

$ echo minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)
minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u 102505152017

The whole thing should be surrounded in single quotes to prevent bash from expanding the date command.

$ echo 'minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)'
minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)

[vincent-dm]

After combining all comments above, I have this setup script:

brew install sleepwatcher
brew services start sleepwatcher
echo "PATH=\$PATH:/usr/local/bin
minikube ssh -- docker run -i --rm --privileged --pid=host debian nsenter -t 1 -m -u -n -i hwclock -s" > ~/.wakeup
chmod +x ~/.wakeup

[coderanger]

Since the minikube distro is already using systemd, might be nice to include timesyncd too. Wouldn't cover fixing drift when not on the internet, but would help for a lot of people I bet. I tried enabling it manually but it looks like that daemon isn't included in the minikube distro currently.

[eecue]

+1 this one is a pain when doing AWS dev as our certs get out of sync and fail: Signature expired: 20171114T003822Z is now earlier than 20171114T004302Z (20171114T004802Z - 5 min.)

[luigi-riefolo]

any updates on this issue?

[Morriz]

please?

I am just running virtualbox, but also have time out of sync...why is this thread just about xhyve when there seems no timesync daemon on the distro?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mildebrandt]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mildebrandt]

/remove-lifecycle stale

[msabramo]

I would add the seconds to the date command, so that it's more precise:

minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y.%S)

but then I wonder what the purpose of docker run of nsenter is? It seems like one could use sudo date -- e.g.:

$ minikube ssh -- date
Wed Jul  4 11:23:15 UTC 2018

$ minikube ssh -- sudo date -u $(date -u +%m%d%H%M%Y.%S)
Tue Jul 10 15:27:28 UTC 2018

$ minikube ssh -- date
Tue Jul 10 15:27:39 UTC 2018

[tstromberg]

I'm not sure when it was added, but the minikube guest VM should include time synchronization. Apparently it works for me in kvm2:

$ timedatectl 
                      Local time: Wed 2018-09-19 22:24:46 UTC
                  Universal time: Wed 2018-09-19 22:24:46 UTC
                        RTC time: Wed 2018-09-19 22:24:46
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: yes
systemd-timesyncd.service active: yes
                 RTC in local TZ: no

YMMV based on your environment.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[metametadata]

/remove-lifecycle stale

[dcecile]

@tstromberg Here's what timedatectl shows inside of minikube ssh for us:

$ timedatectl
                      Local time: Thu 2019-01-10 20:41:50 UTC
                  Universal time: Thu 2019-01-10 20:41:50 UTC
                        RTC time: Thu 2019-01-10 21:03:35
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: no
systemd-timesyncd.service active: n/a
                 RTC in local TZ: no

$ sudo timedatectl set-ntp true
Failed to set ntp: Unit file systemd-timesyncd.service does not exist.

Was timesyncd removed completely with #3476 (for switching to syncing time on startup)?

If so, do you have any recommendations for how to keep the Minikube clock from drifting after startup?

[tstromberg]

@dcecile: It's difficult to say, because each VM environment is different. The critical thing is that the VM should sync to the guest so that SSL access works properly, rather than an external time source.

Since each VM environment handles time differently, I ask that if you are running into this in an environment outside of xhyve, that a new issue get created for it. Thanks!

[tstromberg]

NOTE: The xhyve driver is deprecated, so this bug will likely be closed in the near future when it is finally removed.

[dcecile]

@tstromberg Thanks! OK, I get it now that it's up to the VM to do this properly.

We've seen this issue a couple times with Minikube + the Hyperkit driver. (I missed that this issue is for the xhyve driver.) But we've also seen clock drift with plain old Docker for Mac (also using Hyperkit). And based on the old moby/hyperkit#110, it sounds like Hyperkit relies on guests to do clock sync on their own.

In the meantime, we've switched our setup to Minikube + the VirtualBox driver and haven't seen any clock drift problems at all. I'm not sure what VirtualBox is doing differently, and even though guest additions does clock sync but doesn't seem to be installed, this VirtualBox setup looks to be working fine for us.

[andrei-dascalu]

@tstromberg @dcecile same situation here. MacOS + hyperkit and minikube clock is borked. None of the commands to fix it helped in any way.

[casret]

Same problem here. Should we edit the title and labels to specify that this is still an issue using hyperkit, or open a new issue?

[afbjorklund]

If you (re)start the service, does that resolve the issue ?

sudo systemctl restart systemd-timesyncd

[Downchuck]

@andrei-dascalu The wakeup technique worked out for me - the echo command from @aaron-prindle isn't quite right as the date gets resolved when it writes.

But if you do go ahead and edit a .wakeup file, add the command and install sleepwatcher it seems to address the issue.
~/.wakeup

minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)

For the sleepwatcher service

[alphabt]

+1, also seeing this in hyperkit. @vincent-dm might be best to retitle this mentioning hyperkit so it won't get closed due to xhyve being deprecated.

[andrei-dascalu]

@Downchuck using a wakeup file seems to work, but the problem is that it's not persisted between minikube deletes and starts. Not sure about others, but I'm working in a pretty dynamic manner and often scrap minikube machines and/or change configurations so it's pretty annoying.

At the moment I switched to VB instead of HK. I'm not sure who is in the best position (and in which project) to look into a permanent fix or with the ability to make this workaround persistent? I would definitely try to give it a look if pointed in the right direction.

[webwurst]

I just experienced this on linux with kvm2.

[tly1980]

Here is my one-liner to fix it:

DSTR=$(date -u); minikube ssh "sudo date --set=\"$DSTR\""

[tstromberg]

Related: #4264

[tstromberg]

Here is my one-liner to fix it:

DSTR=$(date -u); minikube ssh "sudo date --set=\"$DSTR\""

This is a good fix, the caution here is that many applications will panic if time moves forward too quickly. That said, this isn't a terrible first approach to consider for #4264, as it would trivially solve 90% of problems.

https://engineering.docker.com/2019/02/addressing-time-drift-in-docker-desktop-for-mac/ has a more robust method of continuous syncing.

[reegnz]

I am still experiencing this issue even though I upgraded to minikube 1.1.1.
I also pulled the latest docker-machine-driver-hyperkit, purged my old cluster (deleted ~/.minikube and all that as well).

Could somebody check if the clock sync feature actually works?

Reproduce: Just launch minikube on MacOS with hyperkit and have it run for half a day or so (or send the machine to sleep).

I still rely on the one-liner above to keep my services not becoming crazy every once in a while.

[tstromberg]

@reegnz - Thanks for raising this. It's quite possible that things aren't quite right with time synchronization yet. Do you mind attaching the logs of minikube start --alsologtostderr -v=1?

It will include the messages from cluster.go that we need to find the bug you are running into. Here's an example (using VirtualBox though, which has a built-in time synchronization):

I0614 08:44:16.202508   84392 ssh_runner.go:182] Transferring 1046 bytes to ca.pem
I0614 08:44:16.202930   84392 ssh_runner.go:195] ca.pem: copied 1046 bytes
I0614 08:44:16.540399   84392 cluster.go:191] guest clock: 1560473056.538974235
I0614 08:44:16.540418   84392 cluster.go:204] Guest: 2019-06-14 08:44:16.538974235 +0800 CST Remote: 2019-06-14 08:44:16.510671 +0800 CST m=+32.671057219 (delta=28.303235ms)
I0614 08:44:16.540439   84392 cluster.go:175] guest clock delta is within tolerance: 28.303235ms

[reegnz]

This morning I checked again:

~ minikube ssh date
Thu Jun 13 17:00:18 UTC 2019
~ date
Fri Jun 14 09:52:00 CEST 2019

So quite a clock skew.

I checked the code yesterday and had a hunch that this sync feature is only on-demand when you execute minikube start, so I ran that:

~ minikube start
💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
🏃  Re-using the currently running hyperkit VM for "minikube" ...
⌛  Waiting for SSH access ...
🐳  Configuring environment for Kubernetes v1.14.3 on Docker 18.09.6
🔄  Relaunching Kubernetes v1.14.3 using kubeadm ...
⌛  Verifying: apiserver proxy etcd scheduler controller dns
🏄  Done! kubectl is now configured to use "minikube"
~ minikube ssh date
Fri Jun 14 07:52:28 UTC 2019
~ date
Fri Jun 14 09:52:25 CEST 2019

Now reading your response it is clear that this sync is only done on-demand by running minikube start, but the docs are not clear at all about this, neither is the issue resolution.

So I'd definitely add this info to the docs somewhere:

You still need to run minikube start periodically to re-sync the clocks, even for a running minikube instance.

Also the issue itself isn't properly fixed, we just got a shorthand for the DSTR=$(date -u); minikube ssh "sudo date --set=\"$DSTR\"" workaround. One still has to periodically deal with the situation.

Another confusing thing was minikube start not telling me about syncing clocks, I think it would be great for the minikube start to at least tell you about syncing clocks, If it actually does that.

Do you think the docker-suggested SNTP-based solution is possible in the future, so the skew is actually avoided by using proper NTP syncing? Maybe put a proper NTPD into the vm?

Also, do you need any help with doing that? I could have a look around in the docker-for-mac code to see how they solve this, then port it over (or guide someone knowing the code better with porting it over).

[reegnz]

Okay, turns out docker-for-mac code isn't opensource. :) But anyway, I will have a look at the buildroot thingy that is used to set up the ISO.

[reegnz]

@tstromberg I see that minikube had systemd-timesyncd enabled but then you disabled it in #3476.
Could you elaborate on why you chose to disable it? Having an sntp daemon constantly running would definitely help in resolving this issue.

[afbjorklund]

One _could_ argue that it is the job of the hypervisor to keep the time synced. Then again, some of them fail...

[tstromberg]

@reegnz - I disabled systemd-timesyncd because it could conflict with hypervisor time synchronization. At the time, I wasn't aware that there were hypervisors which didn't handle time synchronization out-of-band.

I agree though that for hyperkit, we will need an sntp daemon to talk to in order to make this work.

[reegnz]

@afbjorklund one could also argue that a vm should be blind to the hypervisor and take time from another trusted source instead.

The point is, we cannot fix the hypervisor as easily as we can fix this in the minikube ISO however. Once somebody fixes this in hyperkit we could get away with not having an sntpd running.

@tstromberg could we detect what hypervisor we are running in?

[reegnz]

Just checked systemd unit config parameters:
https://www.freedesktop.org/software/systemd/man/systemd.unit.html

One could use ConditionVirtualization to only activate for certain hypervisor types.

[reegnz]

Checked out the systemd-detect-virt code, hyperkit should be identifyable as bhyve (xhyve under hyperkit is a bhyve fork).
https://github.com/systemd/systemd/blob/64ef83139cf84a700b95e8150a458b0bb9f720de/src/basic/virt.c#L132

@tstromberg I think we should enable systemd-timesyncd and add a drop-in unit config with ConditionVirtualization=bhyve.

What do you think, should I give it a go?

[yhekma]

FYI I have encountered the same issue using vmware fusion.

[reegnz]

We could define multiple virtualization technologies in the ConditionVirtualization option, eg. ConditionVirtualization=bhyve|vmware.

I am looking into how I can enable timesyncd with buildroot and build the image on mac (don't have a linux machine at hand).

I will come back with the results.

Then somebody could help out how we get the drop-in file into the buildroot build (maybe with a package 'systemd-timesyncd-conditional'?).

[danielsonjh]

I've encountered the same issue (clock drift when after sleeping) on Windows 10 with the hyper-v driver as well

[reegnz]

@danielsonjh thanks for the input! So we're at 3 hypervisors now...

@tstromberg would it make sense to just reenable the systemd-timesyncd daemon and open a follow-up ticket for the virtualbox issue (if it even exists anymore)?

3 out of 4 hypervisors break because of no ntp in the vm anymore.

We can then experiment with the ConditionVirtualization=!oracle unit file config.

[medyagh]

I think due to the multiple related issues we have seen, we need to address this as an important bug to fix soon.

[cpoole]

I have also encountered this running with kvm2. So that makes 4 hypervisors

minikube v1.2.0

[rkleine]

Just the some workaround but in one line:

ssh -i ~/.minikube/machines/minikube/id_rsa docker@$(minikube ip) "docker run --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)"

[asychev]

Unfortunately, does not work for me. After MacBook wake up today:

$ date
Tue Sep 24 09:35:08 CEST 2019

$ minikube ssh date
Mon Sep 23 19:16:38 UTC 2019

Minikube 1.4.0, hyperkit driver

[gWOLF3]

Yep, I just ran into this issue.

I'm not sure this issue should be closed.

[BatmanAoD]

@reegnz My coworkers and I are also observing this behavior on VirtualBox. Looking through this thread, I don't see why VirtualBox is excluded from the systemd-timesyncd fix; I see that @dcecile said they hadn't observed drift with VirtualBox, but other commenters did.

[phoewass]

I use Minikube on Windows 10 with VirtualBox. Everytime I lock the screen the time in minikube gets out of sync with the host machine.
I tried executing the commands provided by @aaron-prindle and it didn't sync the date.

[rajasaur]

yes, confirmed with VirtualBox and minikube, the drift gets bigger depending on how long the machine is in sleep mode. @msabramo 's comment minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y.%S) works though

[Diggsey]

I use Minikube on Windows 10 with VirtualBox and I see the same issue. The commands provided by @aaron-prindle DO NOT sync the date.