Flux: git repo not ready: git repo has not been cloned yet

[astraldragon]

Describe the bug

• i know there have been previous bugs for this type of error but the solution was to use latest master tag which also didn't seem to work for me
• using 1.19.0 and latest master tag it fails to clone my repo over ssh
• k8s cluster set up using k3s
• i can clone public repo over https fine while shell'd into the pod, can't clone via ssh while in the pod
• i have added the ssh key to the deploy keys with read/write

To Reproduce

Steps to reproduce the behaviour:

1. follow install guide using 1.19.0 or latest master tag
2. run install command
3. monitor the pod
4. see that it can't clone the repo
5. https://github.com/astraldragon/flux-get-started

Expected behavior

1. be able to clone the repo

Logs

ts=2020-07-05T00:29:37.895767093Z caller=main.go:259 version=1.19.0
ts=2020-07-05T00:29:37.895839275Z caller=main.go:412 msg="using kube config: \"/root/.kube/config\" to connect to the cluster"
ts=2020-07-05T00:29:37.934128744Z caller=main.go:492 component=cluster identity=/etc/fluxd/ssh/identity
ts=2020-07-05T00:29:37.934184602Z caller=main.go:493 component=cluster identity.pub="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQClyFPsl2BSz5Fmc5h1k5Vi2h5TJBahHw3iSjHO5+ZSWD8GLM9o80Y6zC0kraOl0p/1Sq58L4uqkH3a2i1O3ZcyepHPZdCZHSuR6eBsmkrf2xiGGxNIy6MpHdC9azyjfY
3FU9G8PT80hmxMNuEjnwHJgU1OWCO5Anp/Z4jVIJDWCoUedfWOb3KVSel5L/y0Ijutypi8Oo23EUK7YOkXElpEw3Sq/BF/niNaddQwz4dQ0pHySm225WYbcYmQkXiyk6K1NInxgQlYeJXwI88AG9k2LzIlh39MxG/siH4mLF8GxZseSMpsHqxcJ87eVf5BzFgZjxGG+8OoWqFSxuLQipsr+j6IilAQAEkUwQF+5kPmWcVbM0jluWDah+pOP
llOv7jmNFSCWT3kAfBsN7h3IHhoM+NcPR/QCkhjCIj5PMmYAiiZAowa5yRtVQGjgTKyJ498IrOG4Eh/KVMF6IVuEbeP6txJ+LZKF4UqmgCJ624RWdEIzvOaLXoFyMBCc6ZvRzU= root@flux-655bfb7cd9-tffvh"
ts=2020-07-05T00:29:37.934221918Z caller=main.go:498 host=https://10.43.0.1:443 version=kubernetes-v1.17.3+k3s1
ts=2020-07-05T00:29:37.934270397Z caller=main.go:510 kubectl=/usr/local/bin/kubectl
ts=2020-07-05T00:29:37.934835149Z caller=main.go:527 ping=true
ts=2020-07-05T00:29:37.935989682Z caller=main.go:666 url=ssh://[email protected]/astraldragon/flux-get-started user=astraldragon [email protected] signing-key= verify-signatures-mode=none sync-tag=flux state=git readonly=false r
egistry-disable-scanning=false notes-ref=flux set-author=false git-secret=false sops=false
ts=2020-07-05T00:29:37.936048892Z caller=main.go:772 upstream="no upstream URL given"
ts=2020-07-05T00:29:37.937005561Z caller=main.go:803 metrics-addr=:3031
ts=2020-07-05T00:29:37.937298947Z caller=loop.go:107 component=sync-loop err="git repo not ready: git repo has not been cloned yet"
ts=2020-07-05T00:29:37.937393073Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2020-07-05T00:29:37.937470854Z caller=images.go:27 component=sync-loop msg="no automated workloads"
ts=2020-07-05T00:29:37.937910336Z caller=main.go:795 addr=:3030
ts=2020-07-05T00:29:38.613084481Z caller=checkpoint.go:24 component=checkpoint msg="up to date" latest=1.19.0

If applicable, please provide logs of fluxd. In a standard stand-alone installation of Flux, you'd get this by running kubectl logs deploy/flux -n flux.

Additional context

• Flux version: 1.19.0, also tried latest master tag
• Kubernetes version: v1.17.3+k3s1
• Git provider: github.com
• Container registry provider:

[irraz]

I have a same issue, when force manual sync I can see:

fluxctl sync --k8s-fwd-ns flux
Error: git repository ssh://[email protected]/myrepo.git is not ready to sync

Full error message: git fetch --tags origin []: running git command: git [fetch --tags origin]: context deadline exceeded
Run 'fluxctl sync --help' for usage.`

To mitigate I deploy versión: fluxcd/flux-prerelease:master-1f0f862b and --sync-timeout=10m
I read this in another issue...

[kvaps]

Hi I had the similar error, after a while I found that my coredns pods were crashed, and flux wasn't able to resolve github.com

[astlock]

the same here

ts=2020-12-22T16:11:28.245150792Z caller=main.go:259 version=1.20.1
ts=2020-12-22T16:11:28.245232952Z caller=main.go:412 msg="using kube config: \"/root/.kube/config\" to connect to the cluster"
ts=2020-12-22T16:11:28.315992473Z caller=main.go:492 component=cluster identity=/etc/fluxd/ssh/identity
ts=2020-12-22T16:11:28.3160425Z caller=main.go:493 component=cluster identity.pub="ssh..."
ts=2020-12-22T16:11:28.316987702Z caller=main.go:498 host=https://10.147.182.155:443 version=kubernetes-v1.16.9-aliyun.1
ts=2020-12-22T16:11:28.317079434Z caller=main.go:510 kubectl=/usr/local/bin/kubectl
ts=2020-12-22T16:11:28.318727134Z caller=main.go:527 ping=true
ts=2020-12-22T16:11:28.319185013Z caller=main.go:666 url=ssh://[email protected]/ORG/ops-repo-io.git user="Weave Flux" [email protected] signing-key= verify-signatures-mode=none sync-tag=ch-v1 state=git readonly=false registry-disable-scanning=true notes-ref=ch-v1 set-author=false git-secret=false sops=false
ts=2020-12-22T16:11:28.319261083Z caller=main.go:772 upstream="no upstream URL given"
ts=2020-12-22T16:11:28.31939696Z caller=main.go:803 metrics-addr=:3031
ts=2020-12-22T16:11:28.389712534Z caller=loop.go:67 component=sync-loop info="Registry scanning is disabled; no image updates will be attempted"
ts=2020-12-22T16:11:28.389821647Z caller=sync.go:51 component=daemon warning="failed to load last-synced resources. sync event may be inaccurate" err="git repo not ready: git repo has not been cloned yet"
ts=2020-12-22T16:11:28.389869286Z caller=loop.go:108 component=sync-loop err="git repo not ready: git repo has not been cloned yet"
ts=2020-12-22T16:11:28.390743821Z caller=main.go:795 addr=:3030
ts=2020-12-22T16:11:29.406947351Z caller=checkpoint.go:24 component=checkpoint msg="up to date" latest=1.20.1
ts=2020-12-22T16:19:40.172422916Z caller=sync.go:51 component=daemon warning="failed to load last-synced resources. sync event may be inaccurate" err="git repo not ready: git clone --mirror: running git command: git [clone --mirror [email protected]:ORG/ops-repo-io.git /tmp/flux-gitclone114229348]: context deadline exceeded"
ts=2020-12-22T16:19:40.172550724Z caller=loop.go:108 component=sync-loop err="git repo not ready: git clone --mirror: running git command: git [clone --mirror [email protected]:ORG/ops-repo-io.git /tmp/flux-gitclone114229348]: context deadline exceeded"
ts=2020-12-22T16:24:40.172664312Z caller=sync.go:51 component=daemon warning="failed to load last-synced resources. sync event may be inaccurate" err="git repo not ready: git fetch --tags origin []: running git command: git [fetch --tags origin]: context deadline exceeded"
ts=2020-12-22T16:24:40.172735363Z caller=loop.go:108 component=sync-loop err="git repo not ready: git fetch --tags origin []: running git command: git [fetch --tags origin]: context deadline exceeded"
ts=2020-12-22T16:29:40.172903509Z caller=sync.go:51 component=daemon warning="failed to load last-synced resources. sync event may be inaccurate" err="git repo not ready: git fetch --tags origin []: running git command: git [fetch --tags origin]: context deadline exceeded"
ts=2020-12-22T16:29:40.172984004Z caller=loop.go:108 component=sync-loop err="git repo not ready: git fetch --tags origin []: running git command: git [fetch --tags origin]: context deadline exceeded"

--git-timeout=60s
--sync-timeout=10m

After
--git-timeout=300s
works like a charm.