Minikube: Minikube connection to internal registry is refused

[kameshsampath]

When I try to push images to internal registry from container (build via Tekton piplines) I get a connection refused error. This is working well with minikube v1.1.1.

The exact command to reproduce the issue:
Following https://github.com/kameshsampath/minikube-helpers/tree/master/registry/README.adoc.

The full output of the command that failed:
[build-greeter-jvm : buildah-build-push] Head http://dev.local/v2/rhdevelopers/pipeline-greeter/blobs/sha256:9ce2ac72ea3a3f426d08fe7ed892b6983f6ded93b9f82d8f87b8083760f26bec: dial tcp 10.111.212.82:80: connect: connection refused

The output of the minikube logs command:

When I inspected registry logs I got this

registry-p8bmz registry 10.1.0.20 - - [25/Jun/2019:08:54:32 +0000] "HEAD /v2/rhdevelopers/pipeline-greeter/blobs/sha256:49d9522d5abfe1bf7e9646b02e1125fec6376add018a3a1a9a5ab875a3e52b68 HTTP/1.1" 404 157 "" "Buildah/1.7"

The operating system version:

• minikube version: v1.2.0
• macOS Mojave 10.14.5
• vm-driver: hyperkit or virtualbox

Related: https://github.com/kubernetes/minikube/issues/4553

[castlemilk]

I've found the issue gets resolved if you remove the registry-proxy DaemonSet. when you enable the registry addon, it creates a Service which is loadbalancing across both the registry-proxy and the registry. Configuration is here: https://github.com/kubernetes/minikube/blob/master/deploy/addons/registry/

[kameshsampath]

thanks @castlemilk - I suspected the same let me try deleting the daemonset

[tstromberg]

I can't tell from #4627 - do we think this issue has been fixed now?

If not, what is required to do so?

[kumarom]

@tstromberg I think, #4627 does not resolves this issue..

[castlemilk]

I was experiencing the exact same issue with Tekton, the merged changes in https://github.com/kubernetes/minikube/pull/4639, should hopefully resolve this and ensure testing validates connectivity from outside the minikube cluster (on port 5000) as well as internal access to the registry (on port 80)

[kumarom]

@tstromberg @castlemilk @kameshsampath and others, I think #4805 should fix this issue.

[kameshsampath]

Thanks all for fixing this looking for the next release of minikube to check this out

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Om Kumar [e-mail]
Sent: Thursday, July 18, 2019 9:16:34 PM
To: kubernetes/minikube [e-mail]
Cc: Kamesh Sampath [e-mail]; Mention [e-mail]
Subject: Re: [kubernetes/minikube] Minikube connection to internal registry is refused (#4604)

@tstromberghttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftstromberg&data=02%7C01%7C%7Cc2dca6bdf0fa4143453d08d70b79c6fe%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636990489959931790&sdata=2aEwSZl0vEDGZnOD8Gi98taGIWFaGHZSkPl%2F9l%2BIILU%3D&reserved=0 @castlemilkhttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcastlemilk&data=02%7C01%7C%7Cc2dca6bdf0fa4143453d08d70b79c6fe%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636990489959931790&sdata=T%2B5NTSQyCuT%2BwkyflwbCP%2BV97xrzg9Z6j239NE7dbP4%3D&reserved=0 @kameshsampathhttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkameshsampath&data=02%7C01%7C%7Cc2dca6bdf0fa4143453d08d70b79c6fe%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636990489959941803&sdata=p0PkYO205fHyImlmntm%2FDgJydtAmv2zett3zTWawt5c%3D&reserved=0 and others, I think #4805https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fminikube%2Fpull%2F4805&data=02%7C01%7C%7Cc2dca6bdf0fa4143453d08d70b79c6fe%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636990489959951799&sdata=BMdL1CiSsnE990nxn50dLiChOVhEdtYQyx%2BWPhfarzg%3D&reserved=0 should fix this issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fminikube%2Fissues%2F4604%3Femail_source%3Dnotifications%26email_token%3DAAHHMIOZSDRGSFMPNJZ3DDTQABNKFA5CNFSM4H3OMJVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2IJBIA%23issuecomment-512790688&data=02%7C01%7C%7Cc2dca6bdf0fa4143453d08d70b79c6fe%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636990489959951799&sdata=dgV2a8Zgkb7h3sU%2FXT%2B6GT9noUqpg9YmVJwHmT%2BvRmg%3D&reserved=0, or mute the threadhttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAHHMIOSHGDXIHL7SVLIJITQABNKFANCNFSM4H3OMJVA&data=02%7C01%7C%7Cc2dca6bdf0fa4143453d08d70b79c6fe%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636990489959961820&sdata=ioE%2BXaG%2Fjp%2BnpsIk6fg%2B98HxIPf1h5Eb2nyz0tIyIkI%3D&reserved=0.

[nicolaferraro]

Deleting the daemonset isn't really a workaround, as it gets recreated by the addon. Does anyone know a proper workaround until the fix gets released?

[kameshsampath]

Nicola Staying with 1.1.1 ;)

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Nicola Ferraro [e-mail]
Sent: Tuesday, July 30, 2019 6:10:05 AM
To: kubernetes/minikube [e-mail]
Cc: Kamesh Sampath [e-mail]; Mention [e-mail]
Subject: Re: [kubernetes/minikube] Minikube connection to internal registry is refused (#4604)

Deleting the daemonset isn't really a workaround, as it gets recreated by the addon. Does anyone know a proper workaround until the fix gets released?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fminikube%2Fissues%2F4604%3Femail_source%3Dnotifications%26email_token%3DAAHHMIPV7LWIQAXSDDCEONDQB5TD3A5CNFSM4H3OMJVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3CE4MQ%23issuecomment-516181554&data=02%7C01%7C%7Cbeb646916787401cc31008d7147182d8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637000350064089960&sdata=1uQHUprTmgrLuBglGg9RoHhkMNEPFkMuNopUAl1njUw%3D&reserved=0, or mute the threadhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAHHMIKRRUZVLT3FIBTTOGLQB5TD3ANCNFSM4H3OMJVA&data=02%7C01%7C%7Cbeb646916787401cc31008d7147182d8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637000350064109976&sdata=EXGLxZ2WEB9ZmWe7pMANSk6gp7gP95k%2BOaWMl6ahoII%3D&reserved=0.

[nicolaferraro]

Nicola Staying with 1.1.1 ;)

That's a good idea ;)

[tstromberg]

Has anyone tested this with v1.3.1? I'm curious if this issue can be closed.

[kameshsampath]

Am doing some work with 1.3 this week will let you all know if it works.

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Thomas Strömberg [e-mail]
Sent: Tuesday, August 20, 2019 2:11:41 AM
To: kubernetes/minikube [e-mail]
Cc: Kamesh Sampath [e-mail]; Mention [e-mail]
Subject: Re: [kubernetes/minikube] Minikube connection to internal registry is refused (#4604)

Has anyone tested this with v1.3.1? I'm curious if this issue can be closed.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fminikube%2Fissues%2F4604%3Femail_source%3Dnotifications%26email_token%3DAAHHMIPFNEIVRSZU76QU2D3QFMAQLA5CNFSM4H3OMJVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4UHLYQ%23issuecomment-522745314&data=02%7C01%7C%7Cc898694f5f504c81ec8108d724e5a42f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637018441023235217&sdata=0Q1hIenIkYoSts%2F0vPzwgLGab8HQY%2FLiTstp1f74TAQ%3D&reserved=0, or mute the threadhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAHHMINWLFNJTYNNYFBVCPDQFMAQLANCNFSM4H3OMJVA&data=02%7C01%7C%7Cc898694f5f504c81ec8108d724e5a42f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637018441023245228&sdata=jAw1Y4y%2FnTNEmWNReGvoKLlz5yFdGWLW%2FxAviTK7yxE%3D&reserved=0.

[tstromberg]

Any luck confirming whether or not this issue still exists with v1.3.1?

[kameshsampath]

@tstromberg - still same issue :( , if you can deploy the demo https://github.com/kameshsampath/pipeline-helloworld after enabling https://github.com/kameshsampath/minikube-helpers/registry you will see it just times out at the stage of pushing the image to local registry. I tried 1.3.1, 1.2.0 both same result. Works as usual with 1.1.1

[castlemilk]

Not sure about those repos, but I can successfully make requests to the internal registry now via say: curl http://registry.kube-system.

Try setting your Tekton PipelineResource as follows:

apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: helloworld-image
spec:
  type: image
  params: 
  - name: url
    value: "registry.kube-system:80/tutorial/helloworld"

[kameshsampath]

@castlemilk ah thats a good tip . I was simply aliasing the same via my helper.

[Javatar81]

@castlemilk I checked your example with Minikube 1.3.1 (registry add-on enabled and registry-proxy active) and Tekton 0.6.0. It worked and uses the registry service. It is important setting port 80 in the url (it does not work without setting any port) because this matches the registry service port.

[kameshsampath]

My registry helper was to avoid those port spec via alisases, but dont know why they do not work on 1.3; need to revisit soon to fix if any issues.

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Benjamin [e-mail]
Sent: Thursday, September 19, 2019 10:41:44 PM
To: kubernetes/minikube [e-mail]
Cc: Kamesh Sampath [e-mail]; Mention [e-mail]
Subject: Re: [kubernetes/minikube] Minikube connection to internal registry is refused (#4604)

@castlemilkhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcastlemilk&data=02%7C01%7C%7Ce254be4871b94968d94808d73d2472c4%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637045099057268748&sdata=R%2BAZ02ZDCAZdtSqihz1AfQQePC%2B3YhoB65auXk8m4VE%3D&reserved=0 I checked your example with Minikube 1.3.1 (registry add-on enabled and registry-proxy active) and Tekton 0.6.0. It worked and uses the registry service. It is important setting port 80 in the url (it does not work without setting any port) because this matches the registry service port.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fminikube%2Fissues%2F4604%3Femail_source%3Dnotifications%26email_token%3DAAHHMIOIHKKQN74QMRWDC5TQKOXFBA5CNFSM4H3OMJVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7EFZUI%23issuecomment-533224657&data=02%7C01%7C%7Ce254be4871b94968d94808d73d2472c4%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637045099057278760&sdata=D7yqdXDPnXX6Yeepvol9B%2BEEmkHk6flzQiB09suF1II%3D&reserved=0, or mute the threadhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAHHMIMXSX42YS3SXCMCOETQKOXFBANCNFSM4H3OMJVA&data=02%7C01%7C%7Ce254be4871b94968d94808d73d2472c4%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637045099057288765&sdata=meoNhXDVxfnoUUxbyBgy1aH2sfx1ByMEeFFfTyGPwqI%3D&reserved=0.

[tstromberg]

I can't tell from the comments - was there a resolution found here?

[kameshsampath]

@castlemilk - were you able to use the image without setting insecure-regsitry in minkube start ?
when run the command like kubectl run greeter --image=registry.kube-system:80/rhdevelopers/knative-tutorial-greeter I get image pull backoff and describing the pod I get the error

Events:
  Type     Reason     Age        From               Message
  ----     ------     ----       ----               -------
  Normal   Scheduled  <unknown>  default-scheduler  Successfully assigned knativetutorial/greeter-78bbb96d8d-4xfk4 to minikube
  Normal   Pulling    11s        kubelet, minikube  Pulling image "registry.kube-system:80/rhdevelopers/knative-tutorial-greeter"
  Warning  Failed     11s        kubelet, minikube  Failed to pull image "registry.kube-system:80/rhdevelopers/knative-tutorial-greeter": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.kube-system:80/v2/: dial tcp: lookup registry.kube-system on 192.168.64.1:53: no such host
  Warning  Failed     11s        kubelet, minikube  Error: ErrImagePull
  Normal   BackOff    11s        kubelet, minikube  Back-off pulling image "registry.kube-system:80/rhdevelopers/knative-tutorial-greeter"
  Warning  Failed     11s        kubelet, minikube  Error: ImagePullBackOff

[kameshsampath]

@tstromberg @castlemilk - I am able to run with localhost:5000/<image-name> but when using registry-kubesystem:80/<image-name> result in lookup errors :(.

[kameshsampath]

@tstromberg - I feel I still need to use my registry helper https://github.com/kameshsampath/minikube-helpers/tree/master/registry to make it resolve the registry.kube-system dns name to go to registry.kube-system.svc.cluster.local. It works with vm-driver virtualbox

[medyagh]

@kameshsampath have you tried with other drivers? could you tell me what your helper does, is there any way we could integrate into minikube?

[kameshsampath]

@medyagh - i see that my helper consistently working with 1.4.0 with both hyperkit and virtualbox.

This is what my helper does

1. Define a list of dns names (via configmap) that you wish to you use for internal registry e.g. dev.local, example.com
2. Run a daemonset to add dns names to /etc/hosts of the node matching the CLUSTER-IP of registry in kube-system that was enabled via addons
3. Finally patch the coredns to resolve the dns names to registry.kube-system.cluster.local address

After these 3 you can now push and pull to the internal registry using the custom domain names.

is there any way we could integrate into minikube?

I was thinking to make this an addon, so we can easily enable/disable this ?

[medyagh]

@kameshsampath that is a good idea ! please feel free to make a PR for it !

[medyagh]

@kameshsampath do you still have interests to make a PR for this and to clarify is the daemonset is inside minikube?

[kameshsampath]

Yes I am interested, since am bit busy with other priorities its taking sometime to push it back.

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Medya Ghazizadeh [e-mail]
Sent: Tuesday, December 17, 2019 3:53:59 AM
To: kubernetes/minikube [e-mail]
Cc: Kamesh Sampath [e-mail]; Mention [e-mail]
Subject: Re: [kubernetes/minikube] Minikube connection to internal registry is refused (#4604)

@kameshsampathhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkameshsampath&data=02%7C01%7C%7C04ac70a23d8243ec40fb08d78276a612%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637121318412032858&sdata=ZMAybfvO39Yhw3bxrUnZnBezV2gfJAJoCwudG3hRmIE%3D&reserved=0 do you still have interests to make a PR for this ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fminikube%2Fissues%2F4604%3Femail_source%3Dnotifications%26email_token%3DAAHHMIPIEYIF3WBEYYENTM3QY75X7A5CNFSM4H3OMJVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHAKKQQ%23issuecomment-566273346&data=02%7C01%7C%7C04ac70a23d8243ec40fb08d78276a612%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637121318412042870&sdata=%2Bm5Wncfu7y9KzURYXxgIk%2Bo50kzoXBLx8CUnU0yAOyk%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAHHMIMMRX4FFHPUFHC4Z23QY75X7ANCNFSM4H3OMJVA&data=02%7C01%7C%7C04ac70a23d8243ec40fb08d78276a612%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637121318412052875&sdata=km7PYSLmHfBlAatR7otAXZfdPv8TFyU5XR4VrEFWf1I%3D&reserved=0.

[kameshsampath]

@medyagh - I have the initial draft for the addon, lets discuss this addon in new issue for clarity

New Issue #6656 and related PR https://github.com/kubernetes/minikube/pull/6657

[tstromberg]

@kameshsampath - PR looks good, but could use a quick README file so that users understand what it is actually doing.

[kameshsampath]

@tstromberg - thinking where is the right place to add that, just in the addons folder ?

I also got another question on whats the best practice to do the following while disabling the addon:

1. I patch the coreDNS CM, wondering if there is way I could do a merge like kubectl does so that if we disable addon the update is reversed ?

2. The /etc/hosts files get updated for entries, just thinking that we need to remove them while disabling

[kameshsampath]

@tstromberg - updates done on the PR. thanks

[kameshsampath]

@tstromberg - I think this issue has popped up again with 1.8.2, after applying the registry-aliases addon or https://github.com/kubernetes/minikube/tree/master/deploy/addons/registry-aliases, then tying to push image from within a pod fails with 404 :(. Any thoughts ?

[tstromberg]

No thoughts, but a 404 does sound like a very different than "connection refused".

[tstromberg]

Anyone still running into this with minikube v1.10.x or newer?

[kameshsampath]

I am not having that issue anymore

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Thomas Strömberg [e-mail]
Sent: Friday, May 29, 2020 12:15:48 AM
To: kubernetes/minikube [e-mail]
Cc: Kamesh Sampath [e-mail]; Mention [e-mail]
Subject: Re: [kubernetes/minikube] Minikube connection to internal registry is refused (#4604)

Anyone still running into this with minikube v1.10.x or newer?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fminikube%2Fissues%2F4604%23issuecomment-635528445&data=02%7C01%7C%7Cda854115e37f451365b608d803375723%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637262883499511860&sdata=%2BnKTEI93fY6iFM5eHJfdD27Fhp32GunQ%2F%2BZ8LsviGGA%3D&reserved=0, or unsubscribehttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAHHMII2S2PTOMFAG2RCLU3RT2WNZANCNFSM4H3OMJVA&data=02%7C01%7C%7Cda854115e37f451365b608d803375723%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637262883499521846&sdata=4qNw2UlA9jCHYTrcN13%2F676DXjwq8AbxoVmSZcDDMUA%3D&reserved=0.

[sharifelgamal]

OK, looks like this is safe to close. Please reopen if the issue return.