Aks-engine: Image pull error in azure-cni-networkmonitor in China 21v

[zhiweiv]

Describe the bug
azure-cni-networkmonitor pods keep crashing due to image pull error.

  Normal   Pulling  91m (x2245 over 8d)   kubelet, k8s-master-13774364-0  Pulling image "dockerhub.azk8s.cn/containernetworking/networkmonitor:v0.0.8"
  Normal   BackOff  11m (x50911 over 8d)  kubelet, k8s-master-13774364-0  Back-off pulling image "dockerhub.azk8s.cn/containernetworking/networkmonitor:v0.0.8"
  Warning  Failed   96s (x50954 over 8d)  kubelet, k8s-master-13774364-0  Error: ImagePullBackOff

AKS Engine version
0.52

Kubernetes version
1.17

We have a working AKS cluster in China 21v, the image azure-cni-networkmonitor used is mcr.azk8s.cn/containernetworking/networkmonitor:v0.0.7.

Seems the registry aks-engine used(_dockerhub.azk8s.cn_) is incorrect.

[mboersma]

@zhiweiv I can't test this but it sounds like a simple bug we could fix here. Could you try this command?

docker pull mcr.azk8s.cn/containernetworking/networkmonitor:v0.0.8

[AbelHu]

cc @andyzhangx

[andyzhangx]

pull image from mcr.azk8s.cn is only allowed in Azure China VM:

$ sudo docker pull mcr.azk8s.cn/containernetworking/networkmonitor:v0.0.8
v0.0.8: Pulling from containernetworking/networkmonitor
5bed26d33875: Pull complete
f11b29a9c730: Pull complete
930bda195c84: Pull complete
78bf9a5ad49e: Pull complete
0b6d2358a5a5: Pull complete
72c6600ae6b6: Pull complete
10015aa5288e: Pull complete
3ae91debff64: Pull complete
0b57c3246021: Pull complete
3228f6f64ae2: Pull complete
Digest: sha256:9e973ec88b31f0d5ecf3d89dfacb4a20d37a067285c6b6ee36db032358506036
Status: Downloaded newer image for mcr.azk8s.cn/containernetworking/networkmonitor:v0.0.8

[andyzhangx]

@zhiweiv pls ping me offline if your VM is already in Azure China.

[zhiweiv]

My aks-engine cluster was deployed to Azure China 21v subscription, I am able to pull image mcr.azk8s.cn/containernetworking/networkmonitor:v0.0.8 in one vm of aks-engine cluster.

[mboersma]

Great, thank you both for the info. So it sounds like the fix for Azure CNI networkmonitor is clear. See #3587.

But after changing that, there are still a few references to dockerhub.azk8s.cn in azenvtypes.go:

        //KubernetesSpecConfig - Due to Chinese firewall issue, the default containers from google is blocked, use the Chinese local mirror instead
        KubernetesSpecConfig: KubernetesSpecConfig{
            KubernetesImageBase:    "gcr.azk8s.cn/google_containers/",
            TillerImageBase:        "mcr.microsoft.com/",
            ACIConnectorImageBase:  "dockerhub.azk8s.cn/microsoft/",
            NVIDIAImageBase:        "dockerhub.azk8s.cn/nvidia/",
            AzureCNIImageBase:      "mcr.azk8s.cn/containernetworking/",
            MCRKubernetesImageBase: "mcr.microsoft.com/",
            CalicoImageBase:        "dockerhub.azk8s.cn/calico/",
            EtcdDownloadURLBase:    "mcr.microsoft.com/oss/etcd-io/",

Do those entries for ACI connector, NVidia, and Calico also need to be changed to mcr.azk8s.cn?

cc: @jchauncey

[andyzhangx]

ACI connector, NVidia, and Calico

that depends on whether we are going to build ACI connector, NVidia, and Calico into MCR, currently they are all in dockerhub? (not sure), so dockerhub.azk8s.cn is the right config, anyway, that's a future work.