istioctl experimental remove-from-mesh deployment istio-egressgateway -n istio-system * failed to update deployment "istio-egressgateway.istio-system" for service "istio-egressgateway.istio-system" due to Deployment.apps "istio-egressgateway" is invalid: spec.template.spec.containers: Required value

[tanjunchen]

(NOTE: This is used to report product bugs:
To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities/
To ask questions about how to use Istio, please visit https://discuss.istio.io
)

Bug description
```
[root@vm10-master ~]# kubectl get pods,deploy -n istio-system
NAME READY STATUS RESTARTS AGE
pod/grafana-74dc798895-hvv9j 1/1 Running 0 19h
pod/istio-egressgateway-69bf865cf8-bs768 1/1 Running 0 19h
pod/istio-ingressgateway-569d44555d-sqh42 1/1 Running 0 19h
pod/istio-tracing-8584b4d7f9-dnl7p 1/1 Running 0 19h
pod/istiod-84cc4dfcd8-vkzsm 1/1 Running 0 19h
pod/kiali-6f457f5964-z5chw 1/1 Running 0 19h
pod/prometheus-79878ff5fd-8ww2t 2/2 Running 0 19h

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/grafana 1/1 1 1 19h
deployment.apps/istio-egressgateway 1/1 1 1 19h
deployment.apps/istio-ingressgateway 1/1 1 1 19h
deployment.apps/istio-tracing 1/1 1 1 19h
deployment.apps/istiod 1/1 1 1 19h
deployment.apps/kiali 1/1 1 1 19h
deployment.apps/prometheus 1/1 1 1 19h

[root@vm10-master ~]# istioctl experimental remove-from-mesh deployment istio-egressgateway -n istio-system
Error: 1 error occurred:
* failed to update deployment "istio-egressgateway.istio-system" for service "istio-egressgateway.istio-system" due to Deployment.apps "istio-egressgateway" is invalid: spec.template.spec.containers: Required value

[root@vm10-master ~]# istioctl experimental remove-from-mesh deployment istio-ingressgateway -n istio-system
Error: 1 error occurred:
* failed to update deployment "istio-ingressgateway.istio-system" for service "istio-ingressgateway.istio-system" due to Deployment.apps "istio-ingressgateway" is invalid: spec.template.spec.containers: Required value

[root@vm10-master ~]# istioctl experimental remove-from-mesh deployment prometheus -n istio-system
Error: 1 error occurred:
* failed to update deployment "prometheus.istio-system" for service "prometheus.istio-system" due to Deployment.apps "prometheus" is invalid: spec.template.spec.containers[0].volumeMounts[1].name: Not found: "istio-certs"

[root@vm10-master ~]# kubectl get deploy istio-ingressgateway -n istio-system -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"istio-ingressgateway","install.operator.istio.io/owning-resource":"example-istiocontrolplane","istio":"ingressgateway","operator.istio.io/component":"IngressGateways","operator.istio.io/managed":"Reconcile","operator.istio.io/version":"1.6.0","release":"istio"},"name":"istio-ingressgateway","namespace":"istio-system"},"spec":{"selector":{"matchLabels":{"app":"istio-ingressgateway","istio":"ingressgateway"}},"strategy":{"rollingUpdate":{"maxSurge":"100%","maxUnavailable":"25%"}},"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"app":"istio-ingressgateway","chart":"gateways","heritage":"Tiller","istio":"ingressgateway","release":"istio","service.istio.io/canonical-name":"istio-ingressgateway","service.istio.io/canonical-revision":"latest"}},"spec":{"affinity":{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"beta.kubernetes.io/arch","operator":"In","values":["amd64"]}]},"weight":2},{"preference":{"matchExpressions":[{"key":"beta.kubernetes.io/arch","operator":"In","values":["ppc64le"]}]},"weight":2},{"preference":{"matchExpressions":[{"key":"beta.kubernetes.io/arch","operator":"In","values":["s390x"]}]},"weight":2}],"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"beta.kubernetes.io/arch","operator":"In","values":["amd64","ppc64le","s390x"]}]}]}}},"containers":[{"args":["proxy","router","--domain","$(POD_NAMESPACE).svc.cluster.local","--proxyLogLevel=warning","--proxyComponentLogLevel=misc:error","--log_output_level=default:info","--serviceCluster","istio-ingressgateway","--trust-domain=cluster.local"],"env":[{"name":"JWT_POLICY","value":"first-party-jwt"},{"name":"PILOT_CERT_PROVIDER","value":"istiod"},{"name":"CA_ADDR","value":"istiod.istio-system.svc:15012"},{"name":"NODE_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.nodeName"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"HOST_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.hostIP"}}},{"name":"SERVICE_ACCOUNT","valueFrom":{"fieldRef":{"fieldPath":"spec.serviceAccountName"}}},{"name":"CANONICAL_SERVICE","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['service.istio.io/canonical-name']"}}},{"name":"CANONICAL_REVISION","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['service.istio.io/canonical-revision']"}}},{"name":"ISTIO_META_WORKLOAD_NAME","value":"istio-ingressgateway"},{"name":"ISTIO_META_OWNER","value":"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-ingressgateway"},{"name":"ISTIO_META_MESH_ID","value":"cluster.local"},{"name":"ISTIO_META_ROUTER_MODE","value":"sni-dnat"},{"name":"ISTIO_META_CLUSTER_ID","value":"Kubernetes"}],"image":"docker.io/istio/proxyv2:1.6.0","name":"istio-proxy","ports":[{"containerPort":15020},{"containerPort":8080},{"containerPort":8443},{"containerPort":31400},{"containerPort":15443},{"containerPort":15011},{"containerPort":15012},{"containerPort":8060},{"containerPort":853},{"containerPort":15090,"name":"http-envoy-prom","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthz/ready","port":15021,"scheme":"HTTP"},"initialDelaySeconds":1,"periodSeconds":2,"successThreshold":1,"timeoutSeconds":1},"resources":{"limits":{"cpu":"2000m","memory":"1024Mi"},"requests":{"cpu":"10m","memory":"40Mi"}},"volumeMounts":[{"mountPath":"/etc/istio/proxy","name":"istio-envoy"},{"mountPath":"/etc/istio/config","name":"config-volume"},{"mountPath":"/var/run/secrets/istio","name":"istiod-ca-cert"},{"mountPath":"/var/run/ingress_gateway","name":"ingressgatewaysdsudspath"},{"mountPath":"/etc/istio/pod","name":"podinfo"},{"mountPath":"/etc/istio/ingressgateway-certs","name":"ingressgateway-certs","readOnly":true},{"mountPath":"/etc/istio/ingressgateway-ca-certs","name":"ingressgateway-ca-certs","readOnly":true}]}],"serviceAccountName":"istio-ingressgateway-service-account","volumes":[{"configMap":{"name":"istio-ca-root-cert"},"name":"istiod-ca-cert"},{"downwardAPI":{"items":[{"fieldRef":{"fieldPath":"metadata.labels"},"path":"labels"},{"fieldRef":{"fieldPath":"metadata.annotations"},"path":"annotations"}]},"name":"podinfo"},{"emptyDir":{},"name":"istio-envoy"},{"emptyDir":{},"name":"ingressgatewaysdsudspath"},{"configMap":{"name":"istio","optional":true},"name":"config-volume"},{"name":"ingressgateway-certs","secret":{"optional":true,"secretName":"istio-ingressgateway-certs"}},{"name":"ingressgateway-ca-certs","secret":{"optional":true,"secretName":"istio-ingressgateway-ca-certs"}}]}}}}
creationTimestamp: "2020-07-24T11:00:29Z"
generation: 1
labels:
app: istio-ingressgateway
install.operator.istio.io/owning-resource: example-istiocontrolplane
istio: ingressgateway
operator.istio.io/component: IngressGateways
operator.istio.io/managed: Reconcile
operator.istio.io/version: 1.6.0
release: istio
name: istio-ingressgateway
namespace: istio-system
resourceVersion: "248689"
selfLink: /apis/apps/v1/namespaces/istio-system/deployments/istio-ingressgateway
uid: 114aa687-4522-409b-a950-29f42a9d6ca1
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: istio-ingressgateway
istio: ingressgateway
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
sidecar.istio.io/inject: "false"
creationTimestamp: null
labels:
app: istio-ingressgateway
chart: gateways
heritage: Tiller
istio: ingressgateway
release: istio
service.istio.io/canonical-name: istio-ingressgateway
service.istio.io/canonical-revision: latest
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
weight: 2
- preference:
matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- ppc64le
weight: 2
- preference:
matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- s390x
weight: 2
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
- ppc64le
- s390x
containers:
- args:
- proxy
- router
- --domain
- $(POD_NAMESPACE).svc.cluster.local
- --proxyLogLevel=warning
- --proxyComponentLogLevel=misc:error
- --log_output_level=default:info
- --serviceCluster
- istio-ingressgateway
- --trust-domain=cluster.local
env:
- name: JWT_POLICY
value: first-party-jwt
- name: PILOT_CERT_PROVIDER
value: istiod
- name: CA_ADDR
value: istiod.istio-system.svc:15012
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: SERVICE_ACCOUNT
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.serviceAccountName
- name: CANONICAL_SERVICE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.labels['service.istio.io/canonical-name']
- name: CANONICAL_REVISION
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.labels['service.istio.io/canonical-revision']
- name: ISTIO_META_WORKLOAD_NAME
value: istio-ingressgateway
- name: ISTIO_META_OWNER
value: kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-ingressgateway
- name: ISTIO_META_MESH_ID
value: cluster.local
- name: ISTIO_META_ROUTER_MODE
value: sni-dnat
- name: ISTIO_META_CLUSTER_ID
value: Kubernetes
image: docker.io/istio/proxyv2:1.6.0
imagePullPolicy: IfNotPresent
name: istio-proxy
ports:
- containerPort: 15020
protocol: TCP
- containerPort: 8080
protocol: TCP
- containerPort: 8443
protocol: TCP
- containerPort: 31400
protocol: TCP
- containerPort: 15443
protocol: TCP
- containerPort: 15011
protocol: TCP
- containerPort: 15012
protocol: TCP
- containerPort: 8060
protocol: TCP
- containerPort: 853
protocol: TCP
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
readinessProbe:
failureThreshold: 30
httpGet:
path: /healthz/ready
port: 15021
scheme: HTTP
initialDelaySeconds: 1
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "2"
memory: 1Gi
requests:
cpu: 10m
memory: 40Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/istio/proxy
name: istio-envoy
- mountPath: /etc/istio/config
name: config-volume
- mountPath: /var/run/secrets/istio
name: istiod-ca-cert
- mountPath: /var/run/ingress_gateway
name: ingressgatewaysdsudspath
- mountPath: /etc/istio/pod
name: podinfo
- mountPath: /etc/istio/ingressgateway-certs
name: ingressgateway-certs
readOnly: true
- mountPath: /etc/istio/ingressgateway-ca-certs
name: ingressgateway-ca-certs
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: istio-ingressgateway-service-account
serviceAccountName: istio-ingressgateway-service-account
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 420
name: istio-ca-root-cert
name: istiod-ca-cert
- downwardAPI:
defaultMode: 420
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.labels
path: labels
- fieldRef:
apiVersion: v1
fieldPath: metadata.annotations
path: annotations
name: podinfo
- emptyDir: {}
name: istio-envoy
- emptyDir: {}
name: ingressgatewaysdsudspath
- configMap:
defaultMode: 420
name: istio
optional: true
name: config-volume
- name: ingressgateway-certs
secret:
defaultMode: 420
optional: true
secretName: istio-ingressgateway-certs
- name: ingressgateway-ca-certs
secret:
defaultMode: 420
optional: true
secretName: istio-ingressgateway-ca-certs
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2020-07-24T11:01:17Z"
lastUpdateTime: "2020-07-24T11:01:17Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2020-07-24T11:00:29Z"
lastUpdateTime: "2020-07-24T11:01:17Z"
message: ReplicaSet "istio-ingressgateway-569d44555d" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1
```

Affected product area (please put an X in all that apply)

[x] Configuration Infrastructure
[x] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[x] User Experience
[x] Developer Infrastructure

Affected features (please put an X in all that apply)

[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane

Expected behavior

Steps to reproduce the bug

1. install istio cluster https://istio.io/latest/docs/setup/install/standalone-operator/
2. 
   

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

How was Istio installed?

IstioOperator

Environment where bug was observed (cloud vendor, OS, etc)

Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.

[tanjunchen]

@esnible

[howardjohn]

You cannot remove the gateway from the mesh, only sidecars. remove-from-mesh just removes the sidecar container. The error message should be better, but its expected this fails

[esnible]

If anyone is still interested in a fix for this let me know and I will update the PR https://github.com/istio/istio/pull/25887 to resolve the merge conflict.