Serving: Stuck in RevisionMissing and "Unknown" state
In what area(s)?
/area networking
What version of Knative?
v0.11.0
Expected Behavior
Service is created successfully
Actual Behavior
Resources stuck in various bad states:
$ k get all
NAME READY STATUS RESTARTS AGE
pod/knative-service-hl52c-deployment-76bd8bfbdb-vphtq 2/2 Running 0 6m7s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/knative-service-hl52c-deployment 1/1 1 1 6m7s
NAME DESIRED CURRENT READY AGE
replicaset.apps/knative-service-hl52c-deployment-76bd8bfbdb 1 1 1 6m7s
NAME URL LATESTCREATED LATESTREADY READY REASON
service.serving.knative.dev/knative-service http://knative-service.knative-1-8760.example.com knative-service-hl52c Unknown RevisionMissing
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON
revision.serving.knative.dev/knative-service-hl52c knative-service 1 Unknown
NAME LATESTCREATED LATESTREADY READY REASON
configuration.serving.knative.dev/knative-service knative-service-hl52c Unknown
NAME URL READY REASON
route.serving.knative.dev/knative-service http://knative-service.knative-1-8760.example.com Unknown RevisionMissing
apiVersion: v1
items:
- apiVersion: serving.knative.dev/v1
kind: Revision
metadata:
creationTimestamp: "2019-12-19T23:19:27Z"
generateName: knative-service-
generation: 1
labels:
serving.knative.dev/configuration: knative-service
serving.knative.dev/configurationGeneration: "1"
serving.knative.dev/service: knative-service
name: knative-service-hl52c
namespace: knative-1-8760
ownerReferences:
- apiVersion: serving.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Configuration
name: knative-service
uid: 3f1f7137-89b6-45bc-88b0-babccd4d607a
resourceVersion: "1065"
selfLink: /apis/serving.knative.dev/v1/namespaces/knative-1-8760/revisions/knative-service-hl52c
uid: 4677f591-05b9-447b-8b29-4abb8d4ff663
spec:
containerConcurrency: 0
containers:
- image: gcr.io/istio-testing/app:latest
name: user-container
readinessProbe:
successThreshold: 1
tcpSocket:
port: 0
resources: {}
timeoutSeconds: 300
status:
conditions:
- lastTransitionTime: "2019-12-19T23:19:28Z"
reason: Deploying
severity: Info
status: Unknown
type: Active
- lastTransitionTime: "2019-12-19T23:19:28Z"
status: Unknown
type: ContainerHealthy
- lastTransitionTime: "2019-12-19T23:19:28Z"
status: Unknown
type: Ready
- lastTransitionTime: "2019-12-19T23:19:28Z"
status: "True"
type: ResourcesAvailable
imageDigest: gcr.io/istio-testing/app@sha256:80790eb8ab6a4453e3fa8c2bc84e2d6b0cef095c4ceec79bc17cbf01fceb72cc
logUrl: http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana#/discover?_a=(query:(match:(kubernetes.labels.knative-dev%2FrevisionUID:(query:'4677f591-05b9-447b-8b29-4abb8d4ff663',type:phrase))))
observedGeneration: 1
kind: List
metadata:
resourceVersion: ""
selfLink: ""
apiVersion: v1
items:
- apiVersion: serving.knative.dev/v1
kind: Configuration
metadata:
annotations:
serving.knative.dev/forceUpgrade: "true"
creationTimestamp: "2019-12-19T23:19:27Z"
generation: 1
labels:
serving.knative.dev/route: knative-service
serving.knative.dev/service: knative-service
name: knative-service
namespace: knative-1-8760
ownerReferences:
- apiVersion: serving.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Service
name: knative-service
uid: 0d7555d6-d266-4886-884a-440e789f1a49
resourceVersion: "1042"
selfLink: /apis/serving.knative.dev/v1/namespaces/knative-1-8760/configurations/knative-service
uid: 3f1f7137-89b6-45bc-88b0-babccd4d607a
spec:
template:
metadata:
creationTimestamp: null
spec:
containerConcurrency: 0
containers:
- image: gcr.io/istio-testing/app:latest
name: user-container
readinessProbe:
successThreshold: 1
tcpSocket:
port: 0
resources: {}
timeoutSeconds: 300
status:
conditions:
- lastTransitionTime: "2019-12-19T23:19:27Z"
status: Unknown
type: Ready
latestCreatedRevisionName: knative-service-hl52c
observedGeneration: 1
kind: List
metadata:
resourceVersion: ""
selfLink: ""
Basically everything is unknown with no real indication about what is going wrong
Steps to Reproduce the Problem
I am trying to get some knative smoke tests integrated into Istio's tests so we don't break things accidentally. See PR https://github.com/istio/istio/pull/19675. It seems fairly reproducible on a fresh cluster, running those steps.
I am probably doing something wrong, but all of the status messages and logs are not leading me in the right direction
howardjohn
All 43 comments
We are working on the stuck in 'Unknown' problem. See #5076
Regarding your current problem,
- Is this 100% reproducible or intermittent? If reproducible can you send the YAML and/or steps you went through?
- What does your deployment look like?
kubectl get deployment -l \ "serving.knative.dev/service=knative-service" --output yaml - Do you see any events from your KService or Deployment?
kubectl describe ksvc knative-serviceorkubectl describe deployment -l "serving.knative.dev/service=knative-service" - Do you see any logs in your controller
kubectl logs deployment/controller -n knative-serving | grep "error" | less
dgerd
on 20 Dec 2019
Is this 100% reproducible or intermittent? If reproducible can you send the YAML and/or steps you went through?
I thought it was but then I just tried it again and it worked :confused:. Seems to be intermittent.
What does your deployment look like? kubectl get deployment -l \ "serving.knative.dev/service=knative-service" --output yaml
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2019-12-20T17:07:46Z"
generation: 1
labels:
app: knative-service-2hgd6
serving.knative.dev/configuration: knative-service
serving.knative.dev/configurationGeneration: "1"
serving.knative.dev/revision: knative-service-2hgd6
serving.knative.dev/revisionUID: 3e310cb3-234b-11ea-bf77-42010a8000a9
serving.knative.dev/service: knative-service
name: knative-service-2hgd6-deployment
namespace: knative-1-49090
ownerReferences:
- apiVersion: serving.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Revision
name: knative-service-2hgd6
uid: 3e310cb3-234b-11ea-bf77-42010a8000a9
resourceVersion: "3733964"
selfLink: /apis/extensions/v1beta1/namespaces/knative-1-49090/deployments/knative-service-2hgd6-deployment
uid: 3e6110bf-234b-11ea-bf77-42010a8000a9
spec:
progressDeadlineSeconds: 120
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
serving.knative.dev/revisionUID: 3e310cb3-234b-11ea-bf77-42010a8000a9
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
traffic.sidecar.istio.io/includeOutboundIPRanges: '*'
creationTimestamp: null
labels:
app: knative-service-2hgd6
serving.knative.dev/configuration: knative-service
serving.knative.dev/configurationGeneration: "1"
serving.knative.dev/revision: knative-service-2hgd6
serving.knative.dev/revisionUID: 3e310cb3-234b-11ea-bf77-42010a8000a9
serving.knative.dev/service: knative-service
spec:
containers:
- env:
- name: PORT
value: "8080"
- name: K_REVISION
value: knative-service-2hgd6
- name: K_CONFIGURATION
value: knative-service
- name: K_SERVICE
value: knative-service
image: gcr.io/istio-testing/app@sha256:1691b71601c9ad4fe7a003cf295ae58bbc01ef753e393ac36acf1c03f6f53d56
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
httpGet:
path: /wait-for-drain
port: 8022
scheme: HTTP
name: user-container
ports:
- containerPort: 8080
name: user-port
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- mountPath: /var/log
name: knative-var-log
- env:
- name: SERVING_NAMESPACE
value: knative-1-49090
- name: SERVING_SERVICE
value: knative-service
- name: SERVING_CONFIGURATION
value: knative-service
- name: SERVING_REVISION
value: knative-service-2hgd6
- name: QUEUE_SERVING_PORT
value: "8012"
- name: CONTAINER_CONCURRENCY
value: "0"
- name: REVISION_TIMEOUT_SECONDS
value: "300"
- name: SERVING_POD
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SERVING_POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: SERVING_LOGGING_CONFIG
value: |-
{
"level": "info",
"development": false,
"outputPaths": ["stdout"],
"errorOutputPaths": ["stderr"],
"encoding": "json",
"encoderConfig": {
"timeKey": "ts",
"levelKey": "level",
"nameKey": "logger",
"callerKey": "caller",
"messageKey": "msg",
"stacktraceKey": "stacktrace",
"lineEnding": "",
"levelEncoder": "",
"timeEncoder": "iso8601",
"durationEncoder": "",
"callerEncoder": ""
}
}
- name: SERVING_LOGGING_LEVEL
- name: SERVING_REQUEST_LOG_TEMPLATE
- name: SERVING_REQUEST_METRICS_BACKEND
value: prometheus
- name: TRACING_CONFIG_BACKEND
value: none
- name: TRACING_CONFIG_ZIPKIN_ENDPOINT
- name: TRACING_CONFIG_STACKDRIVER_PROJECT_ID
- name: TRACING_CONFIG_DEBUG
value: "false"
- name: TRACING_CONFIG_SAMPLE_RATE
value: "0.100000"
- name: USER_PORT
value: "8080"
- name: SYSTEM_NAMESPACE
value: knative-serving
- name: METRICS_DOMAIN
value: knative.dev/internal/serving
- name: USER_CONTAINER_NAME
value: user-container
- name: ENABLE_VAR_LOG_COLLECTION
value: "false"
- name: VAR_LOG_VOLUME_NAME
value: knative-var-log
- name: INTERNAL_VOLUME_PATH
value: /var/knative-internal
- name: SERVING_READINESS_PROBE
value: '{"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}'
- name: ENABLE_PROFILING
value: "false"
- name: SERVING_ENABLE_PROBE_REQUEST_LOG
value: "false"
image: gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:077d82a8f7b3f8c645e95abdf20acf1f1c5ea4d2215aa43ac707920914db5cf8
imagePullPolicy: IfNotPresent
name: queue-proxy
ports:
- containerPort: 8022
name: http-queueadm
protocol: TCP
- containerPort: 9090
name: queue-metrics
protocol: TCP
- containerPort: 9091
name: http-usermetric
protocol: TCP
- containerPort: 8012
name: queue-port
protocol: TCP
readinessProbe:
exec:
command:
- /ko-app/queue
- -probe-period
- "0"
failureThreshold: 3
periodSeconds: 1
successThreshold: 1
timeoutSeconds: 10
resources:
requests:
cpu: 25m
securityContext:
allowPrivilegeEscalation: false
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 300
volumes:
- emptyDir: {}
name: knative-var-log
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2019-12-20T17:07:51Z"
lastUpdateTime: "2019-12-20T17:07:51Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2019-12-20T17:07:46Z"
lastUpdateTime: "2019-12-20T17:07:51Z"
message: ReplicaSet "knative-service-2hgd6-deployment-75757bb8b6" has successfully
progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1
kind: List
metadata:
resourceVersion: ""
selfLink: ""
Do you see any events from your KService or Deployment? kubectl describe ksvc knative-service or kubectl describe deployment -l "serving.knative.dev/service=knative-service"
Name: knative-service
Namespace: knative-1-49090
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"serving.knative.dev/v1alpha1","kind":"Service","metadata":{"annotations":{},"name":"knative-service","namespace":"knative-1...
serving.knative.dev/forceUpgrade: true
API Version: serving.knative.dev/v1
Kind: Service
Metadata:
Creation Timestamp: 2019-12-20T17:07:44Z
Generation: 1
Resource Version: 3733915
Self Link: /apis/serving.knative.dev/v1/namespaces/knative-1-49090/services/knative-service
UID: 3d5e3daf-234b-11ea-bf77-42010a8000a9
Spec:
Run Latest:
Configuration:
Revision Template:
Spec:
Container:
Image: gcr.io/istio-testing/app:latest
Status:
Conditions:
Last Transition Time: 2019-12-20T17:07:46Z
Status: Unknown
Type: ConfigurationsReady
Last Transition Time: 2019-12-20T17:07:46Z
Message: Configuration "knative-service" is waiting for a Revision to become ready.
Reason: RevisionMissing
Status: Unknown
Type: Ready
Last Transition Time: 2019-12-20T17:07:46Z
Message: Configuration "knative-service" is waiting for a Revision to become ready.
Reason: RevisionMissing
Status: Unknown
Type: RoutesReady
Latest Created Revision Name: knative-service-2hgd6
Observed Generation: 1
URL: http://knative-service.knative-1-49090.example.com
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 3m2s service-controller Created Configuration "knative-service"
Normal Created 3m2s service-controller Created Route "knative-service"
Normal Updated 3m2s (x2 over 3m2s) service-controller Updated Service "knative-service"
```
Name: knative-service-2hgd6-deployment
Namespace: knative-1-49090
CreationTimestamp: Fri, 20 Dec 2019 09:07:46 -0800
Labels: app=knative-service-2hgd6
serving.knative.dev/configuration=knative-service
serving.knative.dev/configurationGeneration=1
serving.knative.dev/revision=knative-service-2hgd6
serving.knative.dev/revisionUID=3e310cb3-234b-11ea-bf77-42010a8000a9
serving.knative.dev/service=knative-service
Annotations: deployment.kubernetes.io/revision: 1
Selector: serving.knative.dev/revisionUID=3e310cb3-234b-11ea-bf77-42010a8000a9
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=knative-service-2hgd6
serving.knative.dev/configuration=knative-service
serving.knative.dev/configurationGeneration=1
serving.knative.dev/revision=knative-service-2hgd6
serving.knative.dev/revisionUID=3e310cb3-234b-11ea-bf77-42010a8000a9
serving.knative.dev/service=knative-service
Annotations: traffic.sidecar.istio.io/includeOutboundIPRanges: *
Containers:
user-container:
Image: gcr.io/istio-testing/app@sha256:1691b71601c9ad4fe7a003cf295ae58bbc01ef753e393ac36acf1c03f6f53d56
Port: 8080/TCP
Host Port: 0/TCP
Environment:
PORT: 8080
K_REVISION: knative-service-2hgd6
K_CONFIGURATION: knative-service
K_SERVICE: knative-service
Mounts:
/var/log from knative-var-log (rw)
queue-proxy:
Image: gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:077d82a8f7b3f8c645e95abdf20acf1f1c5ea4d2215aa43ac707920914db5cf8
Ports: 8022/TCP, 9090/TCP, 9091/TCP, 8012/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Requests:
cpu: 25m
Readiness: exec [/ko-app/queue -probe-period 0] delay=0s timeout=10s period=1s #success=1 #failure=3
Environment:
SERVING_NAMESPACE: knative-1-49090
SERVING_SERVICE: knative-service
SERVING_CONFIGURATION: knative-service
SERVING_REVISION: knative-service-2hgd6
QUEUE_SERVING_PORT: 8012
CONTAINER_CONCURRENCY: 0
REVISION_TIMEOUT_SECONDS: 300
SERVING_POD: (v1:metadata.name)
SERVING_POD_IP: (v1:status.podIP)
SERVING_LOGGING_CONFIG: {
"level": "info",
"development": false,
"outputPaths": ["stdout"],
"errorOutputPaths": ["stderr"],
"encoding": "json",
"encoderConfig": {
"timeKey": "ts",
"levelKey": "level",
"nameKey": "logger",
"callerKey": "caller",
"messageKey": "msg",
"stacktraceKey": "stacktrace",
"lineEnding": "",
"levelEncoder": "",
"timeEncoder": "iso8601",
"durationEncoder": "",
"callerEncoder": ""
}
}
SERVING_LOGGING_LEVEL:
SERVING_REQUEST_LOG_TEMPLATE:
SERVING_REQUEST_METRICS_BACKEND: prometheus
TRACING_CONFIG_BACKEND: none
TRACING_CONFIG_ZIPKIN_ENDPOINT:
TRACING_CONFIG_STACKDRIVER_PROJECT_ID:
TRACING_CONFIG_DEBUG: false
TRACING_CONFIG_SAMPLE_RATE: 0.100000
USER_PORT: 8080
SYSTEM_NAMESPACE: knative-serving
METRICS_DOMAIN: knative.dev/internal/serving
USER_CONTAINER_NAME: user-container
ENABLE_VAR_LOG_COLLECTION: false
VAR_LOG_VOLUME_NAME: knative-var-log
INTERNAL_VOLUME_PATH: /var/knative-internal
SERVING_READINESS_PROBE: {"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}
ENABLE_PROFILING: false
SERVING_ENABLE_PROBE_REQUEST_LOG: false
Mounts:
Volumes:
knative-var-log:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets:
NewReplicaSet: knative-service-2hgd6-deployment-75757bb8b6 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 3m17s deployment-controller Scaled up replica set knative-service-2hgd6-deployment-75757bb8b6 to 1
````
Do you see any logs in your controller kubectl logs deployment/controller -n knative-serving | grep "error" | less
With the above info I grabbed the logs but forgot to copy them and the deployment got deleted so I lost the state :slightly_frowning_face:. There were messages about the object has been modified; please apply your changes to the latest version and try again though. I reran the test and everything passed, and the logs show these errors:
{"level":"info","ts":"2019-12-20T17:15:53.534Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-logging\" config was added or updated: &logging.Config{LoggingConfig:\"{\\n \\\"level\\\": \\\"info\\\",\\n \\\"development\\\": false,\\n \\\"outputPaths\\\": [\\\"stdout\\\"],\\n \\\"errorOutputPaths\\\": [\\\"stderr\\\"],\\n \\\"encoding\\\": \\\"json\\\",\\n \\\"encoderConfig\\\": {\\n \\\"timeKey\\\": \\\"ts\\\",\\n \\\"levelKey\\\": \\\"level\\\",\\n \\\"nameKey\\\": \\\"logger\\\",\\n \\\"callerKey\\\": \\\"caller\\\",\\n \\\"messageKey\\\": \\\"msg\\\",\\n \\\"stacktraceKey\\\": \\\"stacktrace\\\",\\n \\\"lineEnding\\\": \\\"\\\",\\n \\\"levelEncoder\\\": \\\"\\\",\\n \\\"timeEncoder\\\": \\\"iso8601\\\",\\n \\\"durationEncoder\\\": \\\"\\\",\\n \\\"callerEncoder\\\": \\\"\\\"\\n }\\n}\", LoggingLevel:map[string]zapcore.Level{}}","commit":"96cffff","knative.dev/controller":"revision-controller"}
{"level":"warn","ts":"2019-12-20T17:15:55.316Z","logger":"controller.configuration-controller","caller":"configuration/configuration.go:91","msg":"Failed to update configuration status","commit":"96cffff","knative.dev/controller":"configuration-controller","knative.dev/traceid":"54809710-3982-41f7-b196-93c7fb2071ef","knative.dev/key":"knative-1-12941/knative-service","error":"Operation cannot be fulfilled on configurations.serving.knative.dev \"knative-service\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"error","ts":"2019-12-20T17:15:55.316Z","logger":"controller.configuration-controller","caller":"controller/controller.go:376","msg":"Reconcile error","commit":"96cffff","knative.dev/controller":"configuration-controller","error":"Operation cannot be fulfilled on configurations.serving.knative.dev \"knative-service\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:376\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:362\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func2\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:310"}
{"level":"warn","ts":"2019-12-20T17:15:55.606Z","logger":"controller.configuration-controller","caller":"configuration/configuration.go:91","msg":"Failed to update configuration status","commit":"96cffff","knative.dev/controller":"configuration-controller","knative.dev/traceid":"d0f80f51-419d-4747-ab13-5cc6cbcd8b50","knative.dev/key":"knative-1-12941/knative-service","error":"Operation cannot be fulfilled on configurations.serving.knative.dev \"knative-service\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"error","ts":"2019-12-20T17:15:55.606Z","logger":"controller.configuration-controller","caller":"controller/controller.go:376","msg":"Reconcile error","commit":"96cffff","knative.dev/controller":"configuration-controller","error":"Operation cannot be fulfilled on configurations.serving.knative.dev \"knative-service\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:376\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:362\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func2\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:310"}
{"level":"warn","ts":"2019-12-20T17:15:55.893Z","logger":"controller.configuration-controller","caller":"configuration/configuration.go:91","msg":"Failed to update configuration status","commit":"96cffff","knative.dev/controller":"configuration-controller","knative.dev/traceid":"653c00a7-54c7-4dc8-aa07-e48c9c91cac4","knative.dev/key":"knative-1-12941/knative-service","error":"Operation cannot be fulfilled on configurations.serving.knative.dev \"knative-service\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"error","ts":"2019-12-20T17:15:55.893Z","logger":"controller.configuration-controller","caller":"controller/controller.go:376","msg":"Reconcile error","commit":"96cffff","knative.dev/controller":"configuration-controller","error":"Operation cannot be fulfilled on configurations.serving.knative.dev \"knative-service\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:376\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:362\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func2\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:310"}
{"level":"warn","ts":"2019-12-20T17:15:56.271Z","logger":"controller.configuration-controller","caller":"configuration/configuration.go:91","msg":"Failed to update configuration status","commit":"96cffff","knative.dev/controller":"configuration-controller","knative.dev/traceid":"d84f96b0-dee3-49a4-ae48-419b06975d36","knative.dev/key":"knative-1-12941/knative-service","error":"Operation cannot be fulfilled on configurations.serving.knative.dev \"knative-service\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"error","ts":"2019-12-20T17:15:56.271Z","logger":"controller.configuration-controller","caller":"controller/controller.go:376","msg":"Reconcile error","commit":"96cffff","knative.dev/controller":"configuration-controller","error":"Operation cannot be fulfilled on configurations.serving.knative.dev \"knative-service\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:376\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:362\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func2\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:310"}
{"level":"warn","ts":"2019-12-20T17:15:56.967Z","logger":"controller.configuration-controller","caller":"configuration/configuration.go:91","msg":"Failed to update configuration status","commit":"96cffff","knative.dev/controller":"configuration-controller","knative.dev/traceid":"9b7d9288-3bf5-49f4-b932-02df9e34175e","knative.dev/key":"knative-1-12941/knative-service","error":"Operation cannot be fulfilled on configurations.serving.knative.dev \"knative-service\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"error","ts":"2019-12-20T17:15:56.967Z","logger":"controller.configuration-controller","caller":"controller/controller.go:376","msg":"Reconcile error","commit":"96cffff","knative.dev/controller":"configuration-controller","error":"Operation cannot be fulfilled on configurations.serving.knative.dev \"knative-service\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:376\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:362\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func2\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:310"}
{"level":"error","ts":"2019-12-20T17:15:59.598Z","logger":"controller.route-controller","caller":"controller/controller.go:376","msg":"Reconcile error","commit":"96cffff","knative.dev/controller":"route-controller","error":"failed to update Ingress: Operation cannot be fulfilled on ingresses.networking.internal.knative.dev \"knative-service\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:376\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:362\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func2\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:310"}
As far as reproducing, the steps are:
- Install istio minimal
- apply knative CRD https://github.com/istio/istio/pull/19675/files#diff-46713653f6fbe067e982048dbd2c9dd3
- apply knative installation https://github.com/istio/istio/pull/19675/files#diff-6cb87a2f954f2216e0b2babf7fbe5b68
- Apply knative service https://github.com/istio/istio/pull/19675/files#diff-2e50ad064a9fe355b66fc4148480c147R61-R72
howardjohn
on 20 Dec 2019
Having the same issue with serving 0.11.1 as well.
guanchzhou
on 30 Dec 2019
I have automation with minikube that creates a bunch of resources right after the cluster is stood up (istio, knative, etc). I'm seeing similar symptoms with a ksvc. I can reproduce it consistently. I'm also using serving 0.11.1. I was able to resolve the issue by adding code to wait for the activator deployment before deploying the ksvc. After this I've tested twice can no longer reproduce the issue. I'm guessing that there's a race condition with standing up knative and creating a ksvc.
tshak
on 16 Jan 2020
Update: After leaving the cluster running all day, the issue eventually came back. The workaround from above seems only reliable for preventing this from a consistent repro in a brand new install. Let me know if there's additional debug data needed.
tshak
on 20 Jan 2020
Update: I just tested with 0.12.1 and the above workaround no longer works. I've verified that the workaround is still reliable on 0.11.1. When investigating on 0.12.1 I noticed that sometimes when I stood up the cluster I got into the stuck Unknown/Deploying and sometimes I got into the IngressNotConfigured state as described in https://github.com/knative/serving/issues/6346. In both cases, this appears to be a state update issue. In the former case the Deployment is rolled out and active, and in the latter case the VirtualService's are created and appear correctly configured with the right gateway.
tshak
on 9 Feb 2020
Same problem on v0.12.0 - totally a showstopper
JCzz
on 22 Feb 2020
Same problem on v0.11.0,k8s v.15.4(binary setup),centos7.7.1908.
but kubeadm(v1.17.0) works fine.
ysjjovo
on 23 Feb 2020
/assign @tcnghia
Assigning to Nghia as this affects our ability to smoke test with Istio, but if folks have an active repro of this, feel free to DM me on slack (mattmoor) as I'd love to be able to debug an active repro of the issue.
mattmoor
on 25 Feb 2020
Same problem on v0.12.0,kubeadm(v.17.0),centos7.7.1908.
Pods info below
[root@kn12 ~]# kubectl get po --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default stock-service-example-first-deployment-5d95c6bc8b-tnzww 2/2 Running 0 14m
istio-system istio-ingressgateway-5c4758f88f-sdh7h 1/1 Running 0 20m
istio-system istio-pilot-55d5864f44-pqtpn 1/1 Running 0 20m
knative-serving activator-6654488fb4-vrrrb 1/1 Running 0 50m
knative-serving autoscaler-7c78fb45b5-hhcmh 1/1 Running 0 50m
knative-serving autoscaler-hpa-84dcb8c78-tsrr5 1/1 Running 0 50m
knative-serving controller-557bf98b95-46zsx 1/1 Running 0 50m
knative-serving networking-istio-7f57947886-v6xm7 1/1 Running 0 50m
knative-serving webhook-bb564cf84-7p2wq 1/1 Running 0 50m
kube-system coredns-6955765f44-fhz9v 1/1 Running 0 17h
kube-system coredns-6955765f44-gn6d2 1/1 Running 0 17h
kube-system etcd-kn12 1/1 Running 1 17h
kube-system kube-apiserver-kn12 1/1 Running 1 17h
kube-system kube-controller-manager-kn12 1/1 Running 1 17h
kube-system kube-flannel-ds-amd64-nlfws 1/1 Running 0 16m
kube-system kube-proxy-kgk2h 1/1 Running 1 17h
kube-system kube-scheduler-kn12 1/1 Running 1 17h
[root@kn12 ~]#
sample app info below
[root@kn12 ~]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/stock-service-example-first-deployment-5d95c6bc8b-tnzww 2/2 Running 0 10m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 17h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/stock-service-example-first-deployment 1/1 1 1 10m
NAME DESIRED CURRENT READY AGE
replicaset.apps/stock-service-example-first-deployment-5d95c6bc8b 1 1 1 10m
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON
revision.serving.knative.dev/stock-service-example-first stock-service-example 1 Unknown Deploying
NAME LATESTCREATED LATESTREADY READY REASON
configuration.serving.knative.dev/stock-service-example stock-service-example-first Unknown
NAME URL READY REASON
route.serving.knative.dev/stock-service-example http://stock-service-example.default.example.com Unknown RevisionMissing
NAME URL LATESTCREATED LATESTREADY READY REASON
service.serving.knative.dev/stock-service-example http://stock-service-example.default.example.com stock-service-example-first Unknown RevisionMissing
[root@kn12 ~]#
Revision describe info
[root@kn12 ~]# kubectl describe revision stock-service-example-first
Name: stock-service-example-first
Namespace: default
Labels: serving.knative.dev/configuration=stock-service-example
serving.knative.dev/configurationGeneration=1
serving.knative.dev/service=stock-service-example
Annotations: <none>
API Version: serving.knative.dev/v1
Kind: Revision
Metadata:
Creation Timestamp: 2020-02-25T02:40:57Z
Generation: 1
Owner References:
API Version: serving.knative.dev/v1alpha1
Block Owner Deletion: true
Controller: true
Kind: Configuration
Name: stock-service-example
UID: 78996df2-33c2-469d-a0f9-772656da4e16
Resource Version: 24611
Self Link: /apis/serving.knative.dev/v1/namespaces/default/revisions/stock-service-example-first
UID: b1414971-19de-43dd-87ed-e50f59bff476
Spec:
Container Concurrency: 0
Containers:
Env:
Name: RESOURCE
Value: stock
Image: registry.cn-chengdu.aliyuncs.com/kn-sample/rest-api-go
Name: user-container
Readiness Probe:
Http Get:
Path: /
Port: 0
Period Seconds: 3
Success Threshold: 1
Resources:
Timeout Seconds: 300
Status:
Conditions:
Last Transition Time: 2020-02-25T02:40:58Z
Reason: Deploying
Severity: Info
Status: Unknown
Type: Active
Last Transition Time: 2020-02-25T02:40:58Z
Reason: Deploying
Status: Unknown
Type: ContainerHealthy
Last Transition Time: 2020-02-25T02:40:58Z
Reason: Deploying
Status: Unknown
Type: Ready
Last Transition Time: 2020-02-25T02:40:58Z
Reason: Deploying
Status: Unknown
Type: ResourcesAvailable
Image Digest: registry.cn-chengdu.aliyuncs.com/kn-sample/rest-api-go@sha256:a12f6f2a6151f19717ec7ba1a91dc1cd2d40acbeed9fd344d3a2a511dc29565e
Log URL: http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana#/discover?_a=(query:(match:(kubernetes.labels.knative-dev%2FrevisionUID:(query:'b1414971-19de-43dd-87ed-e50f59bff476',type:phrase))))
Observed Generation: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning InternalError 12m revision-controller failed to update deployment "stock-service-example-first-deployment": Operation cannot be fulfilled on deployments.apps "stock-service-example-first-deployment": the object has been modified; please apply your changes to the latest version and try again
It would be helpful to share kubectl get knative-internal too. I'm happy to help debug if you can ping me on slack.
mattmoor
on 25 Feb 2020
It would be helpful to share
kubectl get knative-internaltoo. I'm happy to help debug if you can ping me on slack.
[root@kn12 ~]# kubectl get knative-internal
NAME DESIREDSCALE ACTUALSCALE READY REASON
podautoscaler.autoscaling.internal.knative.dev/stock-service-example-first
NAME AGE
image.caching.internal.knative.dev/stock-service-example-first-cache 6h24m
[root@kn12 ~]#
It would be helpful to share
kubectl get knative-internaltoo. I'm happy to help debug if you can ping me on slack.[root@kn12 ~]# kubectl get knative-internal NAME DESIREDSCALE ACTUALSCALE READY REASON podautoscaler.autoscaling.internal.knative.dev/stock-service-example-first NAME AGE image.caching.internal.knative.dev/stock-service-example-first-cache 6h24m [root@kn12 ~]#I don't have an @google.com, @pivotal.io, or @redhat.com email address,my gmail is [email protected]
ysjjovo
on 25 Feb 2020
cc @vagababov KPA not becoming ready.
Anyone should be able to sign up with an invite from slack.knative.dev 馃
mattmoor
on 25 Feb 2020
Can you post the logs from
kubectl logs -n knative-serving $(kubectl get po -n knative-serving | egrep "autoscaler-[^h]" | cut -f1 -d' ') | less
If it's crashing or not starting the file should be short.
vagababov
on 25 Feb 2020
Can you post the logs from
kubectl logs -n knative-serving $(kubectl get po -n knative-serving | egrep "autoscaler-[^h]" | cut -f1 -d' ') | lessIf it's crashing or not starting the file should be short.
That's all logs,seems normal.
[root@kn12 ~]# kubectl logs -n knative-serving $(kubectl get po -n knative-serving | egrep "autoscaler-[^h]" | cut -f1 -d' ')
2020/02/26 01:13:44 Registering 5 clients
2020/02/26 01:13:44 Registering 2 informer factories
2020/02/26 01:13:44 Registering 5 informers
2020/02/26 01:13:44 Registering 2 controllers
{"level":"info","ts":"2020-02-26T01:13:44.872Z","caller":"logging/config.go:108","msg":"Successfully created the logger."}
{"level":"info","ts":"2020-02-26T01:13:44.872Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
{"level":"info","ts":"2020-02-26T01:13:44.925Z","logger":"autoscaler","caller":"profiling/server.go:59","msg":"Profiling enabled: false","commit":"bf0a848"}
{"level":"info","ts":"2020-02-26T01:13:44.926Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/controller.go:72","msg":"Setting up KPA-Class event handlers","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-26T01:13:44.926Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/controller.go:105","msg":"Setting up ConfigMap receivers","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-26T01:13:44.926Z","logger":"autoscaler.metric-controller","caller":"metric/controller.go:50","msg":"Setting up event handlers","commit":"bf0a848","knative.dev/controller":"metric-controller"}
{"level":"info","ts":"2020-02-26T01:13:44.935Z","logger":"autoscaler.kpa-class-podautoscaler-controller.config-store","caller":"configmap/store.go:157","msg":"autoscaler config \"config-autoscaler\" config was added or updated: &autoscaler.Config{EnableScaleToZero:true, EnableGracefulScaledown:false, ContainerConcurrencyTargetFraction:0.7, ContainerConcurrencyTargetDefault:100, TargetUtilization:0.7, RPSTargetDefault:200, TargetBurstCapacity:200, MaxScaleUpRate:1000, MaxScaleDownRate:2, StableWindow:60000000000, PanicWindowPercentage:10, PanicThresholdPercentage:200, TickInterval:2000000000, ScaleToZeroGracePeriod:30000000000}","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-26T01:13:44.936Z","logger":"autoscaler","caller":"metrics/exporter.go:99","msg":"Flushing the existing exporter before setting up the new exporter.","commit":"bf0a848"}
{"level":"info","ts":"2020-02-26T01:13:44.936Z","logger":"autoscaler","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/serving autoscaler prometheus 5000000000 <nil> false 9090 false { false}}. Start the server for Prometheus exporter.","commit":"bf0a848"}
{"level":"info","ts":"2020-02-26T01:13:44.936Z","logger":"autoscaler","caller":"metrics/exporter.go:108","msg":"Successfully updated the metrics exporter; old config: <nil>; new config &{knative.dev/serving autoscaler prometheus 5000000000 <nil> false 9090 false { false}}","commit":"bf0a848"}
{"level":"info","ts":"2020-02-26T01:13:45.431Z","logger":"autoscaler.metric-controller","caller":"controller/controller.go:305","msg":"Starting controller and workers","commit":"bf0a848","knative.dev/controller":"metric-controller"}
{"level":"info","ts":"2020-02-26T01:13:45.434Z","logger":"autoscaler.metric-controller","caller":"controller/controller.go:315","msg":"Started workers","commit":"bf0a848","knative.dev/controller":"metric-controller"}
{"level":"info","ts":"2020-02-26T01:13:45.446Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:305","msg":"Starting controller and workers","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-26T01:13:45.446Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:315","msg":"Started workers","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-26T01:13:45.446Z","logger":"autoscaler.stats-websocket-server","caller":"statserver/server.go:88","msg":"Starting","commit":"bf0a848","address":":8080"}
I0226 01:13:46.586618 1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0226 01:13:49.081547 1 secure_serving.go:123] Serving securely on [::]:8443
[root@kn12 ~]#
Yeah there's no crash here.
So this pod is still never ready?
vagababov
on 26 Feb 2020
Yeah there's no crash here.
So this pod is still never ready?
All pods are in running status.
[root@kn12 ~]# kubectl get po -n knative-serving
NAME READY STATUS RESTARTS AGE
activator-6654488fb4-vrrrb 1/1 Running 2 27h
autoscaler-7c78fb45b5-hhcmh 1/1 Running 2 27h
autoscaler-hpa-84dcb8c78-tsrr5 1/1 Running 2 27h
controller-557bf98b95-46zsx 1/1 Running 2 27h
networking-istio-7f57947886-v6xm7 1/1 Running 2 27h
webhook-bb564cf84-7p2wq 1/1 Running 2 27h
[root@kn12 ~]# kubectl get po
NAME READY STATUS RESTARTS AGE
stock-service-example-first-deployment-5d95c6bc8b-tnzww 2/2 Running 4 27h
[root@kn12 ~]#
/cc @mattmoor
Matt, where did you see autoscaler not starting?
vagababov
on 26 Feb 2020
I just tried another time,It worked!
Context info:kubeadm(k8s v1.17.0),knative v0.12.0.
Normal sample below
[root@kn12 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:17:50Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
[root@kn12 ~]# kubectl get all
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2d1h
service/stock-service-example ExternalName <none> cluster-local-gateway.istio-system.svc.cluster.local <none> 52m
service/stock-service-example-first ClusterIP 10.96.221.229 <none> 80/TCP 52m
service/stock-service-example-first-private ClusterIP 10.96.185.120 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP 52m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/stock-service-example-first-deployment 0/0 0 0 52m
NAME DESIRED CURRENT READY AGE
replicaset.apps/stock-service-example-first-deployment-585568d494 0 0 0 52m
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON
revision.serving.knative.dev/stock-service-example-first stock-service-example stock-service-example-first 1 True
NAME LATESTCREATED LATESTREADY READY REASON
configuration.serving.knative.dev/stock-service-example stock-service-example-first stock-service-example-first True
NAME URL READY REASON
route.serving.knative.dev/stock-service-example http://stock-service-example.default.example.com True
NAME URL LATESTCREATED LATESTREADY READY REASON
service.serving.knative.dev/stock-service-example http://stock-service-example.default.example.com stock-service-example-first stock-service-example-first True
But it's still unknown status with another Environment:
k8s v1.17.3 (binary install)
detail info below:
```
[root@xxx samples]# kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
[root@xxx samples]# kubectl get po --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default stock-service-example-first-deployment-7dd8dfbbcd-rdv6q 2/2 Running 0 3m44s
istio-system istio-ingressgateway-844dd4b69f-s5h2p 1/1 Running 0 109m
istio-system istio-pilot-f95ff47b8-wpgjf 1/1 Running 0 116m
knative-serving activator-858556768f-4m5pk 1/1 Running 0 85m
knative-serving autoscaler-5769f76958-wl26w 1/1 Running 0 85m
knative-serving autoscaler-hpa-5f64f64467-vhs87 1/1 Running 0 85m
knative-serving controller-687466f478-r29jl 1/1 Running 0 85m
knative-serving networking-istio-69fb4fffbc-ldwxz 1/1 Running 0 85m
knative-serving webhook-5d6fc96cf-txhpw 1/1 Running 0 85m
kube-system calico-node-fwpcj 1/1 Running 0 2d18h
kube-system calico-node-xpwvj 1/1 Running 0 2d18h
kube-system calico-typha-7f66875887-8vlmd 1/1 Running 0 2d7h
kube-system calicoctl 1/1 Running 0 2d18h
kube-system coredns-d9459d59c-knvjj 1/1 Running 0 2d7h
kube-system coredns-d9459d59c-mfpnt 1/1 Running 0 2d7h
kube-system tiller-deploy-84456b7d47-hvvqx 1/1 Running 26 2d8h
kube-system tiller-deploy-84456b7d47-kw8kv 1/1 Running 26 2d8h
[root@xxx samples]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/stock-service-example-first-deployment-7dd8dfbbcd-rdv6q 2/2 Running 0 4m1s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 172.254.0.1
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/stock-service-example-first-deployment 1/1 1 1 4m1s
NAME DESIRED CURRENT READY AGE
replicaset.apps/stock-service-example-first-deployment-7dd8dfbbcd 1 1 1 4m1s
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON
revision.serving.knative.dev/stock-service-example-first stock-service-example 1 Unknown Deploying
NAME URL READY REASON
route.serving.knative.dev/stock-service-example http://stock-service-example.default.example.com Unknown RevisionMissing
NAME LATESTCREATED LATESTREADY READY REASON
configuration.serving.knative.dev/stock-service-example stock-service-example-first Unknown
NAME URL LATESTCREATED LATESTREADY READY REASON
service.serving.knative.dev/stock-service-example http://stock-service-example.default.example.com stock-service-example-first Unknown RevisionMissing
```
Compare to normal sample,its service section lack for 'service/stock-service-example-first' and 'service/stock-service-example-first-private',Is there some problem with k8s installation.
ysjjovo
on 26 Feb 2020
where did you see autoscaler not starting
I never said the autoscaler didn't start, I said the KPA didn't become ready. In fact, the resource is never even initialized (from above):
NAME DESIREDSCALE ACTUALSCALE READY REASON
podautoscaler.autoscaling.internal.knative.dev/stock-service-example-first
where did you see autoscaler not starting
I never said the autoscaler didn't start, I said the KPA didn't become ready. In fact, the resource is never even initialized (from above):
NAME DESIREDSCALE ACTUALSCALE READY REASON podautoscaler.autoscaling.internal.knative.dev/stock-service-example-first
Why KPA didn't become ready?
ysjjovo
on 26 Feb 2020
where did you see autoscaler not starting
I never said the autoscaler didn't start, I said the KPA didn't become ready. In fact, the resource is never even initialized (from above):
NAME DESIREDSCALE ACTUALSCALE READY REASON podautoscaler.autoscaling.internal.knative.dev/stock-service-example-firstWhy KPA didn't become ready?
I saw some abnormal messages in k8s binary installation environment.
Last 2th line says ' Cluster doesn't provide requestheader-client-ca-file'.Is there any impact?
[root@xxx ~]# kubectl logs -n knative-serving $(kubectl get po -n knative-serving | egrep "autoscaler-[^h]" | cut -f1 -d' ')
2020/02/26 09:07:05 Registering 5 clients
2020/02/26 09:07:05 Registering 2 informer factories
2020/02/26 09:07:05 Registering 5 informers
2020/02/26 09:07:05 Registering 2 controllers
{"level":"info","ts":"2020-02-26T09:07:05.137Z","caller":"logging/config.go:108","msg":"Successfully created the logger."}
{"level":"info","ts":"2020-02-26T09:07:05.137Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
{"level":"info","ts":"2020-02-26T09:07:05.137Z","logger":"autoscaler","caller":"profiling/server.go:59","msg":"Profiling enabled: false","commit":"bf0a848"}
{"level":"info","ts":"2020-02-26T09:07:05.138Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/controller.go:72","msg":"Setting up KPA-Class event handlers","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-26T09:07:05.138Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/controller.go:105","msg":"Setting up ConfigMap receivers","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-26T09:07:05.138Z","logger":"autoscaler.metric-controller","caller":"metric/controller.go:50","msg":"Setting up event handlers","commit":"bf0a848","knative.dev/controller":"metric-controller"}
{"level":"info","ts":"2020-02-26T09:07:05.144Z","logger":"autoscaler.kpa-class-podautoscaler-controller.config-store","caller":"configmap/store.go:157","msg":"autoscaler config \"config-autoscaler\" config was added or updated: &autoscaler.Config{EnableScaleToZero:true, EnableGracefulScaledown:false, ContainerConcurrencyTargetFraction:0.7, ContainerConcurrencyTargetDefault:100, TargetUtilization:0.7, RPSTargetDefault:200, TargetBurstCapacity:200, MaxScaleUpRate:1000, MaxScaleDownRate:2, StableWindow:60000000000, PanicWindowPercentage:10, PanicThresholdPercentage:200, TickInterval:2000000000, ScaleToZeroGracePeriod:30000000000}","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-26T09:07:05.144Z","logger":"autoscaler","caller":"metrics/exporter.go:99","msg":"Flushing the existing exporter before setting up the new exporter.","commit":"bf0a848"}
{"level":"info","ts":"2020-02-26T09:07:05.144Z","logger":"autoscaler","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/serving autoscaler prometheus 5000000000 <nil> false 9090 false { false}}. Start the server for Prometheus exporter.","commit":"bf0a848"}
{"level":"info","ts":"2020-02-26T09:07:05.144Z","logger":"autoscaler","caller":"metrics/exporter.go:108","msg":"Successfully updated the metrics exporter; old config: <nil>; new config &{knative.dev/serving autoscaler prometheus 5000000000 <nil> false 9090 false { false}}","commit":"bf0a848"}
{"level":"info","ts":"2020-02-26T09:07:05.338Z","logger":"autoscaler.metric-controller","caller":"controller/controller.go:305","msg":"Starting controller and workers","commit":"bf0a848","knative.dev/controller":"metric-controller"}
{"level":"info","ts":"2020-02-26T09:07:05.338Z","logger":"autoscaler.metric-controller","caller":"controller/controller.go:315","msg":"Started workers","commit":"bf0a848","knative.dev/controller":"metric-controller"}
{"level":"info","ts":"2020-02-26T09:07:05.338Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:305","msg":"Starting controller and workers","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-26T09:07:05.338Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:315","msg":"Started workers","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-26T09:07:05.338Z","logger":"autoscaler.stats-websocket-server","caller":"statserver/server.go:88","msg":"Starting","commit":"bf0a848","address":":8080"}
I0226 09:07:06.727868 1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
W0226 09:07:08.827611 1 authentication.go:296] Cluster doesn't provide requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
I0226 09:07:08.831360 1 secure_serving.go:123] Serving securely on [::]:8443
[root@xxx ~]#
Could someone help me out?
ysjjovo
on 28 Feb 2020
@vagababov ping on this since it's the KPA failing to become ready
mattmoor
on 3 Mar 2020
Probably the PA type annotation is missing.
@ysjjovo can you do
kubectl get rev <rev-name> -oyaml and
kubectl get podautoscaler <rev-name> -oyaml
and paste them here please.
(probably knative-service-2hgd6, unless you redeployed).
vagababov
on 3 Mar 2020
Probably the PA type annotation is missing.
@ysjjovo can you do
kubectl get rev <rev-name> -oyamland
kubectl get podautoscaler <rev-name> -oyaml
and paste them here please.
(probably knative-service-2hgd6, unless you redeployed).
Thanks for your reply!
kubectl get all output below
[root@xxx samples]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/stock-service-example-first-deployment-7bcd589f7b-kj5ss 1/2 Running 0 5m1s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 172.254.0.1 <none> 443/TCP 9d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/stock-service-example-first-deployment 0/1 1 0 5m1s
NAME DESIRED CURRENT READY AGE
replicaset.apps/stock-service-example-first-deployment-7bcd589f7b 1 1 0 5m1s
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON
revision.serving.knative.dev/stock-service-example-first stock-service-example 1 Unknown Deploying
NAME URL READY REASON
route.serving.knative.dev/stock-service-example http://stock-service-example.default.example.com Unknown RevisionMissing
NAME LATESTCREATED LATESTREADY READY REASON
configuration.serving.knative.dev/stock-service-example stock-service-example-first Unknown
NAME URL LATESTCREATED LATESTREADY READY REASON
service.serving.knative.dev/stock-service-example http://stock-service-example.default.example.com stock-service-example-first Unknown RevisionMissing
kubectl get rev <rev-name> -oyaml output below
[root@xxx samples]# kubectl get rev stock-service-example-first -oyaml
apiVersion: serving.knative.dev/v1
kind: Revision
metadata:
creationTimestamp: "2020-03-04T08:43:41Z"
generation: 1
labels:
serving.knative.dev/configuration: stock-service-example
serving.knative.dev/configurationGeneration: "1"
serving.knative.dev/service: stock-service-example
name: stock-service-example-first
namespace: default
ownerReferences:
- apiVersion: serving.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Configuration
name: stock-service-example
uid: e5bc55c3-e9e5-4408-89f7-16458c9eb1ff
resourceVersion: "1471608"
selfLink: /apis/serving.knative.dev/v1/namespaces/default/revisions/stock-service-example-first
uid: 9fd497b7-ceba-4d31-86cf-733dbc5bb929
spec:
containerConcurrency: 0
containers:
- env:
- name: RESOURCE
value: stock
image: dev.local/rest-api-go:0.11.0
imagePullPolicy: Never
name: user-container
readinessProbe:
httpGet:
path: /
port: 0
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 1
resources: {}
timeoutSeconds: 300
status:
conditions:
- lastTransitionTime: "2020-03-04T08:43:41Z"
reason: Deploying
severity: Info
status: Unknown
type: Active
- lastTransitionTime: "2020-03-04T08:43:41Z"
reason: Deploying
status: Unknown
type: ContainerHealthy
- lastTransitionTime: "2020-03-04T08:43:41Z"
reason: Deploying
status: Unknown
type: Ready
- lastTransitionTime: "2020-03-04T08:43:41Z"
reason: Deploying
status: Unknown
type: ResourcesAvailable
logUrl: http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana#/discover?_a=(query:(match:(kubernetes.labels.knative-dev%2FrevisionUID:(query:'9fd497b7-ceba-4d31-86cf-733dbc5bb929',type:phrase))))
observedGeneration: 1
kubectl get podautoscaler <rev-name> -oyaml output below
[root@xxx samples]# kubectl get podautoscaler stock-service-example-first -oyaml
apiVersion: autoscaling.internal.knative.dev/v1alpha1
kind: PodAutoscaler
metadata:
creationTimestamp: "2020-03-04T08:43:41Z"
generation: 2
labels:
app: stock-service-example-first
serving.knative.dev/configuration: stock-service-example
serving.knative.dev/configurationGeneration: "1"
serving.knative.dev/revision: stock-service-example-first
serving.knative.dev/revisionUID: 9fd497b7-ceba-4d31-86cf-733dbc5bb929
serving.knative.dev/service: stock-service-example
name: stock-service-example-first
namespace: default
ownerReferences:
- apiVersion: serving.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Revision
name: stock-service-example-first
uid: 9fd497b7-ceba-4d31-86cf-733dbc5bb929
resourceVersion: "1471613"
selfLink: /apis/autoscaling.internal.knative.dev/v1alpha1/namespaces/default/podautoscalers/stock-service-example-first
uid: c8fdc33a-fd9e-4b93-a512-900b9f376b67
spec:
protocolType: http1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: stock-service-example-first-deployment
As I presumed you don't have the annotation:
metadata:
annotations:
autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
and hence PA is not initialized.
- Are your webhooks running?
kubectl get po -n knative-serving
- Are your webhooks configured?
kubectl k get mutatingwebhookconfigurations.admissionregistration.k8s.io
- What is the webhook version?
kubectl get deployments.apps -n knative-serving webhook -oyaml | grep "image:"
vagababov
on 4 Mar 2020
As I presumed you don't have the annotation:
metadata: annotations: autoscaling.knative.dev/class: kpa.autoscaling.knative.devand hence PA is not initialized.
Are your webhooks running?
kubectl get po -n knative-servingAre your webhooks configured?
kubectl k get mutatingwebhookconfigurations.admissionregistration.k8s.ioWhat is the webhook version?
kubectl get deployments.apps -n knative-serving webhook -oyaml | grep "image:"
kubectl commands output below.
[root@SZD-xxx ~]# kubectl get po -n knative-serving
NAME READY STATUS RESTARTS AGE
activator-858556768f-2jlv4 1/1 Running 0 6d3h
autoscaler-5769f76958-8c67q 1/1 Running 0 7d5h
autoscaler-hpa-5f64f64467-rbpjq 1/1 Running 0 6d3h
controller-687466f478-7k64m 1/1 Running 0 6d3h
networking-istio-69fb4fffbc-ldwxz 1/1 Running 0 7d21h
webhook-5d6fc96cf-txhpw 1/1 Running 0 7d21h
[root@SZD-xxx ~]# kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io
NAME CREATED AT
webhook.serving.knative.dev 2020-02-26T09:02:06Z
[root@SZD-xxx ~]# kubectl get deployments.apps -n knative-serving webhook -oyaml | grep "image:"
image: gcr.io/knative-releases/knative.dev/serving/cmd/webhook@sha256:4528b0a54dd4ec91f0398856216b24532566618340c7ef6fd00345b776fb2c10
I tried to add annotation you mentioned.All things remain abnormal state.It seems that it dose not work.Use command kubectl edit rev stock-service-example-firstadd KPA annotation.
[root@SZD-xxx ~]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/stock-service-example-first-deployment-759c98b4c5-m9xbn 1/2 Running 0 6m34s
pod/stock-service-example-first-deployment-7bcd589f7b-kj5ss 1/2 Running 0 21h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 172.254.0.1 <none> 443/TCP 10d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/stock-service-example-first-deployment 0/1 1 0 21h
NAME DESIRED CURRENT READY AGE
replicaset.apps/stock-service-example-first-deployment-759c98b4c5 1 1 0 6m34s
replicaset.apps/stock-service-example-first-deployment-7bcd589f7b 1 1 0 21h
NAME URL LATESTCREATED LATESTREADY READY REASON
service.serving.knative.dev/stock-service-example http://stock-service-example.default.example.com stock-service-example-first Unknown RevisionMissing
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON
revision.serving.knative.dev/stock-service-example-first stock-service-example 1 Unknown Deploying
NAME URL READY REASON
route.serving.knative.dev/stock-service-example http://stock-service-example.default.example.com Unknown RevisionMissing
NAME LATESTCREATED LATESTREADY READY REASON
configuration.serving.knative.dev/stock-service-example stock-service-example-first Unknown
[root@SZD-xxx ~]# kubectl get rev stock-service-example-first -o yaml
apiVersion: serving.knative.dev/v1
kind: Revision
metadata:
annotations:
autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
creationTimestamp: "2020-03-04T08:43:41Z"
generation: 1
labels:
serving.knative.dev/configuration: stock-service-example
serving.knative.dev/configurationGeneration: "1"
serving.knative.dev/service: stock-service-example
name: stock-service-example-first
namespace: default
ownerReferences:
- apiVersion: serving.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Configuration
name: stock-service-example
uid: e5bc55c3-e9e5-4408-89f7-16458c9eb1ff
resourceVersion: "1644448"
selfLink: /apis/serving.knative.dev/v1/namespaces/default/revisions/stock-service-example-first
uid: 9fd497b7-ceba-4d31-86cf-733dbc5bb929
spec:
containerConcurrency: 0
containers:
- env:
- name: RESOURCE
value: stock
image: dev.local/rest-api-go:0.11.0
imagePullPolicy: Never
name: user-container
readinessProbe:
httpGet:
path: /
port: 0
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 1
resources: {}
timeoutSeconds: 300
status:
conditions:
- lastTransitionTime: "2020-03-04T08:43:41Z"
reason: Deploying
severity: Info
status: Unknown
type: Active
- lastTransitionTime: "2020-03-04T08:43:41Z"
reason: Deploying
status: Unknown
type: ContainerHealthy
- lastTransitionTime: "2020-03-04T08:43:41Z"
reason: Deploying
status: Unknown
type: Ready
- lastTransitionTime: "2020-03-04T08:43:41Z"
reason: Deploying
status: Unknown
type: ResourcesAvailable
logUrl: http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana#/discover?_a=(query:(match:(kubernetes.labels.knative-dev%2FrevisionUID:(query:'9fd497b7-ceba-4d31-86cf-733dbc5bb929',type:phrase))))
observedGeneration: 1
Can you get logs from the autoscaler as well?
kubectl logs -n knative-serving <autoscaler-pod-name>
Also kubectl get podautoscaler <rev-name> --oyaml
For some reason it's missing above.
Finally
kubectl describe pod stock-service-example-first-deployment-759c98b4c5-m9xbn is also interesting, since one of the containers doesn't seem to start... I presume it's the user container, rather than QP.
vagababov
on 5 Mar 2020
Can you get logs from the autoscaler as well?
kubectl logs -n knative-serving <autoscaler-pod-name>
Alsokubectl get podautoscaler <rev-name> --oyamlFor some reason it's missing above.
Finally
kubectl describe pod stock-service-example-first-deployment-759c98b4c5-m9xbnis also interesting, since one of the containers doesn't seem to start... I presume it's the user container, rather than QP.
You presumption is right!user container didn't start,because I used a wrong image helloworld-go-eventing.I sorry for that.Now I use a new sample image helloworld-go.
聽Some kubectl commands output below.
kubectl get all
NAME READY STATUS RESTARTS AGE
pod/helloworld-go-vfsdz-deployment-7775fb4d4c-7sr8k 2/2 Running 0 25m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 172.254.0.1 <none> 443/TCP 11d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/helloworld-go-vfsdz-deployment 1/1 1 1 25m
NAME DESIRED CURRENT READY AGE
replicaset.apps/helloworld-go-vfsdz-deployment-7775fb4d4c 1 1 1 25m
NAME LATESTCREATED LATESTREADY READY REASON
configuration.serving.knative.dev/helloworld-go helloworld-go-vfsdz Unknown
NAME URL LATESTCREATED LATESTREADY READY REASON
service.serving.knative.dev/helloworld-go http://helloworld-go.default.example.com helloworld-go-vfsdz Unknown RevisionMissing
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON
revision.serving.knative.dev/helloworld-go-vfsdz helloworld-go 1 Unknown Deploying
NAME URL READY REASON
route.serving.knative.dev/helloworld-go http://helloworld-go.default.example.com Unknown RevisionMissing
kubectl get rev helloworld-go-vfsdz -o yaml
apiVersion: serving.knative.dev/v1
kind: Revision
metadata:
creationTimestamp: "2020-03-06T03:02:04Z"
generateName: helloworld-go-
generation: 1
labels:
serving.knative.dev/configuration: helloworld-go
serving.knative.dev/configurationGeneration: "1"
serving.knative.dev/service: helloworld-go
name: helloworld-go-vfsdz
namespace: default
ownerReferences:
- apiVersion: serving.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Configuration
name: helloworld-go
uid: f003dc1f-fe8b-4a62-b7ca-e9ba07e28ece
resourceVersion: "1808216"
selfLink: /apis/serving.knative.dev/v1/namespaces/default/revisions/helloworld-go-vfsdz
uid: 70d2ecd0-cf74-470e-83bc-569f0467eb3c
spec:
containerConcurrency: 0
containers:
- env:
- name: TARGET
value: Go Sample v1
image: dev.local/helloworld-go
imagePullPolicy: Never
name: user-container
readinessProbe:
successThreshold: 1
tcpSocket:
port: 0
resources: {}
timeoutSeconds: 300
status:
conditions:
- lastTransitionTime: "2020-03-06T03:02:04Z"
reason: Deploying
severity: Info
status: Unknown
type: Active
- lastTransitionTime: "2020-03-06T03:02:04Z"
reason: Deploying
status: Unknown
type: ContainerHealthy
- lastTransitionTime: "2020-03-06T03:02:04Z"
reason: Deploying
status: Unknown
type: Ready
- lastTransitionTime: "2020-03-06T03:02:04Z"
reason: Deploying
status: Unknown
type: ResourcesAvailable
logUrl: http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana#/discover?_a=(query:(match:(kubernetes.labels.knative-dev%2FrevisionUID:(query:'70d2ecd0-cf74-470e-83bc-569f0467eb3c',type:phrase))))
observedGeneration: 1
kubectl logs -n knative-serving autoscaler-5769f76958-8c67q
2020/02/27 01:06:44 Registering 5 clients
2020/02/27 01:06:44 Registering 2 informer factories
2020/02/27 01:06:44 Registering 5 informers
2020/02/27 01:06:44 Registering 2 controllers
{"level":"info","ts":"2020-02-27T01:06:44.074Z","caller":"logging/config.go:108","msg":"Successfully created the logger."}
{"level":"info","ts":"2020-02-27T01:06:44.075Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
{"level":"info","ts":"2020-02-27T01:06:44.075Z","logger":"autoscaler","caller":"profiling/server.go:59","msg":"Profiling enabled: false","commit":"bf0a848"}
{"level":"info","ts":"2020-02-27T01:06:44.075Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/controller.go:72","msg":"Setting up KPA-Class event handlers","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-27T01:06:44.075Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/controller.go:105","msg":"Setting up ConfigMap receivers","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-27T01:06:44.076Z","logger":"autoscaler.metric-controller","caller":"metric/controller.go:50","msg":"Setting up event handlers","commit":"bf0a848","knative.dev/controller":"metric-controller"}
{"level":"info","ts":"2020-02-27T01:06:44.155Z","logger":"autoscaler.kpa-class-podautoscaler-controller.config-store","caller":"configmap/store.go:157","msg":"autoscaler config \"config-autoscaler\" config was added or updated: &autoscaler.Config{EnableScaleToZero:true, EnableGracefulScaledown:false, ContainerConcurrencyTargetFraction:0.7, ContainerConcurrencyTargetDefault:100, TargetUtilization:0.7, RPSTargetDefault:200, TargetBurstCapacity:200, MaxScaleUpRate:1000, MaxScaleDownRate:2, StableWindow:60000000000, PanicWindowPercentage:10, PanicThresholdPercentage:200, TickInterval:2000000000, ScaleToZeroGracePeriod:30000000000}","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-27T01:06:44.155Z","logger":"autoscaler","caller":"metrics/exporter.go:99","msg":"Flushing the existing exporter before setting up the new exporter.","commit":"bf0a848"}
{"level":"info","ts":"2020-02-27T01:06:44.155Z","logger":"autoscaler","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/serving autoscaler prometheus 5000000000 <nil> false 9090 false { false}}. Start the server for Prometheus exporter.","commit":"bf0a848"}
{"level":"info","ts":"2020-02-27T01:06:44.155Z","logger":"autoscaler","caller":"metrics/exporter.go:108","msg":"Successfully updated the metrics exporter; old config: <nil>; new config &{knative.dev/serving autoscaler prometheus 5000000000 <nil> false 9090 false { false}}","commit":"bf0a848"}
{"level":"info","ts":"2020-02-27T01:06:44.277Z","logger":"autoscaler.metric-controller","caller":"controller/controller.go:305","msg":"Starting controller and workers","commit":"bf0a848","knative.dev/controller":"metric-controller"}
{"level":"info","ts":"2020-02-27T01:06:44.277Z","logger":"autoscaler.metric-controller","caller":"controller/controller.go:315","msg":"Started workers","commit":"bf0a848","knative.dev/controller":"metric-controller"}
{"level":"info","ts":"2020-02-27T01:06:44.277Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:305","msg":"Starting controller and workers","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-27T01:06:44.277Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:315","msg":"Started workers","commit":"bf0a848","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2020-02-27T01:06:44.277Z","logger":"autoscaler.stats-websocket-server","caller":"statserver/server.go:88","msg":"Starting","commit":"bf0a848","address":":8080"}
I0227 01:06:45.553648 1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
W0227 01:06:47.675239 1 authentication.go:296] Cluster doesn't provide requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
I0227 01:06:47.750746 1 secure_serving.go:123] Serving securely on [::]:8443
{"level":"error","ts":"2020-02-29T07:43:46.626Z","logger":"autoscaler.stats-websocket-server","caller":"statserver/server.go:152","msg":"Handler exiting on error: &websocket.netError{msg:\"read tcp 172.1.1.35:8080->172.1.0.40:44420: read: connection timed out\", temporary:false, timeout:true}","commit":"bf0a848","address":":8080","stacktrace":"knative.dev/serving/pkg/autoscaler/statserver.(*Server).Handler\n\t/home/prow/go/src/knative.dev/serving/pkg/autoscaler/statserver/server.go:152\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2387\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2802\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1890"}
kubectl get podautoscaler helloworld-go-vfsdz -o yaml
apiVersion: autoscaling.internal.knative.dev/v1alpha1
kind: PodAutoscaler
metadata:
creationTimestamp: "2020-03-06T03:02:04Z"
generation: 2
labels:
app: helloworld-go-vfsdz
serving.knative.dev/configuration: helloworld-go
serving.knative.dev/configurationGeneration: "1"
serving.knative.dev/revision: helloworld-go-vfsdz
serving.knative.dev/revisionUID: 70d2ecd0-cf74-470e-83bc-569f0467eb3c
serving.knative.dev/service: helloworld-go
name: helloworld-go-vfsdz
namespace: default
ownerReferences:
- apiVersion: serving.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Revision
name: helloworld-go-vfsdz
uid: 70d2ecd0-cf74-470e-83bc-569f0467eb3c
resourceVersion: "1808226"
selfLink: /apis/autoscaling.internal.knative.dev/v1alpha1/namespaces/default/podautoscalers/helloworld-go-vfsdz
uid: 3198a6ab-99db-4b42-8075-1643dc85041b
spec:
protocolType: http1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: helloworld-go-vfsdz-deployment
[root@SZD-L0319456 ~]# kubectl get po helloworld-go-vfsdz-deployment-7775fb4d4c-7sr8k -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
cni.projectcalico.org/podIP: 172.1.1.55/32
traffic.sidecar.istio.io/includeOutboundIPRanges: '*'
creationTimestamp: "2020-03-06T03:02:47Z"
generateName: helloworld-go-vfsdz-deployment-7775fb4d4c-
labels:
app: helloworld-go-vfsdz
pod-template-hash: 7775fb4d4c
serving.knative.dev/configuration: helloworld-go
serving.knative.dev/configurationGeneration: "1"
serving.knative.dev/revision: helloworld-go-vfsdz
serving.knative.dev/revisionUID: 70d2ecd0-cf74-470e-83bc-569f0467eb3c
serving.knative.dev/service: helloworld-go
name: helloworld-go-vfsdz-deployment-7775fb4d4c-7sr8k
namespace: default
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: helloworld-go-vfsdz-deployment-7775fb4d4c
uid: a8559cc1-0772-4df2-9baf-6c3605446abb
resourceVersion: "1808367"
selfLink: /api/v1/namespaces/default/pods/helloworld-go-vfsdz-deployment-7775fb4d4c-7sr8k
uid: 71ef6423-d91f-4c52-b06f-70b79810f4d1
spec:
containers:
- env:
- name: TARGET
value: Go Sample v1
- name: PORT
value: "8080"
- name: K_REVISION
value: helloworld-go-vfsdz
- name: K_CONFIGURATION
value: helloworld-go
- name: K_SERVICE
value: helloworld-go
image: dev.local/helloworld-go
imagePullPolicy: Never
lifecycle:
preStop:
httpGet:
path: /wait-for-drain
port: 8022
scheme: HTTP
name: user-container
ports:
- containerPort: 8080
name: user-port
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- mountPath: /var/log
name: knative-var-log
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-vjp62
readOnly: true
- env:
- name: SERVING_NAMESPACE
value: default
- name: SERVING_SERVICE
value: helloworld-go
- name: SERVING_CONFIGURATION
value: helloworld-go
- name: SERVING_REVISION
value: helloworld-go-vfsdz
- name: QUEUE_SERVING_PORT
value: "8012"
- name: CONTAINER_CONCURRENCY
value: "0"
- name: REVISION_TIMEOUT_SECONDS
value: "300"
- name: SERVING_POD
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SERVING_POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: SERVING_LOGGING_CONFIG
value: |-
{
"level": "info",
"development": false,
"outputPaths": ["stdout"],
"errorOutputPaths": ["stderr"],
"encoding": "json",
"encoderConfig": {
"timeKey": "ts",
"levelKey": "level",
"nameKey": "logger",
"callerKey": "caller",
"messageKey": "msg",
"stacktraceKey": "stacktrace",
"lineEnding": "",
"levelEncoder": "",
"timeEncoder": "iso8601",
"durationEncoder": "",
"callerEncoder": ""
}
}
- name: SERVING_LOGGING_LEVEL
- name: SERVING_REQUEST_LOG_TEMPLATE
- name: SERVING_REQUEST_METRICS_BACKEND
value: prometheus
- name: TRACING_CONFIG_BACKEND
value: none
- name: TRACING_CONFIG_ZIPKIN_ENDPOINT
- name: TRACING_CONFIG_STACKDRIVER_PROJECT_ID
- name: TRACING_CONFIG_DEBUG
value: "false"
- name: TRACING_CONFIG_SAMPLE_RATE
value: "0.100000"
- name: USER_PORT
value: "8080"
- name: SYSTEM_NAMESPACE
value: knative-serving
- name: METRICS_DOMAIN
value: knative.dev/internal/serving
- name: USER_CONTAINER_NAME
value: user-container
- name: ENABLE_VAR_LOG_COLLECTION
value: "false"
- name: VAR_LOG_VOLUME_NAME
value: knative-var-log
- name: INTERNAL_VOLUME_PATH
value: /var/knative-internal
- name: SERVING_READINESS_PROBE
value: '{"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}'
- name: ENABLE_PROFILING
value: "false"
- name: SERVING_ENABLE_PROBE_REQUEST_LOG
value: "false"
image: pcr-sz.paic.com.cn/knative-releases/serving-cmd-queue:0.12.0
imagePullPolicy: IfNotPresent
name: queue-proxy
ports:
- containerPort: 8022
name: http-queueadm
protocol: TCP
- containerPort: 9090
name: queue-metrics
protocol: TCP
- containerPort: 9091
name: http-usermetric
protocol: TCP
- containerPort: 8012
name: queue-port
protocol: TCP
readinessProbe:
exec:
command:
- /ko-app/queue
- -probe-period
- "0"
failureThreshold: 3
periodSeconds: 1
successThreshold: 1
timeoutSeconds: 10
resources:
requests:
cpu: 25m
securityContext:
allowPrivilegeEscalation: false
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-vjp62
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: 30.23.17.166
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 300
volumes:
- emptyDir: {}
name: knative-var-log
- name: default-token-vjp62
secret:
defaultMode: 420
secretName: default-token-vjp62
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2020-03-06T03:02:47Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2020-03-06T03:02:49Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2020-03-06T03:02:49Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2020-03-06T03:02:47Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://fc0e60a14481cb8726c074d8eaece0fff514ad44afa364727d398c989a2111df
image: pcr-sz.paic.com.cn/knative-releases/serving-cmd-queue:0.12.0
imageID: docker-pullable://pcr-sz.paic.com.cn/knative-releases/serving-cmd-queue@sha256:59f61b2b20444ddb2aa739f2e7fa5efb3d9cf9fcd2a8ab9155dfe40e9f70e7c2
lastState: {}
name: queue-proxy
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2020-03-06T03:02:48Z"
- containerID: docker://efbce0b2aed5309a23f17695834daa23d5aa972d07a795131e3157dba363af20
image: dev.local/helloworld-go:0.11.0
imageID: docker://sha256:7de72db4aca729a7cc88116e5f8324942c3da2e54135860b421111c6f9a09540
lastState: {}
name: user-container
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2020-03-06T03:02:48Z"
hostIP: 30.23.17.166
phase: Running
podIP: 172.1.1.55
podIPs:
- ip: 172.1.1.55
qosClass: Burstable
startTime: "2020-03-06T03:02:47Z"
Your PA still misses the correct annotation.
It seems that your webhook is somehow broken. May be you can inspect the webhook logs and see if there are errors.
/cc @dprotaso @mattmoor
vagababov
on 6 Mar 2020
Otherwise the system seems to be in order...
vagababov
on 6 Mar 2020
thanks for bearing with us!
Grabbing the webhook logs would be a good next step.
Alternatively if this is something you can reproduce easily - if you could show the script that setup your cluster (or explain it) - we could try to reproduce it ourselves
dprotaso
on 7 Mar 2020
Thanks for your patience!
I restart webhook because of too much logs in it.
kubectl logs -n knative-serving webhook-5d6fc96cf-qqj8b output below.
2020/03/09 03:05:23 Registering 1 clients
2020/03/09 03:05:23 Registering 1 informer factories
2020/03/09 03:05:23 Registering 3 informers
2020/03/09 03:05:23 Registering 4 controllers
{"level":"info","ts":"2020-03-09T03:05:23.162Z","caller":"logging/config.go:108","msg":"Successfully created the logger."}
{"level":"info","ts":"2020-03-09T03:05:23.162Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
{"level":"info","ts":"2020-03-09T03:05:23.163Z","logger":"webhook","caller":"profiling/server.go:59","msg":"Profiling enabled: false","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.175Z","logger":"webhook.config-store","caller":"configmap/store.go:157","msg":"defaults config \"config-defaults\" config was added or updated: &config.Defaults{RevisionTimeoutSeconds:300, MaxRevisionTimeoutSeconds:600, UserContainerNameTemplate:\"user-container\", ContainerConcurrency:0, RevisionCPURequest:(*resource.Quantity)(nil), RevisionCPULimit:(*resource.Quantity)(nil), RevisionMemoryRequest:(*resource.Quantity)(nil), RevisionMemoryLimit:(*resource.Quantity)(nil)}","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.175Z","logger":"webhook","caller":"metrics/exporter.go:99","msg":"Flushing the existing exporter before setting up the new exporter.","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.175Z","logger":"webhook","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/serving webhook prometheus 5000000000 <nil> false 9090 false { false}}. Start the server for Prometheus exporter.","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.175Z","logger":"webhook","caller":"metrics/exporter.go:108","msg":"Successfully updated the metrics exporter; old config: <nil>; new config &{knative.dev/serving webhook prometheus 5000000000 <nil> false 9090 false { false}}","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.269Z","logger":"webhook","caller":"sharedmain/main.go:203","msg":"Starting informers.","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.369Z","logger":"webhook","caller":"sharedmain/main.go:209","msg":"Starting controllers...","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.369Z","logger":"webhook","caller":"controller/controller.go:305","msg":"Starting controller and workers","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.369Z","logger":"webhook","caller":"controller/controller.go:305","msg":"Starting controller and workers","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.369Z","logger":"webhook","caller":"controller/controller.go:315","msg":"Started workers","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.369Z","logger":"webhook","caller":"controller/controller.go:315","msg":"Started workers","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.369Z","logger":"webhook","caller":"controller/controller.go:305","msg":"Starting controller and workers","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.370Z","logger":"webhook","caller":"controller/controller.go:315","msg":"Started workers","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.370Z","logger":"webhook","caller":"controller/controller.go:370","msg":"Reconcile succeeded. Time taken: 25.826渭s.","commit":"bf0a848","knative.dev/traceid":"40dea2f0-f491-49b4-84fa-c1071b9b368e","knative.dev/key":"knative-serving/webhook-certs"}
{"level":"info","ts":"2020-03-09T03:05:23.369Z","logger":"webhook","caller":"webhook/webhook.go:164","msg":"Found certificates for webhook...","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.370Z","logger":"webhook","caller":"controller/controller.go:305","msg":"Starting controller and workers","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.370Z","logger":"webhook","caller":"controller/controller.go:315","msg":"Started workers","commit":"bf0a848"}
{"level":"info","ts":"2020-03-09T03:05:23.552Z","logger":"webhook","caller":"configmaps/configmaps.go:151","msg":"Webhook is valid","commit":"bf0a848","knative.dev/traceid":"0c90a522-9abd-4fad-9a89-5580231d0a3b","knative.dev/key":"knative-serving/webhook-certs"}
{"level":"info","ts":"2020-03-09T03:05:23.552Z","logger":"webhook","caller":"controller/controller.go:370","msg":"Reconcile succeeded. Time taken: 182.244906ms.","commit":"bf0a848","knative.dev/traceid":"0c90a522-9abd-4fad-9a89-5580231d0a3b","knative.dev/key":"knative-serving/webhook-certs"}
{"level":"info","ts":"2020-03-09T03:05:23.553Z","logger":"webhook","caller":"defaulting/defaulting.go:182","msg":"Updating webhook","commit":"bf0a848","knative.dev/traceid":"ce69d45f-813f-4147-b76b-2dc2b6986e0a","knative.dev/key":"knative-serving/webhook-certs"}
{"level":"info","ts":"2020-03-09T03:05:23.553Z","logger":"webhook","caller":"validation/validation.go:171","msg":"Updating webhook","commit":"bf0a848","knative.dev/traceid":"4946b25b-e3bb-42ac-974b-8cd837f11ed2","knative.dev/key":"knative-serving/webhook-certs"}
{"level":"info","ts":"2020-03-09T03:05:23.556Z","logger":"webhook","caller":"configmaps/configmaps.go:151","msg":"Webhook is valid","commit":"bf0a848","knative.dev/traceid":"db26bdee-9414-4bc9-be59-998db160535e","knative.dev/key":"config.webhook.serving.knative.dev"}
{"level":"info","ts":"2020-03-09T03:05:23.556Z","logger":"webhook","caller":"controller/controller.go:370","msg":"Reconcile succeeded. Time taken: 186.171542ms.","commit":"bf0a848","knative.dev/traceid":"db26bdee-9414-4bc9-be59-998db160535e","knative.dev/key":"config.webhook.serving.knative.dev"}
{"level":"info","ts":"2020-03-09T03:05:23.651Z","logger":"webhook","caller":"defaulting/defaulting.go:182","msg":"Updating webhook","commit":"bf0a848","knative.dev/traceid":"e4382b11-714a-4651-99c0-75782582b158","knative.dev/key":"webhook.serving.knative.dev"}
{"level":"info","ts":"2020-03-09T03:05:23.654Z","logger":"webhook","caller":"controller/controller.go:370","msg":"Reconcile succeeded. Time taken: 283.870964ms.","commit":"bf0a848","knative.dev/traceid":"4946b25b-e3bb-42ac-974b-8cd837f11ed2","knative.dev/key":"knative-serving/webhook-certs"}
{"level":"info","ts":"2020-03-09T03:05:23.656Z","logger":"webhook","caller":"controller/controller.go:370","msg":"Reconcile succeeded. Time taken: 285.912894ms.","commit":"bf0a848","knative.dev/traceid":"ce69d45f-813f-4147-b76b-2dc2b6986e0a","knative.dev/key":"knative-serving/webhook-certs"}
{"level":"info","ts":"2020-03-09T03:05:23.657Z","logger":"webhook","caller":"controller/controller.go:370","msg":"Reconcile succeeded. Time taken: 287.140526ms.","commit":"bf0a848","knative.dev/traceid":"e4382b11-714a-4651-99c0-75782582b158","knative.dev/key":"webhook.serving.knative.dev"}
{"level":"info","ts":"2020-03-09T03:05:23.753Z","logger":"webhook","caller":"validation/validation.go:171","msg":"Updating webhook","commit":"bf0a848","knative.dev/traceid":"f95fdcb5-4027-45e4-9694-1d11a9132ff7","knative.dev/key":"validation.webhook.serving.knative.dev"}
{"level":"info","ts":"2020-03-09T03:05:23.758Z","logger":"webhook","caller":"controller/controller.go:370","msg":"Reconcile succeeded. Time taken: 307.715091ms.","commit":"bf0a848","knative.dev/traceid":"f95fdcb5-4027-45e4-9694-1d11a9132ff7","knative.dev/key":"validation.webhook.serving.knative.dev"}
Steps to reproduce it below.
- install istio
# Download and unpack Istio
export ISTIO_VERSION=1.4.3
curl -L https://git.io/getLatestIstio | sh -
cd istio-${ISTIO_VERSION}
# install the Istio CRDs
for i in install/kubernetes/helm/istio-init/files/crd*yaml; do kubectl apply -f $i; done
# Create istio-system namespace
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
name: istio-system
labels:
istio-injection: disabled
EOF
# A lighter template, with just pilot/gateway
helm template --namespace=istio-system \
--set prometheus.enabled=false \
--set mixer.enabled=false \
--set mixer.policy.enabled=false \
--set mixer.telemetry.enabled=false \
`# Pilot doesn't need a sidecar.` \
--set pilot.sidecar=false \
--set pilot.resources.requests.memory=128Mi \
`# Disable galley (and things requiring galley).` \
--set galley.enabled=false \
--set global.useMCP=false \
`# Disable security / policy.` \
--set security.enabled=false \
--set global.disablePolicyChecks=true \
`# Disable sidecar injection.` \
--set sidecarInjectorWebhook.enabled=false \
--set global.proxy.autoInject=disabled \
--set global.omitSidecarInjectorConfigMap=true \
--set gateways.istio-ingressgateway.autoscaleMin=1 \
--set gateways.istio-ingressgateway.autoscaleMax=2 \
`# Set pilot trace sampling to 100%` \
--set pilot.traceSampling=100 \
--set global.mtls.auto=false \
install/kubernetes/helm/istio \
> ./istio-lean.yaml
kubectl apply -f istio-lean.yaml
kubectl get pods --namespace istio-system
- install knative serving
yaml="https://github.com/ysjjovo/knative-tutorial/blob/master/install/3-knative/source/core/0.12.0/serving.yaml"
kubectl apply --selector knative.dev/crd-install=true -f $yaml
echo 'CRDS install completed!'
kubectl apply -f $yaml
My knative installation is some how strange because of network blockade in china.
'serving.yaml' I applied origin from offical site.
I replace images with a reachable url,such as,gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:3932262d4a44284f142f4c49f707526e70dd86317163a88a8cbb6de035a401a9 is replaced with registry.cn-chengdu.aliyuncs.com/ysjjovo/serving-cmd-queue:0.12.0.This image is build from a simple Dockerfile
FROM gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:3932262d4a44284f142f4c49f707526e70dd86317163a88a8cbb6de035a401a9
Then import it to my company internal environment and tag with 'pcr-sz.paic.com.cn/knative-releases/serving-cmd-queue:0.12.0'.
It works fine on kubeadm k8s1.17.3,both 0.11.0 and 0.12.0 version.
But binary installed k8s(1.17.3) do not work.Some environment info below.
[root@xxx ~]# kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default helloworld-go-vfsdz-deployment-7775fb4d4c-7sr8k 2/2 Running 0 3d3h
istio-system istio-ingressgateway-844dd4b69f-2jmzn 1/1 Running 0 10d
istio-system istio-pilot-f95ff47b8-wpgjf 1/1 Running 0 11d
knative-serving activator-858556768f-2jlv4 1/1 Running 0 10d
knative-serving autoscaler-5769f76958-8c67q 1/1 Running 0 11d
knative-serving autoscaler-hpa-5f64f64467-rbpjq 1/1 Running 0 10d
knative-serving controller-687466f478-7k64m 1/1 Running 0 10d
knative-serving networking-istio-69fb4fffbc-ldwxz 1/1 Running 0 11d
knative-serving webhook-5d6fc96cf-qqj8b 1/1 Running 0 3h13m
kube-system calico-node-fwpcj 1/1 Running 0 14d
kube-system calico-typha-7f66875887-8vlmd 1/1 Running 0 14d
kube-system coredns-d9459d59c-bjqb8 1/1 Running 0 10d
kube-system coredns-d9459d59c-knvjj 1/1 Running 0 14d
kube-system tiller-deploy-84456b7d47-kw8kv 1/1 Running 26 14d
kube-system tiller-deploy-84456b7d47-zhmvk 1/1 Running 0
It works fine on kubeadm k8s1.17.3,both 0.11.0 and 0.12.0 version.
But binary installed k8s(1.17.3) do not work.Some environment info below.
Sorry that last sentence there says it _does not_ work on 1.17.3 but the line above says it _works on_ 1.17.3.
Can you clarify which version you're seeing the problem on?
dprotaso
on 9 Mar 2020
FYI - I tried the following with success:
1. cluster setup
minikube start --kubernetes-version=v1.17.3
2. istio setup
same as above
3. knative setup
kubectl apply -l knative.dev/crd-install=true -f https://github.com/knative/serving/releases/download/v0.12.0/serving.yaml
kubectl apply -f https://github.com/knative/serving/releases/download/v0.12.0/serving.yaml
4. knative service installation
cat <<EOF | kubectl apply -f -
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: helloworld-go
namespace: default
spec:
template:
spec:
containers:
- image: gcr.io/knative-samples/helloworld-go
env:
- name: TARGET
value: "Go Sample v1"
EOF
5. wait for service to become ready
$ kubectl get ksvc
NAME URL LATESTCREATED LATESTREADY READY REASON
helloworld-go http://helloworld-go.default.example.com helloworld-go-mrwbr Unknown RevisionMissing
$ kubectl get ksvc
NAME URL LATESTCREATED LATESTREADY READY REASON
helloworld-go http://helloworld-go.default.example.com helloworld-go-mrwbr helloworld-go-mrwbr True
I'am sorry for confusing you.I mean installing k8s from binaries does not work.kubeadm works fine.Maybe,there are some problems with k8s installation from binaries.
ysjjovo
on 10 Mar 2020
@ysjjovo I'm going to close this out - if you have any updates with specific steps that reproduce this error feel free to re-open.
dprotaso
on 8 Apr 2020
I'am sorry for confusing you.I mean installing k8s from binaries does not work.kubeadm works fine.Maybe,there are some problems with k8s installation from binaries.
I had this same issue on a bare-metal installation of K8s v1.15, after several hours of debugging, trying different Knative versions and log checking I found that my K8s installation had the --admission-control flag set on the Kube Api Server and it didn't list the ValidatingAdmissionWebhook (https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/). I've removed the flag to activate the default admission controllers and Knative now works as expected.
jsargiot
on 25 Jul 2020
@jsargiot were you using a tool to deploy K8s? or manually?
dprotaso
on 11 Aug 2020
were you using a tool to deploy K8s? or manually?
Manually.
Actually the missing admission controller was the MutatingAdmissionWebhook, I think this would've helped: https://github.com/knative/serving/pull/109, but it was removed at some point.
jsargiot
on 11 Aug 2020
Related issues
ZhiminXiang
路
3Comments
evankanderson
路
3Comments
scothis
路
3Comments
alexnederlof
路
5Comments
VladimirSmogitel
路
7Comments