Serving: istio-statsd-prom-bridge is in crashloopbackoff in Knative v0.1.1 deployment

[simingweng]

Expected Behavior

Following the v0.1.1 Knative installation guide, the step of installing Istio with command

curl -L https://raw.githubusercontent.com/knative/serving/v0.1.1/third_party/istio-0.8.0/istio.yaml \ | sed 's/LoadBalancer/NodePort/' \ | kubectl apply -f -

should bring up the the Istio pods

Actual Behavior

istio-statsd-prom-bridge pod ends up in crashloopbackoff

Steps to Reproduce the Problem

1. execute the command above
2. kubectl -n istio-system get pods

Additional Info

this is due to the latest v0.7.0 release of prometheus statsd_exporter Docker image, in which there was a breaking change introduced to switch from single dash to double dash for command line arguments.

Knative v0.1.1 istio manifest file is using the "latest" tag, therefore the v0.7.0 statsd_exporter is automatically in, and
args: - '-statsd.mapping-config=/etc/statsd/mapping.conf'
no longer works.

[simingweng]

/assign @mdemirhan

[mattmoor]

cc @yanweiguo @tcnghia

[pzmrzy]

I also get same error when I install kNative with minikube and GKE(Master version
1.10.6-gke.2) by following the GKE guild
Works after change to install istio-1.0.1
kubectl apply -f https://raw.githubusercontent.com/knative/serving/master/third_party/istio-1.0.1/istio.yaml

[yunlzheng]

https://github.com/knative/serving/blob/37c640555a3241ed707962ba6b29d6c7fe68f937/third_party/istio-0.8.0/istio.yaml#L2147

The statsd-prom-bridge args is wrong.

statsd_exporter: error: unknown short flag '-s', try --help

This should be

containers:
      - args:
        - --statsd.mapping-config=/etc/statsd/mapping.conf ## Update this deployment manual

[sdake]

The correct solution is to use what was tested with Istio 0.8.0 - which is statsd v0.6.0. I'll correct this problem in the Istio 0.8 branch. Apologies for the use of :latest tags, we are/have removed them from the upstream templating for Istio.

[tcnghia]

@yanweiguo do you know how this will impact our monitoring components? thanks

[yanweiguo]

@akyyy Yao, could you have a look for this?

[tcnghia]

It looks like this an Istio installation with istio-statsd-prome-bridge crashing doesn't affect Knative functionalities.

We should still fix the issue to avoid confusions from users.

[bobcatfish]

So it seems like this still happens when you follow along with https://github.com/knative/docs/blob/master/install/Knative-with-GKE.md

[cforce]

It is also reproducable with docker for windows edge (statsd_exporter: error: unknown short flag '-s', try --help).When using https://github.com/knative/serving/blob/master/third_party/istio-1.0.2/istio.yaml issue is gone

[mdemirhan]

1922 was submitted to fix this. Will prioritize getting that in soon.

[mdemirhan]

This should be fixed now (#1922). Let me know if you hit issues.

[mdemirhan]

@tcnghia mentioned that we need to cut a new release. Reopening the issue and assigning to @adrcunha to cut a new release.

/assign @adrcunha

[adrcunha]

Synced with Mustafa, better cutting a new release.

/close