Origin: Metrics deployer fails to set up metrics in 1.4.0

[pilhuhn]

Metrics fails to start in v1.4.0. This used to work in v1.4.0-rc1

Version

# oc version
oc v1.5.0-alpha.2+e4b43ee
kubernetes v1.5.2+43a9be4
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://172.31.7.9:8443
openshift v1.4.0+208f053
kubernetes v1.4.0+776c994

Note: I did try this for 1.4.0 and 1.4.0-rc1 both with the above version of oc.
In both cases the needed images were already present on the system (otherwise metric-deployer will also fail on -rc1 as pulling them is taking too long over here).

Steps To Reproduce

oc cluster up --metrics=true --version=v1.4.0 --public-hostname=pintsize

Then log in a system:admin

# oc get pods -n openshift-infra
NAME                         READY     STATUS    RESTARTS   AGE
metrics-deployer-pod-2km6g   0/1       Error     0          2m
metrics-deployer-pod-6tuwk   0/1       Error     0          3m
metrics-deployer-pod-7at81   0/1       Error     0          4m
metrics-deployer-pod-egjsp   0/1       Error     0          4m
metrics-deployer-pod-ij2tx   0/1       Error     0          1m
metrics-deployer-pod-kbcxx   0/1       Error     0          5m
metrics-deployer-pod-mf4ih   0/1       Error     0          1m
metrics-deployer-pod-o29ux   0/1       Error     0          3m
metrics-deployer-pod-oln9m   0/1       Error     0          4m
metrics-deployer-pod-qoqr4   0/1       Error     0          1m
metrics-deployer-pod-st0pu   0/1       Error     0          2m
metrics-deployer-pod-u5gki   0/1       Error     0          3m
metrics-deployer-pod-u7o3r   0/1       Error     0          2m

Current Result

Metrics fails to start

Expected Result

Additional Information

[try to run $ oadm diagnostics command if possible]
[if you are reporting issue related to builds, provide build logs with BUILD_LOGLEVEL=5]
[consider attaching output of the $ oc get all -o json -n <namespace> command to the issue]
[visit https://docs.openshift.org/latest/welcome/index.html]

######## LOG excerpts from 1st and subsequent runs

User "deployer-account" set.
+ oc config set-context deployer-context --cluster=deployer-master --user=deployer-account --namespace=openshift-infra
Context "deployer-context" set.
+ '[' -n 1 ']'
+ oc config use-context deployer-context
Switched to context "deployer-context".
+ case $deployer_mode in
+ '[' false '!=' true ']'
+ validate_preflight
+ set +x
PREFLIGHT CHECK SUCCEEDED
validate_master_accessible: ok
validate_hostname: The HAWKULAR_METRICS_HOSTNAME value is deemed acceptable.
validate_deployer_secret: ok
Generating randomized passwords for the Hawkular Metrics and Cassandra keystores and truststores
Creating the Hawkular Metrics keystore from the PEM file
Entry for alias hawkular-metrics successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled
[Storing /etc/deploy/_output/hawkular-metrics.keystore]
Creating the Hawkular Cassandra keystore from the PEM file
Entry for alias hawkular-cassandra successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled
[Storing /etc/deploy/_output/hawkular-cassandra.keystore]
Creating the Hawkular Metrics Certificate
Certificate stored in file </etc/deploy/_output/hawkular-metrics.cert>
Creating the Hawkular Cassandra Certificate
Certificate stored in file </etc/deploy/_output/hawkular-cassandra.cert>
Importing the Hawkular Metrics Certificate into the Cassandra Truststore
Certificate was added to keystore
[Storing /etc/deploy/_output/hawkular-cassandra.truststore]
Importing the Hawkular Cassandra Certificate into the Hawkular Metrics Truststore
Certificate was added to keystore
[Storing /etc/deploy/_output/hawkular-metrics.truststore]
Importing the Hawkular Cassandra Certificate into the Cassandra Truststore
Certificate was added to keystore
[Storing /etc/deploy/_output/hawkular-cassandra.truststore]
Importing the CA Certificate into the Cassandra Truststore
Certificate was added to keystore
[Storing /etc/deploy/_output/hawkular-cassandra.truststore]
Certificate was added to keystore
[Storing /etc/deploy/_output/hawkular-cassandra.truststore]
Certificate was added to keystore
[Storing /etc/deploy/_output/hawkular-cassandra.truststore]
Importing the CA Certificate into the Hawkular Metrics Truststore
Certificate was added to keystore
[Storing /etc/deploy/_output/hawkular-metrics.truststore]
Certificate was added to keystore
[Storing /etc/deploy/_output/hawkular-metrics.truststore]
Certificate was added to keystore
[Storing /etc/deploy/_output/hawkular-metrics.truststore]
Adding password for user hawkular
Generating the JGroups Keystore
Creating the Hawkular Metrics Secrets configuration json file
Creating the Hawkular Metrics Certificate Secrets configuration json file
Creating the Hawkular Metrics User Account Secrets
Creating the Cassandra Secrets configuration file
Creating the Cassandra Certificate Secrets configuration json file
Creating Hawkular Metrics & Cassandra Secrets
secret "hawkular-metrics-secrets" created
secret "hawkular-metrics-certificate" created
secret "hawkular-metrics-account" created
secret "hawkular-cassandra-secrets" created
secret "hawkular-cassandra-certificate" created
Creating Hawkular Metrics & Cassandra Templates
template "hawkular-metrics" created
template "hawkular-cassandra-services" created
template "hawkular-cassandra-node-pv" created
template "hawkular-cassandra-node-dynamic-pv" created
template "hawkular-cassandra-node-emptydir" created
template "hawkular-support" created
Deploying Hawkular Metrics & Cassandra Components
scripts/hawkular.sh: line 200: STARTUP_TIMEOUT: unbound variable
error: no objects passed to create

---

Next run:

Creating the Hawkular Metrics Secrets configuration json file
Creating the Hawkular Metrics Certificate Secrets configuration json file
Creating the Hawkular Metrics User Account Secrets
Creating the Cassandra Secrets configuration file
Creating the Cassandra Certificate Secrets configuration json file
Creating Hawkular Metrics & Cassandra Secrets
Error from server (AlreadyExists): error when creating "/etc/deploy/_output/hawkular-metrics-secrets.json": secrets "hawkular-metrics-secrets" already exists 

[jorgemoralespou]

We've seen this in 2 environments, one created with "oc cluster up" and another created with ansible installer.

[pilhuhn]

This looks fixed in 1.4.1

[jcantrill]

@pilhuhn Can you please close if you are satisfied the issue is corrected.

[jorgemoralespou]

@pilhuhn Does it completely work hawkular?
I've now tested and even the deployer problem is fixed, Hawkular does not start up correctly for me.

oc cluster up --public-hostname 127.0.0.1 --routing-suffix apps.lcup --host-data-dir /Users/jmorales/.oc/profiles/origin-full/data --host-config-dir /Users/jmorales/.oc/profiles/origin-full/config --use-existing-config -e TZ=CET --metrics --logging --version=v1.4.1

[rafaeltuelho]

Interesting I'm still facing this issue on v1.5.1 (origin) and v3.5.5 (cdk 3.0)...

[slaskawi]

I'm facing it even with the latest build v3.6.0-rc.0