Jx: ERROR: secrets "hmac-token" not found when failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx update webhooks --verbose --warn-on-fail' command in directory 'repositories', output: ''

Created on 7 May 2020  路  11Comments  路  Source: jenkins-x/jx

Summary

When I had same problem "error: fail to acquire the lock: no REPO_OWNER provided" from the output of command 'jx step helm apply' which was reported by issue #7106 yesterday, I saw 'daveconde' merged the new fix for this issue #7106 in the issue #7128 6 days ago. Therefore, this morning I upgraded newest version of jx to v2.1.23 from v2.0.1240, and upgraded newest version of jenkins-x-versions to v1.0.444 from v1.0.406 in the file jx-requirements.yml. Then used 'jx boot' to install Jenkins X with gitlab server and stoped at step "update-webhooks" after successful "STEP: install-pipelines"

STEP: update-webhooks command: /bin/sh -c jx update webhooks --verbose --warn-on-fail in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/repositories

error: secrets "hmac-token" not found
error: failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx update webhooks --verbose --warn-on-fail' command in directory 'repositories', output: ''

Steps to reproduce the behavior

$ jx update webhooks --verbose --warn-on-fail
DEBUG: Finding service url for jenkins in namespace jx
DEBUG: Found service url http://jenkins.jx.jenkinsxgitlabinfra.stg.store.aws.z8s.io
error: secrets "hmac-token" not found

Expected behavior

The command to update webhooks should be successful. And "STEP: update-webhooks" should be completed.

Actual behavior

The "STEP: update-webhooks" in jenkins-x.yml was stopped and the command 'jx boot' was stopped accordingly.

IT-SEA17490-MBP:jenkins-x-boot-config q1447100$ jx boot
Attempting to resolve version for boot config https://github.com/jenkins-x/jenkins-x-boot-config from https://github.com/jenkins-x/jenkins-x-versions.git
Booting Jenkins X
WARNING: failed to load ConfigMap jenkins-x-docker-registry in namespace jx: failed to get configmap jenkins-x-docker-registry in namespace jx, configmaps "jenkins-x-docker-registry" not found
WARNING: failed to load ConfigMap jenkins-x-docker-registry in namespace jx: failed to get configmap jenkins-x-docker-registry in namespace jx, configmaps "jenkins-x-docker-registry" not found

STEP: validate-git command: /bin/sh -c jx step git validate in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/env

Git configured for user: jenkins-x-bot and email [email protected]

STEP: verify-preinstall command: /bin/sh -c jx step verify preinstall --provider-values-dir="kubeProviders" in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config

writing the following to the OWNERS file for the development environment repository:
approvers:
- q1447100
reviewers:
- q1447100
The provided requirements file has 'strictPermissions' enabled but 'provider' is not Openshift.
This feature is only supported on Openshift
WARNING: TLS is not enabled so your webhooks will be called using HTTP. This means your webhook secret will be sent to your cluster in the clear. See https://jenkins-x.io/docs/getting-started/setup/boot/#ingress for more information
? Do you wish to continue? Yes

Verifying the kubernetes cluster before we try to boot Jenkins X in namespace: jx
WARNING: Lazy create of cloud resources is disabled
Verifying Ingress...

Using namespace 'jx' from context named 'arn:aws:eks:us-east-1:282708546392:cluster/stg_k8s_jenkinsxstatic' on server 'https://C62248B050EEFD17B2F7BD5F7F0ED774.sk1.us-east-1.eks.amazonaws.com'.

Verifying the CLI packages using version stream URL: https://github.com/jenkins-x/jenkins-x-versions.git and git ref: v1.0.446
using version 2.1.23 of jx
CLI packages kubectl, git, helm seem to be setup correctly
NAME               VERSION
jx                 2.1.23
Kubernetes cluster v1.15.11-eks-af3caf
kubectl            v1.16.2
git                2.20.1 (Apple Git-117)

Verifying Storage...
WARNING: Your requirements have not enabled cloud storage for logs - we recommend enabling this for kubernetes provider eks
WARNING: Your requirements have not enabled cloud storage for reports - we recommend enabling this for kubernetes provider eks
WARNING: Your requirements have not enabled cloud storage for repository - we recommend enabling this for kubernetes provider eks
WARNING: Your requirements have not enabled cloud storage for backup - we recommend enabling this for kubernetes provider eks
Storage configuration looks good

Helm installed and configured
Ensuring Helm chart repository https://storage.googleapis.com/chartmuseum.jenkins-x.io is configured
Cluster looks good, you are ready to 'jx boot' now!


STEP: install-jx-crds command: /bin/sh -c jx upgrade crd in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config

Jenkins X CRDs upgraded with success

STEP: install-velero command: /bin/sh -c jx step helm apply --boot --remote --no-vault --name velero in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/velero

Modified file /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/velero/Chart.yaml to set the chart to version 1
lock cancelled because not running in tekton
Verifying the helm requirements versions in dir: . using version stream URL: https://github.com/jenkins-x/jenkins-x-versions.git and git ref: v1.0.446

STEP: install-velero-backups command: /bin/sh -c jx step helm apply --boot --remote --no-vault --name velero-backups in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/velero-backups

Modified file /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/velero-backups/Chart.yaml to set the chart to version 1
lock cancelled because not running in tekton
Ignoring templates/default-backup.yaml
No requirements file: /var/folders/n5/k_h05z8j0933frqqr98ch95mmp9_00/T/jx-helm-apply-037819419/velero-backups/requirements.yaml so not checking for missing versions

STEP: install-nginx-controller command: /bin/sh -c jx step helm apply --boot --remote --no-vault --name jxing in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/jxing

Modified file /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/jxing/Chart.yaml to set the chart to version 1
lock cancelled because not running in tekton
Verifying the helm requirements versions in dir: . using version stream URL: https://github.com/jenkins-x/jenkins-x-versions.git and git ref: v1.0.446


STEP: create-install-values command: /bin/sh -c jx step create install values -b in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/env


STEP: install-external-dns command: /bin/sh -c jx step helm apply --boot --remote --no-vault --name exdns in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/external-dns

Modified file /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/external-dns/Chart.yaml to set the chart to version 1
lock cancelled because not running in tekton
Verifying the helm requirements versions in dir: . using version stream URL: https://github.com/jenkins-x/jenkins-x-versions.git and git ref: v1.0.446


STEP: install-cert-manager-crds command: /bin/sh -c kubectl apply --wait --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.11/deploy/manifests/00-crds.yaml in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config

customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io configured

STEP: install-cert-manager command: /bin/sh -c jx step helm apply --boot --remote --no-vault --name cm in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/cm

Modified file /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/cm/Chart.yaml to set the chart to version 1
lock cancelled because not running in tekton
Verifying the helm requirements versions in dir: . using version stream URL: https://github.com/jenkins-x/jenkins-x-versions.git and git ref: v1.0.446

STEP: install-acme-issuer-and-certificate command: /bin/sh -c jx step helm apply --boot --remote --no-vault --name acme in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/acme

Modified file /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/acme/Chart.yaml to set the chart to version 1
lock cancelled because not running in tekton
Ignoring templates/cert-manager-prod-certificate.yaml
Ignoring templates/cert-manager-prod-issuer.yaml
Ignoring templates/cert-manager-staging-certificate.yaml
Ignoring templates/cert-manager-staging-issuer.yaml
No requirements file: /var/folders/n5/k_h05z8j0933frqqr98ch95mmp9_00/T/jx-helm-apply-131397137/acme/requirements.yaml so not checking for missing versions

STEP: install-vault command: /bin/sh -c jx step boot vault --provider-values-dir ../../kubeProviders in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/vault

Not attempting to boot Vault as using secret storage: local

STEP: create-helm-values command: /bin/sh -c jx step create values --name parameters in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/env

defaulting to secret storage scheme local found from requirements file at /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/jx-requirements.yml
defaulting to secret base path to the cluster name stg_k8s_jenkinsxstatic found from requirements file at /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/jx-requirements.yml
generated schema file /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/env/parameters.schema.json from template /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/env/parameters.tmpl.schema.json
? Jenkins X Admin Username admin
? Jenkins X Admin Password [? for help] ************
? Pipeline bot Git username q1447100
? Pipeline bot Git email address [email protected]
? Pipeline bot Git token [? for help] ********************
Generated token ff67bafa630627ef95c3890c6c4e9f39f855d1cfc, to use it press enter.
This is the only time you will be shown it so remember to save it
? HMAC token, used to validate incoming webhooks. Press enter to use the generated token [? for help] 
? Do you want to configure an external Docker Registry? No

STEP: create-jx-auth-config command: /bin/sh -c jx step create templated --parameters-file=../../env/parameters.yaml --requirements-dir=../../ --template-file=jx-auth-configmap.tmpl.yaml --config-file=templates/jx-auth-configmap.yaml in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/jx-auth

Saved the rendered configuration into templates/jx-auth-configmap.yaml file

STEP: install-jx-auth-config command: /bin/sh -c jx step helm apply --boot --remote --no-vault --name jx-auth in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/jx-auth

Modified file /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/systems/jx-auth/Chart.yaml to set the chart to version 1
lock cancelled because not running in tekton
Ignoring templates/jx-auth-configmap.yaml
No requirements file: /var/folders/n5/k_h05z8j0933frqqr98ch95mmp9_00/T/jx-helm-apply-154526009/jx-auth/requirements.yaml so not checking for missing versions

STEP: install-jenkins-x command: /bin/sh -c jx step helm apply --boot --remote --name jenkins-x --provider-values-dir ../kubeProviders in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/env

Modified file /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/env/Chart.yaml to set the chart to version 1
lock cancelled because not running in tekton
Ignoring templates/.gitignore
Applying the kubernetes overrides at ../kubeProviders/eks/values.tmpl.yaml
Verifying the helm requirements versions in dir: . using version stream URL: https://github.com/jenkins-x/jenkins-x-versions.git and git ref: v1.0.446


STEP: verify-jenkins-x-environment command: /bin/sh -c jx step verify env in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config

Storing the requirements in team settings in the dev environment
Validating git repository for dev environment at URL https://gitlab.corp.zulily.com/store-sre/environment-stg-k8s-jenkinsxstatic-dev.git
Duplicated Git repository https://github.com/jenkins-x/jenkins-x-boot-config to https://gitlab.corp.zulily.com/store-sre/environment-stg-k8s-jenkinsxstatic-dev
Setting upstream to https://gitlab.corp.zulily.com/store-sre/environment-stg-k8s-jenkinsxstatic-dev

Pushed Git repository to https://gitlab.corp.zulily.com/store-sre/environment-stg-k8s-jenkinsxstatic-dev
Validating git repository for production environment at URL https://gitlab.corp.zulily.com/store-sre/environment-stg-k8s-jenkinsxstatic-production.git
Using Git provider zulily at https://gitlab.corp.zulily.com
? Using Git user name: q1447100
? Using organisation: store-sre
Creating repository store-sre/environment-stg-k8s-jenkinsxstatic-production
Creating Git repository store-sre/environment-stg-k8s-jenkinsxstatic-production
Pushed Git repository to https://gitlab.corp.zulily.com/store-sre/environment-stg-k8s-jenkinsxstatic-production

Validating git repository for staging environment at URL https://gitlab.corp.zulily.com/store-sre/environment-stg-k8s-jenkinsxstatic-staging.git
Using Git provider zulily at https://gitlab.corp.zulily.com
? Using Git user name: q1447100
? Using organisation: store-sre
Creating repository store-sre/environment-stg-k8s-jenkinsxstatic-staging
Creating Git repository store-sre/environment-stg-k8s-jenkinsxstatic-staging
Pushed Git repository to https://gitlab.corp.zulily.com/store-sre/environment-stg-k8s-jenkinsxstatic-staging

Environment git repositories look good


STEP: install-repositories command: /bin/sh -c jx step helm apply --boot --name repos in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/repositories

Modified file /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/repositories/Chart.yaml to set the chart to version 1
creating the lock configmap jx-lock-jx
lock configmap jx-lock-jx created
Ignoring templates/default-group.yaml
No requirements file: /var/folders/n5/k_h05z8j0933frqqr98ch95mmp9_00/T/jx-helm-apply-106786820/repositories/requirements.yaml so not checking for missing versions

cleaning the lock configmap jx-lock-jx

STEP: install-pipelines command: /bin/sh -c jx step scheduler config apply --direct=true in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/prowConfig


STEP: update-webhooks command: /bin/sh -c jx update webhooks --verbose --warn-on-fail in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/repositories

error: secrets "hmac-token" not found
error: failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx update webhooks --verbose --warn-on-fail' command in directory 'repositories', output: ''

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.1.23
Kubernetes cluster v1.14.9-eks-502bfb
kubectl            v1.16.2
helm client        2.14.3
git                2.20.1 (Apple Git-117)
Operating System   Mac OS X 10.14.6 build 18G103


verifying packages

Jenkins Type

Classic Static Master

Helm Version

IT-SEA17490-MBP:jenkins-x-boot-config.2020-05-06.stg_k8s_jenkinsxstatic.Success q1447100$ helm version
'Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}

Kubernetes cluster

What kind of Kubernetes cluster are you using & how did you create it?
Use Terraform customized AWS EKS module to creat AWS EKS v1.15(newest)

Operating system / Environment

In which environment are you running the jx CLI?
Mac OS X 10.14.6 build 18G103

What is the contents of the file jx-requirements.yml?

IT-SEA17490-MBP:jenkins-x-boot-config q1447100$ cat jx-requirements.yml
autoUpdate:
  enabled: false
  schedule: ""
bootConfigURL: https://github.com/jenkins-x/jenkins-x-boot-config
cluster:
  clusterName: stg_k8s_jenkinsxstatic
  devEnvApprovers:
  - q1447100
  environmentGitOwner: store-sre
  gitKind: gitlab
  gitName: zulily
  gitServer: https://gitlab.xxxx.yyyyyy.com
  namespace: jx
  provider: eks
  region: us-east-1
  registry: 282708546392.dkr.ecr.us-east-1.amazonaws.com
environments:
- ingress:
    domain: jenkinsxstatic.stg.store.aws.z8s.io
    externalDNS: true
    namespaceSubDomain: -jx.
    tls:
      email: ""
      enabled: false
      production: false
  key: dev
  repository: environment-stg-k8s-jenkinsxstatic-dev
- ingress:
    domain: ""
    externalDNS: false
    namespaceSubDomain: ""
    tls:
      email: ""
      enabled: false
      production: false
  key: staging
  repository: environment-stg-k8s-jenkinsxstatic-staging
- ingress:
    domain: ""
    externalDNS: false
    namespaceSubDomain: ""
    tls:
      email: ""
      enabled: false
      production: false
  key: production
  repository: environment-stg-k8s-jenkinsxstatic-production
gitops: true
ingress:
  domain: jenkinsxstatic.stg.store.aws.z8s.io
  externalDNS: true
  namespaceSubDomain: -jx.
  tls:
    email: ""
    enabled: false
    production: false
repository: nexus
secretStorage: local
storage:
  backup:
    enabled: false
    url: ""
  logs:
    enabled: false
    url: ""
  reports:
    enabled: false
    url: ""
  repository:
    enabled: false
    url: ""
terraform: true
vault: {}
velero:
  schedule: ""
  ttl: ""
versionStream:
  ref: v1.0.446
  url: https://github.com/jenkins-x/jenkins-x-versions.git
webhook: jenkins
arelighthouse kinbug prioritcritical-urgent
Source
picture q131172019

Most helpful comment

@q131172019
I just had the same issue. Upgraded JX to 2.1.31 and reran jx boot. Now it works.

picture cprovencher on 11 May 2020
🎉1 👍1

All 11 comments

I'm working on a better fix for this, but a quick workaround is to clone your dev env repo locally and run jx boot from in it - it'll recreate the secret properly. But before you do that, could you do a kubectl get service and see if there's a hook service? Thanks.

abayer picture abayer on 7 May 2020

I'm working on a better fix for this, but a quick workaround is to clone your dev env repo locally and run jx boot from in it - it'll recreate the secret properly.

Good to know. I will create new AWS EKS cluster to try this quick workaround. Last night I had rolled back to jx-v2.0.1258 + jenkins-x-versions-v1.0.406 to move forward to set up Jenkins X CI/CD pipeline environment and tested successfully for our PHP applications due to tight schedule of my project.

But before you do that, could you do a kubectl get service and see if there's a hook service? Thanks.

There is no hook service when I run kubectl get service command. I know if I choose "stateless"
type (Prow + Tekton) installation, the service 'hook' as part of Prow should be running. This weekend I will test whether newest fix resolves the issue. Thanks for your quick response and quick fix! 

IT-SEA17490-MBP:~ q1447100$ kg services
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
exdns-external-dns            ClusterIP   172.20.57.34     <none>        7979/TCP    13h
heapster                      ClusterIP   172.20.239.250   <none>        8082/TCP    13h
jenkins                       ClusterIP   172.20.168.79    <none>        8080/TCP    13h
jenkins-agent                 ClusterIP   172.20.163.66    <none>        50000/TCP   13h
jenkins-x-chartmuseum         ClusterIP   172.20.50.14     <none>        8080/TCP    13h
nexus                         ClusterIP   172.20.24.163    <none>        80/TCP      13h
tekton-pipelines-controller   ClusterIP   172.20.250.26    <none>        9090/TCP    13h
q131172019 picture q131172019 on 7 May 2020

Yup, that means something went wrong when it deleted Prow after installing Lighthouse - they both have a service named hook. I'll see what I can do with that.

abayer picture abayer on 7 May 2020

Yup, that means something went wrong when it deleted Prow after installing Lighthouse - they both have a service named hook. I'll see what I can do with that.

If the configuration is 'webook: jenkins' other than default 'webhook: prow' in jx-requirements.yml, is the service 'hook' still running? Since this Feburary when I tried Jenkins Static Master mode, I have never seen the service 'hook' is running. I check my working logs and never see 'hook' service. this weekend I definitely test your new fix and see happens. Thanks a lot!

q131172019 picture q131172019 on 7 May 2020

If you're using static masters, no, there won't be a hook service.

abayer picture abayer on 7 May 2020

@abayer: I download jx-2.1.30 and jenkins-x-version-v1.0.451 and both are newest version to me. I use 'jx boot' to install Jenkins X static master on another very clean AWS EKS v1.15 cluster. I got the following similar error (lighthouse-hmac-token) in same "STEP: update-webhooks".

STEP: update-webhooks command: /bin/sh -c jx update webhooks --verbose --warn-on-fail in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/jenkins-x-boot-config/repositories

**error: secrets "lighthouse-hmac-token" not found
error: failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx update webhooks --verbose --warn-on-fail' command in directory 'repositories', output: ''**

IT-SEA17490-MBP:jenkins-x-boot-config q1447100$ jx version
NAME               VERSION
jx                 2.1.30
Kubernetes cluster v1.15.11-eks-af3caf
kubectl            v1.16.2
helm client        2.14.3
git                2.20.1 (Apple Git-117)
Operating System   Mac OS X 10.14.6 build 18G103


verifying packages

IT-SEA17490-MBP:jenkins-x-boot-config q1447100$ kg services --all-namespaces
NAMESPACE     NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP                                                                     PORT(S)                      AGE
default       kubernetes                            ClusterIP      172.20.0.1       <none>                                                                          443/TCP                      4d8h
jx            exdns-external-dns                    ClusterIP      172.20.249.4     <none>                                                                          7979/TCP                     22m
jx            heapster                              ClusterIP      172.20.176.40    <none>                                                                          8082/TCP                     19m
jx            jenkins                               ClusterIP      172.20.233.187   <none>                                                                          8080/TCP                     18m
jx            jenkins-agent                         ClusterIP      172.20.90.113    <none>                                                                          50000/TCP                    18m
jx            jenkins-x-chartmuseum                 ClusterIP      172.20.175.18    <none>                                                                          8080/TCP                     19m
jx            nexus                                 ClusterIP      172.20.148.159   <none>                                                                          80/TCP                       18m
jx            tekton-pipelines-controller           ClusterIP      172.20.201.226   <none>                                                                          9090/TCP                     18m
kube-system   jxing-nginx-ingress-controller        LoadBalancer   172.20.146.15    a24a7236a99c7408684f13c339668d86-50bde5423a5337d6.elb.us-east-1.amazonaws.com   80:30736/TCP,443:30454/TCP   4d7h
kube-system   jxing-nginx-ingress-default-backend   ClusterIP      172.20.51.102    <none>                                                                          80/TCP                       4d7h
kube-system   kube-dns                              ClusterIP      172.20.0.10      <none>                                                                          53/UDP,53/TCP                4d8h
kube-system   tiller-deploy                         ClusterIP      172.20.100.75    <none>                                                                          44134/TCP                    4d7h
q131172019 picture q131172019 on 9 May 2020

I'm working on a better fix for this, but a quick workaround is to clone your dev env repo locally and run jx boot from in it - it'll recreate the secret properly.

Good to know. I will create new AWS EKS cluster to try this quick workaround. Last night I had rolled back to jx-v2.0.1258 + jenkins-x-versions-v1.0.406 to move forward to set up Jenkins X CI/CD pipeline environment and tested successfully for our PHP applications due to tight schedule of my project.

But before you do that, could you do a kubectl get service and see if there's a hook service? Thanks.

There is no hook service when I run kubectl get service command. I know if I choose "stateless"
type (Prow + Tekton) installation, the service 'hook' as part of Prow should be running. This weekend I will test whether newest fix resolves the issue. Thanks for your quick response and quick fix!

IT-SEA17490-MBP:~ q1447100$ kg services
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
exdns-external-dns            ClusterIP   172.20.57.34     <none>        7979/TCP    13h
heapster                      ClusterIP   172.20.239.250   <none>        8082/TCP    13h
jenkins                       ClusterIP   172.20.168.79    <none>        8080/TCP    13h
jenkins-agent                 ClusterIP   172.20.163.66    <none>        50000/TCP   13h
jenkins-x-chartmuseum         ClusterIP   172.20.50.14     <none>        8080/TCP    13h
nexus                         ClusterIP   172.20.24.163    <none>        80/TCP      13h
tekton-pipelines-controller   ClusterIP   172.20.250.26    <none>        9090/TCP    13h

I clone my dev env repo locally and run 'jx boot' from in it - it is not recreated the secret properly with the same error.

STEP: update-webhooks command: /bin/sh -c jx update webhooks --verbose --warn-on-fail in dir: /Users/q1447100/SREProjects/JenkinsX-TMP/environment-jenkinsx-eks-115-dev/repositories

error: secrets "lighthouse-hmac-token" not found
error: failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx update webhooks --verbose --warn-on-fail' command in directory 'repositories', output: ''
IT-SEA17490-MBP:environment-jenkinsx-eks-115-dev q1447100$ pwd
/Users/q1447100/SREProjects/JenkinsX-TMP/environment-jenkinsx-eks-115-dev
q131172019 picture q131172019 on 9 May 2020

@q131172019
I just had the same issue. Upgraded JX to 2.1.31 and reran jx boot. Now it works.

cprovencher picture cprovencher on 11 May 2020
🎉1 👍1

I have the same error if i try to run jx boot with this jx-requirements.yml

autoUpdate:
  enabled: false
  schedule: ""
bootConfigURL: https://github.com/***/jx-boot-config
cluster:
  clusterName: coon
  devEnvApprovers:
  - luca-dichiro
  environmentGitOwner: ***
  environmentGitPublic: true
  gitKind: github
  gitName: github
  gitServer: https://github.com
  namespace: jx
  project: norse-fiber-275614
  provider: gke
  zone: europe-west2-a
environments:
- ingress:
    domain: *******.nip.io
    externalDNS: false
    namespaceSubDomain: -jx.
    tls:
      email: ""
      enabled: false
      production: false
  key: dev
  repository: environment-coon-dev
- ingress:
    domain: ""
    externalDNS: false
    namespaceSubDomain: ""
    tls:
      email: ""
      enabled: false
      production: false
  key: staging
  repository: environment-coon-staging
- ingress:
    domain: ""
    externalDNS: false
    namespaceSubDomain: ""
    tls:
      email: ""
      enabled: false
      production: false
  key: production
  repository: environment-coon-production
gitops: true
ingress:
  domain: ********.nip.io
  externalDNS: false
  namespaceSubDomain: -jx.
  tls:
    email: ""
    enabled: false
    production: false
kaniko: true
repository: nexus
secretStorage: local
storage:
  backup:
    enabled: false
    url: ""
  logs:
    enabled: false
    url: ""
  reports:
    enabled: false
    url: ""
  repository:
    enabled: false
    url: ""
vault: {}
velero:
  schedule: ""
  ttl: ""
versionStream:
  ref: v1.0.465
  url: https://github.com/jenkins-x/jenkins-x-versions.git
webhook: jenkins

do you have any update on this?

luca-dichiro picture luca-dichiro on 20 May 2020

Do you have any update on this?
I am using jx version 2.01139. Downgraded from 2.01286.
I am using Classic Jenkins.
I get the same error when my environment-cluster-staging runs to promote.
WARNING: no REPO_OWNER provided error: fail to acquire the lock: no REPO_OWNER provided

Does jx version 2.1.31 supports classic Jenkins? cprovencher mentioned upgrading to 2.1.31 fixed this issue for him but I am not sure if I can upgrade because I am using Classic Jenkins.

@abayer

zakswindow picture zakswindow on 22 May 2020

I had the same issue.

jx version
version: 3.1.15

error as below

jx gitops postprocess
there is no post processing Secret jx-post-process in namespace default so not performing any additional post processing steps
changing to the jx namespace to verify
jx ns jx --quiet
Now using namespace 'jx' on server ''.
jx verify ingress --ingress-service ingress-nginx-controller
now verifying docker registry ingress setup
jx gitops webhook update --warn-on-fail
Error: failed to validate options: failed to find hmac token from secret: could not find lighthouse hmac token lighthouse-hmac-token in namespace jx: secrets "lighthouse-hmac-token" not found
Usage:
  update [flags]

Examples:
  # update all the webhooks for all SourceRepository and Environment resource:
  jx-gitops update

  # only update the webhooks for a given owner
  jx-gitops update --org=mycorp

  # use a custom hook webhook endpoint (e.g. if you are on premise using node ports or something)
  jx-gitops update --endpoint http://mything.com

Flags:
  -b, --batch-mode                 Runs in batch mode without prompting for user input
      --endpoint string            Don't use the endpoint from the cluster, use the provided endpoint
      --exact-hook-url-match       Whether to exactly match the hook based on the URL (default true)
      --git-kind string            the kind of git server to connect to
      --git-server string          the git server URL to create the scm client
      --git-token string           the git token used to operate on the git repository. If not specified it's loaded from the git credentials file
      --git-username string        the git username used to operate on the git repository. If not specified it's loaded from the git credentials file
  -h, --help                       help for update
      --hmac string                Don't use the HMAC token from the cluster, use the provided token
      --log-level string           Sets the logging level. If not specified defaults to $JX_LOG_LEVEL
  -o, --owner string               The name of the git organisation or user to filter on
      --previous-hook-url string   Whether to match based on an another URL
  -r, --repo string                The name of the repository to filter on
      --verbose                    Enables verbose output. The environment variable JX_LOG_LEVEL has precedence over this flag and allows setting the logging level to any value of: panic, fatal, error, warn, info, debug, trace
      --warn-on-fail               If enabled lets just log a warning that we could not update the webhook

error: failed to validate options: failed to find hmac token from secret: could not find lighthouse hmac token lighthouse-hmac-token in namespace jx: secrets "lighthouse-hmac-token" not found
make: *** [versionStream/src/Makefile.mk:145: verify] Error 1
boot Job pod jx-boot-b12ae738-a383-4eb7-9a87-703d8c4f838f-nb29j has Failed
IamGabrielWu picture IamGabrielWu on 9 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

tdcox picture tdcox  路  3Comments
tdcox picture tdcox  路  3Comments
abayer picture abayer  路  4Comments
ccojocar picture ccojocar  路  3Comments
empath picture empath  路  3Comments