jx boot fails at step to create vault

[gazal-k]

Summary

jx boot fails at step to create vault

Steps to reproduce the behavior

1. changes in jx-requirements.yml

   cluster:
     provider: eks
     gitKind: bitbucketcloud
     gitName: bc
     gitServer: https://bitbucket.org
   secretStorage: vault
   vault:
     name: clustername-vault
   webhook: lighthouse
   

2. Run jx boot

Expected behavior

Installs Jenkins X in the EKS cluster

Actual behavior

while installing vault, jx boot errors out

vault-operator addon succesfully installed.
vault operator installed in namespace jx
finding vault 'clustername-vault' in namespace jx
Creating new system vault
error: creating vault: missing S3 bucket flag
error: failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx step boot vault' command in directory 'systems/vault', output: ''

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.0.798
Kubernetes cluster v1.13.10-eks-5ac0f1
kubectl            v1.16.0
git                2.21.0
Operating System   "Fedora release 30 (Thirty)"

Jenkins type

• [x] Serverless Jenkins X Pipelines (Tekton + lighthouse)
• [ ] Classic Jenkins

Kubernetes cluster

EKS created using https://eksctl.io/gitops-quickstart/setup-gitops/

Operating system / Environment

Operating System "Fedora release 30 (Thirty)"

[gazal-k]

Is this related to #5113 ?

[TarekAS]

Having the same problem. The cluster is in a subaccount, if that helps.

[gazal-k]

This terraform snippet: https://jenkins-x.io/getting-started/aws-terraform-install-gitops/ sets up S3, dynamo db and KMS for vault. Is that only applicable when using jx install?

With jx boot, did I follow the recommended steps, creating the cluster with eksctl and running jx boot on a fork of the boot configuration repo?

[code-is-art]

got the same problem

vault-operator addon successfully installed.
vault operator installed in namespace jx
finding vault 'xxx-clstr-01-vault' in namespace jx
Creating new system vault
error: creating vault: missing S3 bucket flag

will not work with jx boot

[martijnburger]

Here's the failing step in my env with verbose logging:

martijnburger@martijn-lt02:~/environment-apollon-dev$ jx step boot vault --provider-values-dir ../../kubeProviders --verbose
No --namespace option specified or $DEPLOY_NAMESPACE environment variable available so defaulting to using namespace jx
DEBUG: Setting the dev namespace to: jx
DEBUG: Using vault name 'jx-vault-paston'
WARNING: failed to find file vault-ing.tmpl.yaml
DEBUG: Setting the dev namespace to: jx
Region not specified for DynamoDB, defaulting to eu-west-1
Region not specified for KMS, defaulting to eu-west-1
Region not specified for S3, defaulting to eu-west-1
DEBUG: Current configuration dir: /home/martijnburger/.jx
DEBUG: VersionRepository: https://github.com/jenkins-x/jenkins-x-versions.git git ref: v1.0.209
? A local Jenkins X versions repository already exists, pulling the latest: Yes
installing vault operator with helm values: image.repository=banzaicloud/vault-operator,image.tag=0.5.3
DEBUG: Using helmBinary helm with feature flag: none
DEBUG: getting vault name for namespace jx
DEBUG: system vault name from config jx-vault-paston
DEBUG: vault jx-vault-paston not found in namespace jx, err is vaults.vault.banzaicloud.com "jx-vault-paston" not found
Installing vault-operator...
DEBUG: Current configuration dir: /home/martijnburger/.jx
DEBUG: VersionRepository: https://github.com/jenkins-x/jenkins-x-versions.git git ref: v1.0.209
? A local Jenkins X versions repository already exists, pulling the latest: Yes
DEBUG: getting vault name for namespace jx
DEBUG: system vault name from config jx-vault-paston
DEBUG: vault jx-vault-paston not found in namespace jx, err is vaults.vault.banzaicloud.com "jx-vault-paston" not found
DEBUG: using stable version 0.3.1 from charts of jenkins-x/vault-operator from /home/martijnburger/.jx/jenkins-x-versions
Upgrading Chart 'upgrade --namespace jx --install --force --timeout 6000 --version 0.3.1 --set image.repository=banzaicloud/vault-operator --set image.tag=0.5.3 vault-operator jenkins-x/vault-operator'
vault-operator addon successfully installed.
vault operator installed in namespace jx
finding vault 'jx-vault-paston' in namespace jx
DEBUG: vault jx-vault-paston not found in namespace jx, err is vaults.vault.banzaicloud.com "jx-vault-paston" not found
Creating new system vault
DEBUG: vault jx-vault-paston not found in namespace jx, err is vaults.vault.banzaicloud.com "jx-vault-paston" not found
DEBUG: cluster short name for vault naming: 'paston'
DEBUG: Created service account 'jx-vault-paston-auth-sa' for Vault authentication
error: creating vault: missing S3 bucket flag

[martijnburger]

Looks like it's in this piece of code where the error is:

https://github.com/jenkins-x/jx/blob/db587464c93de338afefa16acd616f665b97a879/pkg/cmd/create/create_vault.go#L368

@dgozalo can you maybe take a look, as you last touched this code, I am not proficient enough in Go to be of any more help, I am afraid.

[gazal-k]

Looks like this change maybe related; jenkins-x/jenkins-x-boot-config#101

[uny]

This error will be solved by modifying jx-requirements.yml,

vault:
  name: clustername-vault
  aws:
    autoCreate: true

or

vault:
  name: clustername-vault
  aws:
    s3Bucket: xxxxx

Reference

Then, however, I am facing the next error:

Creating vault resources with following values, ap-northeast-1, vault_ap-northeast-1, vault-data, vault-unseal.ap-northeast-1.jenkins-x-domain
DEBUG: Attempting to read Vault CloudFormation template from path ../../kubeProviders/eks/templates/vault_cf_tmpl.yml
error: creating vault: an error occurred while creating the vault resources: executing the Vault CloudFormation : unable to create vault prerequisite resources: MissingRegion: could not find region configuration
error: failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx step boot vault --provider-values-dir ../../kubeProviders' command in directory 'systems/vault', output: ''

It seems that Region option is needed in somewhere. For example, here:
https://github.com/jenkins-x/jx/blob/6193d12664ba91fbe68fd02649a76d916b211feb/pkg/cloud/amazon/vault/vault_resources.go#L449

Reference

[martijnburger]

I have solved this problem by adding the following:

vault:
  aws:
    autoCreate: true
    dynamoDBTable: jx-vault
    iamUserName: martijnburger
    kmsKeyId: jx-vault
    s3Bucket: jx-vault
    s3Region: eu-west-1

The username is my own IAM username with all premissions, should probably better be an IAM account created with S3 and KMS access or something. Be aware that the IAM account needs a free access key "slot" for the IAM user, as a user can have maximum of two access keys active at the same moment.

[anilkumarpasupuleti]

@martijnburger , i had the same problem, but i was able to solve it by adding the above code. but the problem is that i can run jx boot maximum of two times with above code, after that if i want to run jx boot, it is failing to update or create vault. is there any solution that, i can create vault once, run jx boot any number of times?. Thanks in advance

[dgozalo]

The problem lies within the way we handle Vault.

We need you to provide an IAM User in order to use its access keys to authenticate the Vault pod.

You can only have a maximum of 2 access key pairs per IAM User, so you'll have to delete one of the pairs before running boot again.

You can also provide an already existing key pair through env variables:

VAULT_AWS_ACCESS_KEY_ID and VAULT_AWS_SECRET_ACCESS_KEY.

I hope this helps.

[anilkumarpasupuleti]

@dgozalo, thanks, setting up vault environmental variable solved the issue. but there is another problem the way you handle vault,each time I run jx boot, its creating new vault resources, isn't that going to problem.

[deanesmith]

Closing per original issue report.

[martijnburger]

Closing per original issue report.

Sorry, I am not familiar with this terminology. What do you mean?

[deanesmith]

Hi @martijnburger, the original issue that was reported should now be addressed. If there is a separate issue when re-running boot, it should be captured and described in a new issue. FWIW, I am not able to duplicate that at this moment.