Jx: JX Boot Fails IRSA On Existing EKS Cluster

[Callumccr]

Summary

Having issues running a jx boot. We've followed the documentation and cloned a fresh jenkins-x-boot-config from branch master against an existing EKS Cluster v1.14.0. Failing at the verify-preinstall STEP when it validates and deploys the IRSA ClouformationStacks.

It successfully deploys the AWS IAM Policies, Roles and configures each respective role correct for the IRSA, however, when it tries to validate the k8s namespaces and service accounts, it fails and stops the jx boot

Based on the error output, my assumption is that the RBAC I'm using to access the cluster is not sufficient to pass this pre-validate STEP in jx boot. My reasoning behind this is because if I run jx boot using the --start-step flag on any of the helm deployments, it deploys the services as expected outside of pre-validating the cluster.

Interestingly, I've deployed JX in an EKS cluster successfully using the jx create cluster eks and then jx boot to deploy the cluster based on my specifics in the jx-requirements.yml file using the credentials in my default AWS profile.

We would like to continue to use our existing Clouformation templates to deploy our EKS cluster have JX only manage the IRSA roles required for JX boot to work with the cluster, any thoughts?

Steps to reproduce the behavior

• Have an existing EKS Cluster 1.14.0
• Have that EKS cluster deployed using eksctl (not jx create cluster eks) and passing it a config file with the desired configuration
• We have associated tags that map access to the cluster access via an assumed role with the permissions RBAC as system:masters
• Set our K8s and JX context and namespace to the desired cluster
• Can run kubectl get ns successfully
• Able to deploy K8s resources using jx compliance run

Expected behavior

Because we want JX to manage our IRSA and K8's objects, we expect the jx boot to complete all steps in the jenkins-x pipeline file to run successfully.

Actual behavior

It successfully deploys the IAM Policies, Roles and configures each respective role correctly using IRSA clouformationn, however, when it tries to validate the k8s Service accounts, it fails with the following:

[ℹ]  deploying stack "eksctl-REDACTED-addon-iamserviceaccount-jx-jxui"
[ℹ]  deploying stack "eksctl-REDACTED-addon-iamserviceaccount-cert-manager-cm-cainjector"
[ℹ]  deploying stack "eksctl-REDACTED-addon-iamserviceaccount-jx-jenkins-x-controllerbuild"
[ℹ]  deploying stack "eksctl-REDACTED-addon-iamserviceaccount-jx-tekton-bot"
[ℹ]  deploying stack "eksctl-REDACTED-addon-iamserviceaccount-cert-manager-cm-cert-manager"
[ℹ]  deploying stack "eksctl-REDACTED-addon-iamserviceaccount-jx-exdns-external-dns"
[ℹ]  6 error(s) occurred and IAM Role stacks haven't been created properly, you may wish to check CloudFormation console
[✖]  checking whether namespace "cert-manager" exists: Unauthorized
[✖]  checking whether namespace "jx" exists: Unauthorized
[✖]  checking whether namespace "cert-manager" exists: Unauthorized
[✖]  checking whether namespace "jx" exists: Unauthorized
[✖]  checking whether namespace "jx" exists: Unauthorized
[✖]  checking whether namespace "jx" exists: Unauthorized
Error: failed to create iamserviceaccount(s)
error: error creating the IRSA managed Service Accounts: failure creating the IRSA managed service accounts: there was a problem executing the IRSA ConfigFile: there was a problem calling eksctl with the provided args: failed to run 'eksctl create iamserviceaccount --override-existing-serviceaccounts --config-file /var/folders/m9/357z1flj7kz70c5rs0p77_fc0000gn/T/irsa-template-729583734 --include="*" --approve' command in directory '', output: ''
error: failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx step verify preinstall --provider-values-dir="kubeProviders"' command in directory '.', output: ''

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.0.1234
Kubernetes cluster v1.14.9-eks-502bfb
kubectl            v1.14.7-eks-1861c5
git                2.20.1 (Apple Git-117)
Operating System   Mac OS X 10.14.6 build 18G3020

Jenkins type

Select which installation type are you using:

• Serverless Jenkins X Pipelines (Tekton + Prow)

Kubernetes cluster

• EKS 1.14.0, created as above

Operating system / Environment

Operating System:

• Mac OS X 10.14.6 build 18G3020

[deanesmith]

Hi @Callumccr, just to confirm, were you referencing this document https://jenkins-x.io/docs/getting-started/setup/boot/clouds/amazon/#iam-policies-for-cluster-creation-and-jenkins-x-boot ?

[Callumccr]

hey @deanesmith, that's correct, it looks like the jx boot will configure and deploy IRSA when you run it against the type eks cluster

[Callumccr]

I managed to get this working, it looks like it was deploying the cloud formation stacks successfully but potentially couldn't validate them? I run the jx boot -s install-jx-crds and rm -rf ~/.jx for it to continue with a successful installation.

I was deploying this using my own personal credentials with AdministratorAccess for AWS services.

My thoughts are how the aws-iam-authenticator works for EKS using mapped roles, potentially JX assumes that I'm going to be accessing the cluster with the same credentials as it was deployed with which is why it fails the K8's namespaces?

My recommendation is to manage IRSA outside of JX boot for EKS right now and use terraform or cloud formation and set the terraform: false in the jx-requirements.yml if anyone else is having this problem

[mlasalmo]

Facing the same issue following same steps on an existing EKS cluster when installing jx with the boot command.

jx version output

NAME               VERSION
jx                 2.0.1278
Kubernetes cluster v1.14.9-eks-502bfb
kubectl            v1.16.3
git                2.26.1
Operating System   Mac OS X 10.14.6 build 18G3020

Actual behavior

Error: existing iamserviceaccount "iamserviceaccountpoc/iamserviceaccountpoc" should be excluded, but matches include filter: *
error: error creating the IRSA managed Service Accounts: failure creating the IRSA managed service accounts: there was a problem executing the IRSA ConfigFile: there was a problem calling eksctl with the
provided args: failed to run 'eksctl create iamserviceaccount --override-existing-serviceaccounts --config-file /var/folders/fh/wkn_b0191tj3cc2pp_s755k80000gn/T/irsa-template-451005005 --include="*" --app
rove' command in directory '', output: ''
error: failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx step verify preinstall --provider-values-dir="kubeProviders"' command in directory '.', output: ''

Note: Setting terraform: false in the jx-requirements.yaml file didn't solve the issue. After running jx boot the terraform setting is removed from the file.

[jenkins-x-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle stale

[jenkins-x-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle rotten

[jenkins-x-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://jenkins-x.io/community.
/close

[jenkins-x-bot]

@jenkins-x-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://jenkins-x.io/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository.