Terraform-provider-aws: Retrieve OIDC provider thumbprint not populating which is required for EKS service accounts

[marcincuber]

Description

New or Affected Resource(s)

Currently I can specify the following:

resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = []
  url             = aws_eks_cluster.cluster.identity.0.oidc.0.issuer
}

There is no way to retrieve thumbprint for that OIDC provider using terraform.

Note that if you create the same OIDC provider in the console, it will automatically populate the thumbprint which is required for EKS service accounts to assume correct IAM Role.

References

Current way of getting thumbprint is documented here -> https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html#thumbstep2

[gkrizek]

+1

I went through the example here:
https://www.terraform.io/docs/providers/aws/r/eks_cluster.html#enabling-iam-roles-for-service-accounts

In which thumbprint_list is an empty list. Everything worked until I tried to actually do an AWS operation:

An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint

I really wish AWS offered an easier way to get the thumbprint of the certificate.

[gkrizek]

After some searching it looks like this could solve our issue if implemented:
https://github.com/terraform-providers/terraform-provider-tls/issues/52

[zzh8829]

In the mean time here is a quick hack to get around it, not ideal but tested working

echo | openssl s_client -connect oidc.eks.us-west-2.amazonaws.com:443 2>&- | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}'

locals {
  eks-oidc-thumbprint = "$OUTPUT_FROM_ABOVE"
}

resource "aws_iam_openid_connect_provider" "eks-oidc" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [local.eks-oidc-thumbprint]
  url             = "${aws_eks_cluster.cluster.identity.0.oidc.0.issuer}"
}

[marcincuber]

There is an automated way that I received from kubernetes slack which is using external source:

### External cli kubergrunt
data "external" "thumb" {
  program = ["kubergrunt", "eks", "oidc-thumbprint", "--issuer-url", aws_eks_cluster.cluster.identity.0.oidc.0.issuer]
}
### OIDC config
resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumb.result.thumbprint]
  url             = aws_eks_cluster.cluster.identity.0.oidc.0.issuer
}

[zzh8829]

I personally think this thumbprint thing should live in terraform-provider-aws repo.
SHA1 finger print are usually in the form of

SHA1 Fingerprint=63:96:61:30:76:16:08:20:97:18:C5:04:5C:FF:B4:85:6F:B5:39:76

But aws_iam_openid_connect_provider needs it to be like (yes lowercase)

I don't know if this specific transform is general enough to live in Terraform TLS provider.

[dogzzdogzz]

For anyone prefer not to install kubergrunt, I merge both @zzh8829 and @marcincuber workaround to make terraform automatically retrieve thumbprint from external script.

thumbprint.sh

#!/bin/bash

THUMBPRINT=$(echo | openssl s_client -connect oidc.eks.$1.amazonaws.com:443 2>&- | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

terraform.tf

data "external" "thumbprint" {
  program = ["thumbprint.sh", data.aws_region.current.name]
}

resource "aws_iam_openid_connect_provider" "this" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumbprint.result.thumbprint]
  url             = "${aws_eks_cluster.this.identity.0.oidc.0.issuer}"
}

[matteosilv]

@dogzzdogzz thanks for your solution! Unfortunately i tried it without any luck. It is giving me an incorrect thumbprint.

From my understanding of the aws tutorial we need to extract only the last certificate between BEGIN CERTIFICATE and END CERTIFICATE, that one is the certificate of the root CA.

Then we can give it to openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}' to extract the thumbprint

[JamiKarvanen]

Yep, that produces a non-working thumbprint. Here's my horrible oneliner to produce a working thumbprint:
echo | openssl s_client -servername oidc.eks.${REGION}.amazonaws.com -showcerts -connect oidc.eks.${REGION}.amazonaws.com:443 2>&- | tail -r | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tail -r | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}'

Edit: this is for OSX, for GNU/Linux replace tail -r with tac

[matteosilv]

Thanks! I combined the two solutions to get a working terraform resource

[chiefy]

So the solutions above don't work for our setup, we use an https proxy and thus the CA chain won't be correct if I got through that. I can tunnel through a bastion, but am I missing something here, that CA root will be mostly static for each AWS region, no? Wouldn't it be viable to just keep the fingerprints for each region in a TF map until a better solution presents itself?

[willthames]

@chiefy you don't even need a map - it looks like the root CA is the same for the two regions I tried, and I imagine that's the case for all the regions.

Setting thumbprints to ["9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"] should suffice :)

[chiefy]

@willthames thanks for confirming my sanity.

[dayglojesus]

For anybody who is running into issues with differing or inconsistent thumbprints, this might help ...

~When running openssl s_client -servername oidc.eks.${REGION}.amazonaws.com etc. from "inside" the cluster (from one of your EKS workers), you get a cert like:~
When running openssl s_client -servername oidc.eks.${REGION}.amazonaws.com etc. from "inside" the pod, you get a cert like:

subject=CN = oidc.eks.us-west-2.amazonaws.com
issuer=C = US, O = Amazon, OU = Server CA 1B, CN = Amazon

☝️ This is the thumbprint you want.

But, if you are performing this same operation from outside the cluster, you will likely receive something like:

subject=/CN=*.execute-api.us-west-2.amazonaws.com
issuer=/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon

HTH

[Grejeru]

@dayglojesus yes, it helps, but still there is a problem with automating this. If we need to run command from inside Pod to get thumbprint required to automate IAM role policies for ServiceAccounts makes it impossible to work :-/
Nevertheless thx, as it solve my issue why script generated thumbprint didn't worked and had to find it other way. Which is even funnier, there are more then 1 cert, to be exact 4 of them and with this list I got it working.

[dayglojesus]

@Grejeru I agree, this shouldn't be a hard requirement, but per @chiefy's comments ... You should be able to construct a map of the correct thumbs, one for each required region.

[rastakajakwanna]

Setting thumbprints to ["9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"] should suffice :)

@willthames well, this will result in constant drift because the thumbprint needs to be lowercase (at least that's my experience).
However, all our eu regions have the same thumbprint. I tried the script from outside of VPC as well as from Gitlab runner in EKS and I've got the same results/thumbprint.

I merged my previous ugly one-liner with the suggestions here and made this (for tf v.0.11.x):

<module_name>/bin/get_thumbprint.sh

#!/bin/bash
set -e 

XR="${1:-eu-west-1}"
XT=`mktemp`
XXT=`mktemp`

function cleanup {
  rm -f ${XT} ${XXT}
}

trap cleanup SIGHUP SIGINT SIGTERM EXIT

THUMBPRINT=$(echo QUIT | openssl s_client -showcerts -connect oidc.eks.${XR}.amazonaws.com:443 2>/dev/null > ${XT}; cat ${XT} | sed -n '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p' | tac | awk '/-----BEGIN CERTIFICATE-----/ {exit} 1' > ${XXT} && echo '-----BEGIN CERTIFICATE-----' >> ${XXT} && tac ${XXT} > ${XT}; openssl x509 -in ${XT} -fingerprint -noout | sed -r 's|.*+?=(.*)|\1|g' | sed 's|:||g' | awk '{print tolower($1)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

<module_name>/data.tf

data "aws_region" "current" {}
data "external" "thumbprint" {
  program = ["${path.module}/bin/get_thumbprint.sh", "${data.aws_region.current.name}"]
}

<module_name>/main.tf

resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = ["${data.external.thumbprint.result.thumbprint}"]
  url             = "${data.aws_eks_cluster.this.identity.0.oidc.0.issuer}"
}

It seems that this works perfectly fine for the kube-system aws-node service account. At least as much as I can say after two days....

[willthames]

@rastakajakwanna yes, you're quite right, I think I wrote my comment after realising that it worked, but before realising about the drift you mention.

But I'm literally just hardcoding the (lowercase) value into a variable. I don't see the benefit of having terraform run a script to generate a value that so far is consistent across all regions, and given it's a root CA, likely will remain so for a decade.

[mzupan]

This works with terraform cloud

#!/bin/bash

THUMBPRINT=$(echo | openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | tac | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tac | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

data "external" "thumbprint" {
  program = [format("%s/bin/get_thumbprint.sh", path.module), data.aws_region.current.name]
}

resource "aws_iam_openid_connect_provider" "this" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumbprint.result.thumbprint]
  url             = module.eks.cluster_oidc_issuer_url
}

[jujugrrr]

Should this issue be marked as a bug? The terraform provider documentation is misleading, as an empty thumbprint doesn't allow to use EKS IAM with service account.

[marcincuber]

@jujugrrr I don't think this is a bug. Terraform is simply missing a way to retrieve the thumbprint. I do agree that documentation is not perfect.

[jujugrrr]

@jujugrrr I don't think this is a bug. Terraform is simply missing a way to retrieve the thumbprint. I do agree that documentation is not perfect.

Fair enough 😄 , I've raised it with AWS, but it feels it's a sane behavior not to expose the thumbprint through the API response on their side, which is probably why the AWS Console auto populate it for convenience.

Let's follow https://github.com/terraform-providers/terraform-provider-aws/pull/10217

[cippaciong]

If anyone needs to run @mzupan example on macOS/BSD/others, please consider that tac is not available there by default and you need to either install coreutils or replace it with tail -r, which is what I did:

#/bin/bash

THUMBPRINT=$(echo | \
    openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | \
    tail -r | \
    sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | \
    tail -r | \
    openssl x509 -fingerprint -noout | \
    sed 's/://g' | awk -F= '{print tolower($2)}')

THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"

echo $THUMBPRINT_JSON

Thanks @mzupan ❤️

[s0enke]

~When running openssl s_client -servername oidc.eks.${REGION}.amazonaws.com etc. from "inside" the cluster (from one of your EKS workers), you get a cert like:~
When running openssl s_client -servername oidc.eks.${REGION}.amazonaws.com etc. from "inside" the pod, you get a cert like:

@dayglojesus @Grejeru I think I found the root issue for different certificates from different clients:

It depends on the implementation/version of the openssl command. In OpenSSL (since 1.1?) the connect will automatically send an servername with the connection (Server Name Indication). See [1]

Other implementations like LibreSSL (on Mac) or CodeBuild/AmazonLinux 2 (which use OpenSSL 1.0.2) don't automatically set the servername so we have to provide it.

The following command should work independently from OpenSSL versions/implemenentations:

openssl s_client -connect oidc.eks.eu-central-1.amazonaws.com:443 -servername oidc.eks.eu-central-1.amazonaws.com

[1] https://www.openssl.org/docs/man1.1.1/man1/openssl-s_client.html

[mzupan]

@cippaciong

I have an updated version that tests for a env var used by tf cloud

#!/bin/bash



if [[ -z "${ATLAS_WORKSPACE_SLUG}" ]]; then
  APP="tail -r"
else
  APP="tac"
fi

THUMBPRINT=$(echo | openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | ${APP} | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | ${APP} | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

[christian-marie]

If your openssl's x509 fingerprint behaviour defaults to something other than sha1 you may get invalid fingerprints:

Member must satisfy constraint: [Member must have length less than or equal to 40, Member must have length greater than or equal to 40]

Being explicit fixes this:

#!/bin/sh
# https://github.com/terraform-providers/terraform-provider-aws/issues/10104

THUMBPRINT=$(echo | openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | tac | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tac | openssl x509 -fingerprint -sha1 -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

[groodt]

The thumbprint even seems to be the same cn-northwest-1

echo | openssl s_client -servername oidc.eks.cn-northwest-1.amazonaws.com.cn -showcerts -connect oidc.eks.cn-northwest-1.amazonaws.com.cn:443 2>&- | tail -r | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tail -r | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}'

[scardena]

For anyone prefer not to install kubergrunt, I merge both @zzh8829 and @marcincuber workaround to make terraform automatically retrieve thumbprint from external script.

thumbprint.sh

#!/bin/bash

THUMBPRINT=$(echo | openssl s_client -connect oidc.eks.$1.amazonaws.com:443 2>&- | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

terraform.tf

data "external" "thumbprint" {
  program = ["thumbprint.sh", data.aws_region.current.name]
}

resource "aws_iam_openid_connect_provider" "this" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumbprint.result.thumbprint]
  url             = "${aws_eks_cluster.this.identity.0.oidc.0.issuer}"
}

@mzupan I had to modify this answer to work for me, as I am using it in a module:
program = ["${path.module}/thumbprint.sh", var.aws_region]

[Grejeru]

@mzupan I had to modify this answer to work for me, as I am using it in a module:
program = ["${path.module}/thumbprint.sh", var.aws_region]

You can use data "aws_region" "current" {} in the module too, it assumes the region in which Your provider works, don't need to pass aws_region into module then.

[romaryoricardo]

You can generate fingerprint and set in your thumbprint_list follow how to this link: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html

Those are all CA's fingerprint by region:

3ACE16CA6BAE7B16AAE3707096D1DE7D29093AD8
ap-south-1
59ADE3A3A6039BBA092E920FEE413466493F409C
eu-west-3
3FB881BCACBD420928168C739B07EEF47555946B
eu-west-2
7CA6BE9F14E20973CB2C58452DA9B1E2BEB7236B
eu-west-1
CAB073498D7558FEC3B2C414C006ACBA30805431
ap-northeast-2
2EAFC197C15CDEE5426BFD4D27D3321A685F3B78
ap-northeast-1
B715DC079832DA5FC1D4706515BE48BE79A1C871
sa-east-1
CB454452665937052981CA118417B7A162A25F54
ca-central-1
1C8B5245E80A6B7A0E8BF5FFDAB032273D7D5DF1
ap-southeast-1
F719C49FEA86549E159818880E392C1570C953B6
ap-southeast-2
0148872FA02F3A7D6B38AA88FA5397228B28E08B
eu-central-1
9884072430220E6253011B88F940E4F20F53D0CC
us-east-1
598ADECB9A3E6CC70AA53D64BD5EE4704300382A
us-east-2
750B948515281953BC6F3D717A1E1654ECBFA852
us-west-1
89BABC6D46502653516CC0BA38B14A2B7864D161
us-west-2
63966130761608209718C5045CFFB4856FB53976

You can define your variable like this:

thumbprint_list = ["3ACE16CA6BAE7B16AAE3707096D1DE7D29093AD8","59ADE3A3A6039BBA092E920FEE413466493F409C","3FB881BCACBD420928168C739B07EEF47555946B","7CA6BE9F14E20973CB2C58452DA9B1E2BEB7236B","CAB073498D7558FEC3B2C414C006ACBA30805431","2EAFC197C15CDEE5426BFD4D27D3321A685F3B78","B715DC079832DA5FC1D4706515BE48BE79A1C871","CB454452665937052981CA118417B7A162A25F54","1C8B5245E80A6B7A0E8BF5FFDAB032273D7D5DF1","F719C49FEA86549E159818880E392C1570C953B6","0148872FA02F3A7D6B38AA88FA5397228B28E08B","9884072430220E6253011B88F940E4F20F53D0CC","598ADECB9A3E6CC70AA53D64BD5EE4704300382A","750B948515281953BC6F3D717A1E1654ECBFA852","89BABC6D46502653516CC0BA38B14A2B7864D161","63966130761608209718C5045CFFB4856FB53976"]

command for get fingerprint :

echo | openssl s_client -servername oidc.eks.${region_name}.amazonaws.com -connect oidc.eks.${region_name}.amazonaws.com:443 2>&- | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print $2}'

[scardena]

@romaryoricardo I haven't test this yet, but I recommend @mzupan (or @cippaciong for macos) workaround over yours. I am not sure why this happens, but the fingerprints generated using your command are not the same as the ones generated by @mzupan commands (I am going to throw the guess that multiple certificates are being spit out for each command, but only 1 is processed, in @mzupan case is the most recent one, and in your case is the oldest one, although i can be completely wrong). It doesn't mean your solutions shouldn't work, but if we inspect the certificate of the fingerprint that your command spits, we can see that this certificate is about to expire (check the validity, it expires september 2020):

echo | openssl s_client -servername oidc.eks.${region_name}.amazonaws.com -connect oidc.eks.${region_name}.amazonaws.com:443 2>&- | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | openssl x509 -fingerprint -noout -text

whereas in the other, it expires on 2034.
More over, I ran @mzupan script for all regions, and the output fingerprint seems to be the same for absolutely all regions (as @willthames suggests), so in my case, we decided to simply hardcode the fingerprint in the terraform module, although we still have the workaround commented out for future reference. We know hardcoding stuff sucks, but the workarounds presented are not too portable.

[wojciesze]

For those who are not happy with dirty tricks in shell script: get_thumbprint.sh (e.g: tac twice).
Here is a working python script: (tested with python3)

import socket
from OpenSSL import SSL
import certifi
import sys, json

hostname = f'oidc.eks.{sys.argv[1]}.amazonaws.com'
port = 443


context = SSL.Context(method=SSL.TLSv1_METHOD)
context.load_verify_locations(cafile=certifi.where())

conn = SSL.Connection(context, socket=socket.socket(socket.AF_INET, socket.SOCK_STREAM))
conn.settimeout(5)
conn.connect((hostname, port))
conn.setblocking(1)
conn.do_handshake()
conn.set_tlsext_host_name(hostname.encode())

thumbprint = conn.get_peer_cert_chain()[-1].digest("sha1")
obj = {"thumbprint": thumbprint.decode("utf-8").replace(":", "").lower() }
print(json.dumps(obj))
conn.close()

For alpine docker images it enough to install:

apk add py3-openssl
pip3 install certifi

For python:3 docker image:

pip3 install pyopenssl certifi

[guitarrapc]

https://github.com/terraform-providers/terraform-provider-aws/issues/10104#issuecomment-632309246 used to be show longer certificate, which expire 2034.

endpoint: oidc.eks.ap-northeast-1.amazonaws.com

{"thumbprint": "9e99a48a9960b14926bb7f3b02e22da2b0ab7280"}

However I've ran same script and it shows shorter certificate, it expires on Sep 2020. (And is match to certificate show on browser.)

endpoint: oidc.eks.ap-northeast-1.amazonaws.com

{"thumbprint": "b715dc079832da5fc1d4706515be48be79a1c871"}

Certificate Chain

, CN=oidc.eks.ap-northeast-1.amazonaws.com, B715DC079832DA5FC1D4706515BE48BE79A1C871, 08/09/2020 21:00:00
, CN=Amazon, OU=Server CA 1B, O=Amazon, C=US, 917E732D330F9A12404F73D8BEA36948B929DFFC, 19/10/2025 09:00:00
, CN=Amazon Root CA 1, O=Amazon, C=US, 06B25927C42A721631C1EFD9431E648FA62E1E39, 31/12/2037 10:00:00
, CN=Starfield Services Root Certificate Authority - G2, O="Starfield Technologies, Inc.", L=Scottsdale, S=Arizona, C=US, 9E99A48A9960B14926BB7F3B02E22DA2B0AB7280, 29/06/2034 02:39:16
Starfield Class 2 Certification Authority, OU=Starfield Class 2 Certification Authority, O="Starfield Technologies, Inc.", C=US, AD7E1C28B064EF8F6003402014C3D0E3370EB58A, 30/06/2034 02:39:16

[marcincuber]

@guitarrapc this is a correct value for your region. B715DC079832DA5FC1D4706515BE48BE79A1C871. The initial value is for a different region.
See previous comment https://github.com/terraform-providers/terraform-provider-aws/issues/10104#issuecomment-633130751

I feel like this is an issue that can now be closed. There are numerous solutions suggested that will solve problems with OIDC provider.

[nickatsegment]

Oof, I just created a new cluster today and it has a new cert for us-west-2:

$ region_name=us-west-2; echo | openssl s_client -servername oidc.eks.${region_name}.amazonaws.com -connect oidc.eks.${region_name}.amazonaws.com:443 2>&- | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | openssl x509 -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0f:a2:dc:f6:0e:ff:f7:41:2f:76:c8:3c:e2:71:0d:6f
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Amazon, OU=Server CA 1B, CN=Amazon
        Validity
            Not Before: Jul 10 00:00:00 2020 GMT
            Not After : Aug 10 12:00:00 2021 GMT

$ region_name=us-west-2; echo | openssl s_client -servername oidc.eks.${region_name}.amazonaws.com -connect oidc.eks.${region_name}.amazonaws.com:443 2>&- | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print $2}'

7FF5BDFA36CD78827820278B4DABB5E1E97C1476

[tyrken]

@nickatsegment I'm thinking this is as you've made the same mistake I made and are using the "leaf" server cert which gets updated owing to it's short expiry. Instead use the root Certificate Authority cert which expires in 2034 or similar.

Switch to use the commands above which have -showcerts in the openssl command line to show all certs in the chain, but beware most force you to use a line reordering utility to pick out the right cert, which changes depending on the OS you run: tac (Linux) or tail -r OS/X.

[justcompile]

Taken from a number of sources but with explicit help from this article https://medium.com/@michael.kandelaars/did-your-eks-iam-service-account-roles-break-today-2ea50c869aee, this thread & of course Stackoverflow for bash-fu, I've got this script which grabs the root ca thumbprint.

Not the prettiest, but function over form right?

#!/usr/bin/env bash

echo | openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | awk '/-----BEGIN/{f="cert."(n++)} f{print>f} /-----END/{f=""}'

certificates=()

for c in cert.*; do
   certificates+=($(openssl x509 <$c -noout -fingerprint))
done
rm cert.* 

thumbprint=$(echo ${certificates[${#certificates[@]}-1]} | sed 's/://g' | awk -F= '{print tolower($2)}')
thumbprint_json="{\"thumbprint\": \"${thumbprint}\"}"
echo $thumbprint_json

./this_script.sh <aws-region-code>

[nickatsegment]

@tyrken ah yep, you're right. Thanks for the pointer.

Here's hoping that TLS provider PR lands sometime in the next century so we can all dispense with the janky awk scripts

[olfway]

My external for this

external/thumbprint

#!/bin/bash

set -euo pipefail

HOST="oidc.eks.$1.amazonaws.com"

# https://github.com/terraform-providers/terraform-provider-aws/issues/10104

echo | openssl s_client -servername "$HOST" -showcerts -connect "$HOST:443" 2>&- \
  | sed -n '/-----BEGIN CERTIFICATE-----/h;//!H;$!d;x;s/\(.*-----END CERTIFICATE-----\).*/\1/p' \
  | openssl x509 -fingerprint -sha1 -noout \
  | tr '[:upper:]' '[:lower:]' \
  | sed 's/://g; s/.*=\(.*\)/{"thumbprint": "\1"}/'

Use it like this:

data "external" "thumbprint" {
  program = [
    "/bin/sh",
    "${path.module}/external/thumbprint",
    data.aws_region.current.name,
  ]
}

resource "aws_iam_openid_connect_provider" "this" {
  url = aws_eks_cluster.this.identity[0]["oidc"][0]["issuer"]
  client_id_list = [
    "sts.amazonaws.com",
  ]
  thumbprint_list = [
    data.external.thumbprint.result.thumbprint,
  ]
}

For eu-central-1 region it returns fingerprint for this cert:

Issuer: C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
Validity
    Not Before: Sep  2 00:00:00 2009 GMT
    Not After : Jun 28 17:39:16 2034 GMT
Subject: C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!