Installer: AWS Credentials Validation Fails in 0.13

[wkulhanek]

@wking As requested.

Version

$ openshift-install version
openshift-install v0.13.0

Platform (aws|libvirt|openstack):

aws

What happened?

Tried to install a cluster with AWS credentials that only have permissions for region 'us-east-1'. Installer checks for permissions and fails:

[wkulhane-redhat.com@bastion .aws]$ openshift-install create cluster --dir=$HOME/wktest
? SSH Public Key /home/wkulhane-redhat.com/.ssh/cluster-ede2-key.pub
? Platform aws
? Region us-east-1
? Base Domain ede2.ocp4.opentlc.com
? Cluster Name wktest
? Pull Secret [? for help] *******************************************************************WARN[0019] Action not allowed with tested creds action="ec2:AllocateAddress"************************************************WARN[0019] Action not allowed with tested creds action="ec2:AssociateAddress"***********************************************WARN[0019] Action not allowed with tested creds action="ec2:AssociateDhcpOptions"**********************************************WARN[0019] Action not allowed with tested creds action="ec2:AssociateRouteTable"**********************************************WARN[0019] Action not allowed with tested creds action="ec2:AttachInternetGateway"*********************************************WARN[0019] Action not allowed with tested creds action="ec2:AuthorizeSecurityGroupEgress"*******************************************WARN[0019] Action not allowed with tested creds action="ec2:AuthorizeSecurityGroupIngress"*******************************************WARN[0019] Action not allowed with tested creds action="ec2:CreateDhcpOptions"***********************************************WARN[0019] Action not allowed with tested creds action="ec2:CreateInternetGateway"**********************************************WARN[0019] Action not allowed with tested creds action="ec2:CreateNatGateway"***********************************************WARN[0019] Action not allowed with tested creds action="ec2:CreateRoute"*************************************************WARN[0019] Action not allowed with tested creds action="ec2:CreateRouteTable"
WARN[0019] Action not allowed with tested creds action="ec2:CreateSecurityGroup"
WARN[0019] Action not allowed with tested creds action="ec2:CreateSubnet"
WARN[0019] Action not allowed with tested creds action="ec2:CreateTags"
WARN[0019] Action not allowed with tested creds action="ec2:CreateVpc"
WARN[0019] Action not allowed with tested creds action="ec2:CreateVpcEndpoint"
WARN[0019] Action not allowed with tested creds action="ec2:CreateVolume"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeAccountAttributes"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeAddresses"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeAvailabilityZones"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeDhcpOptions"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeImages"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeInstanceAttribute"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeInstanceCreditSpecifications"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeInstances"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeInternetGateways"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeKeyPairs"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeNatGateways"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeNetworkAcls"
WARN[0019] Action not allowed with tested creds action="ec2:DescribePrefixLists"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeRegions"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeRouteTables"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeSecurityGroups"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeSubnets"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeTags"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeVpcEndpoints"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeVpcs"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeVpcAttribute"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeVolumes"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeVpcClassicLink"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeVpcClassicLinkDnsSupport"
WARN[0019] Action not allowed with tested creds action="ec2:ModifyInstanceAttribute"
WARN[0019] Action not allowed with tested creds action="ec2:ModifySubnetAttribute"
WARN[0019] Action not allowed with tested creds action="ec2:ModifyVpcAttribute"
WARN[0019] Action not allowed with tested creds action="ec2:RevokeSecurityGroupEgress"
WARN[0019] Action not allowed with tested creds action="ec2:RunInstances"
WARN[0019] Action not allowed with tested creds action="ec2:TerminateInstances"
WARN[0019] Action not allowed with tested creds action="ec2:DeleteDhcpOptions"
WARN[0019] Action not allowed with tested creds action="ec2:DeleteRoute"
WARN[0019] Action not allowed with tested creds action="ec2:RevokeSecurityGroupIngress"
WARN[0019] Action not allowed with tested creds action="ec2:DisassociateRouteTable"
WARN[0019] Action not allowed with tested creds action="ec2:ReplaceRouteTableAssociation"
WARN[0019] Action not allowed with tested creds action="ec2:DeleteRouteTable"
WARN[0019] Action not allowed with tested creds action="ec2:DeleteSubnet"
WARN[0019] Action not allowed with tested creds action="ec2:DescribeNetworkInterfaces"
WARN[0019] Action not allowed with tested creds action="ec2:ModifyNetworkInterfaceAttribute"
WARN[0019] Action not allowed with tested creds action="ec2:DeleteNatGateway"
WARN[0019] Action not allowed with tested creds action="ec2:DeleteSecurityGroup"
WARN[0019] Action not allowed with tested creds action="ec2:DetachInternetGateway"
WARN[0019] Action not allowed with tested creds action="ec2:DeleteInternetGateway"
WARN[0019] Action not allowed with tested creds action="ec2:ReleaseAddress"
WARN[0019] Action not allowed with tested creds action="ec2:DeleteVpc"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:AddTags"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:ApplySecurityGroupsToLoadBalancer"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:AttachLoadBalancerToSubnets"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:CreateListener"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:CreateLoadBalancer"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:CreateLoadBalancerListeners"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:CreateTargetGroup"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:ConfigureHealthCheck"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:DeleteLoadBalancer"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:DeregisterInstancesFromLoadBalancer"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:DeregisterTargets"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:DescribeInstanceHealth"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:DescribeListeners"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:DescribeLoadBalancers"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:DescribeLoadBalancerAttributes"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:DescribeTags"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:DescribeTargetGroupAttributes"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:DescribeTargetHealth"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:ModifyLoadBalancerAttributes"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:ModifyTargetGroup"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:ModifyTargetGroupAttributes"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:RegisterTargets"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:RegisterInstancesWithLoadBalancer"
WARN[0019] Action not allowed with tested creds action="elasticloadbalancing:SetLoadBalancerPoliciesOfListener"
WARN[0019] Tested creds not able to perform all requested actions
FATAL failed to fetch Terraform Variables: failed to fetch dependency of "Terraform Variables": failed to fetch dependency of "Bootstrap Ignition Config": failed to fetch dependency of "Master Machines": failed to generate asset "Platform Credentials Check": validate AWS credentials: current credentials insufficient for performing cluster installation

.openshift_install.log:
time="2019-02-27T14:53:58Z" level=debug msg="OpenShift Installer v0.13.0"
time="2019-02-27T14:53:58Z" level=debug msg="Fetching \"Terraform Variables\"..."
time="2019-02-27T14:53:58Z" level=debug msg="Loading \"Terraform Variables\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Cluster ID\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"SSH Key\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Base Domain\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Platform\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Cluster Name\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Base Domain\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Pull Secret\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Platform\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Image\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Bootstrap Ignition Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Kubeconfig Admin Client\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (admin-kubeconfig-client)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (admin-kubeconfig-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-complete-server-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-ca)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-localhost-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-localhost-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-service-network-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-service-network-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-lb-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-lb-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Kubeconfig Kubelet\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-ca)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (system:serviceaccount:openshift-machine-config-operator:node-bootstrapper)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-ca)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Kubeconfig Kubelet Client\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-complete-server-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kubelet-client)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kubelet-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-lb-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-localhost-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Master Machines\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Cluster ID\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Platform Credentials Check\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Image\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Master Ignition Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Root CA\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Common Manifests\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Cluster ID\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Ingress Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"DNS Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Cluster ID\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Platform Credentials Check\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Infrastructure Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Network Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Network CRDs\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Root CA\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (etcd)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Root CA\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (etcd)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (etcd)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (mcs)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Root CA\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"KubeCloudConfig\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"MachineConfigServerTLSSecret\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Pull\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"CVOOverrides\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"HostEtcdServiceEndpointsKubeSystem\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"KubeSystemConfigmapEtcdServingCA\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"KubeSystemConfigmapRootCA\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"KubeSystemSecretEtcdClient\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"OpenshiftMachineConfigOperator\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"EtcdServiceKubeSystem\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"HostEtcdServiceKubeSystem\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Openshift Manifests\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Cluster.machine.openshift.io/v1beta1\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Cluster ID\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Network Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Worker Machines\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Cluster ID\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Platform Credentials Check\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Image\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Worker Ignition Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Root CA\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Kubeadmin Password\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"BindingDiscovery\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"CloudCredsSecret\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"KubeadminPasswordSecret\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"RoleCloudCredsSecretReader\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (admin-kubeconfig-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (admin-kubeconfig-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (aggregator)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Root CA\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (aggregator-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (aggregator-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (system:kube-apiserver-proxy)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (aggregator-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (aggregator-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiaserver)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-ca)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (system:kube-apiserver-proxy)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (aggregator)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (etcd)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (etcd-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (etcd-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (etcd)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (etcd-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (etcd-signer-client)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (etcd-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (journal-gatewayd)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Root CA\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-lb-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-lb-server)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-lb-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-lb-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-localhost-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-localhost-server)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-localhost-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-localhost-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-service-network-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-service-network-server)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-service-network-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-service-network-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-complete-server-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-complete-client-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-ca)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (admin-kubeconfig-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kubelet-client-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kubelet-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-control-plane-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-control-plane-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-lb-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-localhost-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-service-network-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-to-kubelet-client)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-to-kubelet-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-to-kubelet-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-to-kubelet-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-to-kubelet-client)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-apiserver-to-kubelet-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-ca)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-control-plane-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-control-plane-kube-controller-manager-client)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-control-plane-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-control-plane-kube-scheduler-client)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-control-plane-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kube-control-plane-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (system:serviceaccount:openshift-machine-config-operator:node-bootstrapper)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kubelet-client-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kubelet-client)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kubelet-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kubelet-serving-ca-bundle)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (kubelet-signer)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Certificate (mcs)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Root CA\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Key Pair (service-account.pub)\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Master Ignition Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Loading \"Master Machines\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Fetching \"Cluster ID\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Fetching \"Install Config\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Fetching \"SSH Key\"..."
time="2019-02-27T14:53:58Z" level=debug msg=" Generating \"SSH Key\"..."
time="2019-02-27T14:53:59Z" level=debug msg=" Fetching \"Base Domain\"..."
time="2019-02-27T14:53:59Z" level=debug msg=" Fetching \"Platform\"..."
time="2019-02-27T14:53:59Z" level=debug msg=" Generating \"Platform\"..."
time="2019-02-27T14:54:01Z" level=debug msg=" Generating \"Base Domain\"..."
time="2019-02-27T14:54:01Z" level=debug msg="listing AWS hosted zones (page 0)"
time="2019-02-27T14:54:01Z" level=debug msg="listing AWS hosted zones (page 1)"
time="2019-02-27T14:54:05Z" level=debug msg=" Fetching \"Cluster Name\"..."
time="2019-02-27T14:54:05Z" level=debug msg=" Fetching \"Base Domain\"..."
time="2019-02-27T14:54:05Z" level=debug msg=" Reusing previously-fetched \"Base Domain\""
time="2019-02-27T14:54:05Z" level=debug msg=" Generating \"Cluster Name\"..."
time="2019-02-27T14:54:08Z" level=debug msg=" Fetching \"Pull Secret\"..."
time="2019-02-27T14:54:08Z" level=debug msg=" Generating \"Pull Secret\"..."
time="2019-02-27T14:54:13Z" level=debug msg=" Fetching \"Platform\"..."
time="2019-02-27T14:54:13Z" level=debug msg=" Reusing previously-fetched \"Platform\""
time="2019-02-27T14:54:13Z" level=debug msg=" Generating \"Install Config\"..."
time="2019-02-27T14:54:13Z" level=debug msg=" Generating \"Cluster ID\"..."
time="2019-02-27T14:54:13Z" level=debug msg=" Fetching \"Install Config\"..."
time="2019-02-27T14:54:13Z" level=debug msg=" Reusing previously-fetched \"Install Config\""
time="2019-02-27T14:54:13Z" level=debug msg=" Fetching \"Image\"..."
time="2019-02-27T14:54:13Z" level=debug msg=" Fetching \"Install Config\"..."
time="2019-02-27T14:54:13Z" level=debug msg=" Reusing previously-fetched \"Install Config\""
time="2019-02-27T14:54:13Z" level=debug msg=" Generating \"Image\"..."
time="2019-02-27T14:54:13Z" level=debug msg="Fetching RHCOS metadata from \"https://releases-rhcos.svc.ci.openshift.org/storage/releases/maipo/47.330/meta.json\""
time="2019-02-27T14:54:14Z" level=debug msg=" Fetching \"Bootstrap Ignition Config\"..."
time="2019-02-27T14:54:14Z" level=debug msg=" Fetching \"Install Config\"..."
time="2019-02-27T14:54:14Z" level=debug msg=" Reusing previously-fetched \"Install Config\""
time="2019-02-27T14:54:14Z" level=debug msg=" Fetching \"Kubeconfig Admin Client\"..."
time="2019-02-27T14:54:14Z" level=debug msg=" Fetching \"Certificate (admin-kubeconfig-client)\"..."
time="2019-02-27T14:54:14Z" level=debug msg=" Fetching \"Certificate (admin-kubeconfig-signer)\"..."
time="2019-02-27T14:54:14Z" level=debug msg=" Generating \"Certificate (admin-kubeconfig-signer)\"..."
time="2019-02-27T14:54:15Z" level=debug msg=" Generating \"Certificate (admin-kubeconfig-client)\"..."
time="2019-02-27T14:54:15Z" level=debug msg=" Fetching \"Certificate (kube-apiserver-complete-server-ca-bundle)\"..."
time="2019-02-27T14:54:15Z" level=debug msg=" Fetching \"Certificate (kube-ca)\"..."
time="2019-02-27T14:54:15Z" level=debug msg=" Generating \"Certificate (kube-ca)\"..."
time="2019-02-27T14:54:15Z" level=debug msg=" Fetching \"Certificate (kube-apiserver-localhost-ca-bundle)\"..."
time="2019-02-27T14:54:15Z" level=debug msg=" Fetching \"Certificate (kube-apiserver-localhost-signer)\"..."
time="2019-02-27T14:54:15Z" level=debug msg=" Generating \"Certificate (kube-apiserver-localhost-signer)\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Generating \"Certificate (kube-apiserver-localhost-ca-bundle)\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Fetching \"Certificate (kube-apiserver-service-network-ca-bundle)\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Fetching \"Certificate (kube-apiserver-service-network-signer)\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Generating \"Certificate (kube-apiserver-service-network-signer)\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Generating \"Certificate (kube-apiserver-service-network-ca-bundle)\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Fetching \"Certificate (kube-apiserver-lb-ca-bundle)\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Fetching \"Certificate (kube-apiserver-lb-signer)\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Generating \"Certificate (kube-apiserver-lb-signer)\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Generating \"Certificate (kube-apiserver-lb-ca-bundle)\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Generating \"Certificate (kube-apiserver-complete-server-ca-bundle)\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Fetching \"Install Config\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Reusing previously-fetched \"Install Config\""
time="2019-02-27T14:54:16Z" level=debug msg=" Generating \"Kubeconfig Admin Client\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Fetching \"Kubeconfig Kubelet\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Fetching \"Certificate (kube-ca)\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Reusing previously-fetched \"Certificate (kube-ca)\""
time="2019-02-27T14:54:16Z" level=debug msg=" Fetching \"Certificate (system:serviceaccount:openshift-machine-config-operator:node-bootstrapper)\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Fetching \"Certificate (kube-ca)\"..."
time="2019-02-27T14:54:16Z" level=debug msg=" Reusing previously-fetched \"Certificate (kube-ca)\""
time="2019-02-27T14:54:16Z" level=debug msg=" Generating \"Certificate (system:serviceaccount:openshift-machine-config-operator:node-bootstrapper)\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Fetching \"Install Config\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Reusing previously-fetched \"Install Config\""
time="2019-02-27T14:54:17Z" level=debug msg=" Generating \"Kubeconfig Kubelet\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Fetching \"Kubeconfig Kubelet Client\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Fetching \"Certificate (kube-apiserver-complete-server-ca-bundle)\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Reusing previously-fetched \"Certificate (kube-apiserver-complete-server-ca-bundle)\""
time="2019-02-27T14:54:17Z" level=debug msg=" Fetching \"Certificate (kubelet-client)\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Fetching \"Certificate (kubelet-signer)\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Generating \"Certificate (kubelet-signer)\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Fetching \"Certificate (kube-apiserver-lb-signer)\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Reusing previously-fetched \"Certificate (kube-apiserver-lb-signer)\""
time="2019-02-27T14:54:17Z" level=debug msg=" Fetching \"Certificate (kube-apiserver-localhost-signer)\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Reusing previously-fetched \"Certificate (kube-apiserver-localhost-signer)\""
time="2019-02-27T14:54:17Z" level=debug msg=" Generating \"Certificate (kubelet-client)\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Fetching \"Install Config\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Reusing previously-fetched \"Install Config\""
time="2019-02-27T14:54:17Z" level=debug msg=" Generating \"Kubeconfig Kubelet Client\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Fetching \"Master Machines\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Fetching \"Cluster ID\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Reusing previously-fetched \"Cluster ID\""
time="2019-02-27T14:54:17Z" level=debug msg=" Fetching \"Platform Credentials Check\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Fetching \"Install Config\"..."
time="2019-02-27T14:54:17Z" level=debug msg=" Reusing previously-fetched \"Install Config\""
time="2019-02-27T14:54:17Z" level=debug msg=" Generating \"Platform Credentials Check\"..."
time="2019-02-27T14:54:17Z" level=fatal msg="failed to fetch Terraform Variables: failed to fetch dependency of \"Terraform Variables\": failed to fetch dependency of \"Bootstrap Ignition Config\": failed to fetch dependency of \"Master Machines\": failed to generate asset \"Platform Credentials Check\": validate AWS credentials: current credentials insufficient for performing cluster installation"

What you expected to happen?

Installation to succeed. :-)

How to reproduce it (as minimally and precisely as possible)?

Get an environment from labs.opentlc.com (OpenTLC ID necessary, request OpenShift Labs/OCP 4 HA Lab). Use provided credentials to install cluster.

Ping me for help needed?

Anything else we need to know?

My own "admin" credentials work fine.

Here is the description of the AWS Permissions of the access key:

{
    "Statement": [
        {
            "Condition": {
                "StringEquals": {
                    "aws:RequestedRegion": "us-east-1"
                }
            },
            "Action": "*",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": "route53:*",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": "iam:*",
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

[davivcgarcia]

AFAIK, this is expected in the current version of the installer since it requires highly privileged credentials (admin level) on AWS. This is the official statement:

Today we require highly privileged credentials. We recommend using a dedicated account to try this out. We are working to reduce and improve how we use your credentials.

[wkulhanek]

@davivcgarcia Well yes.... but the installer changelog for 0.13 specifically mentions that it'll check the permissions required. I suspect it may not check in the region specified in the install-config.yaml. These credentials worked for 0.12. Point is it should check the region that the cluster is being set up in.

[davivcgarcia]

@wkulhanek hum... got it! I though the validation mentioned in the v0.13.0 changelog was related to the message seen in your log:

"Platform Credentials Check": validate AWS credentials: current credentials insufficient for performing cluster installation"

[wkulhanek]

@davivcgarcia Right. My point is that I believe that the credentials I am using DO have sufficient permissions. But the installer reports they don't.

[davivcgarcia]

@wkulhanek I'm not an AWS expert, so I'm not sure if your permission rule is covering all the access tested by the installer. Looks like the logic is implemented at asset/installconfig/aws/permissions.go.

Never mind, the very same information is displayed in the debug logs... :smile:

[maci0]

same error here.

Credentials work fine with v0.12.0, fail to validate with v0.13.0

[wkulhanek]

@maci0 Are you using an OpenTLC environment? If so I'm fixing this as we speak... until the installer does this correctly.

[maci0]

@wkulhanek yes, it's OpenTLC GUID is 0fb6

[wkulhanek]

@maci0 Fixed your access. Try again.

[maci0]

@wkulhanek thanks. will give it a shot once the destroy is complete

[maci0]

ok, the validation worked now

[wking]

Can you try to reproduce with the aws CLI? Something like:

USER_ARN="$(aws sts get-caller-identity --query Arn --output text)"
aws configure get region
aws iam simulate-principal-policy --action-names ec2:AllocateAddress --policy-source-arn "${USER_ARN}"
aws --region us-east-1 iam simulate-principal-policy --action-names ec2:AllocateAddress --policy-source-arn "${USER_ARN}"
aws --region us-west-1 iam simulate-principal-policy --action-names ec2:AllocateAddress --policy-source-arn "${USER_ARN}"

[wkulhanek]

@wking

Output here (policy as above: explicitly limiting to us-east-1):

[wkulhane-redhat.com@bastion ~]$ USER_ARN="$(aws sts get-caller-identity --query Arn --output text)"
[wkulhane-redhat.com@bastion ~]$ aws configure get region
[wkulhane-redhat.com@bastion ~]$ aws iam simulate-principal-policy --action-names ec2:AllocateAddress --policy-source-arn "${USER_ARN}"
{
    "EvaluationResults": [
        {
            "EvalDecision": "implicitDeny",
            "MissingContextValues": [
                "aws:RequestedRegion"
            ],
            "EvalActionName": "ec2:AllocateAddress",
            "MatchedStatements": [],
            "EvalResourceName": "*"
        }
    ]
}
[wkulhane-redhat.com@bastion ~]$ aws --region us-east-1 iam simulate-principal-policy --action-names ec2:AllocateAddress --policy-source-arn "${USER_ARN}"
{
    "EvaluationResults": [
        {
            "EvalDecision": "implicitDeny",
            "MissingContextValues": [
                "aws:RequestedRegion"
            ],
            "EvalActionName": "ec2:AllocateAddress",
            "MatchedStatements": [],
            "EvalResourceName": "*"
        }
    ]
}
[wkulhane-redhat.com@bastion ~]$ aws --region us-west-1 iam simulate-principal-policy --action-names ec2:AllocateAddress --policy-source-arn "${USER_ARN}"
{
    "EvaluationResults": [
        {
            "EvalDecision": "implicitDeny",
            "MissingContextValues": [
                "aws:RequestedRegion"
            ],
            "EvalActionName": "ec2:AllocateAddress",
            "MatchedStatements": [],
            "EvalResourceName": "*"
        }
    ]
}

[wking]

We've made a number of improvements since February, so I'm going to close this as obsolete. If you can still reproduce it, please open issues against the credential operator (whose code the installer is vendoring).

/close

[openshift-ci-robot]

@wking: Closing this issue.

In response to this:

We've made a number of improvements since February, so I'm going to close this as obsolete. If you can still reproduce it, please open issues against [the credential operator][1] (whose code the installer is vendoring).

/close

[1]: https://github.com/openshift/cloud-credential-operator/issues

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.