Nextflow: Cloud Create Error: Invalid IAM Instance Profile name

[crashGoBoom]

Hi @pditommaso !
Just trying to get a cloud cluster up for testing purposes but run into the following error:

$ nextflow cloud create test-cluster -c 1
> cluster name: test-cluster
> instances count: 1
> Launch configuration:
 - driver: 'aws'
 - imageId: 'ami-SNIP'
 - instanceRole: 'my-IAMRole-SNIP'
 - instanceType: 'm4.xlarge'
 - keyFile:SNIP/.ssh/id_rsa.pub
 - sharedStorageId: 'fs-SNIP'
 - sharedStorageMount: '/mnt'
 - subnetId: 'subnet-SNIP'
 - userName: 'SNIP'

Please confirm you really want to launch the cluster with above configuration [y/n] y
Launching master node -- Waiting for `running` status.. ERROR ~ Value (my-IAMRole-SNIP) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: ####)

It seems to be using the same name as the IAM Instance Role for the Profile name as found here perhaps? It is saying that the parameter for the iamInstanceProfile name is invalid but the name is not using any invalid characters. Have you run into this before?

My nextflow.config:

cloud {
    imageId = 'ami-SNIP'
    instanceType = 'm4.xlarge'
    subnetId = 'subnet-SNIP'
    sharedStorageId = 'fs-SNIP'
    sharedStorageMount = '/mnt'
}

[pditommaso]

Does that IAM Instance Role name exists ?

[crashGoBoom]

@pditommaso Thanks for the quick response! Yes it does exist already.

[pditommaso]

IAM is quite tricky, there's should something wrong. Double check it verifying that you are able to launch an instance using the profile role name from the Ec2 dashboard.

[crashGoBoom]

@pditommaso haha yeah iam is silly sometimes...
So I just double checked and I was able to get an instance up just fine with that role and instance profile. I do notice that here it is providing the IAM Role Name whereas it might need to be providing the the IAM Instance Profile Name instead. Anything else I can check? Thanks for your help!

[pditommaso]

Amazon EC2 uses an instance profile as a container for an IAM role. When you create an IAM role using the IAM console, the console creates an instance profile automatically and gives it the same name as the role to which it corresponds. If you use the Amazon EC2 console to launch an instance with an IAM role or to attach an IAM role to an instance, you choose the instance based on a list of instance profile names.

If you use the AWS CLI, API, or an AWS SDK to create a role, you create the role and instance profile as separate actions, with potentially different names. If you then use the AWS CLI, API, or an AWS SDK to launch an instance with an IAM role or to attach an IAM role to an instance, specify the instance profile name.

Make sure to read this.

[crashGoBoom]

When you create an IAM role using the IAM console, the console creates an instance profile automatically and gives it the same name as the role to which it corresponds.

@pditommaso Ah ok this makes sense. We are using cloudformation and the IAM Role and IAM Instance Profile names that are generated are different in order to allow an update of the resource. Would you be interested in a pull request that allows the user to specify the IAM Instance Profile in the nextflow.config? Or is this diverging from the use case of this cloud mode?
Thanks!

[pditommaso]

How would that work in practice? I fear that it would make the role configuration even more confusing.

[pditommaso]

Is this issue fixed? You may want to share the solution for future reference.

[crashGoBoom]

Hi @pditommaso,

I believe the work around right now is to make sure the IAM Role and IAM Instance Profile share the same name. Either by creating the role using the cli or explicitly naming the role and profile in the cloudFormation template. This is acceptable for now since we are just using it for testing purposes but we may need NF to support pulling in both the IAM Role name and the IAM Instance Profile names separately at some point in the future.

Thanks!