Aws-sdk-ruby: Gem install aws-sdk-resources conflicts with aws-sdk-core

[marcoadkins]

gem install aws-sdk-resources has a conflict with one of its dependencies aws-adk-core

This causes a prompt to happen in the middle of gem install and is causing some issues for my automated build process. Seems like this should not be desired behavior.

gem install aws-sdk-resources

Fetching: aws-partitions-1.16.0.gem (100%)
Successfully installed aws-partitions-1.16.0
Fetching: aws-sdk-core-3.0.0.gem (100%)
Successfully installed aws-sdk-core-3.0.0
Fetching: aws-sdk-acm-1.0.0.gem (100%)
Successfully installed aws-sdk-acm-1.0.0
Fetching: aws-sdk-apigateway-1.0.0.gem (100%)
Successfully installed aws-sdk-apigateway-1.0.0
Fetching: aws-sdk-appstream-1.0.0.gem (100%)
Successfully installed aws-sdk-appstream-1.0.0
Fetching: aws-sdk-applicationautoscaling-1.0.0.gem (100%)
Successfully installed aws-sdk-applicationautoscaling-1.0.0
Fetching: aws-sdk-applicationdiscoveryservice-1.0.0.gem (100%)
Successfully installed aws-sdk-applicationdiscoveryservice-1.0.0
Fetching: aws-sdk-athena-1.0.0.gem (100%)
Successfully installed aws-sdk-athena-1.0.0
Fetching: aws-sdk-autoscaling-1.0.0.gem (100%)
Successfully installed aws-sdk-autoscaling-1.0.0
Fetching: aws-sdk-batch-1.0.0.gem (100%)
Successfully installed aws-sdk-batch-1.0.0
Fetching: aws-sdk-budgets-1.0.0.gem (100%)
Successfully installed aws-sdk-budgets-1.0.0
Fetching: aws-sdk-clouddirectory-1.0.0.gem (100%)
Successfully installed aws-sdk-clouddirectory-1.0.0
Fetching: aws-sdk-cloudformation-1.0.0.gem (100%)
Successfully installed aws-sdk-cloudformation-1.0.0
Fetching: aws-sdk-cloudfront-1.0.0.gem (100%)
Successfully installed aws-sdk-cloudfront-1.0.0
Fetching: aws-sdk-cloudhsm-1.0.0.gem (100%)
Successfully installed aws-sdk-cloudhsm-1.0.0
Fetching: aws-sdk-cloudhsmv2-1.0.0.gem (100%)
Successfully installed aws-sdk-cloudhsmv2-1.0.0
Fetching: aws-sdk-cloudsearch-1.0.0.gem (100%)
Successfully installed aws-sdk-cloudsearch-1.0.0
Fetching: aws-sdk-cloudsearchdomain-1.0.0.gem (100%)
Successfully installed aws-sdk-cloudsearchdomain-1.0.0
Fetching: aws-sdk-cloudtrail-1.0.0.gem (100%)
Successfully installed aws-sdk-cloudtrail-1.0.0
Fetching: aws-sdk-cloudwatch-1.0.0.gem (100%)
Successfully installed aws-sdk-cloudwatch-1.0.0
Fetching: aws-sdk-cloudwatchevents-1.0.0.gem (100%)
Successfully installed aws-sdk-cloudwatchevents-1.0.0
Fetching: aws-sdk-cloudwatchlogs-1.0.0.gem (100%)
Successfully installed aws-sdk-cloudwatchlogs-1.0.0
Fetching: aws-sdk-codebuild-1.0.0.gem (100%)
Successfully installed aws-sdk-codebuild-1.0.0
Fetching: aws-sdk-codecommit-1.0.0.gem (100%)
Successfully installed aws-sdk-codecommit-1.0.0
Fetching: aws-sdk-codedeploy-1.0.0.gem (100%)
Successfully installed aws-sdk-codedeploy-1.0.0
Fetching: aws-sdk-codepipeline-1.0.0.gem (100%)
Successfully installed aws-sdk-codepipeline-1.0.0
Fetching: aws-sdk-codestar-1.0.0.gem (100%)
Successfully installed aws-sdk-codestar-1.0.0
Fetching: aws-sdk-cognitoidentity-1.0.0.gem (100%)
Successfully installed aws-sdk-cognitoidentity-1.0.0
Fetching: aws-sdk-cognitoidentityprovider-1.0.0.gem (100%)
Successfully installed aws-sdk-cognitoidentityprovider-1.0.0
Fetching: aws-sdk-cognitosync-1.0.0.gem (100%)
Successfully installed aws-sdk-cognitosync-1.0.0
Fetching: aws-sdk-configservice-1.0.0.gem (100%)
Successfully installed aws-sdk-configservice-1.0.0
Fetching: aws-sdk-costandusagereportservice-1.0.0.gem (100%)
Successfully installed aws-sdk-costandusagereportservice-1.0.0
Fetching: aws-sdk-dax-1.0.0.gem (100%)
Successfully installed aws-sdk-dax-1.0.0
Fetching: aws-sdk-datapipeline-1.0.0.gem (100%)
Successfully installed aws-sdk-datapipeline-1.0.0
Fetching: aws-sdk-databasemigrationservice-1.0.0.gem (100%)
Successfully installed aws-sdk-databasemigrationservice-1.0.0
Fetching: aws-sdk-devicefarm-1.0.0.gem (100%)
Successfully installed aws-sdk-devicefarm-1.0.0
Fetching: aws-sdk-directconnect-1.0.0.gem (100%)
Successfully installed aws-sdk-directconnect-1.0.0
Fetching: aws-sdk-directoryservice-1.0.0.gem (100%)
Successfully installed aws-sdk-directoryservice-1.0.0
Fetching: aws-sdk-dynamodb-1.0.0.gem (100%)
Successfully installed aws-sdk-dynamodb-1.0.0
Fetching: aws-sdk-dynamodbstreams-1.0.0.gem (100%)
Successfully installed aws-sdk-dynamodbstreams-1.0.0
Fetching: aws-sdk-ec2-1.0.0.gem (100%)
Successfully installed aws-sdk-ec2-1.0.0
Fetching: aws-sdk-ecr-1.0.0.gem (100%)
Successfully installed aws-sdk-ecr-1.0.0
Fetching: aws-sdk-ecs-1.0.0.gem (100%)
Successfully installed aws-sdk-ecs-1.0.0
Fetching: aws-sdk-efs-1.0.0.gem (100%)
Successfully installed aws-sdk-efs-1.0.0
Fetching: aws-sdk-emr-1.0.0.gem (100%)
Successfully installed aws-sdk-emr-1.0.0
Fetching: aws-sdk-elasticache-1.0.0.gem (100%)
Successfully installed aws-sdk-elasticache-1.0.0
Fetching: aws-sdk-elasticbeanstalk-1.0.0.gem (100%)
Successfully installed aws-sdk-elasticbeanstalk-1.0.0
Fetching: aws-sdk-elasticloadbalancing-1.0.0.gem (100%)
Successfully installed aws-sdk-elasticloadbalancing-1.0.0
Fetching: aws-sdk-elasticloadbalancingv2-1.0.0.gem (100%)
Successfully installed aws-sdk-elasticloadbalancingv2-1.0.0
Fetching: aws-sdk-elastictranscoder-1.0.0.gem (100%)
Successfully installed aws-sdk-elastictranscoder-1.0.0
Fetching: aws-sdk-elasticsearchservice-1.0.0.gem (100%)
Successfully installed aws-sdk-elasticsearchservice-1.0.0
Fetching: aws-sdk-firehose-1.0.0.gem (100%)
Successfully installed aws-sdk-firehose-1.0.0
Fetching: aws-sdk-gamelift-1.0.0.gem (100%)
Successfully installed aws-sdk-gamelift-1.0.0
Fetching: aws-sdk-glacier-1.0.0.gem (100%)
Successfully installed aws-sdk-glacier-1.0.0
Fetching: aws-sdk-glue-1.0.0.gem (100%)
Successfully installed aws-sdk-glue-1.0.0
Fetching: aws-sdk-greengrass-1.0.0.gem (100%)
Successfully installed aws-sdk-greengrass-1.0.0
Fetching: aws-sdk-health-1.0.0.gem (100%)
Successfully installed aws-sdk-health-1.0.0
Fetching: aws-sdk-iam-1.0.0.gem (100%)
Successfully installed aws-sdk-iam-1.0.0
Fetching: aws-sigv2-1.0.0.gem (100%)
Successfully installed aws-sigv2-1.0.0
Fetching: aws-sdk-importexport-1.0.0.gem (100%)
Successfully installed aws-sdk-importexport-1.0.0
Fetching: aws-sdk-inspector-1.0.0.gem (100%)
Successfully installed aws-sdk-inspector-1.0.0
Fetching: aws-sdk-iot-1.0.0.gem (100%)
Successfully installed aws-sdk-iot-1.0.0
Fetching: aws-sdk-iotdataplane-1.0.0.gem (100%)
Successfully installed aws-sdk-iotdataplane-1.0.0
Fetching: aws-sdk-kms-1.0.0.gem (100%)
Successfully installed aws-sdk-kms-1.0.0
Fetching: aws-sdk-kinesis-1.0.0.gem (100%)
Successfully installed aws-sdk-kinesis-1.0.0
Fetching: aws-sdk-kinesisanalytics-1.0.0.gem (100%)
Successfully installed aws-sdk-kinesisanalytics-1.0.0
Fetching: aws-sdk-lambda-1.0.0.gem (100%)
Successfully installed aws-sdk-lambda-1.0.0
Fetching: aws-sdk-lambdapreview-1.0.0.gem (100%)
Successfully installed aws-sdk-lambdapreview-1.0.0
Fetching: aws-sdk-lex-1.0.0.gem (100%)
Successfully installed aws-sdk-lex-1.0.0
Fetching: aws-sdk-lexmodelbuildingservice-1.0.0.gem (100%)
Successfully installed aws-sdk-lexmodelbuildingservice-1.0.0
Fetching: aws-sdk-lightsail-1.0.0.gem (100%)
Successfully installed aws-sdk-lightsail-1.0.0
Fetching: aws-sdk-mturk-1.0.0.gem (100%)
Successfully installed aws-sdk-mturk-1.0.0
Fetching: aws-sdk-machinelearning-1.0.0.gem (100%)
Successfully installed aws-sdk-machinelearning-1.0.0
Fetching: aws-sdk-marketplacecommerceanalytics-1.0.0.gem (100%)
Successfully installed aws-sdk-marketplacecommerceanalytics-1.0.0
Fetching: aws-sdk-marketplaceentitlementservice-1.0.0.gem (100%)
Successfully installed aws-sdk-marketplaceentitlementservice-1.0.0
Fetching: aws-sdk-marketplacemetering-1.0.0.gem (100%)
Successfully installed aws-sdk-marketplacemetering-1.0.0
Fetching: aws-sdk-migrationhub-1.0.0.gem (100%)
Successfully installed aws-sdk-migrationhub-1.0.0
Fetching: aws-sdk-opsworks-1.0.0.gem (100%)
Successfully installed aws-sdk-opsworks-1.0.0
Fetching: aws-sdk-opsworkscm-1.0.0.gem (100%)
Successfully installed aws-sdk-opsworkscm-1.0.0
Fetching: aws-sdk-organizations-1.0.0.gem (100%)
Successfully installed aws-sdk-organizations-1.0.0
Fetching: aws-sdk-pinpoint-1.0.0.gem (100%)
Successfully installed aws-sdk-pinpoint-1.0.0
Fetching: aws-sdk-polly-1.0.0.gem (100%)
Successfully installed aws-sdk-polly-1.0.0
Fetching: aws-sdk-rds-1.0.0.gem (100%)
Successfully installed aws-sdk-rds-1.0.0
Fetching: aws-sdk-redshift-1.0.0.gem (100%)
Successfully installed aws-sdk-redshift-1.0.0
Fetching: aws-sdk-rekognition-1.0.0.gem (100%)
Successfully installed aws-sdk-rekognition-1.0.0
Fetching: aws-sdk-resourcegroupstaggingapi-1.0.0.gem (100%)
Successfully installed aws-sdk-resourcegroupstaggingapi-1.0.0
Fetching: aws-sdk-route53-1.0.0.gem (100%)
Successfully installed aws-sdk-route53-1.0.0
Fetching: aws-sdk-route53domains-1.0.0.gem (100%)
Successfully installed aws-sdk-route53domains-1.0.0
Fetching: aws-sdk-s3-1.0.0.gem (100%)
Successfully installed aws-sdk-s3-1.0.0
Fetching: aws-sdk-ses-1.0.0.gem (100%)
Successfully installed aws-sdk-ses-1.0.0
Fetching: aws-sdk-sms-1.0.0.gem (100%)
Successfully installed aws-sdk-sms-1.0.0
Fetching: aws-sdk-sns-1.0.0.gem (100%)
Successfully installed aws-sdk-sns-1.0.0
Fetching: aws-sdk-sqs-1.0.0.gem (100%)
Successfully installed aws-sdk-sqs-1.0.0
Fetching: aws-sdk-ssm-1.0.0.gem (100%)
Successfully installed aws-sdk-ssm-1.0.0
Fetching: aws-sdk-swf-1.0.0.gem (100%)
Successfully installed aws-sdk-swf-1.0.0
Fetching: aws-sdk-servicecatalog-1.0.0.gem (100%)
Successfully installed aws-sdk-servicecatalog-1.0.0
Fetching: aws-sdk-shield-1.0.0.gem (100%)
Successfully installed aws-sdk-shield-1.0.0
Fetching: aws-sdk-simpledb-1.0.0.gem (100%)
Successfully installed aws-sdk-simpledb-1.0.0
Fetching: aws-sdk-snowball-1.0.0.gem (100%)
Successfully installed aws-sdk-snowball-1.0.0
Fetching: aws-sdk-states-1.0.0.gem (100%)
Successfully installed aws-sdk-states-1.0.0
Fetching: aws-sdk-storagegateway-1.0.0.gem (100%)
Successfully installed aws-sdk-storagegateway-1.0.0
Fetching: aws-sdk-support-1.0.0.gem (100%)
Successfully installed aws-sdk-support-1.0.0
Fetching: aws-sdk-waf-1.0.0.gem (100%)
Successfully installed aws-sdk-waf-1.0.0
Fetching: aws-sdk-wafregional-1.0.0.gem (100%)
Successfully installed aws-sdk-wafregional-1.0.0
Fetching: aws-sdk-workdocs-1.0.0.gem (100%)
Successfully installed aws-sdk-workdocs-1.0.0
Fetching: aws-sdk-workspaces-1.0.0.gem (100%)
Successfully installed aws-sdk-workspaces-1.0.0
Fetching: aws-sdk-xray-1.0.0.gem (100%)
Successfully installed aws-sdk-xray-1.0.0
Fetching: aws-sdk-resources-3.0.0.gem (100%)
aws-sdk-resources's executable "aws.rb" conflicts with aws-sdk-core
Overwrite the executable? [yN]  

[cjyclaire]

@madkin10 Thanks for the info.! Quick question to help me reproduce this, what's your aws-sdk-core version? I just tried to run gem install aws-sdk-resources from scratch and there is no issue for me, the installed aws-sdk-core version is 3.0.0 as well

[cjyclaire]

This might likely has to do with your aws-sdk-core version is still 2.*, aws-sdk-core 3.*doesn't contain executables.

[marcoadkins]

@cjyclaire Your theory was correct. However, This is still undesired behavior that you cannot install two separate version of a gem without them conflicting. Why are they trying to write an executable to the same location?

[awood45]

In this case, it is intended behavior that V2 and V3 cannot coexist. I do see where build automation processes breaking due to the same executable is an issue, and we can take a look at ways to alleviate that.

[autarchprinceps]

This cannot possibly be the desired behaviour. Not only do certain rubygems depend on e.g. 'aws-sdk ~> 2.3' which will keep installing aws-sdk v2, while others may have a >= which would install v3, conflict and crash, it isn't even possible to do a gem update.
I don't know why aws-sdk v2 and v3 cannot coexist in your mind, but in rubygems in general you should and can install many different versions of the same gem. This is very much a desired behaviour. You cannot possibly expect all ruby gems to update in the exact same moment as you release a major update, not that even that would work, since as I mentioned it requires a clean install, a update is broken as well.

[awood45]

The executable conflict is not desired behavior, though also difficult to avoid.

However, major versions 2 and 3 not existing in the same project is a tradeoff we've made consciously. The upgrading guide walks through the use cases, but essentially the major versions are code compatible, so you're only changing packaging to upgrade. Unfortunately, with modularization, if we allowed you to have V2 and V3 in the same project, there was no way to ensure the underlying implementations didn't collide and cause unpredictable runtime behavior, while still providing code compatibility.

As for libraries which lock you to aws-sdk ~> 2.3, can you point me to these libraries? We'd love to help provide an upgrade path for them.

[awood45]

The case of a library which is hard locking on old aws-sdk minor versions would raise other questions too, since that minor version is, I think, over a year old at this point and hasn't been receiving updates for new services and features for some time.

[alvises]

I think that's could be related to this. I've started to have chef scripts failing. This scripts are used in aws opsworks.

Expected process to exit with [0], but received '1' ---- Begin output of /opt/chef/embedded/bin/gem install aws-sdk -q --no-rdoc --no-ri -v "~> 2.2" ---- STDOUT: STDERR: ERROR: Error installing aws-sdk: "aws.rb" from aws-sdk-core conflicts with installed executable from aws-sdk-resources ---- End output of /opt/chef/embedded/bin/gem install aws-sdk -q --no-rdoc --no-ri -v "~> 2.2" ---- Ran /opt/chef/embedded/bin/gem install aws-sdk -q --no-rdoc --no-ri -v "~> 2.2" returned 1

[awood45]

@alvises Can you tell me a bit more about that environment? I'm inclined at this point to rename the executable for V3 and end that issue.

[alvises]

@awood45 they are all ec2 machines with ubuntu 14.04 under aws opsworks. Using aws-sdk-ruby library with Chef 12.13.37

[awood45]

Thanks for the info. Working on a fix now.

[alvises]

@awood45 thanks!

[awood45]

I've pushed a fix to rename aws.rb to aws-v3.rb and we are working on getting it out this morning. This issue has discussed two different topics (the executable conflict and the V2/V3 packaging decisions) but I'm going to focus on the executable question here.

[awood45]

The executable issue should be fixed as of aws-sdk-resources version 3.0.1, which has the rename. Closing this, but feel free to reopen if the issue persists.

As for the discussion of why V2 and V3 can't coexist in general, if we want to discuss that we should do so in a different issue to avoid conflating this specific issue with that discussion.