Masstransit: Access to resource denied after upgrading from 6.0.2 to 6.2.5

[gsaadeh]

After upgrading an application that uses MassTransit with SQS transport (and a hosted service) from 6.0.2 to 6.2.5, I am getting the following exception in AWS.

Application startup exception: System.AggregateException: One or more errors occurred. (Access to the resource https://sqs.eu-west-1.amazonaws.com/<account id>/dev-quartz-sqs is denied.)

Amazon.SQS.AmazonSQSException: Access to the resource https://sqs.eu-west-1.amazonaws.com/<account id>/dev-quartz-sqs is denied.

Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.

The error happens when the application starts.

I am unable to reproduce the error locally as it works fine with localstack.

Can you also reproduce the problem with the latest version?

Tested with both 6.2.4 and 6.2.5

Environment

1. Operating system: Linux
2. Dotnet version: 3.1.0

Steps to Reproduce

This is the startup code

        public async Task StartAsync(CancellationToken cancellationToken)
        {
            cancellationToken.ThrowIfCancellationRequested();

            _logger.LogInformation("Starting the bus...");

            _busControl.ConnectReceiveEndpointObserver(_receiveEndpointCheck);

            try
            {
                await _busControl.StartAsync(cancellationToken).ConfigureAwait(false);

                _scheduler.JobFactory = new MassTransitJobFactory(_busControl);

                _logger.LogInformation("Starting scheduler");

                await _scheduler.Start(cancellationToken);
            }
            catch (Exception)
            {
                await _scheduler.Shutdown();
                throw;
            }

            _simplifiedBusCheck.ReportBusStarted();

            _logger.LogInformation("Bus started");

        }

There is also one recurring job that we are setting up with on application start

Expected Behavior

Access to SQS should work as it's working normally with version 6.0.2

Actual Behavior

Getting an Amazon.SQS.AmazonSQSException exception

[phatboyg]

Hey @Crozin have you seen anything like this in SQS?

[Crozin]

I've some similar problems in the past. It was caused by invalid IAM permission setup. Maybe MT started using something extra around 6.2.x, hence the issue? You have to add some extra logs of AWS SDK - they contain to exact reason.

// assuming you're using NLog

AWSConfigs.LoggingConfig.LogResponses = ResponseLoggingOption.Always;
AWSConfigs.LoggingConfig.LogTo = LoggingOptions.SystemDiagnostics;
AWSConfigs.AddTraceListener("Amazon", new NLogTraceListener());

Once this setup is in place you should get tone of AWS logs including the response of failed request.

[gsaadeh]

Thanks @Crozin I am not sure what new permissions MT requires. At the moment I tried to give the ECS task a full permission on sqs (sqs.*) and it still fails. I will try the AWS logging to see what's the root cause.

[phatboyg]

Did you figure this one out @gsaadeh

[gsaadeh]

@phatboyg sorry for the late response. I think i will hold off on upgrading until the SQS changes (that are currently in develop) are released. I will open a new issue then if this persists. Thanks a lot for your help @phatboyg