Ckan: [Bug] netkan.exe no longer able to successfully use IAM

[techman83]

Background

Noted nothing had been checked in 15 hours when I was curious to see if the GitHub outage was impacting indexing.

Problem

I am trying to get some more info, this applies specifically to IAM via EC2, testing locally the latest image is able to use token auth just fine.

754 [1] INFO CKAN.KSPManager (null) - Loading KSP instances from registry
790 [1] INFO CKAN.NetKAN.Processors.Inflator (null) - Using user-supplied cache at ckan_cache
32747 [1] FATAL CKAN.NetKAN.Program (null) - Unable to get IAM security credentials from EC2 Instance Metadata Service.

It does line up with a merge that update the AWSSDK. One thing that is making it awkward is we have to use 'Shell Form` for our entrypoint, making it quite difficult to pass in a debug flag via the task definition.

Debug output doesn't tell us a whole lot more

650 [1] DEBUG CKAN.NetKAN.Processors.QueueHandler (null) - Initializing SQS queue handler
681 [1] DEBUG CKAN.NetKAN.Processors.Inflator (null) - Initializing inflator
861 [1] INFO CKAN.KSPManager (null) - Loading KSP instances from registry
881 [1] INFO CKAN.NetKAN.Processors.Inflator (null) - Using main CKAN meta-cache at /home/netkan/.local/share/CKAN/downloads
1302 [1] DEBUG CKAN.NetKAN.Processors.QueueHandler (null) - Looking up URL for queue Inbound.fifo
32828 [1] FATAL CKAN.NetKAN.Program (null) - Unable to get IAM security credentials from EC2 Instance Metadata Service.
32830 [1] FATAL CKAN.NetKAN.Program (null) -   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.FetchCredentials () [0x00013] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentials () [0x0001b] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.CredentialsRetriever.PreInvoke (Amazon.Runtime.IExecutionContext executionContext) [0x00030] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.CredentialsRetriever.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.RetryHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00050] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.CallbackHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00007] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.EndpointResolver.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00007] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.SQS.Internal.ProcessRequestHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00007] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.Marshaller.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00006] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.CallbackHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00007] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00012] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.MetricsHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0002b] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.Internal.RuntimePipeline.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00006] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.Runtime.AmazonServiceClient.Invoke[TResponse] (Amazon.Runtime.AmazonWebServiceRequest request, Amazon.Runtime.Internal.InvokeOptionsBase options) [0x0007d] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at Amazon.SQS.AmazonSQSClient.GetQueueUrl (Amazon.SQS.Model.GetQueueUrlRequest request) [0x0001c] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at CKAN.NetKAN.Processors.QueueHandler.getQueueUrl (System.String name) [0x00022] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at CKAN.NetKAN.Processors.QueueHandler..ctor (System.String inputQueueName, System.String outputQueueName, System.String cacheDir, System.Boolean overwriteCache, System.String githubToken, System.Boolean prerelease) [0x0005e] in <7bf78a1a90b146f1b4841c36916192cf>:0 
  at CKAN.NetKAN.Program.Run (CKAN.NetKAN.CmdLineOptions options) [0x0014b] in <7bf78a1a90b146f1b4841c36916192cf>:0 

[HebaruSan]

Maybe they changed the $AWS_REGION env var to $AWS_DEFAULT_REGION for consistency with everything else. Can you try that?

EDIT: My testing script sets both vars, and it doesn't help. :-1:

EDIT II: Hmm, even a pre-#3108 netkan.exe doesn't work with my test script anymore, so I may not be very much help in this...

[HebaruSan]

BTW WTH is IAM?

[7ranceaddic7]

Identity Access Management
https://aws.amazon.com/iam/?nc=bc&pg=f-mc

[7ranceaddic7]

Also, this may have caused a bit of problem with mods that had updated during this outage.

I know of two that are not correctly reporting in CKAN
Persistent Thrust 1.6.2 => 1.6.4
TweakScale 2.4.3.17 => 2.4.3.18

Last known good updates were to:
Waypoint Manager 2.8.2.5 (updated 2020-07-12 23:20:09 PT)
EarlyBird 0.2.1.0 (updated 2020-07-12 23:20:09 PT

[HebaruSan]

Yeah, the bot's broken. Please bear with us.

[HebaruSan]

Is it proxy related?
https://github.com/aws/aws-sdk-net/issues/1609#issuecomment-628140989