Aws-parallelcluster: Encoded message getting truncated when passed back to cli

[tom-dyar]

Not sure how to reproduce, but this happens reliably for me when creating a cluster, due to permissions error on my account regarding EC2 AMI's. This is how the error and encoded message is reported back:

~ tdyar$ cfncluster create sge3
Beginning cluster creation for cluster: sge3
Creating stack named: cfncluster-sge3
Status: cfncluster-sge3 - ROLLBACK_IN_PROGRESS
Cluster creation failed.  Failed events:
  - AWS::IAM::Policy CfnClusterPolicies Resource creation cancelled
  - AWS::EC2::Instance MasterServer API: ec2:RunInstances You are not authorized to perform this operation. Encoded authorization failure message: rHU-pVHMSckD0OAc5l3r9-gKN1XZjZ2X9J3w0fZ31licaTpAVwlD9ix3fLSABy3XTEkRZRz0cNrXAz1L9EuYD42cSrVIrEBEd_PLpYkwcK8H4IpFKmbdAbSv8ckmleW3wXOez7Z1zQmB5RfzX8rGGme0aV8zpj7W_BEloqdHBTd0nmWAYDWuHF5d-a7BnDA5PDOB-tuDs3qEwRDB7tHgmAR3xGFrapNDFasNJm2ossb7xeOKQ2GhFREwmf823IC7PE_ASxCMKKvFdwf585E0zURw6yaEo991FkPYeoZDyRVvYGyoi4FyKH1hpaT0ss3FYfLcecRy_8--Vl8llOphwA0RfgmFofm7aybHecr9PZF7xpD9Cp9R6orpM71iE-ubnb8oDDz3Mca6sdSO7xTQLf2u_QWqY_ptEsGBWQPdytPCyIqlpge6yHDbj00giWESuyeZf5nV0Lv6byyv1LWxPK7FSNyYmJqn8qZ8pts7Hd_LhWd_dXBDRRI3yeMrt9W2tNLqY7VGqU_TwwHgnsiu05Ic5FjU0yhpX3LpQYS_8DnpvbN-4TO4weNesABoqZiDLSPWXpFlTKIIkFXFUxlZz9nI1lf71nMy9O5BbDlB4UK5NzXd7S_zKCZSw2nlMRAVMBWBwASLYvT8umnrlPBqXZdwEIgEQqiEvkKSFlofa3jJVqBn3HzaQCYQve7nWVoJKwcTt2gX8Lyd7X8AT5SZTRL0CKiah4nTPk8Bu1l-mSxGLkc67dNPdAS8fC7VK3BylrG1fUyTmk7K_Q5E8rjnaRrPhFfTI22b6MCGkXT3MJ8ymPXEGcABITqdTTYSmsDtN3dtlLfBPw7NQxlIK9B-W74VyTAcT8SCIp5-K_f1xqCY0bdnJ555gEjgp65BodpJ6bQWgeJ6P-cxc24yW

However, when trying to decode the message it was not valid:

~ tdyar$ aws sts decode-authorization-message --encoded-message rHU-pVHMSckD0OAc5l3r9-gKN1XZjZ2X9J3w0fZ31licaTpAVwlD9ix3fLSABy3XTEkRZRz0cNrXAz1L9EuYD42cSrVIrEBEd_PLpYkwcK8H4IpFKmbdAbSv8ckmleW3wXOez7Z1zQmB5RfzX8rGGme0aV8zpj7W_BEloqdHBTd0nmWAYDWuHF5d-a7BnDA5PDOB-tuDs3qEwRDB7tHgmAR3xGFrapNDFasNJm2ossb7xeOKQ2GhFREwmf823IC7PE_ASxCMKKvFdwf585E0zURw6yaEo991FkPYeoZDyRVvYGyoi4FyKH1hpaT0ss3FYfLcecRy_8--Vl8llOphwA0RfgmFofm7aybHecr9PZF7xpD9Cp9R6orpM71iE-ubnb8oDDz3Mca6sdSO7xTQLf2u_QWqY_ptEsGBWQPdytPCyIqlpge6yHDbj00giWESuyeZf5nV0Lv6byyv1LWxPK7FSNyYmJqn8qZ8pts7Hd_LhWd_dXBDRRI3yeMrt9W2tNLqY7VGqU_TwwHgnsiu05Ic5FjU0yhpX3LpQYS_8DnpvbN-4TO4weNesABoqZiDLSPWXpFlTKIIkFXFUxlZz9nI1lf71nMy9O5BbDlB4UK5NzXd7S_zKCZSw2nlMRAVMBWBwASLYvT8umnrlPBqXZdwEIgEQqiEvkKSFlofa3jJVqBn3HzaQCYQve7nWVoJKwcTt2gX8Lyd7X8AT5SZTRL0CKiah4nTPk8Bu1l-mSxGLkc67dNPdAS8fC7VK3BylrG1fUyTmk7K_Q5E8rjnaRrPhFfTI22b6MCGkXT3MJ8ymPXEGcABITqdTTYSmsDtN3dtlLfBPw7NQxlIK9B-W74VyTAcT8SCIp5-K_f1xqCY0bdnJ555gEjgp65BodpJ6bQWgeJ6P-cxc24yW

An error occurred (InvalidAuthorizationMessageException) when calling the DecodeAuthorizationMessage operation: Message is not valid: [-84, 117, 62, -91, 81, -52, 73, -55, 3, -48, -32, 28, -26, 93, -21, -9, -24, 10, 55, 85, -39, -115, -99, -105, -12, -99, -16, -47, -10, 119, -42, 88, -100, 105, 58, 64, 87, 9, 67, -10, 44, 119, 124, -76, -128, 7, 45, -41, 76, 73, 17, 101, 28, -12, 112, -38, -41, 3, 61, 75, -12, 75, -104, 15, -115, -100, 74, -75, 72, -84, 64, 68, 119, -13, -53, -91, -119, 48, 112, -81, 7, -32, -118, 69, 42, 102, -35, 1, -76, -81, -15, -55, 38, -107, -27, -73, -63, 115, -98, -49, -74, 117, -51, 9, -127, -27, 23, -13, 95, -54, -58, 26, 103, -76, 105, 95, 51, -90, 62, -42, -4, 17, 37, -94, -89, 71, 5, 55, 116, -98, 101, -128, 96, 53, -82, 28, 94, 93, -7, -82, -63, -100, 48, 57, 60, 51, -127, -6, -37, -125, -77, 122, -124, -63, 16, -63, -18, -47, -32, -104, 4, 119, -60, 97, 107, 106, -109, 67, 21, -85, 13, 38, 109, -88, -78, -58, -5, -59, -29, -118, 67, 97, -95, 21, 17, 48, -103, -1, 54, -36, -128, -69, 60, 79, -64, 75, 16, -116, 40, -85, -59, 119, 7, -7, -13, -111, 52, -51, 68, 112, -21, 38, -124, -93, -33, 117, 22, 67, -40, 122, -122, 67, -55, 21, 111, 96, 108, -88, -117, -127, 114, 40, 125, 97, -91, -92, -12, -78, -51, -59, 97, -14, -36, 121, -60, 114, -1, -49, -66, 86, 95, 37, -108, -22, 97, -64, 13, 17, 126, 9, -123, -95, -7, -69, 107, 38, -57, 121, -54, -3, 61, -111, 123, -58, -112, -3, 10, -97, 81, -22, -118, -23, 51, -67, 98, 19, -21, -101, -99, -65, 40, 12, 60, -9, 49, -58, -70, -79, -44, -114, -17, 20, -48, 45, -3, -82, -3, 5, -86, 99, -6, 109, 18, -63, -127, 89, 3, -35, -54, -45, -62, -56, -118, -91, -90, 7, -70, -56, 112, -37, -113, 77, 32, -119, 97, 18, -69, 39, -103, 127, -103, -43, -48, -69, -6, 111, 44, -81, -44, -75, -79, 60, -82, -59, 72, -36, -104, -104, -102, -89, -14, -90, 124, -90, -37, 59, 29, -33, -53, -123, 103, 127, 117, 112, 67, 69, 18, 55, -55, -29, 43, -73, -43, -74, -76, -46, -22, 99, -75, 70, -87, 79, -45, -61, 1, -32, -98, -56, -82, -45, -110, 28, -28, 88, -44, -45, 40, 105, 95, 114, -23, 65, -124, -65, -16, 57, -23, -67, -77, 126, -31, 51, -72, -63, -29, 94, -80, 0, 104, -87, -104, -125, 45, 35, -42, 94, -111, 101, 76, -94, 8, -112, 85, -59, 83, 25, 89, -49, -39, -56, -42, 87, -5, -42, 115, 50, -12, -18, 65, 108, 57, 65, -31, 66, -71, 55, 53, -35, -19, 47, -13, 40, 38, 82, -61, 105, -27, 49, 16, 21, 48, 21, -127, -64, 4, -117, 98, -12, -4, -70, 105, -21, -108, -16, 106, 93, -105, 112, 16, -120, 4, 66, -88, -124, -66, 66, -110, 22, 90, 31, 107, 120, -55, 86, -96, 103, -36, 124, -38, 64, 38, 16, -67, -18, -25, 89, 90, 9, 43, 7, 19, -73, 104, 23, -16, -68, -99, -19, 127, 0, 79, -108, -103, 77, 18, -12, 8, -88, -102, -121, -119, -45, 62, 79, 1, -69, 89, 126, -103, 44, 70, 46, 71, 58, -19, -45, 79, 116, 4, -68, 124, 46, -43, 43, 112, 114, -106, -79, -75, 125, 76, -109, -102, 78, -54, -3, 14, 68, -14, -72, -25, 105, 26, -49, -124, 87, -45, 35, 109, -101, -24, -64, -122, -111, 116, -9, 48, -97, 50, -104, -11, -60, 25, -64, 1, 33, 58, -99, 77, 54, 18, -102, -64, -19, 55, 119, 109, -108, -73, -63, 63, 14, -51, 67, 25, 72, 43, -48, 126, 91, -66, 21, -55, 48, 28, 79, -60, -126, 34, -98, 126, 43, -9, -11, -58, -96, -104, -47, -73, 103, 39, -98, 121, -128, 72, -32, -89, -82, 65, -95, -38, 73, -23, -76, 22, -127, -30, 122, 63, -25, 49, 115, 110, 50]

I checked CloudTrail, and the encoded message was actually longer, and decoding worked on this message:

[lukeseawalker]

Hi,
please follow the guide here to configure the proper permissions.
About the truncated error message, I wasn't able to reproduce that behaviour even setting the "ec2:RunInstances" to "Deny": the error message I got wasn't truncated. Are you using the latest cfncluster version?

[bronislav]

@tom-dyar This is known issue for CF. I just had a chat with AWS Support. They don't have ETA for now.

[DwivediShashikant]

@tom-dyar
I am trying to create an EMR stack using a cloudformation stack, in the stack events I got a similar encoded-message which is also truncated, and when I am trying to decode it using aws-cli its leads to message not valid error. however, I checked the cloudtrails and there also I do not see the full message in the JSON body :

"eventTime": "2020-10-21T04:10:24Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "CreateFleet",
    "awsRegion": "us-west-2",
    "sourceIPAddress": "cloudformation.amazonaws.com",
    "userAgent": "cloudformation.amazonaws.com",
    "errorCode": "Client.UnauthorizedOperation",
    "errorMessage": "You are not authorized to perform this operation. Encoded authorization failure message: 71yyWWrahXh_YKwpIBSCczkbuFqVpi40x5rWBVsW3NtgKcIx8ByqaNdHoTVFeEtilil-SqNq7eG2-SnuBvA5NI07AXO_TyKPweg2vmHyXTPDXldJutGxZ80_WRWnlJLf9v9Ftudn0ddJoXWEmCLNMmvHPvIouKxO3taaSaULhvdV-3VPHLwTBCWOi41hZeL_WaAImSpIy4hrgY5l0pU7fG-5-zlpImfFmm8t-4w2wQtzE_cDzKaDXd5OKlGSe_qUl8xCHuD66r2kRstX-17pQlfhLIhwNZVBGgmiSsnGa7lK_UoAJv2Mt1qznl-eFqUc1GIUrk3cxhFcvTUEPrmVnCbJ2qi4uyBSnJQhjJ5GUKLp62CLLneGMis7HXYVw8b3TJwvFWQ1ejGLBSIk4KwzP731CZ8XzttiuBhcOiiuUm3M5Q0aZ_EhNq10MYzZ7Qwgpm_0s_USqgBSlq9c_V2xO-qtN9mijguDCHRcaIxmPauREKRLIJnlyTDAyPugDG6Y98o9VlWLdo8qFVJYfKxf4DEUYBzbaAD3dckRMsd3Glx97Sc6gYlGU5zbpcgzBCFc1u46OdDmYwP29d7BZx_upRhzgFKVJfE-UKAIGihZoI5ugcI5oQDUaImPnWbYP0wBDhJiWdtmC-uKquQJs4-WA9o_cNOp9PhrlV0J79akT_xVG_kzdwc38eQUR5jXXlXRDzy8ZtrFPSCu9CeCcnJChrhm2uI4yGit4ns0O1BGp-VJ0oWr0CscKgyTfgWUDzeQLDKoNF3B7aVhz99RJSs6HO0aRMbJP_RSL8yNqzaxMg09wNW3GC7x_DzhbRVLqKA3qaHX6ozD9eMJUrwVvIvK_gC6AUE8pRsBRuG2V7aPmdCp_KjXjMw0I1_DVwi-blSVH6tKbpprvhH0JsawKYTUZ2WmlOOwn9r9PYxP...",
    "requestParameters": {
        "CreateFleetRequest": {
            "TargetCapacitySpecification": {
                "DefaultTargetCapacityType": "spot",
                "TotalTargetCapacity": 200
            },
            "SpotOptions": {
                "AllocationStrategy": "lowestPrice",
                "InstancePoolsToUseCount": 2
            }

Can you please tell me how did you see your full encoded message in cloudtrail.

[da-edra]

I'm also having this issue, I can't believe this hasn't been fixed after 2 years.

[lukeseawalker]

Hi @da-edra,
can you please open a new ticket so we can help you in solving the issue you are experiencing?

Thanks