Terraform-provider-aws: aws_codepipeline with Github OAuth still breaking auth

[ybron]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.12.28
+ provider.archive v1.3.0
+ provider.aws v3.0.0
+ provider.template v2.1.2

Affected Resource(s)

• aws_codepipeline

Terraform Configuration Files

resource "aws_codepipeline" "pipeline" {
    name      = "${var.ecs_service_name}-codepipeline"
    role_arn  = var.codepipeline_role

    artifact_store {
        location = var.artifact_store_bucket
        type     = "S3"
    }

    stage {
        name = "Source"

        action {
            name             = "SourceAction"
            category         = "Source"
            owner            = "ThirdParty"
            provider         = "GitHub"
            version          = "1"
            output_artifacts = ["SourceArtifact"]

            configuration = {
                Owner                  = "XXXXXX"
                Repo                   = var.repository_name
                Branch                 = "main"
                PollForSourceChanges   = "true"
                #OAuthToken             = "*"
            }
        }
     }

     stage {
         name = "Build"

         action {
             name             = "Build"
             category         = "Build"
             owner            = "AWS"
             provider         = "CodeBuild"
             version          = "1"
             input_artifacts  = ["SourceArtifact"]
             output_artifacts = ["BuildArtifact"]

             configuration = {
                ProjectName             = "${var.ecs_service_name}-codebuild"
             }
          }
      }
    stage {
        name = "Deploy"

        dynamic action {
        for_each = var.dedup_conf
            content {
                category         = "Deploy"
                configuration    = {
                    ClusterName  = var.ecs_cluster_name
                    ServiceName  = "${var.ecs_service_name}-${action.key}"
                    FileName     = "imagedefinitions.json"
                }
                input_artifacts  = [
                    "BuildArtifact",
                ]
                name             = "${var.ecs_service_name}-${action.key}"
                output_artifacts = []
                owner            = "AWS"
                provider         = "ECS"
                run_order        = 1
                version          = "1"
            }
        }
    }
}

Debug Output

N/A

Panic Output

N/A

Expected Behavior

The existing OAuthToken should have been left in-place when the aws_codepipeline resource was updated.

Actual Behavior

Same behaviour as before, when you were forced to push a broken placeholder OAuthToken resource in order to update it. Only now you don't have to comment-in the OAuthToken placeholder to update the resource.

Steps to Reproduce

1. terraform apply (after triggering authorizing to github, and forcing an update to the aws_codepipeline resource

Important Factoids

N/A

References

• #2854

[gdavison]

Hi @ybron. I'm not sure I fully understand the error you're encountering. I have a few questions that should help clarify this.

1. When you say you're updating the aws_codepipeline, is that updating from the AWS provider 2.x to 3.x, or is it modifying the resource using the same version of the provider?

2. I notice that you have the OAuthToken field commented out and with a dummy value. With the changes introduced in v3.0 of the AWS provider, the field needs to be kept in the Terraform configuration. If it is not kept in the configuration, the existing value will be removed. (I've just noticed that there is some missing text in the upgrade instructions. Before the line $ TF_VAR_github_token=<token> terraform apply, it should say that the next two blocks are the v3.0 approach.)

If it is neither of these cases, can you provide the before-and-after Terraform configurations as well as indicate how the Github token is being passed to Terraform in both cases?

[luhn]

I'm experiencing this too.

1. Updating the resource, to trigger a change.
2. Here's my configuration, field not commented:

data "aws_ssm_parameter" "token" {
  name = "/github/personal_access_token"
}

resource "aws_codepipeline" "main" {
  name     = local.name
  role_arn = aws_iam_role.codepipeline.arn

  artifact_store {
    location = aws_s3_bucket.artifacts.bucket
    type     = "S3"
  }

  stage {
    name = "Source"

    action {
      name             = "Source"
      category         = "Source"
      owner            = "ThirdParty"
      provider         = "GitHub"
      version          = "1"
      namespace        = "SourceVariables"
      output_artifacts = ["source_output"]

      configuration = {
        Owner      = "owner"
        Repo       = "repo"
        Branch     = "master"
        OAuthToken = data.aws_ssm_parameter.token.value
      }
    }
  }

    # ...
}

On initial creation, source works fine. Change to resource and apply the update, now source is broken:

If I taint and recreate the pipeline, the source works again.

[ybron]

Hi @gdavison, thanks for commenting.

Re: #1, This is after I have updated the provider to 3.0.0 and re-initialized. It is when I update the resource (by making changes to a step, adding actions, etc.) that the OAuthToken is removed.

Re: #2, that would explain why #1 happens... I had hoped that not providing it would leave it in-place as already established, or at least a notice that it was clearing it (maybe I just missed this in the output though?). Nobody wants to dig up the/an OAuthToken any/every time they make a change to the pipeline, nor check it into the code. From that perspective, it sounds like @luhn has a great solution, utilizing a data block. However, it seems like that may have a hitch or two remaining as well.

I probably misunderstood what was being addressed in https://github.com/terraform-providers/terraform-provider-aws/issues/2854 - I thought that the hashed value of the token (or some similar non-plaintext version) would be kept in the state, without needing to keep providing it on every single apply.

[mchoi-truework]

Hello folks. I wanted to report a bug regarding this issue.

Below is a TF_LOG=TRACE of me going through an exercise of creating a CodePipeline via Terraform and then making an update to it (changing a name of an action). In the process, I'm seeing that Terraform module is sending the literal "hash-<value>" as the OAuthToken request parameter. Don't worry about the GitHub token value in the logs that token was only used for this exercise and has since been deleted.

Step 1: I'm creating a CodePipeline

2020-09-24T19:37:03.635-0700 [DEBUG] plugin.terraform-provider-aws_v3.8.0_x5: 2020/09/24 19:37:03 [DEBUG] [aws-sdk-go] DEBUG: Request codepipeline/CreatePipeline Details:
2020-09-24T19:37:03.636-0700 [DEBUG] plugin.terraform-provider-aws_v3.8.0_x5: {"pipeline":{"artifactStore":{"location":"mchoi-test-bucket","type":"S3"},"name":"mchoi-test","roleArn":"arn:aws:iam::<REDACTED>:role/mchoi-test-role","stages":[{"actions":[{"actionTypeId":{"category":"Source","owner":"ThirdParty","provider":"GitHub","version":"1"},"configuration":{"Branch":"master","OAuthToken":"4ead838fabf7f7474617d54dbd4d66c27bd5e31c","Owner":"mchoi-truework","PollForSourceChanges":"true","Repo":"mchoi-test"},"name":"Source","outputArtifacts":[{"name":"MyApp"}],"runOrder":1}],"name":"Source"},{"actions":[{"actionTypeId":{"category":"Approval","owner":"AWS","provider":"Manual","version":"1"},"configuration":{"CustomData":"n/a","ExternalEntityLink":"http://example.com","NotificationArn":"arn:aws:sns:us-east-2:<REDACTED>:mchoi-test-topic"},"name":"Approve","runOrder":1}],"name":"Approve"}]},"tags":[]}

It seems to do the correct thing and my CodePipeline is working well.

Step 2: Update the name of an action from "Approve" to "Approve2"

2020-09-24T19:44:44.834-0700 [DEBUG] plugin.terraform-provider-aws_v3.8.0_x5: 2020/09/24 19:44:44 [DEBUG] [aws-sdk-go] DEBUG: Request codepipeline/UpdatePipeline Details:
2020-09-24T19:44:44.834-0700 [DEBUG] plugin.terraform-provider-aws_v3.8.0_x5: {"pipeline":{"artifactStore":{"location":"mchoi-test-bucket","type":"S3"},"name":"mchoi-test","roleArn":"arn:aws:iam::<REDACTED>:role/mchoi-test-role","stages":[{"actions":[{"actionTypeId":{"category":"Source","owner":"ThirdParty","provider":"GitHub","version":"1"},"configuration":{"Branch":"master","OAuthToken":"hash-ddc67fffbc06433cf61b8d87cfa1b6ad7f88cd00d2b60a305f7643736d9ecfcb","Owner":"mchoi-truework","PollForSourceChanges":"true","Repo":"mchoi-test"},"name":"Source","outputArtifacts":[{"name":"MyApp"}],"runOrder":1}],"name":"Source"},{"actions":[{"actionTypeId":{"category":"Approval","owner":"AWS","provider":"Manual","version":"1"},"configuration":{"CustomData":"n/a","ExternalEntityLink":"http://example.com","NotificationArn":"arn:aws:sns:us-east-2:<REDACTED>:mchoi-test-topic"},"name":"Approve2","runOrder":1}],"name":"Approve"}]}}

As you can see it's sending the "hash-<value>" literal as the OAuthToken parameter. Here's a check:

$ echo -n '4ead838fabf7f7474617d54dbd4d66c27bd5e31c' | openssl sha256
ddc67fffbc06433cf61b8d87cfa1b6ad7f88cd00d2b60a305f7643736d9ecfcb

Step 3: Run terraform plan

No changes. Infrastructure is up-to-date.

I'm not sure what the correct fix would be but I wanted demonstrate that this is a bug and consequently when you configure a bogus token in AWS CodePipeline, the GitHub Source stage/action will not work. I look forward to what TF folks think. Thank you!

[CarloSimacan]

I face similar problems as described above.

Also, I noticed that Terraform shows the OAuthToken in its CLI/log output, but I think it should be marked as a sensitive value. Is this a small enough issue that can be addressed at the same time or should I make a new issue for it?

[woodrow]

I've also been affected by the bug mentioned by @mchoi-truework and @luhn. Please let us know if we should create a separate issue for it.

This bug unfortunately seems difficult to work around based on the current design of the aws_codepipeline resource provider. The unhashed OAuthToken value is needed for the AWS UpdatePipeline API call, but by Terraform's design it's not possible to access both the config value (unhashed token) and state value (hashed token) via the resourceData struct. Thus as @mchoi-truework called out above in https://github.com/terraform-providers/terraform-provider-aws/issues/15200#issuecomment-698695759, once the aws_codepipeline resource is created and the hashed OAuthToken value is stored in the statefile, any further updates to the resource will pass the hashed token to UpdatePipeline and cause the CodePipeline's Github SourceAction to fail with the error shown in @luhn's comment https://github.com/terraform-providers/terraform-provider-aws/issues/15200#issuecomment-696914822.

It seems like a useful fix for this bug could be removing the hashing and instead creating provider-specific configuration blocks to represent configuration objects, as mentioned here https://github.com/terraform-providers/terraform-provider-aws/pull/14175#discussion_r460880340, rather than treating the config as a map. This would then allow the OAuthToken to be marked as sensitive and hidden from output, which seems like it was the intent of #14175 in the first place. The token would still be stored in cleartext in the statefile, but that seems unavoidable based on the current design of Terraform.

[mwarkentin]

AWS CodeStarConnections now support Github: https://github.com/terraform-providers/terraform-provider-aws/issues/15453

Once you have a connection set up there you can use that as your source connection to bypass the entire OAuth song and dance:

stage {
    name = "Source"

    action {
      name             = "Source"
      category         = "Source"
      owner            = "AWS"
      provider         = "CodeStarSourceConnection"
      version          = "1"
      output_artifacts = ["source_output"]

      configuration = {
        ConnectionArn    = var.codestar_connection_arn
        FullRepositoryId = "${var.github_organization}/${var.repo_name}"
        BranchName       = var.branch_name
      }
    }
}

Edit: According the the AWS Docs this old way of connecting to Github is deprecated and will be unsupported "soon" (not sure what that means).

[CarloSimacan]

AWS CodeStarConnections now support Github: #15453

Once you have a connection set up there you can use that as your source connection to bypass the entire OAuth song and dance:

stage {
    name = "Source"

    action {
      name             = "Source"
      category         = "Source"
      owner            = "AWS"
      provider         = "CodeStarSourceConnection"
      version          = "1"
      output_artifacts = ["source_output"]

      configuration = {
        ConnectionArn    = var.codestar_connection_arn
        FullRepositoryId = "${var.github_organization}/${var.repo_name}"
        BranchName       = var.branch_name
      }
    }
}

Edit: According the the AWS Docs this old way of connecting to Github is deprecated and will be unsupported "soon" (not sure what that means).

Well, can't argue with AWS here, but this solution requires some manual steps to set up the connection. This is might be a minor inconvenience for one AWS account + connection. It gets troublesome when dealing with automation across multiple AWS accounts (and connections), which is common in larger organizations/enterprises.

[mwarkentin]

You can reuse the one installed app on your GitHub org across all of your AWS accounts. Not sure if that's what you meant? If it's the manual set up process of the connection each one takes a minute or two, so even large-ish orgs should be doable.

Maybe Oauth Device Flow could simplify the initial set up if they supported that?

[CarloSimacan]

You can reuse the one installed app on your GitHub org across all of your AWS accounts. Not sure if that's what you meant? If it's the manual set up process of the connection each one takes a minute or two, so even large-ish orgs should be doable.

Maybe Oauth Device Flow could simplify the initial set up if they supported that?

Yes, it's the manual set-up process. Formerly, I worked at an enterprise where AWS accounts were created on-demand, in full automation and multiple times per day. In such an environment the central team managing AWS can quickly become a bottleneck if any manual steps get involved in the process. It's true that I describe a very specific situation here, but I think a central repository in GitHub with multiple CodePipeline 'listeners' would scale very well to do maintenance over many accounts in (such) an enterprise.

I had never heard of Oauth Device Flow tbh. I looked into it a bit and it seems quite some hassle to set-up. Especially considering that the same functionality could 'just work' out of the box by providing a token to CodePipeline.

[bhudgens]

Is there a mechanism to taint a record if TF detects a change or a lifecycle declaration we can set that forces a 'delete' on change to overcome this bug?

[gauravachatrath]

Hi Any update on fix of this bug ?

[bhudgens]

It looks like the Github V1 provider in codepipeline is being deprecated which will likely result in this bug going away:

https://docs.aws.amazon.com/codepipeline/latest/userguide/update-github-action-connections.html

Related Issue(s):

16042

15453

15960