Sops: Deprecate storing aws profile within the encrypted file / with the key

Created on 20 Feb 2020 · 13Comments · Source: mozilla/sops

There seems to have been a number of issues surrounding our use of AWS profiles:

Storing the AWS Profile in the encrypted file breaks expectations because it cannot be decrypted on systems where that profile doesn't exist. This is very different from how folks are used to interacting with AWS.

I'd propose that we deprecate storing the aws profile within the key and instead require users to pass --aws-profile every time they want to use an AWS profile.

AWS Priority - Medium bug enhancement

Source

ajvb

👍17

Most helpful comment

It's really useful to pass the profile using --aws-profile. I can understand reducing confusion but is it possible to just deprecate storing the aws profile in the encrypted file and keep --aws-profile?

thanks

pnguyen-okta on 20 Feb 2020

👍3

All 13 comments

@autrilla What do you think?

ajvb on 20 Feb 2020

Given all the confusion around it I think we should probably deprecate it, and also deprecate --aws-profile. The AWS lets you use environment variables to control what profile is used, users should just use that.

autrilla on 20 Feb 2020

👎3 👍2

thanks

pnguyen-okta on 20 Feb 2020

👍3

I really like how we implemented this in frost: https://github.com/mozilla/frost/blob/master/aws/client.py#L17-L32

You can pass in a profile using --aws-profiles (though for sops it would be --aws-profile) and it gets passed to the session creation, otherwise we just do the default AWS auth flow.

I'd also consider expanding this to removing support for setting AWS roles in the config. Most tools that interact with AWS via the command line require you to authenticate to AWS outside of the context of the tool and it makes sense to me that sops functions the same way. It seems like most of the issues are around people being confused it doesn't work that way (i.e. https://github.com/mozilla/sops/issues/555).

ajvb on 20 Feb 2020

Why do you prefer using a flag vs an environment variable? It’s kind of the
same thing with different syntax, no?

Using a flag would let us simplify the code a bit...

On Thu, 20 Feb 2020 at 23:21, Paul Nguyen notifications@github.com wrote:

It's really useful to pass the profile using --aws-profile. I can
understand reducing confusion but is it possible to just deprecate storing
the aws profile in the encrypted file and keep --aws-profile?

thanks

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/mozilla/sops/issues/634?email_source=notifications&email_token=AARH4V4XN2QM5AOAC5UTFVDRD366JA5CNFSM4KYXW3E2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMQT26Y#issuecomment-589380987,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AARH4V7JG5KURGVFRD6LHWDRD366JANCNFSM4KYXW3EQ
.

autrilla on 21 Feb 2020

I'd also consider expanding this to removing support for setting AWS roles in the config.

Original support for that came from a requirement to invoke KMS from various accounts transparently, by having Sops assume roles in those accounts. I believe this feature is still in use in our own teams.

jvehent on 21 Feb 2020

👍2

I agree with @jvehent, we use sops in the company I work for and being able to use multiple accounts transparently on the same shell is very important to us. Changing environment variables before running a sops command is pretty dirty, especially in automation scripts.

dsalaza4 on 21 Feb 2020

I agree with @jvehent, we use sops in the company I work for and being able to use multiple accounts transparently on the same shell is very important to us. Changing environment variables before running a sops command is pretty dirty, especially in automation scripts.

I assume you mean the removal of the feature from the config file.

I personally find it very hard to care about the difference in spelling between:

AWS_PROFILE=foo sops foo.yaml

and

sops --aws-profile=foo foo.yaml

But if there's some specific reason you care about this, I'd love to hear about it.

I'd also consider expanding this to removing support for setting AWS roles in the config.

Original support for that came from a requirement to invoke KMS from various accounts transparently, by having Sops assume roles in those accounts. I believe this feature is still in use in our own teams.

Hm, I wonder if it is? I don't think the .sops.yaml config file is used at all when decrypting or editing. My understanding is that it'll only be used when creating new files. IIRC, I had to change profiles manually back then, but maybe things have changed. I think if you use MFA, SOPS's profile switching is not going to work anyway, because we don't support token input.

autrilla on 21 Feb 2020

I agree with ajvb.
When I decrypt a file encrypted by other member of our team, I have to manually delete profile entry in the file every time.

nakatsuchi on 26 Feb 2020

👍2

One vote to deprecate - it's standard practice in our pipelines and locally to set up the AWS env for each CI/CD step. We've used SOPS for a while now and never touched (maybe never even knew about) embedding the aws_profile in the encrypted file.

casey-robertson on 28 Feb 2020

I'd add that, although the AWS CLI could just pick one of AWS_PROFILE or --profile, it supports both and I've used both. It also seems like SOPS follows this pattern for most other flags/env vars, so I'm not sure why this one should be different.

scjudd on 4 Apr 2020

Not having the --aws-profile makes things more difficult for beginners - how do you communicate that the AWS_PROFILE env and AWS_SDK_LOAD_CONFIG=1 are needed? Not having the profile in the encrypted file makes sense.

I initially had expected the aws profile to be settable in the .sops.yaml file -- since you specify the kms id there.

Not specifying the role makes in the .sops.yaml makes sense, I created a policy that allowed access to the sops key. I then assigned the policy to the user. This means, you don't directly have to specify permissions in the key policy.

If you keep the role - then the aws user specified by the AWS_PROFILE must have sts:AssumeRole to be able to use it -- I find this more complicated.

I found the docs to be difficult to use, and had to dig before I got things to work with AWS keys.

Prerequisite -- aws-cli installed, aws configure has run, aws s3 ls --profile myprofile works
You will want to create a new KMS key in the AWS console, assign a policy when creating it to allow admins to use manage and use it.

{
    "Version": "2012-10-17",
    "Id": "key-consolepolicy-3",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam:::root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow access for Key Administrators",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::[AWS_ACCOUNT]:user/joe",
                ]
            },
            "Action": [
                "kms:Create*",
                "kms:Describe*",
                "kms:Enable*",
                "kms:List*",
                "kms:Put*",
                "kms:Update*",
                "kms:Revoke*",
                "kms:Disable*",
                "kms:Get*",
                "kms:Delete*",
                "kms:TagResource",
                "kms:UntagResource",
                "kms:ScheduleKeyDeletion",
                "kms:CancelKeyDeletion"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::[AWS_ACCOUNT]:user/devops",

                ]
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow attachment of persistent resources",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::[AWS_ACCOUNT]:user/devops",
                ]
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "*",
            "Condition": {
                "Bool": {
                    "kms:GrantIsForAWSResource": "true"
                }
            }
        }
    ]
}

You need to create an IAM policy, and name it. (you may want to restrict the action a bit more)

{
    "Version": "2012-10-17",
    "Id": "kms-key-policy-sops",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "kms:*",
            "Resource": "arn:aws:kms:us-east-2:[AWS_ACCOUNT]:key/[ID]"
        }
    ]
}

Assign the policy to one or more users (not the ones you gave access to in the key policy -- they already have access)
Switch to a git repo that you want to store secrets in, go to the root folder of repo
Create a new file called .sops.yaml (root of repo) with content

creation_rules:
  - path-regex: '*/**/*'
    kms: 'arn:aws:kms:us-east-2:[AWS_ACCOUNT]:key/[ID]'

Install the sops tool with brew, apt, yum, apk etc.
Export env variables to determine the AWS_PROFILE used. If you are not sure what AWS account the profile (in ~/.aws/credentials the thing in square brackets) is, run aws sts get-caller-identity. This must match the user you assigned the policy to in step 3

export AWS_PROFILE=myprofile
export AWS_SDK_LOAD_CONFIG=1
# instead of doing this you can also invoke sops with
# AWS_PROFILE=myprofile sops ..., but that can be more typing.

You're done.

sops test.yaml # should open vim, change something and exit
cat test.yaml.  # should show encrypted
sops -d test.yaml # should show original unencrypted

It would also be good to have a terraform example to create the key / policy etc, like https://github.com/minamijoyo/terraform-kms-example, so that this can be done in code rather than with the AWS console.

brunzefb on 13 Apr 2020

This is also an issue with the role that is used to encrypt the secret is being persisted into the encrypted metadata and attempted to always be used in decryption.

SOPS config

creation_rules:
    - kms: 'arn:aws:kms:us-east-1:123456789:key/123456+arn:aws:iam::123456789:role/encrypt-role'

The encryption works fine, but it fails to decrypt because the role is persisted to the enc.yaml metadata file and in the where I am running the decrypt step does not have access to the original role used for encryption (it does have access to the KMS key using a separate role).

Example SOPS encrypted metadata with role hardcoded.

super_secret:
    token: ENC[AES256_GCM,data:asdasdasdasdasdas/wM=,iv:asdasdasdasdasdasd=,tag:asdasdasdasdasdasd==,type:str]
sops:
    kms:
    -   arn: arn:aws:kms:us-east-1:123456789:key/123456
        role: arn:aws:iam::123456789:role/encrypt-role
        created_at: '2020-07-14T18:28:18Z'
        enc: stuff
        aws_profile: ""
    gcp_kms: []
    azure_kv: []
    lastmodified: '2020-07-14T18:28:19Z'
    mac: mac-stuff
    pgp: []
    unencrypted_suffix: _unencrypted
    version: 3.5.0