Kops: Specify AWS tags on create cluster as a CLI argument

[AlexLast]

Currently tags can only be added by editing instance groups and adding cloudLabels.

For billing purposes we need to tag all resources, would be great if this could be done as a command line flag when creating the cluster. e.g --tags {"tag_key": "tag_value", "tag_key2": "tag_value2"}

[kris-nova]

So unfortunately one cannot tag "all" resources. The AWS API only supports tagging on some resources. Usually in kops we handle this issue by cross tagging resources on other resources, and looking them up that way.

I think we can use cloud labels to do this, and a command line flag implementation certainly could happen.

Would it be helpful if we provided a master list of resource IDs that a user could look up my cluster name? There are a few issues floating around that touch on that.

[AlexLast]

@kris-nova Sorry, when I said all resources I didn't mean all resources :)

Ideally we would just want to tag Master, node and bastion EC2 instances/EBS volumes (the things that are mainly costing money) if this flag could add our additional tags to those resources it would be ideal!

[kris-nova]

I'm not sure I'm sold on the JSON - but I think this is low hanging fruit.

We just need a design and we can start the PR.

Any volunteers for coding? I know @frodopwns has been working with the command logic recently

[robinpercy]

@kris-nova I'd like to take this one on. Should I create a new issue to kick off the use case/design discussion, or add it to this one? This also seems to duplicate most of https://github.com/kubernetes/kops/issues/729

[robinpercy]

Posting this here for now. Though this issue should probably be closed as a dup of #729.

__Use case__:
Users want to define custom tags for as many cloud resources as possible on the CLI. I propose to add three new options to 'kops create cluster:

• --master-cloud-labels
• --node-cloud-labels
• --global-cloud-labels

The first two just expose existing variables in InstanceGroupSpec. The global option would allow custom tags on all kops-created resources. Global labels would also apply to master and node groups, so shared tags don't need to be defined twice. In the event of collision, global labels would win, based on experiments with the design below.

The proposed format for all three options would be a CSV list of key=value pairs, eg --global-cloud-labels "Owner=Robin,Team=Dev". I'm not married to this, but it's consistent with how zones are passed in and edge cases can be easily supported with quoted fields in encoding/csv.

(Preliminary) __Design__

1. Add a GlobalClusterLabels property to ClusterSpec (all versions of it)
2. Add the three new options to CreateClusterOptions.
3. Set the labels on the ClusterSpec and respective InstanceGroupSpecs in RunCreateCluster
4. Copy the GlobalClusterLabels from ClusterSpec to cloudTags in BuildCloud

cc @kris-nova @justinsb

[AlexLast]

@robinpercy Would be good to apply --global-cloud-labels to bastions as well.

[robinpercy]

@kris-nova I've implemented the above design (if nothing else as a "getting to know kops" exercise): https://github.com/robinpercy/kops/commit/4887bf54bd5f005dda203cafa1a2b23522572eb9

I suspect everyone is slammed with 1.5 priorites, so whenever you get a chance. :)

[justinsb]

I am going to move this to 1.5.1, but mark it as a P1 (super important) there. @robinpercy if you PR-ed that I think it looks great. I had a few suggestions:

1) We can probably use StringSliceVar and avoid having to split values on commas. So in Options the type would be GlobalCloudLabels []string. Yes, I messed up here in the past e.g. zones
2) My guess would be --cloud-labels and then maybe --master-cloud-labels and --node-cloud-labels and --bastion-cloud-labels. I personally would start with only --cloud-labels first, and wait for the need for the per-role specifier, but if you or others have that need, go for it!
3) I'm not sure if we still need the CSV library for parsing.
4) (and this is kinda contrary to some of my other points)... apiserver has RuntimeConfig https://github.com/kubernetes/kubernetes/blob/master/pkg/genericapiserver/server/options/server_run_options.go#L56 I think if we used the type config.ConfigurationMap (we already vendor kubernetes) we would then have consistent parsing with other k8s tools.

[robinpercy]

Thanks @justinsb, I'll make those first two changes and create a WIP PR.

The only problem with using RuntimeConfig is that AWS tags can contain '=' but config.ConfigurationMap doesn't support escaping them (AFAICT). So I think the CSV library is still required for that bit.

[justinsb]

Ah OK @robinpercy - good choice on CSV then! Ignore me on RuntimeConfig :-)

[stp-ip]

Seems like this could be Closed. Labels/billing tags on EBS is also supported since 1.6.2.
A new issue was opened to track ELB tagging: https://github.com/kubernetes/kops/issues/2894

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close