Terraform-provider-aws: Feature Request: Managed Streaming for Kafka
Community Note
- Please vote on this issue by adding a 馃憤 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Description
AWS now offers managed Kafka clusters
New or Affected Resource(s)
- aws_msk_cluster
Potential Terraform Configuration
# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.
References
gazoakley
All 40 comments
I have a basic implementation of this feature working in PR #6809. This is my first contribution to this project, so I am not sure what the acceptance criteria for PRs is.
One issue with MSK right now is Update operations are not yet implemented in the AWS MSK api. I am unsure how to handle updating resources in this case? Seems dangerous to ForceNew for all changes to the resource.
jrefi
on 12 Dec 2018
@jrefi any updates? I'm looking forward for this function. Can I help you?
random1st
on 19 Feb 2019
@random1st Sorry, got caught up in my day to day work. Just fixed the unit tests for my MR. I am not quite sure what all remains, so if you don't mind glancing at it, that would be great.
jrefi
on 19 Feb 2019
@jrefi
- Ask to reformat code
make fmt - Wrong cluster name test_cluster - valid is like "test-cluster"
- "Broker node group information is invalid. Security groups list is empty. See https://docs.aws.amazon.com/en_us/msk/latest/developerguide/create-cluster.html - it is required param
random1st
on 20 Feb 2019
@random1st
- Ran that command resulted in no code changes.
- Why is
test_clusterinvalid? - Made security groups required.
jrefi
on 21 Feb 2019
@jrefi test_cluster is invalid because underscore is not supported
random1st
on 22 Feb 2019
@random1st Where is that specified? Seems odd, as I am seeing underscores used extensively for Terraform resource names.
jrefi
on 22 Feb 2019
resource "aws_msk_cluster" "test_cluster" {
name = "%s"
broker_count = 3
broker_instance_type = "kafka.m5.large"
broker_volume_size = 10
broker_security_groups =["${aws_security_group.test_sg_a.id}"]
kafka_version = "1.1.1"
client_subnets = ["${aws_subnet.test_subnet_a.id}", "${aws_subnet.test_subnet_b.id}", "${aws_subnet.test_subnet_c.id}"]
}
@jrefi
random1st
on 24 Feb 2019
@jrefi amazing work! I think maybe if you just hardcode "test-cluster" instead of using "%s", that's all that's needed? Would be so happy to be able to use this. If there's a chance I can help move this over the line, let me know!
dthtvwls
on 1 Mar 2019
@dthtvwls Thanks. Yeah Id be open to help with whatever is left.
I am still confused why I need to change that test. I am passing in a cluster name so Im not sure what hard-coding would accomplish.
jrefi
on 1 Mar 2019
You know, it seems I mistakenly inferred that you were passing in _test_cluster_ as a name, which MSK would not have accepted, but a name like _terraform-msk-test-00000_ should be fine.
@random1st would you mind verifying again? I'm very eager to get this merged if possible and it looks so painfully close.
dthtvwls
on 1 Mar 2019
@dthtvwls I've checked an example configuration. Everything is ok. Let's fix the markdown description of example usage and merge it. broker_security_groups is not optional param and test_cluster is invalid name.
random1st
on 14 Mar 2019
resource "aws_msk_cluster" "test_cluster" {
name = "test_cluster"
broker_count = 3
broker_instance_type = "kafka.m5.large"
broker_volume_size = 10
broker_security_groups =["${aws_security_group.test_sg_a.id}"]
kafka_version = "1.1.1"
client_subnets = ["${aws_subnet.test_subnet_a.id}", "${aws_subnet.test_subnet_b.id}", "${aws_subnet.test_subnet_c.id}"]
}
Trace:
2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: GET /v1/clusters?clusterNameFilter=test_cluster HTTP/1.1
2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: Host: kafka.us-east-1.amazonaws.com
2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: User-Agent: aws-sdk-go/1.16.19 (go1.10.4; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.9-beta1
2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: Authorization: AWS4-HMAC-SHA256 Credential=AKIAIT6SVQ2D6HAOBIFQ/20190314/us-east-1/kafka/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=31473150a38a3cd4b8d4f5dbdc0e4fb3a27ee0cf35f36588170a51f6e4ecd6e7
2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: Content-Type: application/x-amz-json-1.1
2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: X-Amz-Date: 20190314T072016Z
2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: Accept-Encoding: gzip
2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws:
2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws:
2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: -----------------------------------------------------
2019-03-14T10:20:16.675+0300 [DEBUG] plugin.terraform-provider-aws: 2019/03/14 10:20:16 [DEBUG] [aws-sdk-go] DEBUG: Response Kafka/ListClusters Details:
2019-03-14T10:20:16.675+0300 [DEBUG] plugin.terraform-provider-aws: ---[ RESPONSE ]--------------------------------------
2019-03-14T10:20:16.675+0300 [DEBUG] plugin.terraform-provider-aws: HTTP/2.0 400 Bad Request
2019-03-14T10:20:16.675+0300 [DEBUG] plugin.terraform-provider-aws: Content-Length: 124
2019-03-14T10:20:16.675+0300 [DEBUG] plugin.terraform-provider-aws: Content-Type: application/json
2019-03-14T10:20:16.675+0300 [DEBUG] plugin.terraform-provider-aws: Date: Thu, 14 Mar 2019 07:20:16 GMT
2019-03-14T10:20:16.675+0300 [DEBUG] plugin.terraform-provider-aws: Via: 1.1 029f15a661be82d29f31e88713b71d65.cloudfront.net (CloudFront)
random1st
on 14 Mar 2019
@jrefi Please, let's fix it asap.
https://github.com/jrefi/terraform-provider-aws/pull/2
random1st
on 14 Mar 2019
@random1st Merged
jrefi
on 14 Mar 2019
There is a conflict with root repository
random1st
on 14 Mar 2019
hello, its great to see that you have already developing support for MSK. Can I get some estimation when would it be released?
rohitshrivastava04
on 14 Mar 2019
@random1st Resolved
jrefi
on 14 Mar 2019
Guys, when can we merge it to master?
random1st
on 15 Mar 2019
Guys, any plans on delivering MSK?
stsukrov
on 18 Mar 2019
I believe my MR is about ready to go. If there is anything else I need to add, someone please let me know. (https://github.com/terraform-providers/terraform-provider-aws/pull/6809)
My organization has actually decided to transition away from MSK to self-hosted Kafka, so this work is no longer a priority for us. I am happy to get it across the finish line, but I don't have the bandwidth to hunt down approvers and get them to merge this in.
jrefi
on 18 Mar 2019
@mrf @tomelliff Can we get any guidance on what's needed for final reviews of this resource? #6809
Additionally, thank you for all your progress @jrefi. I and I'm sure many others are eagerly awaiting this functionality
kesensoy
on 20 Mar 2019
+1 for #6809 it would be nice to have the implementation soon.
hExPY
on 26 Mar 2019
@hExPY Unfortunately I'm having little success in encouraging reviewers to check this code, even though it passes the tests and seems to be ready for a merge. The more people that show interest, the better chance we have!
kesensoy
on 26 Mar 2019
@kesensoy in that case, allow me to pile on and +1 this as well!
bploetz
on 26 Mar 2019
@bploetz me too! +1
landondao1
on 26 Mar 2019
If @bploetz is in, i'm in. +1
billcrook
on 26 Mar 2019
Can鈥檛 wait 馃憤
nullobject
on 26 Mar 2019
Looking forward to this!!!! What does it take to get this merged already!?
gavinmbell
on 28 Mar 2019
+1
AnishJS
on 28 Mar 2019
+1
alessio-torelli
on 8 Apr 2019
@jrefi Great work! Can't wait to see it merged in master.
Posted a tweet with the hope more people will get involved and the PR will get the needed attention.
/offtopic
@jrefi Mind if I ask why your organization moved away from MSK in respect to self-hosted Kafka? In case you are allowed disclose such information.
hristoyankov
on 11 Apr 2019
Waiting for merge
serelk
on 14 Apr 2019
Thanks so much for the great work! I really look forward to seeing this merged 馃殌
renatoargh
on 16 Apr 2019
Please see #8357 which supersedes the previous PR #6809
dthtvwls
on 17 Apr 2019
A new aws_msk_cluster resource has been merged and will release with version 2.12.0 of the Terraform AWS Provider, likely tomorrow. Special thanks to @jrefi, @dthtvwls, and @jesseaukeman for making this possible! 馃殌
For future bug reports and feature requests with MSK, please submit new GitHub issues. Thanks!
bflad
on 22 May 2019
This has been released in version 2.12.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.
bflad
on 24 May 2019
This has been released in version 2.12.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.
Hi , I have tried creating AWS MSK cluster with reference from the document shared. But I am getting the error "Error: error creating MSK cluster: BadRequestException: The parameter value contains one or more characters that are not valid.
status code: 400, request id: e2589e6a-8161-11e9-8f31-6f8877605e30". Below is the terraform code used. Please help me to fix the error.
resource "aws_msk_cluster" "msk_cluster" {
cluster_name = "Testing_Cluster"
kafka_version = "2.1.0"
number_of_broker_nodes = 3
broker_node_group_info {
instance_type = "kafka.m5.large"
client_subnets = [
"${aws_subnet.subnet_a.id}",
"${aws_subnet.subnet_b.id}",
"${aws_subnet.subnet_c.id}",
]
ebs_volume_size = 5
security_groups = [ "${aws_security_group.MSK_Sg.id}" ]
}
tags = {
Name = "Cluster_MSK"
}
}
navinsnn53
on 28 May 2019
Hi @navinsnn53 :wave: in general please submit a new GitHub issue so the maintainers and community can appropriately triage your issue instead of commenting on a closed issue, which may get lost/locked over time.
That said, my hunch would be that the MSK service does not allow the usage of underscores (_) in cluster names. The MSK API Reference, does not provide too much guidance for string validation on fields such as the cluster name. We currently only implement the known string length check within the Terraform resource for the cluster_name argument: https://github.com/terraform-providers/terraform-provider-aws/blob/5619c0e0ac94f4dd645dd2f388c19795b5800341/aws/resource_aws_msk_cluster.go#L83
bflad
on 28 May 2019
Locking to encourage new GitHub issues.
bflad
on 28 May 2019
Related issues
carmas
路
3Comments
andywirv
路
3Comments
modax
路
3Comments
hashibot
路
3Comments
hazmeister
路
3Comments
Most helpful comment
@hExPY Unfortunately I'm having little success in encouraging reviewers to check this code, even though it passes the tests and seems to be ready for a merge. The more people that show interest, the better chance we have!