Terraform: vSphere Provider: Mapping out the Next Steps

[phinze]

Wanted to kick off a higher level discussion of what needs to be done on the vSphere provider and in what order.

• What are the important missing resources?
• Are there any enhancements that need to be made to the existing functionality?
• What do we need to do to ensure the provider works with all common versions of vSphere in the wild?

Pinging @tkak and @mkuzmin to chime in as well as anybody else with interest/knowledge in the community.

[v1gnesh]

• To handle the NIC problem, the code could be updated to drop the NIC(s) from the template (if any) and do its usual business of creating NIC(s). This can be introduced as an optional parameter so that those with no NIC(s) in their template won't be affected
• Ability to build using a CustSpec that's available in vCenter (with the option of overriding specific parameters if needed)
• Steps to format the vmdk and make it accessible within the guest OS (guest scripts). I _think_ that the only returnable value when initiating a program within the guest is the process ID, so success/failure must be dealth with within the initiated script itself and shouldn't be tracked by the vSphere provider

[mkuzmin]

@v1gnesh NIC issue is discussed in https://github.com/hashicorp/terraform/issues/3559.
For other two items - create separate issues please.

[mkuzmin]

Indeed, we need to discuss many things, but at the moment the plugin is broken, and I wasn't able to launch any VMs yet. https://github.com/hashicorp/terraform/issues/3537 is a show-stopper issue: until it's fixed I cannot finish my testing, so cannot talk about future tasks.
@tkak, can we expect some support from you?

[tkak]

• Add flexible parameters support in acceptance test with environment value like openstack provider
• Windows template support
• Memory reservation parameter support
• Test VMware vSphere 6
• Add update feature for changing a number of CPU, memory size, network interfaces and disks

[chrislovecnm]

@tkak once @phinze gives me more info I will be running vSphere 6 testing. Also you noticed that govmomi is supporting 6.0 now ... https://github.com/vmware/govmomi/issues/356

@phinze what is the minimal viable components that a provider is to support? Nics, Disk, CPU, ram, etc...

[phinze]

Okay, got some basic acceptance test docs over in #3683. We got @tkak's crasher fix in #3636 merged. There's a candidate fix for #3559 in review over at #3652.

what is the minimal viable components that a provider is to support?

Because each provider is so different, it's difficult to define much in the way of standards across providers. Generally it's up to the community using the provider to help guide the priorities, as they are the ones who understand best which features are most crucial to have.

[sbarie]

Support for Linked Clones would be a great addition!

[chrislovecnm]

So here is the tally so far:

• Add flexible parameters support in acceptance test with environment value like openstack provider
• Windows template support
• Memory reservation parameter support
• Test VMware vSphere 6
• Add update feature for changing a number of CPU, memory size, network interfaces and disks
• Support for different vmdk disk types: thin, thick, eager zeroed thick, etc
• better test support
• linked clones
• Ability to build using a CustSpec that's available in vCenter (with the option of overriding specific parameters if needed)
• Steps to format the vmdk and make it accessible within the guest OS (guest scripts). I think that the only returnable value when initiating a program within the guest is the process ID, so success/failure must be dealth with within the initiated script itself and shouldn't be tracked by the vSphere provider
• add config value to handle 'insecure' SSL connections. For example when vSphere url is using a self-signed certificate.

What about networking? For instance: vlans, ports groups, ip pools, virtual switches?

@tkak / @phinze do you want to add priorities? I am thinking we can divide and conquer a bit. Also in your opinion do we have a minimal viable product, or do we have some more work to do? I think we are really close ...

Please folks we need to vote on the P1's in this group!!

Also @tkak we appreciate you getting this code in and doing the work that you have done!! どうもありがとうございます

[chrislovecnm]

Question for the group. Who a google hangout next week be good to review this stuff? We have a bunch of different TZ's here, but I would probably recommend working it around @tkak's schedule

[phinze]

@chrislovecnm Thanks for the summary!

As I am not a vSphere user, I can't really provide any input as to which features are most important to do first, nor can I be of much help in determining what defines a "minimum viable product" here.

I _can_ say that the SSL connection issue in #3521 is important for security and an easy win for whoever wants to pick it up.

Re: Hangout - I'm not sure a synchronous meeting is really necessary at this point. It seems to me that anything we'd video chat about could be easily handled using GitHub as we've been doing.

Rather than individual feature requests - what we could really use from a high level is for the folks with an understanding of the vSphere domain model to propose potential Terraform resources and the backing APIs that would support those resources.

I can provide an equivalent example from AWS, and perhaps a vSphere expert can chime in using this pattern.

In AWS we have:

• aws_instance - a resource representing an instance in EC2
  
  
  
  • Created with an EC2 RunInstances API call
  
  
  • Read with EC2 DescribeInstances
  
  
  • Updated via calls to EC2 ModifyInstanceAttribute as well as a few other specific calls.
  
  
  • Deleted by calling EC2 TerminateInstances
  
  
• aws_ebs_volume - represents a block storage volume
  
  
  
  • Created with a EC2 CreateVolume API call
  
  
  • Read with EC2 DescribeVolumes
  
  
  • Currently can only update its tags via {Create,Update,Delete}Tags API calls, other modifications require replacement
  
  
  • Deleted by calling EC2 DeleteVolume
  
  
• aws_ebs_volume_attachment - associates a volume with an instance
  
  
  
  • Created with AttachVolume
  
  
  • Read from the details of DescribeVolumes
  
  
  • Removed with DetachVolume
  
  

Currently in vSphere we have:

• vsphere_virtual_machine - represents a running VM
  
  
  
  • Created via DeployVM if a template is provided or CreateVM if not?
  
  
  • Deleted with PowerOff followed by Destroy
  
  

So - what else could we use in vSphere?

@tkak - As I dig through the code looking for models - I see that there are a lot of functions named create* being referenced that are really building up data structures which will end up being shoved into the CreateVM/ DeployVM API calls. It might be nice to flop those to use build instead of create to help distinguish the fact that they are not calling any API methods.

[tkak]

I'm sorry for the late response. I tried to list potential Terraform resources for VMware vSphere. But, I'm not sure how much demand is there in the resources except for vsphere_virtual_machine. Because I think the resources are used by the VMware vSphere admins, not users. @mkuzmin and @chrislovecnm, what do you think?

• vsphere_virtual_machine - represents a running VM
  
  
  
  • Created via DeployVM if a template is provided or CreateVM if not?
  
  
  • Deleted with PowerOff followed by Destroy
  
  
• vsphere_datacenter - represents a datacenter with https://github.com/vmware/govmomi/tree/master/govc/datacenter
  
  
  
  • Created with CreateDatacenter
  
  
  • Deleted with Destroy
  
  
• vsphere_datastore - represents a datacenter
  
  
  
  • Created? - There is no methods in govmomi now.
  
  
  • Deleted? - There is no methods in govmomi now.
  
  
• vsphere_cluster - represents a cluster compute resource with https://github.com/vmware/govmomi/tree/master/govc/cluster
  
  
  
  • Created with CreateCluster
  
  
  • Deleted with Destroy
  
  
• vsphere_resource_pool - represents a resource pool with https://github.com/vmware/govmomi/tree/master/govc/pool
  
  
  
  • Created with Create
  
  
  • Deleted with Destroy
  
  
• vsphere_virtual_switch - represents a virtual switch with https://github.com/vmware/govmomi/tree/master/govc/dvs
  
  
  
  • Created with CreateDVS
  
  
  • Deleted with Destroy
  
  
• vsphere_virtual_portgroup - represents a virtual port group with https://github.com/vmware/govmomi/blob/master/govc/dvs/portgroup/add.go
  
  
  
  • Created with AddPortgroup
  
  
  • Deleted? - There is no methods in govmomi now.
  
  

@tkak - As I dig through the code looking for models - I see that there are a lot of functions named create* being referenced that are really building up data structures which will end up being shoved into the CreateVM/ DeployVM API calls. It might be nice to flop those to use build instead of create to help distinguish the fact that they are not calling any API methods.

OK. I'll fix that. :+1:

[chrislovecnm]

@tkak awesome! We also may want to look at the ability to use govmomi to exec commands on the vm's as well.

[talarczykco]

Admittedly low priority, but the ability to set annotations on the VM in vsphere_virtual_machine would be nice. I have minimal Go-fu but a strong desire to learn; feel free to contact me if I can help.

[chrislovecnm]

@mijit help is ALWAYS appreciated!!! Email is on the way

[chrislovecnm]

A separate conversation with @mijit he mentioned:

• multiple data centers
• multiple VLANS/dSwitches

@mijit can you provide more details?

[ringods]

@chrislovecnm you can always @-mention me if you need testing of new Terraform vSphere resources. I have a vSphere 6 single machine at home and wanting to get this Terraform managed.

[matt-deboer]

Any interest in adding cloud-init capability? I looked at this only briefly, and it seems that people doing this on esxi have to do this via config-drive in an ISO. The idea would be to allow a user_data string attribute on the vm, similar to the AWS provider (and others).
Is anybody already doing this, or have some other route to leverage cloud-init on vsphere (maybe deltacloud)?

[Preskton]

@tkak @mkuzmin @chrislovecnm Windows template support (#5490) is currently critical on my list. I have folks willing to poke at it to try to get basic support going to enable a workload we need to deploy. I'd like to see who, if anyone, has ideas/guidance/support for what needs to be done to get it working.

[aheeren]

@Preskton This works on my machine, looking for feedback to get it ready for prime-time:
https://github.com/hashicorp/terraform/pull/6087

[Preskton]

@aheeren We're cloning & pulling it in now. Will update you later today. Looks like great stuff - thanks for hopping in the discussion and letting us know what you've been up to! cc: @rossedman

[chrislovecnm]

@ringods you mind giving my branch a spin? I tweaked the WaitForIp call, hopefully fixing the timing problem with getting ip addresses. I modified @tkak's PR, because I thought we needed the call sooner.

It is on this branch. I can submit as a PR as well. https://github.com/chrislovecnm/terraform/tree/vsphere-ip-wait-issue

If any other go gurus can comment on moving the WaitForIp sooner in the function call, please pipe in.

[ringods]

@chrislovecnm maybe I can find some time coming weekend. Will let you know the outcome.

[rossedman]

@chrislovecnm @phinze @mkuzmin Hey guys, @aheeren branch is working really well. I know this doesn't add new resources but this is a pretty serious outstanding bug. Would someone mind reviewing this? We have tested it on vSphere 5.5 and vSphere 6.0.

Also, I have vSphere 6.0 at home if you need me to test some things

[Preskton]

@chrislovecnm @phinze @mkuzmin /second @rossedman's request above. Having good success with @aheeren's changes and would like to see these reviewed asap so we can then get it merged in.

cc @burzinpatel - this is the PR we have been discussing over email.

[chrislovecnm]

@rossedman might you test drive https://github.com/chrislovecnm/terraform/tree/vsphere-ip-wait-issue for me

I have not done a PR cause we may kill another PR that is in ;) I don't have a test bed and won't put in a PR that has not been smoked tested.

[rossedman]

@chrislovecnm Yes I will! I need to set this up at home. May be a few days but I can do it.

[chrislovecnm]

Yo peeps we have a new issue open for documentation improvements - what are we missing in the docs!???

https://github.com/hashicorp/terraform/issues/6198

[chrislovecnm]

Ladies and gentlemen, please take a peek at https://github.com/hashicorp/terraform/pull/6204 trying to create onboarding / dev docs

[ringods]

@chrislovecnm didn't find the time unfortunately. Currently swamped with my home renovation, so very scarce on time for the coming weeks.

[chrislovecnm]

@ringods understand completely. Hope the renovations are awesome once they are done! Appreciate the follow-up!

[chrislovecnm]

Hey vSphere gurus!

Please let us know which PR's are the most important right now. I would like to help Hashicorp with a priority list, so they can get this done!

Thanks

[ringods]

@chrislovecnm regarding pull requests: #4243 for me. And documentation is always helpful, so add #6200 to it. :smile:

[aheeren]

@chrislovecnm Pull request https://github.com/hashicorp/terraform/pull/6283 fixes issue https://github.com/hashicorp/terraform/issues/3966

[chrislovecnm]

Hey all.

First off we really appreciate all of the work that a bunch of people are doing with this provider. I always Love it when people break code. It is the only way that we can make it better.

How about some good news! Been talking to (they would probably call it mild harassment) @phinze and a few others over at Hashicorp, and they don't mind me sharing some news. I will temper this with an ask for more patience, as all of us want more velocity, but that does take time!

Let me introduce a couple of new folks who are going to be helping out with this project! @markpeek and @frapposelli. The cool thing is that they work for VMware 😄

Again let me set expectations, as it will take time for the n00bs to get up to speed. But we have now have VMware taking an active role in the project!

So here is my ask! File issues. Break the code. Ask for improvements. Be active when you can. If you can code, then code and submit PRs. Unfortunately I gotta ask for it again, but be patient and kind. This is a open source provider that has not been around for a very long time. Again hearty welcome to @markpeek and @frapposelli!

[chrislovecnm]

Also a big thanks to @jen20 and @stack72 for recently merging PRs for this provider!

[markpeek]

@chrislovecnm thanks for the welcome! Yes, we are very interested to understand the prioritized list of issues and help improve the vSphere provider. Looking forward to working with the community on it.

[chrislovecnm]

@markpeek - off the top of my head here would be a good start with priorities in the following order:

1. Issues like this were the provider is returning before IP addreses are assigned to the nic(s) - https://github.com/hashicorp/terraform/issues/6174
2. Issues where we multiple networks found - https://github.com/hashicorp/terraform/issues/5607
3. Any crash issues that have been reported
4. New feature, datastore support - https://github.com/hashicorp/terraform/issues/3721

How else would you recommend creating a list of issues / features?

Anyone else opinions?

[rossedman]

@markpeek @chrislovecnm I think solidifying the vsphere_virtual_machine provider is a good first step. There are lots of different resources that could be created but making the ones we have more stable would be great. I would love to see linked clones completed and merged with a blessing from VMWare. There have been lots of Windows issues. I think that is hard to tackle because it requires us having licenses for multiple products to even debug.

[chrislovecnm]

@rossedman thanks and great feedback. Questions:

1. "Lots of Windows issues." We do not have any Windows specific issues open, unless I missed them, please report em if you got em ;) I will bow out to the Windows gurus, cause I try not to do Windows.
2. Agreed the code we have needs more testing and stability! I am concerned about the network device stuff, both lookups, as in finding networks, and issues were we do not get IP addresses back. We have open issues for both of those.

@markpeek do you have what u need? A better road map of new features should be setup, but right now stability, testing and docs are vital.

[rossedman]

@chrislovecnm Hmm, yeah I guess I could add on to that. Here were some gotchas I noticed. I think these may fall outside the realm of the vsphere provider and some even outside the realm of Terraform. Not sure.

1. Powershell is not supported (atleast from what I can tell, this actually seems to be a global Terraform issue because it relates to provisioner types)
2. When using the chef provisioner, sometimes the Chef installer will kick off before Windows is done receiving its IP address. This can cause failures so you have to trick Terraform by giving it a local-exec command to sleep for 60 seconds or something. Sometimes when assigning a static IP to a Windows machine, it receives a 169 address first. Then it receives its next address. When that 169 address is received it sends a messages back to Terraform saying "I have received my IP" which can cause problems with provisioners.
3. Windows has to restart quite a bit for stuff like AD joins. When using the chef provisioner, if a Windows box restarts on vSphere it sends a completed resource creation. Its kind of a false positive, but not totally. At that point Chef should be handling the box but it leaves a dark window in some ways or can give false positives. I am testing to see if this reaches beyond vSphere.

That's kind of longwinded. Tried my best to keep it concise. Was trying to formulate more substantial examples.

[chrislovecnm]

@rossedman - long winded is good ;) list item (2 is a symptom of the networking issues we are having. The provider is not waiting for the ip to be accessible to the API. Multiple bugs filed :(

(1 is a new feature - is it supported in the API we use?
(3 what can I say about MS having to reboot ;) Not sure how to fix that one. Wait in Windows to have a domain name?? I would defer to Windows guru. To me this is an added feature, but I hear yah.

The two above need issues. Also you don't want a Linux guy to write Windows issues ... Lol

[rossedman]

@chrislovecnm I am not a Windows guy either which is why I prefaced it the way I did. Ha. I do however have access to some great Windows minds so I will see if they can help me formulate something and help me decide what can be fixed in Terraform.

[chrislovecnm]

One thing that @stack72 pointed out is that we do not have update methods. We have Create, Read, and Delete from CRUD. But no Update ...

[rossedman]

@chrislovecnm Yeah, I can see that being important. There are certain changes that force a destroy on a resource instead of an update. For instance maybe RAM or CPU can be an update rather than forcing a new resource.

[chrislovecnm]

@stack72 @jen20 does you team need to bless this product direction, or are you good with the community laying this out?

[stack72]

you guys go for your life :) I just contribute some code once in a while - i don't work for the company

[mixacha]

@rossedman do have in mind that compute resource has to be powered down before changing cpu/memory allocation (users can use hot-plug but it disables v-numa and all other shenanigans). Just to keep in mind this regarding U operations. I believe that adding additional disks/increasing size of already provisioned disks are good candidates for update operations as they can be performed with compute resource online.

I'll let the smarter people talk and sit back and read the disucssion :-)

[dkalleg]

In my testing I've come across a similar issue to @rossedman 's #2. I'd spin up multiple ubuntu VMs in vSphere and then do provisioning steps on then which required their IPs, but I get intermittent failures due to the IP var not being set. FYI, I added d.Set("network_interface.0.ipv4_address", ip) at the end of the VM provisioning steps then try to get that value back in my .tf script ( I plan to submit a pull request for this too). Roughly 50% of the time I get failures telling me its not set. I'm very interested in seeing resolution to that. I plan to spend some time to look into it. Going to try out the local-exec sleep as a temporary work around.

I'm also planning on working on resource_vsphere_virtual_machine update this week. If anyone else has/is working on this, please let me know so I'm not duplicating work. Going to start off with a single update use case (probably adding a disk) to get whatever frameworks setup, then should be minimal effort to add other resource/config updates to it.

Is anybody else currently working on any other features? Give me a shout if my plans conflict with anyone else.

[rossedman]

@mixacha Yes it does require a power down but does that require a complete destroy? Hot swap can work with memory but I am not familiar with the v-numa stuff. You are right it is probably easier to implement additional block storage first.

[chrislovecnm]

@mixacha I might be wrong but, I think some os's allow for hot plug of cpu and mem. This allows for increase of mem and CPU without a reboot. Add and subtract. I have done this with Linux flavors. So we should be able to do some hot adds. You can add a new disk, but not resize a disk on a running VM.

We probably should drop this to a design issue, because we need to determine what govmomi supports as well. I think I am going to cut a roadmap issue and a update VM issue.

[talarczykco]

This might be scope creep since it probably means changes to core Terraform, but I've been using govc to guest.upload and guest.start scripts to my vSphere resources (to adhere to a no-ssh-as-root policy and avoid sudo setup.) It would be nice if the provider could provide those functions as options to, say, the connection block of a file provisioner. I probably lack sufficient Go-fu or round tuits to contribute code to that, but I'm hoping to follow threads that come out of this mention of govmomi in order to learn.

[chrislovecnm]

@mijit not really. I would eventually expect defining a start script as part of vm creation.

@dkalleg I know you have been very active. Do you mind sharing what features you are planning on working on? I am drafting a road map issue, and would like to include that information.

[dkalleg]

@chrislovecnm Sure.

My current tasks in prio order:

• Look into an issue I'm seeing where a d.Set() after a waitForIp() seems to not be properly setting the value about 50% of the time.
• Start work on virtual_machine update (starting with adding a disk to a running vm)
• File resource. Pushing a file to a datastore. Not the same as the provisioner "file" command to push a file to a vm.
• Start adding various other virtual_machine updates (need to come up with a list of plausible updatable fields)
• Look into Delete. Seems like it fails to delete a VM that in a Folder if they were created together. May result in working on folder deletion. (Still need to pinpoint any/all issues around Delete)
• Template resource. Probably start with CRD, submit PR for review, then work on Update.

Big picture, I want to go from 0 to Deployed-App with as little manual interaction as possible. End goal is to allow user flow of Push a vmdk to vSphere -> Optional create template from it -> Boot VM with this vmdk/template. Also want to have a story on injecting ssh keys into the image to give the user access. Currently baking my own cloudInit iso's with my ssh keys and using the cdrom feature to mount it. Intending to look into a way the user can provide ssh keys in the tf script and bake an iso on the fly or find another (faster/more efficient) way to inject those keys.

FYI, I already have PRs for specifying a bootable vmdk to boot from, as an alternative to booting from a template:
https://github.com/hashicorp/terraform/pull/6146
As well as a new resource to create a new non-bootable vmdk, which leads into my story about updating a vm to mount a new disk:
https://github.com/hashicorp/terraform/pull/6273
I still need to address some comments on each, will get to that soon, I promise :)

[chrislovecnm]

@dkalleg

Look into an issue I'm seeing where a d.Set() after a waitForIp() seems to not be properly setting the value about 50% of the time.

See this issue: https://github.com/hashicorp/terraform/issues/6300 - It has a PR in it mentioned by govmomi team https://github.com/vmware/govmomi/pull/501 - please try that

Start work on virtual_machine update (starting with adding a disk to a running vm)

See here https://github.com/hashicorp/terraform/issues/6341 - please link PR's / issue to there

File resource. Pushing a file to a datastore. Not the same as the provisioner "file" command to
push a file to a vm.

Not sure where to put this - either on roadmap or separate issue

Start adding various other virtual_machine updates (need to come up with a list of plausible updatable fields)

Awesome

Look into Delete. Seems like it fails to delete a VM that in a Folder if they were created together. May result in working on folder deletion. (Still need to pinpoint any/all issues around Delete)

This should be filed as an issue

Template resource. Probably start with CRD, submit PR for review, then work on Update.

cool

[chrislovecnm]

Hey all ... just openned an issue to track the work on updating VMs.https://github.com/hashicorp/terraform/issues/6341

Will start pulling out components out of this issue into a roadmap ... will take a few days. Currently I believe that the network issues of both waiting for the nics to come up and the way the provider looks up the networks are the two critical things that need to be addressed.

Not waiting for networks properly:

https://github.com/hashicorp/terraform/issues/6300

Not finding networks and duplicate networks:

https://github.com/hashicorp/terraform/issues/6266
https://github.com/hashicorp/terraform/issues/5607

[dkalleg]

@chrislovecnm Just wanted to voice a distinction with my case. I see the waitForIp() call blocking while I watch vSphere UI booting up my vm, when I see my ip show up in vSphere (usually < 30 seconds) the waitForIp() stops blocking and continues on. At this time I'm writing the ip to d.SetConnInfo.host as well as network_interface.0.ipv4_address.

My understanding is that the remote-exec calls read the ConnInfo.host in order to make the ssh connection; I've never had a problem with the remote-exec not knowing what the host ip was. I've only ever had problems when I try to dereference ${self.network_interface.0.ipv4_address} in my .tf file. Considering this, I not suspicious of the waitForIp() not doing what its expected to. I'm suspecting some sort of race condition between the Set and Get of network_interface.0.ipv4.address.. or something else dumb I've done in my fork :)

Anywho, I'll make sure I update here if I come to any conclusion.

[Preskton]

@chrislovecnm Other than this thread, do you have somewhere where you're compiling the "official" (at least as official as we'll get) roadmap?

[chrislovecnm]

@preskton - on my todo list :)

[mixacha]

@rossedman we can make update operation work by powering down the instance and then increasing memory and cpu - we just have to cover it with appropriate docs so people are aware that this will happen. Hot-plug option is possible but it requires setting certain advanced parameters on virtual machine creation for both memory and cpu - then we can plainly add new resources to it (maybe some pre-flight checking if this is enabled, if supported by govc). Otherwise if a virtual machine doesn't have these options set, action should fall back to fore mentioned behavior of powering off a resource and changing resource allocation. If my memory serves me well these should be vcpu.hotadd and mem.hotadd inside a .vmx file.

@chrislovecnm You can actually resize an running disk but you still have to do in-guest actions of rescaning scsi bus to re-read geometry and all that follows. You're correct, hot-add is supported by majority of Linux OSes today, however for people that care about performance this option is usually not enabled as it disables v-numa (https://cto.vmware.com/vnuma-what-it-is-and-why-it-matters/).

I guess adding additional disk to existing resource might be the easiest thing to start working on.

@mijit I've worked around this by creating a special account and then using a local provisioner to do any post configuration steps within a guest operating system, deleting the account itself as a last step. You might also consider cloud-init once iso mounting feature is merged to terraform vsphere provider for any additional post-configuration.

[vipulsabhaya]

@chrislovecnm Is there any interest in supporting NSX in Terraform?

[chrislovecnm]

@vipulsabhaya do not know. At the point we are mostly concentrating on vsphere. Does NSX have a go API?

[vipulsabhaya]

@chrislovecnm Not that I have seen, NSX has a REST api, so it may need to be added to govmomi or similar

[chrislovecnm]

@vipulsabhaya a supporting API is the first step. You many want to put in a request to the govmomi maintainers, if this support is of interest to you. If you develop you may want to work on that support. Thanks Chris

[rossedman]

@vipulsabhaya I would like to see NSX. There is definitely a need there but would love to see more vSphere resources covered first.

[dagnello]

@chrislovecnm starting to work towards adding file resource, I have created the following issue (#6366) for tracking.

[chrislovecnm]

@rossedman we are of like minds.

It may be till next week before I have a chance to create a roadmap issue. Busy busy 😄

[dkalleg]

Not sure if this is the right forum for this question.. pl point me elsewhere if needed. Regarding vm update to add a disk (I'm hoping some vmware folks can chime in here):

From vSphere I can add a disk to a running vm, but I cannot see that disk from within the vm; fdisk -l doesn't show it and nothing new under /dev/sd*. On the vm there are various ways to rescan the SCSI controller for new disks, but does anybody know a way to handle hot-attach without having to run commands on the guest OS? For now I'm find focusing on Ubuntu. I'm told in IRC that RHEL is hot-attaching disks, not sure how.

I can think of 3 potential solutions:
1) Some vm configuration parameter to enable hot-plugging that I don't know aobut
2) some govmomi api calls to do scsi rescan I don't know about
3) document for users that they need to do a remote-exec to rescan scsi when adding a new disk.

Thoughts?

[chrislovecnm]

@mixacha you seemed to have done disk stuff before. Can you comment on @dkalleg's question??

@dkalleg I think we need to assume that a remote-exec needs to be run, and probably a reboot. At the least they have to format or add the disk to lvm for instance. What @mixacha mentioned is that hot add functionality is often not supported because it is a performance hit. I recommend that we proceed assuming that hot add has been turned off.

The following is a pretty good write up about adding a disk hot: http://ryandlane.com/blog/2010/03/12/adding-new-hard-drives-live-on-an-ubuntu-guest-with-vmware-esx/

But again a lot of this depends on OS and settings at an esx level.

[dkalleg]

After some internal discussions, it sounds like leaving the scsi rescan up to the user (whether via remote-exec or other means) is the preferable option. This is a guest OS level operation, which Terraform should be agnostic to. Maybe we can add a parameter to the vm resources disk definition like hot-add which would determine whether we force a guest OS reboot on new disk update or not. Thoughts?

[chrislovecnm]

@dkalleg I would say that we should default to reboot, and have an override functionality. Not sure how the other providers handle it, but would model them. I know for instance to update sizing on a gce instance, you have to power it down first.

[dkalleg]

@chrislovecnm @phinze @tkak et al. Are there plans to move up our govmomi version? I've learned that there is a fix/feature thats come in since 0.3.0 that I'd be interested in taking advantage of. In short, with 0.3.0, we cannot detach a disk from a vm without destroying that disk. I'm sure theres other improvements/features too.

[chrislovecnm]

@dkalleg yes we need the latest for another bug 👍

[dkalleg]

@chrislovecnm Is anyone working on it? Do we have an eta?

[chrislovecnm]

@thetuxkeeper actually is having some issues with it. I cc'ed you on another issue. If you guys want to figure it out and get a PR in ... we can try to expedite it

[chrislovecnm]

@thetuxkeeper can you try to bump the version of govmomi again? I am getting an error doing it ;(

[chrislovecnm]

@thetuxkeeper @dkalleg the version bump is not trivial :( We have breaking changes ... working on it. Guys let me know if you want to cut a separate PR on this or if I need to focus on it.

Thanks

[chrislovecnm]

@thetuxkeeper @dkalleg @markpeek working on getting the version bump

[chrislovecnm]

Roadmap started ... please comment!!

Issue: https://github.com/hashicorp/terraform/issues/6565

[chrislovecnm]

Hey all need some reviews please, documentation improvments.

https://github.com/hashicorp/terraform/pull/6661

@markpeek pretty please.

[chrislovecnm]

If anyone has DRS and a few minutes to test the new DRS code, please holler!!

Code is here: https://github.com/hashicorp/terraform/pull/7031

Initial support for VMs. Disks, Files, and CDs are not yet supported.

[ringods]

And to make good progress on all of this: proper test automation.
@chrislovecnm can you have a look at my findings in #7973 to see if this helps in testing the terraform plugin against a real vsphere?

[mrjcleaver]

Is this initiative dead? Who from Hashicorp or vmWare is championing or sponsoring this effort to make sure it sees the light of day?

@phinze do you own it?

[mrjcleaver]

@markpeek - is VMware committed to helping progress the Terraform driver?

[ktham]

Is there a plan forward for #5843? it's causing terraform to force new resources one every VM.

We're doing an evaluation this month to see if we'd want to switch our provisioning system from our homegrown tooling written with rbvmomi and replace it with terraform. We love terraform so far! We don't usually have opportunities to evaluate new tooling and we have the opportunity/bandwidth to do so in the next month-ish before we have to scrap this exploration and move on.

Will gladly help where we can!! :-)

[mitchellh]

@mrjcleaver We're actively communicating with VMware as well as other interested companies to help push forward the vSphere work. Nobody at HashiCorp works on vSphere full time and it isn't an official HashiCorp provider at the moment (only AWS, Azure, Google at the time of writing this comment).

I'm going to close this since we're trying to go away from bigger "question" issues since they're too open ended and end up sitting open for years. Individual vsphere bugs exist and most releases still include vsphere improvements. Internally at HC, we're working hard to find a better long term solution and we'll share as soon as we can. Externally, we welcome any contributors.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.