Salt: Module request: PowerShell DSC resource

[absolutejam]

I'm not sure if it's acceptable to request modules or features, but I think that the win_dsc would benefit from the ability to apply DSC resources in addition to applying a whole MOF.

This would function like module.run, where each kwarg is passed directly to the module, leveraging Invoke-DSCResource. This allows for full utilisation of DSC resources, bolstering what SaltStack can manage.

I'd really like to start on this, but I currently don't have time. If we could perhaps start a discussion, it may be easier for me or somebody else to pick up on this?

The idea for this was taken from Ansible's recent implementation, although I'm not sure how to best implement this as Salt leverages python not Powershell on Windows hosts. Cross-calling cmd.powershell could work, but I'm not sure if there's a more succinct implementation?

[damon-atkins]

The issue is the overhead of starting powershell. Myself and few others have written code to prevent the need to run powershell to find out basic information about it, due to how slow it is to start.

See also #33003 Enhancement: For Windows Add Python Dot Net (.net) interface which is the only way I see to get over the time it takes to start powershell. Others might see an alternative.

[Ch3LL]

@twangboy do you have any suggestions here or direction for this?

[Ch3LL]

ping @saltstack/team-windows

[twangboy]

@absolutejam The win_dsc module is written to allow you to define your DSC stuff with powershell. You create your powershell, then compile it with dsc.compile_config which creates the .mof file. Then you apply the .mof with dsc.apply_config. The dsc.run_config is a convenience function that basically just does both of those.

That being said, couldn't you add whatever Invoke-DSCResource commands in your powershell script before it's compiled? Or am I not understanding?

[absolutejam]

This method bypasses the need to write any Powershell or generate a MOF and directly runs the DSC module's get/set/test methods, meaning it can run a DSC resource from parameters alone and not requiring a full configuration. This could essentially allow us to use DSC modules alongside native salt modules, assuming we can report changes successfully (no reason why not).

[absolutejam]

@damon-atkins I can appreciate this is a concern, especially if a Powershell process has to be fired up multiple times (To validate before running).

Regarding #33003 - if there was a way to have Powershell as a first-class citizen on Windows, it'd be a geeky dream come true. While I appreciate SaltStack is a Python tool, and Python is my go-to tool for unix-like systems, Powershell is the sysadmin tool for Windows, and it's management treasure-trove ready to be tapped into. I'm pretty confident in writing Python to extend SaltStack for my Windows hosts, but if I could leverage parts of Powershell more directly directly, without having to constantly shell out, it would be awesome. While that may be a pipe dream, any improved integration, should as for this usage, would be a big improvement for Windows hosts.

[damon-atkins]

Currently its run power shell one liner to check the current status (4 secs), then another power shell one liner to change the item if required (4 secs). For example salt would start powershell to check its version multiple times (different modules), so I developed cmd.shell_info which gets the information from the registry instead, which takes less than a second. Calling powershell was slowing down Salt on Windows initialisation.

I am not against powershell, we just need a smart way to get around the startup overhead.

It would be nice to have a powershell slave which is started once per state run for all powershell tasks and then closed down at the end of the state run. Maybe using #33003 I do not know if this is possible.

Maybe something like this, using Named Pipes in Python to talk to a PowerShell Process listing to the Named Pipe.
https://gbegerow.wordpress.com/2012/04/09/interprocess-communication-in-powershell/

[absolutejam]

Just revisiting this - I tried a few ways of having persisting Powershell process/sessions, but had to resort to named pipes. I have made some progress with named pipes, but it still seems a flaky, indirect way to do it.

Looks like Chef are just invoking PowerShell directly, and they're well known for top-tier DSC support. I really think that letting the PowerShell performance overhead stop SaltStack from including this feature is a net loss, because using DSC opens SaltStack up to a tonne of more functionality with little effort (Well, a lot less effort than reinventing the wheel). While Salt has good Windows support, this is what takes that to the next level.

[damon-atkins]

33003 Enhancement: For Windows Add Python Dot Net (.net) interface Did you try this, I personal do not have the skills to do this in a reasonable time frame

[absolutejam]

@damon-atkins That looks ideal, but it's another hard dependency that would have to ship with SaltStack right, as IronPython is completely separate to CPython?

[absolutejam]

@damon-atkins @twangboy Okay, a while back I really wanted to take a stab at creating the dsc resource module (Basically, exactly what Ansible does where it passes kwargs to Invoke-DSCResource), but I've had a change in job role and therefore not enough time to play.

However, I did check back on this recently as I've been diving back into Salt for a new project and saw reference to #33003. I had a little play this morning and managed to get PowerShell invocation working from within Python, which could be a good start.

• First, I downloaded the PowerShell 5.0 nuget package and extracted it because I couldn't find System.Management.Automation.dll
• I also found out afterwards, that running ([PSObject].Assembly.Location) shows the location of the current PowerShell dlls, so it should be okay to piggyback off the machine's.

import clr
import sys
sys.path.append(r"Path\to\extracted")
clr.AddReference("System.Management.Automation")
from System.Management.Automation import PowerShell

ps = PowerShell.Create()

# Create a pipeline (Bad example)
ps.AddCommand("Get-Service").AddCommand("Out-String")
services = ps.Invoke()
for line in services:
  print(line)

# Clear the commands
ps.Commands.Clear()

# Another example
ps.AddCommand("Get-Process")
procs = ps.Invoke()
print(procs[0].BaseObject.ProcessName)

The best part about this, is that it allows interaction with objects (including methods) without a lot of converting to and from JSON.

Could this be utllised for PowerShell processing instead of spawning PowerShell instances? This would definitely make the process of multiple PowerShell runs slightly faster/less demanding, and I know the idea for an Invoke-DSCResource equivalent would require a few for validation.

[twangboy]

@absolutejam This looks like a pretty sweet approach. @dwoz What do you think.

[damon-atkins]

@absolutejam Thats the first use of clr
https://github.com/saltstack/salt/pull/45304 Add pythonnet dependency for Windows Jan 2018
https://github.com/saltstack/salt/issues/33003 RFE: For Windows Add Python Dot Net (.net) interface Original request

[kinwolfqc]

Anything new to report on this? Starting to learn Salt here to see if it might be a better fit for us than ansible(mostly for speed). but the lack of proper DSC ressources support is a showstopper for us(we are a windows shop at 90%). There are thousands of DSC resources to be used on Windows and they are kept up to date, so I am really surprised to find that Salt does not support using/calling them.

[damon-atkins]

Its not that it does not have DSC support its just slow. I would have thought other tools would be in the same boat.

[kinwolfqc]

Ansible and Puppet can use the whole range of DSC ressources. I think Chef too but I never checked myself on that one.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[mchugh19]

Still valid feature request

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[kinwolfqc]

As long as salt won't be able to use all the DSCRessources like Ansible or Puppet can, it will never be a contender in the windows world.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[damon-atkins]

@twangboy Some time need to be invested in this, maybe someone in the working group would like to look into it who has more experiance with clr

[twangboy]

@damon-atkins I think you're right

[damon-atkins]

@twangboy Got powershell calling working the other day the key is setting sys.path.append(r"important path") correctly
Where important path is the value of
powershell "([PSObject].Assembly.Location)"
Have not been able to find the above value in the registry.
And it responds in less than a second, I suspect its quicker than calling powershell command because python does not need to start a thread or fork(). And starting thread or fork is slow on Windows.

[twangboy]

What do you mean by Got powershell calling working? Is the path a requirement?

I implemented a version of @absolutejam 's example above (https://github.com/saltstack/salt/issues/43718#issuecomment-410968388) to get interface info using .NET instead of WMI. It was an attempt to speed up the creation of grains. I didn't need to specify a path. I think the clr library handles all that.

https://github.com/saltstack/salt/blob/master/salt/utils/win_network.py#L228

How would this avoid starting a new fork()? Isn't the new fork started when a module or state is called? That would occur before the .NET code is called, right?

Sorry, I should know better where all that happens in Salt.

[damon-atkins]

Here is a full example. cmd.run etc fork and then run the command e.g. powershell.
This code loads a DLL and executes a function within it, I believe no new process is started.

import clr
import sys
# PS > ([PSObject].Assembly.Location)
# Prints C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Management.Automation\v4.0_3.0.0.0__31bf3856ad364e35\System.Management.Automation.dll
sys.path.append(r'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Management.Automation\v4.0_3.0.0.0__31bf3856ad364e35') # WORKS
clr.AddReference("System.Management.Automation")
from System.Management.Automation import PowerShell
ps = PowerShell.Create()  
# Another example
ps.AddCommand("Get-Process")
procs = ps.Invoke()
for item in procs:
   print(item.BaseObject.ProcessName)

See https://stackoverflow.com/questions/52465237/multiprocessing-slower-than-serial-processing-in-windows-but-not-in-linux