Packer: Suppressing powershell output streams in packer output

[mrhockeymonkey]

Packer version: 0.12.0
PSVersion: 5.0
OS: Windows 10

Currently when using the "powershell" provisioner i keep getting a bunch of clixml output from packer. it looks to be describing all the output streams powershell has (i.e. verbose, information, progress, etc). is there a way to suppress this? I have added for instance $ProgressPreferance = 'SilentlyContinue' to some of my scripts but it only removes that stream information.

Here is an example of such output - this example is really small but when i do something that takes a long time with lots of progress like windows updates i seem endless clixml streaming to my console.

> virtualbox-iso: Restarting Machine
==> virtualbox-iso: Waiting for machine to restart...
    virtualbox-iso: A system shutdown is in progress.(1115)
    virtualbox-iso: #< CLIXML
    virtualbox-iso: <Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><Obj S="progress" RefId="0"><TN RefId="0"><T>System.Management.Automation.PSCustomObject</T><T>System.Object</T></TN><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj></Objs>
    virtualbox-iso: WIN-8F5U52D6M2U restarted.
==> virtualbox-iso: Machine successfully restarted, moving on
==> virtualbox-iso: Provisioning with Powershell...
==> virtualbox-iso: Provisioning with shell script: scripts/provisioning/InstallPSModules.ps1
    virtualbox-iso: Installing PackageProvider: Nuget
    virtualbox-iso:

example of windows update packer output of clicml (scroll right)

    virtualbox-iso: 4 Installed  KB3095701 102 KB Update for Windows Server 2012 R2 (KB3095701)
    virtualbox-iso: 4 Installed  KB3094486  71 KB Update for Windows Server 2012 R2 (KB3094486)
    virtualbox-iso: 4 Installed  KB3091297 192 KB Update for Windows Server 2012 R2 (KB3091297)
    virtualbox-iso: 4 Installed  KB3078405 102 KB Update for Windows Server 2012 R2 (KB3078405)
    virtualbox-iso: 4 Installed  KB3081320   4 MB Security Update for Windows Server 2012 R2 (KB...
    virtualbox-iso: 4 Installed  KB3102939   2 MB Security Update for Windows Server 2012 R2 (KB...
    virtualbox-iso: for Windows Server 2012 R2 (KB3086255)</SD></PR></MS></Obj><S S="debug">Show update to install: Security Update for Windows Server 2012 R2 (KB3086255)</S><S S="debug">Send update to install collection</S><S S="debug">Try install update</S><S S="debug">Add to log collection</S><Obj S="progress" RefId="650"><TNRef RefId="3" /><MS><I64 N="SourceId">0</I64><PR N="Record"><AV>[4/4] Installing updates</AV><AI>0</AI><Nil /><PI>-1</PI><PC>45</PC><T>Processing</T><SR>-1</SR><SD>[84/185] Update for Windows Server 2012 R2 (KB3084905)</SD></PR></MS></Obj><S S="debug">Show update to install: Update for Windows Server 2012 R2 (KB3084905)</S><S S="debug">Send update to install collection</S><S S="debug">Try install update</S><S S="debug">Add to log collection</S><Obj S="progress" RefId="651"><TNRef RefId="3" /><MS><I64 N="SourceId">0</I64><PR N="Record"><AV>[4/4] Installing updates</AV><AI>0</AI><Nil /><PI>-1</PI><PC>46</PC><T>Processing</T><SR>-1</SR><SD>[85/185] Update for Windows Server 2012 R2 (KB3087390)</SD></PR></MS></Obj><S S="debug">Show update to install: Update for Windows Server 2012 R2 (KB3087390)</S><S S="debug">Send update to install collection</S><S S="debug">Try install update</S><S S="debug">Add to log collection</S><Obj S="progress" RefId="652"><TNRef RefId="3" /><MS><I64 N="SourceId">0</I64><PR N="Record"><AV>[4/4] Installing updates</AV><AI>0</AI><Nil /><PI>-1</PI><PC>46</PC><T>Processing</T><SR>-1</SR><SD>[86/185] Update for Windows Server 2012 R2 (KB3087041)</SD></PR></MS></Obj><S S="debug">Show update to install: Update for Windows Server 2012 R2 (KB3087041)</S><S S="debug">Send update to install collection</S><S S="debug">Try install update</S><S S="debug">Add to log collection</S><Obj S="progress" RefId="653"><TNRef RefId="3" /><MS><I64 N="SourceId">0</I64><PR N="Record"><AV>[4/4] Installing updates</AV><AI>0</AI><Nil /><PI>-1</PI><PC>47</PC><T>Processing</T><SR>-1</SR><SD>[87/185] Update for Windows Server 2012 R2 (KB3087137)</SD></PR></MS></Obj><S S="debug">Show update to install: Update for Windows Server 2012 R2 (KB3087137)</S><S S="debug">Send update to install collection</S><S S="debug">Try install update</S><S S="debug">Add to log collection</S><Obj S="progress" RefId="654"><TNRef RefId="3" /><MS><I64 N="SourceId">0</I64><PR N="Record"><AV>[4/4] Installing updates</AV><AI>0</AI><Nil /><PI>-1</PI><PC>48</PC><T>Processing</T><SR>-1</SR><SD>[88/185] Update for Windows Server 2012 R2 (KB3080042)</SD></PR></MS></Obj><S S="debug">Show update to install: Update for Windows Server 2012 R2 (KB3080042)</S><S S="debug">Send update to install collection</S><S S="debug">Try install update</S><S S="debug">Add to log collection</S><Obj S="progress" RefId="655"><TNRef RefId="3" /><MS><I64 N="SourceId">0</I64><PR N="Record"><AV>[4/4] Installing updates</AV><AI>0</AI><Nil /><PI>-1</PI><PC>48</PC><T>Processing</T><SR>-1</SR><SD>[89/185] Update for Windows Server 2012 R2 (KB3096433)</SD></PR></MS></Obj><S S="debug">Show update to install: Update for Windows Server 2012 R2 (KB3096433)</S><S S="debug">Send update to install collection</S><S S="debug">Try install update</S><S S="debug">Add to log collection</S><Obj S="progress" RefId="656"><TNRef RefId="3" /><MS><I64 N="SourceId">0</I64><PR N="Record"><AV>[4/4] Installing updates</AV><AI>0</AI><Nil /><PI>-1</PI><PC>49</PC><T>Processing</T><SR>-1</SR><SD>[90/185] Update for Windows Server 2012 R2 (KB3095701)</SD></PR></MS></Obj><S S="debug">Show update to install: Update for Windows Server 2012 R2 (KB3095701)</S><S S="debug">Send update to install collection</S><S S="debug">Try install update</S><S S="debug">Add to log collection</S><Obj S="progress" RefId="657"><TNRef RefId="3" /><MS><I64 N="SourceId">0</I64><PR N="Record"><AV>[4/4] Installing updates</AV><AI>0</AI><Nil /><PI>-1</PI><PC>49</PC><T>Processing</T><SR>-1</SR><SD>[91/185] Update for Windows Server 2012 R2 (KB3094486)</SD></PR></MS></Obj><S S="debug">Show update to install: Update for Windows Server 2012 R2 (KB3094486)</S><S S="debug">Send update to install collection</S><S S="debug">Try install update</S><S S="debug">Add to log collection</S><Obj S="progress" RefId="658"><TNRef RefId="3" /><MS><I64 N="SourceId">0</I64><PR N="Record"><AV>[4/4] Installing updates</AV><AI>0</AI><Nil /><PI>-1</PI><PC>50</PC><T>Processing</T><SR>-1</SR><SD>[92/185] Update for Windows Server 2012 R2 (KB3091297)</SD></PR></MS></Obj><S S="debug">Show update to install: Update for Windows Server 2012 R2 (KB3091297)</S><S S="debug">Send update to install collection</S><S S="debug">Try install update</S><S S="debug">Add to log collection</S><Obj S="progress" RefId="659"><TNRef RefId="3" /><MS><I64 N="SourceId">0</I64><PR N="Record"><AV>[4/4] Installing updates</AV><AI>0</AI><Nil /><PI>-1</PI><PC>50</PC><T>Processing</T><SR>-1</SR><SD>[93/185] Update for Windows Server 2012 R2 (KB3078405)</SD></PR></MS></Obj><S S="debug">Show update to install: Update for Windows Server 2012 R2 (KB3078405)</S><S S="debug">Send update to install collection</S><S S="debug">Try install update</S><S S="debug">Add to log collection</S><Obj S="progress" RefId="660"><TNRef RefId="3" /><MS><I64 N="SourceId">0</I64><PR N="Record"><AV>[4/4] Installing updates</AV><AI>0</AI><Nil /><PI>-1</PI><PC>51</PC><T>Processing</T><SR>-1</SR><SD>[94/185] Security Update for Windows Server 2012 R2 (KB3081320)</SD></PR></MS></Obj><S S="debug">Show update to install: Security Update for Windows Server 2012 R2 (KB3081320)</S><S S="debug">Send update to install collection</S><S S="debug">Try install update</S><S S="debug">Add to log collection</S><Obj S="progress" RefId="661"><TNRef RefId="3" /><MS><I64 N="SourceId">0</I64><PR N="Record"><AV>[4/4] Installing updates</AV><AI>0</AI><Nil /><PI>-1</PI><PC>51</PC><T>Processing</T><SR>-1</SR><SD>[95/185] Security Update for Windows Server 2012 R2 (KB3102939)</SD></PR></MS></Obj><S S="debug">Show update to install: Security Update for Windows Server 2012 R2 (KB3102939)</S><S S="debug">Send update to install collection</S><S S="debug">Try install update</S><S S="debug">Add to log collection</S><Obj S="progress" RefId="662"><TNRef RefId="3" /><MS><I64
    virtualbox-iso: 4 Installed  KB3034348   3 MB Update for Windows Server 2012 R2 (KB3034348)
    virtualbox-iso: 4 Installed  KB3098779   7 MB Security Update for Microsoft .NET Framework 4...
    virtualbox-iso: 4 Installed  KB3097997   1 MB Security Update for Microsoft .NET Framework 4...
    virtualbox-iso: 4 Installed  KB3108381   7 MB Security Update for Windows Server 2012 R2 (KB...

Hopefully someone has run into this before but im happy to provide anything else needed to answer this.

[wv-jbaldwin]

Could you suppress it in your PowerShell script using | Out-Null after the commands that are generating the long output?

[mrhockeymonkey]

No that doesn't work. I already do that on a few commands i dont want to see output for. The issue is Powershell has different output streams (seven i think; Verbose, Warning, Information,Debug, Progress, Error & Standard). Out-Null will mute Standard out for each command it is used on but verbose, etc message still come through in the xml show above.

[sneal]

As already indicated this is from the additional PowerShell streams, mainly the progress stream. In Vagrant all script are prepended with $ProgressPreference = 'SilentlyContinue', I would suggest we do the same for Packer.

Currently Packer makes direct use of winrm.PowerShell to build a runnable command over WinRM. I propose we wrap winrm.PowerShell with a Packer specific function that prepends the progress pref.

[rgl]

does the following output has anything to do with this issue? pay special attention to the <Objs ... element.

==> windows-2016-amd64-virtualbox: Provisioning with shell script: provision.ps1
    windows-2016-amd64-virtualbox: #< CLIXML
    windows-2016-amd64-virtualbox: <Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><Obj S="progress" RefId="0"><TN RefId="0"><T>System.Management.Automation.PSCustomObject</T><T>System.Object</T></TN><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj><Obj S="information" RefId="1"><TN RefId="1"><T>System.Management.Automation.InformationRecord</T><T>System.Object</T></TN><ToString>Importing the Oracle (for VirtualBox) certificate as a Trusted Publisher...</ToString><Props><Obj N="MessageData" RefId="2"><TN RefId="2"><T>System.Management.Automation.HostInformationMessage</T><T>System.Object</T></TN><ToString>Importing the Oracle (for VirtualBox) certificate as a Trusted Publisher...</ToString><Props><S N="Message">Importing the Oracle (for VirtualBox) certificate as a Trusted Publisher...</S><B N="NoNewLine">false</B><S N="ForegroundColor">Gray</S><S N="BackgroundColor">Black</S></Props></Obj><S N="Source">C:\Windows\Temp\script.ps1</S><DT N="TimeGenerated">2017-01-02T19:28:13.2935515+00:00</DT><Obj N="Tags" RefId="3"><TN RefId="3"><T>System.Collections.Generic.List`1[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]</T><T>System.Object</T></TN><LST><S>PSHOST</S></LST></Obj><S N="User">VAGRANT-HFFPH0G\vagrant</S><S N="Computer">VAGRANT-HFFPH0G</S><U32 N="ProcessId">4780</U32><U32 N="NativeThreadId">4676</U32><U32 N="ManagedThreadId">7</U32></Props></Obj><Obj S="information" RefId="4"><TNRef RefId="1" /><ToString>Installing the VirtualBox Guest Additions...</ToString><Props><Obj N="MessageData" RefId="5"><TNRef RefId="2" /><ToString>Installing the VirtualBox Guest Additions...</ToString><Props><S N="Message">Installing the VirtualBox Guest Additions...</S><B N="NoNewLine">false</B><S N="ForegroundColor">Gray</S><S N="BackgroundColor">Black</S></Props></Obj><S N="Source">C:\Windows\Temp\script.ps1</S><DT N="TimeGenerated">2017-01-02T19:28:13.9427583+00:00</DT><Obj N="Tags" RefId="6"><TNRef RefId="3" /><LST><S>PSHOST</S></LST></Obj><S N="User">VAGRANT-HFFPH0G\vagrant</S><S N="Computer">VAGRANT-HFFPH0G</S><U32 N="ProcessId">4780</U32><U32 N="NativeThreadId">4676</U32><U32 N="ManagedThreadId">7</U32></Props></Obj><Obj S="information" RefId="7"><TNRef RefId="1" /><ToString>Done.</ToString><Props><Obj N="MessageData" RefId="8"><TNRef RefId="2" /><ToString>Done.</ToString><Props><S N="Message">Done.</S><B N="NoNewLine">false</B><S N="ForegroundColor">Gray</S><S N="BackgroundColor">Black</S></Props></Obj><S N="Source">C:\Windows\Temp\script.ps1</S><DT N="TimeGenerated">2017-01-02T19:28:33.8651457+00:00</DT><Obj N="Tags" RefId="9"><TNRef RefId="3" /><LST><S>PSHOST</S></LST></Obj><S N="User">VAGRANT-HFFPH0G\vagrant</S><S N="Computer">VAGRANT-HFFPH0G</S><U32 N="ProcessId">4780</U32><U32 N="NativeThreadId">4676</U32><U32 N="ManagedThreadId">7</U32></Props></Obj></Objs>
==> windows-2016-amd64-virtualbox: Gracefully halting virtual machine...

[DanHam]

@rgl Yes - same issue...

[dayglojesus]

Seeing the same thing. Prepending ... $ProgressPreference = "SilentlyContinue" in your script or as an environment variable does nothing to curb this when you are executing the Powershell with elevated privileges.

[kalpik]

So is there a way to suppress this? Reading up on the comments, it seems there was no workaround?

[DanHam]

@kalpik Unfortunately not - I found that there was nothing that could be done as an end user to prevent the unwanted output.
I found that Powershell _always_ leaked its Progress stream to stderr when the -EncodedCommand flag is used and this is currently hard coded into Packers Powershell provisioner.
I submitted a PR that fix this back in Jan. Unfortunately, despite having put a fair bit of effort in to get the PR merged, I ended up closing (and blanking) the PR - see my comment here.

However, it does look as though @SwampDragons has been busy making some changes to the Powershell provisioner of late though so hopefully a fix should be working its way in to the code shortly!

[kalpik]

Thanks a lot @DanHam. Waiting for a proper fix for this.

[SwampDragons]

I hadn't planned on getting around to this very soon. That said, @DanHam if you would be willing to reopen your PR, we'll take another look. The timing on your PR was really bad originally through no fault of your own; @mwhooker was on leave and I was really new; I wasn't comfortable tackling it because I hadn't yet worked on the powershell provisioner. But Hooker is back and as you noticed I've been putting some energy into this provisioner lately. So we'd love to get your work across the finish line if you're willing to bear with us again.

[DanHam]

@SwampDragons I was not supposed to be spending any more time on this! When I blanked all the commits my intention was to destroy the code and prevent me from being tempted with further tinkering. However, I realised I had the code backed up in a Time Machine backup... and now I've been sucked right back in!! 😄
I've rebased, reworked due to the latest changes that have been made and resubmitted...

[SwampDragons]

Hah! Nothing ever dies in git. Thanks for being tempted back into it.

[kalpik]

Haha! I'm now glad I left that comment :P

[tlmartins]

Hello, I've solved a similar issue junt definig the use of version 2.0 :

powershell.exe -version 2.0 -noprofile -executionpolicy bypass -outputformat Text -nologo -encodedcommand....

[DanHam]

Unfortunately, version 2 of PowerShell is not installed by default.

From the latest Amazon Windows 2016 Base AMI:

PS C:\Users\Administrator> Get-WindowsFeature -Name "PowerShell*"

Display Name                                            Name                       Install State
------------                                            ----                       -------------
[X] Windows PowerShell                                  PowerShellRoot                 Installed
    [X] Windows PowerShell 5.1                          PowerShell                     Installed
    [ ] Windows PowerShell 2.0 Engine                   PowerShell-V2                    Removed
    [X] Windows PowerShell ISE                          PowerShell-ISE                 Installed

Clearly, adding -version 2 to the PowerShell command without version 2 being installed will result in an error:

PS C:\Users\Administrator> powershell.exe -version 2 -encodedCommand $base64command
Version v2.0.50727 of the .NET Framework is not installed and it is required to run 
version 2 of Windows PowerShell. 

However, out of curiosity, I had a quick look at this...

• I checked out the code at 80c0f47cf19f93298c1f6dc1e2bec226a60b5e5f - this was the last point at which we used base64 encoding.
• I then added the -version 2 arguments at the required points in the code, and rebuilt Packer.
• Using the amazon-ebs builder I ensured version 2 of PowerShell was installed via a command in the userdata script

With the set up above I was able to get to the point where I was able to run a PowerShell provisioner with the -version 2 argument. Although Packer then hung for some reason, it does look as though the leaking of the Progress stream has been fixed in version 2 of PowerShell.

I didn't take this any further, but this seems to confirm to me that the issues we were seeing here are caused by a bug in v1 of PowerShell. As @tlmartins has shown, the problem appears to have been fixed with v2 of PowerShell...

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.