Azure-pipelines-tasks: Write-Host "##vso[task.setvariable variable=someVar;]someValue" sets are (sometimes) overwritten with 'default' values in subsequent PowerShell -Argument steps

[JoeBrockhaus]

In a Meta-Task, we've got a Property/Variable defined that needs to be set from within an AzurePowerShell script.

We've verified that the script is writing the correct resulting string with Write-Host, however the next step in the Meta-Task (executed during a Release) is not getting the new value set in the previous step.

In a normal PowerShell script task, the setvariable command works fine.
The steps in this particular Meta-Task were all created automatically by the tool from a build definition where this was working previously.

[ericsciple]

what is your agent version?

[ericsciple]

Are you updating a variable that was defined already on your definition? Or is the ##setvariable command defining a new variable?

[JoeBrockhaus]

[e] @ericsciple thanks for the reply. Agent Version is 1.104.1 [/e]

I've spent a lot of time in the weeds on this in the last 24 hours, and have found some odd inconsistencies. Hopefully I didn't miss anything obvious :)
It appears this is at least partially-related to Meta-Task execution and the 'magic' that happens behind the scenes with Variables, Parameters, and Environment Variables.

Also in our scenario, the Release in question should not have to specify a value for the variable we are trying to set and then use as a param in a subsequent task. (This might be a feature request for Meta-Tasks?)

Scenario 1:

Use a Variable set from a previous PoSh step in a later PoSh step, _without using the -Arguments task parameter_

1. Release: no Variables, single Meta-Task Step
2. Meta-Task: no Properties (no $(variable)s used in task params, yet)
   
   
   
   1. AzurePowerShell: execute .ps1 (no arguments)
      
      
      
      
      
      1. write-host "##vso[task.setvariable variable=TestVar]TestVal"
      
      
      
   
   
   2. AzurePowerShell: execute .ps1 (no arguments)
      
      
      
      
      
      1. write-host ("TestVar: {0}" -f $env:TestVar)
      
      
      
   
   

This results in expected output in the 2.ii : "TestVar: TestVal"

Scenario 2:

Use a Variable set from a previous PoSh step in a later PoSh step, _as -Argument -TestVar $(TestVar), without editing the Release (trying to edit the Release will force you to enter a value for the new MT Property.)_

1. Release:
   
   
   
   1. no Variables, single Meta-Task Step (don't update Release after updating MT.)
   
   
2. Meta-Task: 1 Property: TestVar, DefaultValue: {empty}
   
   
   
   1. AzurePowerShell: execute .ps1 (no arguments)
      
      
      
      
      
      1. write-host "##vso[task.setvariable variable=TestVar]TestVal"
      
      
      
   
   
   2. AzurePowerShell: execute .ps1 (1 argument: "-myTestVar $(TestVar)")
      
      
      
      
      
      1. Param( [string]$myTestVar )
      
      
      
      2. write-host ("TestVar: {0}" -f $myTestVar)
      
      
      
   
   

This results in expected output in the 2.ii : "TestVar: TestVal"
Also, in the Log for that Step, you can see the platform make the call to the .ps1, passing the value set from 2.i: timestamp ##[command]& 'C:\apps\buildagent\_work\9384aa42c\folder\myScript.ps1' -myTestVar TestVal

Scenario 3:

Use a Variable set from a previous PoSh step in a later PoSh step, _as -Argument -myTestVar $(TestVar), after editing the Release to supply a default value (because it does not allow empty string)_

1. Release:
   
   
   
   1. Meta-Task Step,
      
      
      
      
      
      1. TestVar Property: $(TestVar) DefaultValue: "will overwrite"
      
      
      
   
   
2. Meta-Task: 1 Property: TestVar, DefaultValue: {empty}
   
   
   
   1. AzurePowerShell: execute .ps1 (no arguments)
      
      
      
      
      
      1. write-host "##vso[task.setvariable variable=TestVar]TestVal"
      
      
      
   
   
   2. AzurePowerShell: execute .ps1 (no arguments)
      
      
      
      
      
      1. write-host ("TestVar: {0}" -f $env:TestVar)
      
      
      
   
   
   3. AzurePowerShell: execute .ps1 (1 argument: "-myTestVar $(TestVar)")
      
      
      
      
      
      1. Param( [string]$myTestVar )
      
      
      
      2. write-host ("TestVar: {0}" -f $myTestVar)
      
      
      
      3. write-host ("TestVar: {0}" -f $env:TestVar)
      
      
      
   
   

This results in an expected output in the 2.ii AND 3.iii : "TestVar: TestVal"
However, in 3.ii when the platform makes the call to the script, the value being passed for myTestVar is "will overwrite", not "TestVal".

[bryanmacfarlane]

Adding RM who did meta tasks

[ericsciple]

it sounds like in scenario 3 the variable is getting expanded server side before the job message is sent down to the agent.

[JoeBrockhaus]

Thanks for the replies.

I've found a workaround for this.

1. In the Build or Release where the MT is used, create a variable with no value.
2. In the Meta-Task configuration, specify that variable for the MT Property (which is ultimately not needed in the parent).

I did not test if the variable/property names in the consumer (build/release) or MT can be named differently - in my case they are the same.

[JoeBrockhaus]

Related to this, I've also had trouble retrieving the value of a Property set in a MT, in a subsequent step in the consuming Build or Release. (ie: a MT sets a version number or something similar that would be used in another MT, for instance.) I didn't spend really any time on a thorough investigation though.

[harshil93]

@JoeBrockhaus Thanks for a detailed write up of the scenarios that you have tested. I reproduced scenario 3. I think Eric is right that the variable replacement is happening server side where meta tasks are expanded to tasks because of which the variables are replaced before even the build/release has started.

The workaround that you have mentioned circumvents this by using a variable in MT property in a release/build. So when the replacement has happened it has replaced with the value which itself contains a variable.

Instead of directly creating a release, create a draft release, then open that draft release. The draft release will show all the meta tasks expanded, you can easily inspect the value substitutions.

About retrieving the value of a variable (not property) set in a MT, this should work properly. If you set something in MT1 by using WriteHost setvariable and then you can use that value in subsequent tasks by using it like $(varName).

As a side thought, I don't know whether its a good idea to have meta task which is dependent on another meta task for setting a variable as it is not explicit. We do have a story coming up which will allow tasks to explicitly set output variables but I don't have the timelines.

Let me know if this solves your issue or not.

[JoeBrockhaus]

@harshil93 Thanks for the feedback. Creating a Draft immediately de-mystified the Meta-Task dereferencing, so that was helpful. This allows us to modify those values when creating manual releases. The caveat of needing to create the (extraneous) variable in the Build/Release is an easy-enough workaround for this.

It would be really helpful if the MetaTask Properties could be configured so that they are not required to have a value configured when added to a build or release, and even be hidden.

For instance, if inside the MetaTask I create $(MetaTaskCalculatedProp) property from a concatenation of $(BuildInputProp1) and $(BuildInputProp2) and then use that in another step, the consuming build or release must now go and create a variable to hold an empty value and use that in the MetaTask configuration for that calculated property. I cannot leave it empty because you cannot save the build or release. I cannot give is a default (bogus) value because of the issue described here. Since $(MetaTaskCalculatedProp) is not relevant to the build or release adding the MetaTask, it should not have to configure it. (Though it would make sense for a resulting Draft Release to contain a 'variable' for that auto-detected-property.)

[JoeBrockhaus]

Also just to clarify what I meant about (dynamically created) variables from a MetaTask being available in subsequent build tasks - the dependency was not between multiple Meta-Tasks, but just another task in the Build/Release.

For instance, when using the GitVersion task, in a following task you may want to leverage the variables it creates (FullSemVer, etc). Similarly, for Meta-Tasks, there are variables, like in our use case which should not be required for the build/release to know about (or provide a value for during configuration), but should be able to use if they know about them in later steps. Perhaps I was doing something wrong (and I did not test this thoroughly), but I was unable to in this case. I'm pretty sure I was 'creating' and using the variables dynamically inside a powershell step, so the MT did not auto-create a property. (I will acknowledge that knowing a bit more how Meta-Tasks are dereferenced makes me question the behavior I saw, but I just thought it was worth mentioning in case it's related to the same bug that causes the wrong -Argument variable expansion in this instance.)

[davidlbyrne]

I'm trying to set a variable from my inline powershell script Write-Host "##vsotask.setvariable variable=git_date[2]"

What I'm getting is the value of git _data is the string literal "(Get-Content commit.log)[2]" and not the value that should be evaluated by the expression. Any ideas? I can do this easily in bash but ps is new to me.

[GitHubSriramB]

@TingluoHuang @ericsciple - is executing PS commands supported within ##vso commands?

[ericsciple]

@davidlbyrne this is the syntax you want: Write-Host "##vso[task.setvariable variable=git_date]$((Get-Content commit.log)[2])"

Please open a separate thread if that doesn't work. This thread is too long and more about task groups. Also this sounds more like a generic powershell question.

[ericsciple]

here is a blog post I found about powershell string expansion

[manishsat]

I have 2 powershell scripts
On macOS
PowerShell script 1
Write-Host "##vso[task.setvariable variable=myvar;]$value"
The above line worked under macOS.
I could see ##[debug]Processed: ##vso[task.setvariable variable=myvar;]/Users/abc/ in the log file
PowerShell script 2
$env:myvar ----> This does not get any value

On Windows it works fine and the PowerShell script 2 does get the proper value of "myvar"

[manishsat]

Both PowerShell scripts are added as a inline PowerShell scripts in vsts build definition.
PowerShell script 2 added after PowerShell script 1

[manishsat]

Fixed !!
the variable name was in all CAPS in this case it was MYVAR
on windows I could access by $env:myvar but not in the case of macOS

[W0GER]

Starting 11/20 we started seeing the same issue as originally reported:

In a Meta-Task, we've got a Property/Variable defined that needs to be set from within an AzurePowerShell script.
We've verified that the script is writing the correct resulting string with Write-Host, however the next step in the Meta-Task (executed during a Release) is not getting the new value set in the previous step.

[tmakin]

I've just run in to this and I can confirm that the workaround recommended by @JoeBrockhaus fixes it.

In my case I have a variable called $(ContentContainer) which is set by powershell inside the task group. I created a release variable called $(ContentContainer) with no value and passed that into the task group from the pipeline. Messy, but it works.

I suggest that task groups should allow parameters to be marked as Internal, meaning that they not exposed in the pipeline parameter dialog.

[LockTar]

Workaround of @JoeBrockhaus works!

[Lovakumar]

Starting 11/20 we started seeing the same issue as originally reported:

In a Meta-Task, we've got a Property/Variable defined that needs to be set from within an AzurePowerShell script.
We've verified that the script is writing the correct resulting string with Write-Host, however the next step in the Meta-Task (executed during a Release) is not getting the new value set in the previous step.

This is an enhancement request hence added enhancement label.

[ddivita]

I am not having luck with the workaround, but I may have a different problem.

I setup a release pipeline with multiple stages. One stage sets a task variable I need in subsequent stages. I tried setting an empty variable with release scope, but the variable is only available in the stage that creates it. I then tried setting the scope to the stage that requires it, but that did not work either. The variable is getting set and the value is correct in the stage that creates it.

One thing to note, each stage is made up of task groups, so I then tired to set the variable outside the task group, but again it was only available to the stage that created it,

[AndyP2]

I think I've found a simpler workaround. Assuming you have a TaskGroup (a.k.a. MetaTask) with:

• a task that sets $(myvariable) with something like
  Write-Host "##vso[task.setvariable variable=myvariable;]banana"
• another task that consumes the $(myvariable) value later

The TaskGroup will add a parameter for $(myvariable); give it a default value of the literal string $(myvariable). When the release is created, the calling plan or task group won't recognise it to substitute it, leaving $(myvariable) as the value of the parameter.
When the second task is executed the desired value banana _is_ substituted into this variable, and the caller has not needed to know anything of this at all.

[michaelpisula]

@AndyP2 That workaround does not work on Azure DevOps Server for me for some reason. Even though I set the variable in a task, the second task uses the literal value $(myvariable). Have you had that problem? Or maybe it is a different behavior between versions.
Edit: turns out, it does work, for some reason I had the isOutput option in my Powershell script where I set the line, and that worked fine in normal releases, but not when using task groups shrug

[AndyP2]

I'm using Azure DevOps Service, so that might be the difference, but I have just recreated the scenario and it is working as expected:

• TaskGroup
  
  
  
  • one parameter 'myvariable' with default value $(myvariable)
  
  
  • "task 1: set" powershell inline script:
    
    Write-Host 'setting myvariable=banana'

Write-Host '##vso[task.setvariable variable=myvariable;]banana'
  
  
  • "task 2: display" powershell inline script:
    
    Write-Host 'myvariable is now $(myvariable)'
  
  
• Release pipeline with one stage, one agent, one task which is the taskgroup:
  
  
  
  • one parameter 'myvariable' with value $(myvariable) as supplied by the default
  
  
• Output:
  
  
  
  • "task 1: set"
    
    setting myvariable=banana
  
  
  • "task 2: display"
    
    myvariable is now banana
  
  

[bmccabe]

I just ran into this "Problem" and it took me a while to find the cause and read this thread.
I have a lot of Task Groups in both Build and Release pipelines and the work-arounds noted here require an increasing number of Build and Release variables. Being able to mark Task Group variables as "Internal" would solve this. Is there any consideration to implementing this?

[leantk]

This issue is stale because it has been open for a year with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days