Azure-pipelines-tasks: FileTransformV1 doesn't support using variables starting with "Build." or "System."

[johncrim]

Question, Bug, or Feature?
Bug

Task Name: FileTransformV1

Environment

• Server: Azure Pipelines
• Agent - Hosted, ubuntu-16.04

Issue Description

Using the File Transform task to replace values in a JSON file:

  jobs:
  - job: CI_Build
    pool:
      vmImage: 'ubuntu-16.04'

    variables:
      # Set build variables written to build-metadata.json file
      Build.Id: $(Build.BuildId)
      Build.PipelineName: $(System.DefinitionName)
      Build.JobName: $(System.JobDisplayName)
      Build.Timestamp: $(System.PipelineStartTime)

    steps:
    - task: FileTransform@1
      displayName: 'Update build-meta.json file'
      inputs:
        folderPath: 'build'
        fileType: json
        targetFiles: 'build-metadata.json'

And a build-metadata.json JSON file like:

{
  "Build": {
    "Id": "-1",
    "PipelineName": "??",
    "JobName": "??",
    "Timestamp": "2018-08-01 16:13:00-07:00"
  },
  "GitVersion": {
    "SemVer": "0.1.0-dev",
    "Sha": "??????????????????????????????????",
    "ShortSha": "???????",
    "BranchName": "??"
  }
}

I expected all the JSON property values to be filled in. The values within my GitVersion block were filled in as expected; the values within my Build block were not.

There is no apparent good reason to prevent the use of Build or System variables in the FileTransform task. If there is, then the reason should be documented, and a task property/setting should exist that enables replacing properties with "well known" prefixes.

Task logs

No log output shows variables that aren't replaced, so logging isn't helpful here.

I determined that a bug exists by reviewing the source:

https://github.com/microsoft/azure-pipelines-tasks/blob/ccb5852d8d2cd714879559da2013b9211c381966/Tasks/Common/webdeployment-common-v2/jsonvariablesubstitutionutility.ts#L21

        if(varUtility.isPredefinedVariable(envVariable.name)) {
            continue;
        } 

and https://github.com/microsoft/azure-pipelines-tasks/blob/ccb5852d8d2cd714879559da2013b9211c381966/Tasks/Common/webdeployment-common/variableutility.ts#L3

export function isPredefinedVariable(variable: string): boolean {
    var predefinedVarPrefix = ['agent.', 'azure_http_user_agent', 'build.', 'common.', 'release.', 'system', 'tf_'];
    for(let varPrefix of predefinedVarPrefix) {
        if(variable.toLowerCase().startsWith(varPrefix)) {
            return true;
        }
    }
    return false;
}

shows that any variables with this set of common prefixes are dropped on the ground, and not logged. From the standpoint of developers trying to insert values into files, these exclusions are arbitrary, mysterious, and problematic.

[SumiranAgg]

@johncrim Built-in variables are excluded in JSON variable substitution.
image

We exclude such variables that has the same prefix as pre-defined variables.
@vincentdass @N-Usha Please share your thoughts on how can we make this behaviour more transparent. Also should we introduce some toggle to callout that this is custom defined variable?

[johncrim]

The same note is shown on the File Transform docs page, if you look hard enough:

Note: Only custom variables defined in build and release pipelines are used in substitution. Default and system pipeline variables are excluded. If the same variables are defined in the release pipeline and in a stage, the stage-defined variables supersede the pipeline-defined variables.

The first problem with this is that system variables can be useful in an output - so this decision is arbitrary. Consider this issue a plea to reconsider this - developers should be able to use system variables in files.

The second problem is that there is nothing in Azure Pipelines preventing users from defining variables that start with "Build.*" or "Release.*". The implementation doesn't actually exclude the standard predefined variables, it excludes everything with a prefix used by a standard predefined variable. These name prefixes are useful for developers trying to make accurate or descriptive hierarchical variable names. However, if they use these names, they will not be included in the file transform output.

If the restricted namespace prefixes were "System." or "AzurePipelines." or "VSTS.", then this restriction would be more acceptable.

My recommendation (for backward compatibility) is to add a task option to enable these prefixes, and all system variables, to be included.

[TK-Dev-1]

Is there a reason why we shouldn't be exposing the predefined variables in this task?

I'd like to request that we are provided a toggle in the parameters to allow predefined variable substitution. At this time I'm having to build a large manual mapping of predefined variables to custom variables with new prefixes to allow my devs to use these variables in their file transforms.

[github-actions[bot]]

This issue is stale because it has been open for a year with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[johncrim]

There's no good reason that this issue is stale, besides a lack of movement on the AzP side. Again, the request, is not to prevent substitution of system variables, and not to prevent developers from using values like:

    variables:
      # Set build variables written to build-metadata.json file
      Build.Id: $(Build.BuildId)
      Build.PipelineName: $(System.DefinitionName)
      Build.JobName: $(System.JobDisplayName)
      Build.Timestamp: $(System.PipelineStartTime)

And, if you are going to prevent using values with a Build or Release prefix, you should dump warnings to the console when they are skipped.