Azure-pipelines-tasks: FileTransform task not working as expected for appsettings.json file variable substitution

[itorian]

I know how variable substitution works in release pipeline, just check "XML variable substitution" for XML files or write json filename in "JSON variable substitution" textbox. This works well if I'm deploying to "Web App on Windows" app service type. But as soon as I select "Web App on Linux" app service type "File Transforms & Variable Substitution Options" group disappears.

Then I decided to move variable substitution in build pipeline in YAML file (azure-pipelines.yml), that's where problem started for me.

So, i'm trying to achieve appsettings.json file variable substitution in build pipeline. I already listed variables in library:

Here's the json file:

{
  "Logging": {
    "LogLevel": {
      "Default": "Warning"
    }
  },
  "AllowedHosts": "*",
  "name": "value"
}

Here's the code i have in azure-pipelines.yml:-

trigger:
 - master

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'

steps:
- task: DotNetCoreCLI@2
  displayName: 'Restore project'
  inputs:
    command: 'restore'
    projects: |
      **/WebApplication_Li_CI_CD.csproj
      **/WebApplication_Li_CI_CD_1.csproj
    feedsToUse: 'select'

- task: DotNetCoreCLI@2
  displayName: 'Build project'
  inputs:
    command: 'build'
    projects: |
      **/WebApplication_Li_CI_CD.csproj
      **/WebApplication_Li_CI_CD_1.csproj

- task: DotNetCoreCLI@2
  displayName: 'Publish project'
  inputs:
    command: publish
    projects: |
      **/WebApplication_Li_CI_CD.csproj
      **/WebApplication_Li_CI_CD_1.csproj
    publishWebProjects: True
    arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
    zipAfterPublish: True

- task: CopyFiles@2
  displayName: 'Copy zip files'
  inputs:
    SourceFolder: '$(Build.SourcesDirectory)'
    Contents: '**\*.zip'
    TargetFolder: '$(Build.ArtifactStagingDirectory)'

- task: PublishBuildArtifacts@1
  displayName: 'Publish artifacts'


- task: FileTransform@1
  displayName: 'Update appsettings.json file'
  inputs:
    folderPath: '$(Build.ArtifactStagingDirectory)'
    fileType: 'json'
    targetFiles: '**/appsettings.json'

I tried to adjust folderPath in FileTransformation task, and received errors:

#folderPath: '$(System.DefaultWorkingDirectory)/**/*.zip'
Error: Nopackagefoundwithspecifiedpattern /home/vsts/work/1/s/**/*.zip

#folderPath: '$(System.DefaultWorkingDirectory)/_WebApplication_Linux_CI_CD/drop/WebApplication_Li_CI_CD.zip'
Error: Nopackagefoundwithspecifiedpattern /home/vsts/work/1/s/_WebApplication_Linux_CI_CD/drop/WebApplication_Li_CI_CD.zip

#folderPath: '$(Build.ArtifactStagingDirectory)/**/*.zip'
Error: More than one package matched with specified pattern: /home/vsts/work/1/a/**/*.zip. Please restrain the search pattern.

#folderPath: '$(Build.ArtifactStagingDirectory)/**/WebApplication_Linux_CI_CD.zip'
Error: Nopackagefoundwithspecifiedpattern /home/vsts/work/1/a/**/WebApplication_Linux_CI_CD.zip

#folderPath: '$(Build.ArtifactStagingDirectory)/_WebApplication_Linux_CI_CD/drop/WebApplication_Li_CI_CD.zip'
Error: Nopackagefoundwithspecifiedpattern /home/vsts/work/1/a/_WebApplication_Linux_CI_CD/drop/WebApplication_Li_CI_CD.zip

#folderPath: '$(Build.ArtifactStagingDirectory)/WebApplication_Linux_CI_CD.zip'
Error: Nopackagefoundwithspecifiedpattern /home/vsts/work/1/a/WebApplication_Linux_CI_CD.zip

#folderPath: '$(Build.ArtifactStagingDirectory)/drop/WebApplication_Linux_CI_CD.zip'
Error: Nopackagefoundwithspecifiedpattern /home/vsts/work/1/a/drop/WebApplication_Linux_CI_CD.zip

#folderPath: '$(Build.ArtifactStagingDirectory)/drop/**/WebApplication_Linux_CI_CD.zip'
Error: Nopackagefoundwithspecifiedpattern /home/vsts/work/1/a/drop/**/WebApplication_Linux_CI_CD.zip

And none of the combination works for me.

[20shivangi]

@itorian We are working on the fix. Will let you know once the fix is done

[itorian]

@20shivangi can you please confirm if this is a product issue ?

[eaarora-ms]

@itorian Hi, Azure App Service deploy does not support Variable Substitution for Linux apps. Also, we have deprecated that functionality in the task. We have a new task: File Transformation (https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/FileTransformV2) for file transformation and variable substitution now which works with all app types since it takes a package as it's input.
As for your query, I will check the file path error and get back to you.

[itorian]

@eaarora-ms i'm not using Variable Substitution, what i'm using is FileTransform task to replace appsettings.json variables.

"name": "value" should become "name": "value-updated" in released package.

You can very easily reproduce this issue using code snippet which I shared above.

[itorian]

Any update ?

[eaarora-ms]

@itorian I am sorry for the delay in response. I just tried to run the task and I was able to do that successfully. Sharing the YAML script and a few points due to which the issue could be occurring.

YAML snippet:

steps:
- task: FileTransform@1
  displayName: 'Update appsettings.json file'
  inputs:
    folderPath: '$(System.DefaultWorkingDirectory)/_adcf/Application.zip'
    fileType: json
    targetFiles: Application/appsettings.json

My folder format:

- Application.zip
    - appsettings.json
    - ...

Can you tell me if this was of any help?

[eaarora-ms]

@itorian I am sorry for the delay in response. I just tried to run the task and I was able to do that successfully. Sharing the YAML script and a few points due to which the issue could be occurring.

YAML snippet:

steps:
- task: FileTransform@1
  displayName: 'Update appsettings.json file'
  inputs:
    folderPath: '$(System.DefaultWorkingDirectory)/_adcf/Application.zip'
    fileType: json
    targetFiles: Application/appsettings.json

My folder format:

- Application.zip
    - appsettings.json
    - ...

Can you tell me if this was of any help?

So basically the project path should be: $(System.DefaultWorkingDirectory)/{your_project_name}/{your_zip_folder_name}

[eaarora-ms]

2019-11-20T11:29:01.1408454Z Task : File transform
2019-11-20T11:29:01.1408490Z Description : Replace tokens with variable values in XML or JSON configuration files
2019-11-20T11:29:01.1408514Z Version : 1.156.3
2019-11-20T11:29:01.1408551Z Author : Microsoft Corporation
2019-11-20T11:29:01.1408574Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/file-transform
2019-11-20T11:29:01.1408601Z ==============================================================================
2019-11-20T11:29:01.7685420Z Applying JSON variable substitution for Application/appsettings.json
2019-11-20T11:29:01.7692899Z Applying JSON variable substitution for C:\Users\eaarora\myagent_work_temp\temp_web_package_9938779557808703\Application\appsettings.json
2019-11-20T11:29:01.7694874Z Substituting value on key testString with (string) value: testValue
2019-11-20T11:29:01.7718134Z JSON variable substitution applied successfully.
2019-11-20T11:29:01.8990343Z ##[section]Finishing: Update appsettings.json file

[itorian]

Thanks for the update.

We had complex deployment (if you can re-read my question) like one YAML file for .NET Framework apps, and another for .NET Core apps for which we used projects filtering in restore command task, build command task and publish command task. In this scenario, i still can't see it working.

So, we started using Azure App Service which can override appsettings.json keys and now we don't need FileTransform task.

[mliesmons]

@itorian Hi, Azure App Service deploy does not support Variable Substitution for Linux apps. Also, we have deprecated that functionality in the task. We have a new task: File Transformation (https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/FileTransformV2) for file transformation and variable substitution now which works with all app types since it takes a package as it's input.
As for your query, I will check the file path error and get back to you.

Hi Arora,
There is still a bug in the new FileTransform.
I was already worried to see in the log only the leaf keys, and not the full hierarchy of which key is being replaced.
I have the key ApiKey twice in my JSON but in a different hierarchy.
This fails, the last ApiKey found is replaced with its current value, a noop in fact.
Another key, with the same parent, is simply not logged/replaced, no clue why.
Honestly, I am wasting precious time on what should be a simple task...
Michel.

[mliesmons]

Arora,
After another couple of hours wasted...
It seems when a your JSON contains a child called 'Common', its children get never substituted.
I renamed it to 'Commons' and everything works as expected, go figure.
Story is the same for both the FileTransform task as well as for this option in the App Service Deploy task, same code base?
{
"Commons": {
"Child": "substitution works"
},
"Common": {
"Child": "substitution does NOT work"
}
}

[eaarora-ms]

@mliesmons Built-in variables are excluded in JSON variable substitution.

We exclude such variables that have the same prefix as pre-defined variables.

Reference: https://github.com/microsoft/azure-pipelines-tasks/blob/e6b2859cadd9e69b964dbb5b425720b7172d1abf/Tasks/Common/webdeployment-common-v2/variableutility.ts#L3

[mliesmons]

Hi Arora,

My variable entered as the environment variable [COMMON_CHILD], please explain how this is considered a built-in?

'azure_http_user_agent', 'common.', 'tf_' are not mentioned in; https://docs.microsoft.com/en-us/azure/devops/pipelines/release/variables?view=azure-devops&tabs=batch

Kr,
Michel.

From: Eavanshi Arora [e-mail]
Sent: zaterdag 4 januari 2020 17:07
To: microsoft/azure-pipelines-tasks [e-mail]
Cc: Michel Liesmons [e-mail]; Mention [e-mail]
Subject: Re: [microsoft/azure-pipelines-tasks] FileTransform task not working as expected for appsettings.json file variable substitution (#11493)

@mliesmonshttps://github.com/mliesmons Built-in variables are excluded in JSON variable substitution.
[62449643-4ace9a80-b788-11e9-842d-5738608f6cfc]https://user-images.githubusercontent.com/52486630/71768217-37dc1400-2f3a-11ea-8e52-df4a3a6bdcb4.png

We exclude such variables that have the same prefix as pre-defined variables.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/azure-pipelines-tasks/issues/11493?email_source=notifications&email_token=ABAUWCDD2KZQYV2TC5MNV2LQ4CXYVA5CNFSM4I4772A2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIC2Y6I#issuecomment-570797177, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABAUWCCTDHDFJDSFZWUNKDLQ4CXYVANCNFSM4I4772AQ.

[eaarora-ms]

@mliesmons The documentation is definitely pending behind and it is in our backlogs to update the same. However as per the code here :

https://github.com/microsoft/azure-pipelines-tasks/blob/e6b2859cadd9e69b964dbb5b425720b7172d1abf/Tasks/Common/webdeployment-common-v2/variableutility.ts#L3

The words starting with common, build etc. are not replaced in variable substitution.

[mliesmons]

Hi Arora,

Thanks, I see it in the code yes…

3 questions:

• You mentioned somewhere in this thread that the option to specify substitutions within the App Service Deployment task will be deprecated in favor of the specific task. How should we know? Could this be indicated in the current task, e.g. when you click the ‘i’nfo button?
• I have a hard time consulting my customers on which tasks to use. E.g. Azure Functions: should we use AzureRmWebAppDeploymentV4 or AzureFunctionAppV1?
• What is the difference between App Settings and Configuration Settings, the help link brings us to the same page…
  https://stackoverflow.com/questions/55330076/what-is-the-difference-between-app-settings-and-configuration-settings-in-azure

Kr,
Michel.

From: Eavanshi Arora [e-mail]
Sent: zaterdag 4 januari 2020 19:42
To: microsoft/azure-pipelines-tasks [e-mail]
Cc: Michel Liesmons [e-mail]; Mention [e-mail]
Subject: Re: [microsoft/azure-pipelines-tasks] FileTransform task not working as expected for appsettings.json file variable substitution (#11493)

@mliesmonshttps://github.com/mliesmons The documentation is definitely pending behind and it is in our backlogs to update the same. However as per the code here :

https://github.com/microsoft/azure-pipelines-tasks/blob/e6b2859cadd9e69b964dbb5b425720b7172d1abf/Tasks/Common/webdeployment-common-v2/variableutility.ts#L3

The words starting with common, build etc. are not replaced in variable substitution.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/azure-pipelines-tasks/issues/11493?email_source=notifications&email_token=ABAUWCBKXT7NNVWQYRPWJTTQ4DKAJA5CNFSM4I4772A2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIC5UMY#issuecomment-570808883, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABAUWCDCUTY6YCUFP6L2273Q4DKAJANCNFSM4I4772AQ.

[eaarora-ms]

@mliesmons
You mentioned somewhere in this thread that the option to specify substitutions within the App Service Deployment task will be deprecated in favor of the specific task. How should we know? Could this be indicated in the current task, e.g. when you click the ‘i’nfo button?

We will add this to our documentation thanks for pointing out.

I have a hard time consulting my customers on which tasks to use. E.g. Azure Functions: should we use AzureRmWebAppDeploymentV4 or AzureFunctionAppV1?

AzureRmWebAppDeploymentV4 task is being deprecated and hence we would suggest you use newer tasks like AzureFunctionAppV1

What is the difference between App Settings and Configuration Settings, the help link brings us to the same page… https://stackoverflow.com/questions/55330076/what-is-the-difference-between-app-settings-and-configuration-settings-in-azure

Thanks for pointing us to the question. App settings are used to set the key value in app settings on the Azure portal however configuration settings are used for General settings. The Azure App Service Deploy does not support configuring connection strings. The configuration settings do not have a syntax because they can have a variety of values like a single value or an array, etc.

However, to clear the misconceptions we have added a new task called Azure App Service Settings which has three sections namely App Settings, General Settings and Connection Strings which clears all your doubts asked in this question.

[mliesmons]

Thanks for the clarification.
If the App Service Deployment task will be deprecated, will there be new tasks for Web and API App deployment?

I see the Azure App Service Settings has no appsettings.json variable substitution capability.
We often have large, hierarchical settings specified in an appsettings.json, substitution using variables worked pretty well here.
Copying them into a pipeline task is tedious and error prone.

Is this feature deprecated too?
What is the advised way to handle appsettings.json?

Kr,
Michel.

From: Eavanshi Arora [e-mail]
Sent: Monday, January 06, 2020 7:21 AM
To: microsoft/azure-pipelines-tasks [e-mail]
Cc: Michel Liesmons [e-mail]; Mention [e-mail]
Subject: Re: [microsoft/azure-pipelines-tasks] FileTransform task not working as expected for appsettings.json file variable substitution (#11493)

@mliesmonshttps://github.com/mliesmons
'''You mentioned somewhere in this thread that the option to specify substitutions within the App Service Deployment task will be deprecated in favor of the specific task. How should we know? Could this be indicated in the current task, e.g. when you click the ‘i’nfo button?'''

We will add this to our documentation thanks for pointing out.

''' I have a hard time consulting my customers on which tasks to use. E.g. Azure Functions: should we use AzureRmWebAppDeploymentV4 or AzureFunctionAppV1? '''

AzureRmWebAppDeploymentV4 task is being deprecated and hence we would suggest you use newer tasks like AzureFunctionAppV1

''' What is the difference between App Settings and Configuration Settings, the help link brings us to the same page… https://stackoverflow.com/questions/55330076/what-is-the-difference-between-app-settings-and-configuration-settings-in-azure '''

Thanks for pointing us to the question. App settings are used to set the key value in app settings on the Azure portal however configuration settings are used for General settings. The Azure App Service Deploy does not support configuring connection strings. The configuration settings do not have a syntax because they can have a variety of values like a single value or an array, etc.

However, to clear the misconceptions we have added a new task called Azure App Service Settings which has three sections namely App Settings, General Settings and Connection Strings which clears all your doubts asked in this question.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/azure-pipelines-tasks/issues/11493?email_source=notifications&email_token=ABAUWCCLUBLMXUX6LH7RX6TQ4LEVBA5CNFSM4I4772A2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIERFLA#issuecomment-571019948, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABAUWCFCJPHLP7X2SQHAZ7DQ4LEVBANCNFSM4I4772AQ.

[eaarora-ms]

@mliesmons We already have new tasks for the same viz Azure Web App, Azure Function App, etc.,

And variable substitution is not deprecated, it has moved to File Transform Task .

I hope you use these tasks in the future instead.

[mliesmons]

I will 😉

From: Eavanshi Arora [e-mail]
Sent: Monday, January 06, 2020 1:31 PM
To: microsoft/azure-pipelines-tasks [e-mail]
Cc: Michel Liesmons [e-mail]; Mention [e-mail]
Subject: Re: [microsoft/azure-pipelines-tasks] FileTransform task not working as expected for appsettings.json file variable substitution (#11493)

@mliesmonshttps://github.com/mliesmons We already have new tasks for the same viz Azure Web Apphttps://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/AzureWebAppV1/README.md, Azure Function App, etc.,

And variable substitution is not deprecated, it has moved to File Transform Task https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/FileTransformV2/README.md .

I hope you use these tasks in the future instead.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/azure-pipelines-tasks/issues/11493?email_source=notifications&email_token=ABAUWCHCOQDUG5W6VEN5HO3Q4MQAVA5CNFSM4I4772A2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIFJ7VA#issuecomment-571121620, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABAUWCDY4CIY3CI3IPSOZR3Q4MQAVANCNFSM4I4772AQ.