Azure-pipelines-agent: Some Pipeline variables not being resolved on Agent 2.165.x, works up through 2.164.8

[OranguTech]

Have you tried troubleshooting?

Troubleshooting doc
Yes, system.debug=true

Agent Version and Platform

Version of your agent? 2.165.0/2.165.1

OS of the machine running the agent?
Windows, mostly Server 2016 but tested on Windows 10 1809 as well.

Azure DevOps Type and Version

dev.azure.com

If dev.azure.com, what is your organization name? https://dev.azure.com/{organization} or

'REDACTED' for now.

What's not working?

"A lot" of variables from our libraries aren't getting enumerated/expanded upon initialization, where they were before. I've narrowed down the issue to version of agent (on-prem), between 2.164.8 and 2.165.0.

Examples (potentially sensitive data obfuscated):
Working:

2020-03-06T00:56:37.2355860Z ##[section]Starting: Initialize job
2020-03-06T00:56:37.2356368Z Agent name: 'ORG7IT11919 - 2.164.8'
2020-03-06T00:56:37.2356606Z Agent machine name: 'ORG7IT11919'
2020-03-06T00:56:37.2356759Z Current agent version: '2.164.8'
2020-03-06T00:56:37.2371109Z Agent running as: 'ORG7IT11919$'
2020-03-06T00:56:37.2394436Z Prepare release directory.
2020-03-06T00:56:37.2407267Z ReleaseId=7315, TeamProjectId=40854903-a7a9-4867-95a8-d5cd31de25d0, ReleaseDefinitionName=XKCD.Configuration
2020-03-06T00:56:37.2694988Z Release folder: C:\xfer\vsts-agent-win-x64-2.164.8\_work\r1\a
2020-03-06T00:56:37.2927325Z Environment variables available are below.  Note that these environment variables can be referred to in the task (in the ReleaseDefinition) by replacing "_" with "." e.g. AGENT_NAME environment variable can be referenced using Agent.Name in the ReleaseDefinition: 
                [AGENT_DISABLELOGPLUGIN_TESTFILEPUBLISHERPLUGIN] --> [true]
                [AGENT_DISABLELOGPLUGIN_TESTRESULTLOGPLUGIN] --> [true]
                [AGENT_HOMEDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.164.8]
                [AGENT_ID] --> [226]
                [AGENT_JOBNAME] --> [Agent job]
                [AGENT_MACHINENAME] --> [ORG7IT11919]
                [AGENT_NAME] --> [ORG7IT11919 - 2.164.8]
                [AGENT_OS] --> [Windows_NT]
                [AGENT_OSARCHITECTURE] --> [X64]
                [AGENT_RELEASEDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.164.8\_work\r1\a]
                [AGENT_RETAINDEFAULTENCODING] --> [false]
                [AGENT_ROOTDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.164.8\_work]
                [AGENT_SERVEROMDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.164.8\externals\vstsom]
                [AGENT_TEMPDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.164.8\_work\_temp]
                [AGENT_TOOLSDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.164.8\_work\_tool]
                [AGENT_VERSION] --> [2.164.8]
                [AGENT_WORKFOLDER] --> [C:\xfer\vsts-agent-win-x64-2.164.8\_work]
                [AUTH_AD_GROUP_DEVELOPERS] --> [ORG7\G-S-ORG7 XKCD Information Technology User]
                [AUTH_AD_GROUP_TESTERS] --> [ORG7\G-S-ORG7 XKCD Testing User]
                [AUTH_AM_ACTIVEDIRECTORYTENANTID] --> [https://login.microsoftonline.com/org7.onmicrosoft.com]
                [AUTH_AM_AD_ROOTURL] --> [https://login.windows.net]
                [AUTH_AM_ADFS_ROOTURL] --> [https://xkcd.sts.org7.gov]
                [AUTH_AM_ADTENANT] --> [https://login.windows.net/AGUID]
                [AUTH_AM_ADTENANTID_ADFS] --> [https://xkcd.sts.org7.gov/adfs]
                [AUTH_AM_ADTENANTID_AZUREAD] --> [https://login.windows.net/AGUID]
                [AUTH_AM_DATASYNC_USERNAME] --> [ORG7\svcAMDataSyncTORG]
                [AUTH_AM_TENANTID] --> [dadb1cce-67d9-457a-940f-b1a34e248fd6]
                [AUTH_BATCH_AMCREDENTIALSOURCE] --> [AzureActiveDirectory]
                [AUTH_BATCH_AOLCLIENTAPPID] --> [e0d70f83-c116-4bcb-ada1-7b0714928224]
                [AUTH_BATCH_DORAPI_USERNAME] --> [ORGtestaccount]
                [AUTH_BENEFITACCOUNT_HOST_ENCRYPTION_IDENTIFIER] --> [10-29-2019-17-40-16]
                [AUTH_BENEFITS_QUEUESERVICE_USERNAME] --> [ORG7\svcXKCDQSvcTORG]
                [AUTH_AOL_ADMIN_USERNAME] --> [ORG7\svcAOLOrgAdmTORG]
                [AUTH_AOL_AM_CLIENTAPPID] --> []
                [AUTH_AOL_AM_CLIENTSECRET] --> []
                [AUTH_AOL_AMTENANTID] --> [https://login.windows.net/c3dd5aab-8028-4b27-8234-5ef09229feca]
                [AUTH_AOL_DATASYNC_USERNAME] --> [ORG7\svcAOLDataSyncTORG]
                [AUTH_AOL_TESTING_ADMIN_USERNAME] --> [ORG7\svcXKCDDynTest0ORG]
                [AUTH_AOL_TESTING_INTEGRATIONS_USERNAME] --> [ORG7\svcXKCDDynTest6ORG]

Example not working/change in behavior:

2020-03-06T01:09:02.0484645Z ##[section]Starting: Initialize job
2020-03-06T01:09:02.0486050Z Agent name: 'ORG7IT11919 - 2.165.1'
2020-03-06T01:09:02.0486738Z Agent machine name: 'ORG7IT11919'
2020-03-06T01:09:02.0487199Z Current agent version: '2.165.1'
2020-03-06T01:09:02.0499850Z Agent running as: 'ORG7IT11919$'
2020-03-06T01:09:02.0509420Z Prepare release directory.
2020-03-06T01:09:02.0515576Z ReleaseId=7315, TeamProjectId=40854903-a7a9-4867-95a8-d5cd31de25d0, ReleaseDefinitionName=XKCD.Configuration
2020-03-06T01:09:02.0711345Z Release folder: C:\xfer\vsts-agent-win-x64-2.165.1\_work\r1\a
2020-03-06T01:09:02.1000680Z Environment variables available are below.  Note that these environment variables can be referred to in the task (in the ReleaseDefinition) by replacing "_" with "." e.g. AGENT_NAME environment variable can be referenced using Agent.Name in the ReleaseDefinition: 
                [AGENT_DISABLELOGPLUGIN_TESTFILEPUBLISHERPLUGIN] --> [true]
                [AGENT_DISABLELOGPLUGIN_TESTRESULTLOGPLUGIN] --> [true]
                [AGENT_HOMEDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.165.1]
                [AGENT_ID] --> [227]
                [AGENT_JOBNAME] --> [Agent job]
                [AGENT_MACHINENAME] --> [ORG7IT11919]
                [AGENT_NAME] --> [ORG7IT11919 - 2.165.1]
                [AGENT_OS] --> [Windows_NT]
                [AGENT_OSARCHITECTURE] --> [X64]
                [AGENT_RELEASEDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.165.1\_work\r1\a]
                [AGENT_RETAINDEFAULTENCODING] --> [false]
                [AGENT_ROOTDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.165.1\_work]
                [AGENT_SERVEROMDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.165.1\externals\vstsom]
                [AGENT_TEMPDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.165.1\_work\_temp]
                [AGENT_TOOLSDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.165.1\_work\_tool]
                [AGENT_VERSION] --> [2.165.1]
                [AGENT_WORKFOLDER] --> [C:\xfer\vsts-agent-win-x64-2.165.1\_work]
                [AUTH_AD_GROUP_DEVELOPERS] --> [ORG7\G-S-ORG7 XKCD Information Technology User]
                [AUTH_AD_GROUP_TESTERS] --> [ORG7\G-S-ORG7 XKCD Testing User]
                [AUTH_AM_ACTIVEDIRECTORYTENANTID] --> [https://login.microsoftonline.com/org7.onmicrosoft.com]
                [AUTH_AM_AD_ROOTURL] --> [https://login.windows.net]
                [AUTH_AM_ADFS_ROOTURL] --> [https:***@ORG.xkcd.gov]
                [AOL_ILINX_URL] --> [https://testImaging/?mode=internalopen&appname=XKCD&docid=]
                [AOL_ORGANIZATION] --> [test2]
                [AOL_ORGANIZATION_ADMIN_ID] --> [ORG7\svcAOLOrgAdmTORG]
                [AOL_PORTAL_BASE_DEEPLINK] --> [https://test2.ORG7.ca.lcl/Staff/Routing?]

Agent and Worker's Diagnostic Logs

Local agent logs available on request, likely will have to sanitize them before posting publicly.

This looks likely related to issue #2839 . Possibly due to .NET core 3.1?

[damccorm]

Hey @OranguTech - would definitely be helpful if you're able to post logs. From your description, it looks like the issue you're seeing is that variables aren't getting propogated through - how are you setting those variables?

Would also be helpful to have your organization name if possible so that I can determine what bits you were getting from the service. If you're not comfortable sharing any of this publicly, feel free to email me at [email protected]

[OranguTech]

I replied in email earlier with more details and logs.
(Seems to be only an issue in the “initialize job” section of the “Agent job” task, it is not outputting all the variables as it did before, but the values are getting evaluated in the pipeline correctly.)

Further testing and better looking at logs, I think it's worthwhile to look at PR #2791 "mask secrets in URLs" .

Example previously
[ENV_ADFS_OAUTHTOKENSERVERURL] --> [https://abc.xyz.xkcd.gov/adfs/oauth2/token]

is now obfuscated to [ENV_ADFS_OAUTHTOKENSERVERURL] --> [https:***@xkcd.gov], and then a whole giant chunk of vars aren't enumerated.

Similary:
[ENV_CRM_SOAP_URL] --> [https://test2.dumbcrm-test.org7.xkcd.lcl/XRMServices/2011/Organization.svc] --> [ENV_CRM_SOAP_URL] --> [https:***@org7.xkcd.gov] then a whole new chunk missing.

It does this 3 times in our set of variables. I don't know if the issue is the regex or what it does once the regex is tripped.

[damccorm]

That definitely looks like the issue - I think the regex is overly aggressive here. For example, if you have:

https://abc.def
https://ghi.jkl
https://mno.pqr
https://[email protected]

the following gets masked by our regex ((?<=//[^:/?#]+:)[^@]+(?=@))

//ghi.jkl
https://mno.pqr
https://password

I think the regex should be (?<=//[^:/?#\n]+:)[^@]+(?=@) (note the added \n which stops looking for a match if we reach the EOL)

I'll add a PR

[OranguTech]

Thanks for the quick response.

If I wanted to try compiling my own copy, would compiling from Master be likely to work, or should I just make your change on current release branch?

[damccorm]

Either should work.

[OranguTech]

For completeness, linking PR #2844

[damccorm]

@OranguTech Thanks! I should've linked them.

Were you able to compile an agent and verify this fixes the issue?

[OranguTech]

I have not - opening the solution file immediately have errors about nuget
package versions that a simple restore or update doesn't fix. I'm pretty
new to the VS world, and it's not a blocking issue, so left it alone.

On Tue, Mar 10, 2020 at 7:19 PM Danny McCormick [e-mail]
wrote:

@OranguTech https://github.com/OranguTech Thanks! I should've linked
them.

Were you able to compile an agent and verify this fixes the issue?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/azure-pipelines-agent/issues/2842?email_source=notifications&email_token=AAM44DNS4D3SCK5YRER3WYDRG3YJVA5CNFSM4LCWTLYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEON36AY#issuecomment-597409539,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAM44DPX6DBC4OWSHYTDOLDRG3YJVANCNFSM4LCWTLYA
.

[OranguTech]

(I'm assuming the fix will be in the next release)

[OranguTech]

@damccorm - Is there a general release schedule? I know this got merged into Master, I assume that will be branched into 2.165.3, but not sure on your dev/branch process. ( I'm just trying to get an ETA on this.)

[damccorm]

@OranguTech normally we deploy at least once every 3 weeks. Right now is an exception though, we are currently in a deployment freeze due to Cornavirus, so it may be a little bit before this fix gets out.

[OranguTech]

Newer agents finally rolled out to our instance, verified that as of 2.166.4, issue appears fixed. Thank you @damccorm !