Azure-pipelines-tasks: VsTestV2 -> Unable to get feature flag: TestExecution.EnableTranslationApi

[sebastienlabine]

Issue Description

When running the task: VsTest - testAssemblies version 2.138.11, the agent is unable to find the feature flag: TestExecution.EnableTranslationApi.

This causes the 'effectiveState' to be undefined and thus to throw the error:
Unhandled: Cannot read property 'effectiveState' of undefined which fails task and our release.

Code

The function affected is this one. Somehow, it enters the second if() and because the flag was not found, faModel is undefined.

const enableApiExecution = await isFeatureFlagEnabled(tl.getVariable('System.TeamFoundationCollectionUri'),
'TestExecution.EnableTranslationApi', tl.getEndpointAuthorization('SystemVssConnection', true).parameters.AccessToken);

function isFeatureFlagEnabled(collectionUri: string, featureFlag: string, token: string): Promise<boolean> {
    let state = false;
    const options = {
        url: collectionUri + '/_apis/FeatureFlags/' + featureFlag,
        json: true,
        headers: {
            'Content-Type': 'application/json',
            'Authorization': 'Bearer ' + token
        }
        return new Promise((resolve, reject) => {
        request(options, (err, res, faModel) => {
            if (err) {
                tl.warning(tl.loc('UnableToGetFeatureFlag', featureFlag));
                tl.debug('Unable to get feature flag ' + featureFlag + ' Error:' + err.message);
                resolve(state);
            }
            // faModel is undefined
            if (faModel.effectiveState) {
                state = ('on' === faModel.effectiveState.toLowerCase());
                tl.debug(' Final feature flag state: ' + state);
            }
            resolve(state);
        });
    });

Error logs

2018-08-08T13:22:33.8285663Z ##[section]Starting: VsTest - testAssemblies
2018-08-08T13:22:33.8295617Z ==============================================================================
2018-08-08T13:22:33.8295762Z Task : Visual Studio Test
2018-08-08T13:22:33.8295974Z Description : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
2018-08-08T13:22:33.8296210Z Version : 2.138.11
2018-08-08T13:22:33.8296306Z Author : Microsoft Corporation
2018-08-08T13:22:33.8296412Z Help : More Information
2018-08-08T13:22:33.8296542Z ==============================================================================
2018-08-08T13:22:41.9551990Z SystemVssConnection exists true
2018-08-08T13:23:02.8956112Z ##[warning]Unable to get feature flag: TestExecution.EnableTranslationApi
2018-08-08T13:23:02.8967415Z ##[error]Unhandled: Cannot read property 'effectiveState' of undefined

[singhsarab]

@nigurr Please take a look

[dblain1987]

Any news on this? All my builds are failing due to this error.

[ACE786786]

Affected also, issue started on the 8th also.

[nigurr]

Hi,

We are rolling out hotfix. Sorry for the inconvenience. It should be fixed as early as possible.

[ohadschn]

Until the hotfix is deployed (and in similar future cases) I wonder if your admin can work around this by setting the flag himself: https://blogs.msdn.microsoft.com/buckh/2016/09/30/controlling-exposure-through-feature-flags-in-vs-team-services/

[dblain1987]

Looks like this has been resolved. Builds are working as of 13th.

[richardzaat]

Today, 14th August 09:50 CET we are still experiencing this issue. Could it be because we are in the 4th ring and the deployment is not yet complete?

[nigurr]

@richardzaat can you please let us know the task version (it will be printed in build logs)

[richardzaat]

@nigurr Thanks for the quick response:
Yesterday everything was fine (2.137.9, see below), today we are experiencing the issue (2.138.12)

Today:
2018-08-14T06:54:06.6544518Z Version : 2.138.12
2018-08-14T06:54:06.6544625Z Author : Microsoft Corporation
2018-08-14T06:54:06.6544746Z Help : More Information
2018-08-14T06:54:06.6544888Z ==============================================================================
2018-08-14T06:54:08.7577659Z SystemVssConnection exists true
2018-08-14T06:54:29.6910454Z ##[warning]Unable to get feature flag: TestExecution.EnableTranslationApi
2018-08-14T06:54:29.6918143Z ##[error]Unhandled: Cannot read property 'effectiveState' of undefined

Yesterday:

2018-08-13T12:57:59.8890938Z Version : 2.137.9
2018-08-13T12:57:59.8891059Z Author : Microsoft Corporation
2018-08-13T12:57:59.8891175Z Help : More Information
2018-08-13T12:57:59.8891317Z ==============================================================================
2018-08-13T12:58:01.7955496Z Run the tests locally using vstest.console.exe
2018-08-13T12:58:01.7955692Z ========================================================
2018-08-13T12:58:01.7955996Z Test selector : Test assemblies
2018-08-13T12:58:01.7956187Z Test assemblies : *Tests.dll,!obj*,!*DLL.VF.SSP.Base.Tests.dll

[nigurr]

@richardzaat we will hot fix your ring now. We rolled out hot fix for all rings except one ring which was running under M137.

[balayoglu]

We are also experiencing the issue. Current task version is 2.138.12.

[nigurr]

@richardzaat @balayoglu sorry for the trouble. It's hot fixed. can you please check now?

[richardzaat]

@nigurr @balayoglu
It seems to find the effectiveState, now I do see another one popping up (TestExecution.EnableTranslationApi).
PS: I am working on it, but it seems the hosted agents are running fine, and the issue occurs especially on our onprem agents (still on agent version 126). Still working on confirmation for this by running a few test builds.

2018-08-14T11:55:26.1117296Z Task : Visual Studio Test
2018-08-14T11:55:26.1117492Z Description : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
2018-08-14T11:55:26.1117706Z Version : 2.138.14
2018-08-14T11:55:26.1117831Z Author : Microsoft Corporation
2018-08-14T11:55:26.1117947Z Help : More Information
2018-08-14T11:55:26.1118072Z ==============================================================================
2018-08-14T11:55:28.1890818Z SystemVssConnection exists true
2018-08-14T11:55:49.1216496Z ##[warning]Unable to get feature flag: TestExecution.EnableTranslationApi
2018-08-14T11:55:49.2400967Z Running tests using vstest.console.exe runner.

[nigurr]

@richardzaat the warning will disappear after a while (probably within 24hrs) but it's safe to ignore.

[richardzaat]

@nigurr @balayoglu It is working, both onprem and hosted. Thanks for the quick response and fix!

[balayoglu]

@nigurr it is working on my agent as well. Thanks for your help.

[illumination7272]

Hello after the hotfix the issue seem to be resolved , but am starting to get another issue

An exception occurred while executing the task. So the task is not executed. System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct.

for the record i'm running Nunit tests via nunit test adapters , and it was working just fine until the fixed issue started

[Xyclade]

As of Today I am receiving this issue again, with version 2.140.4

[calebcartwright]

Getting this too. Rolling back to v1.x of the task allows the tests to run, but hope there's another hotfix/solution coming soon (please let us know if you need our account names)

[nigurr]

Hi, the hotfix will be rolled out soon. sorry for the regression, we are adding an action item to address this.

[richardzaat]

Same issue here (again), on the 2.140.4 version. @nigurr Any ETA on the hotfix deployment?

[balayoglu]

Is there any progress about the hotfix, we are also facing the issue. I tried v1.x but did not help.

[nigurr]

Sorry for the delay. It will be rolled out within 24hrs. We already comitted the fix to release bits

[JonathanJacksonAO]

Are we still on track for the hotfix? Just tried and still have the issue...

[JonathanJacksonAO]

Hold that thought... I'm trying again.

[PhilStanton]

Still an issue here.

[PHPSoft]

Yeah... same here as well.

[dblain1987]

Confirmed issue has come back.

Version : 2.140.4
2018-09-28T11:15:23.7779209Z SystemVssConnection exists true
2018-09-28T11:15:23.8247212Z ##[warning]Unable to get feature flag: TestExecution.EnableTranslationApi
2018-09-28T11:15:23.8403213Z ##[error]Unhandled: Cannot read property 'effectiveState' of undefined
2018-09-28T11:15:23.8403213Z SystemVssConnection exists true
2018-09-28T11:15:23.8715215Z ##[warning]Unable to get feature flag: TestExecution.EnableDiagnostics
2018-09-28T11:15:23.8715215Z ##[error]Unhandled: Cannot read property 'effectiveState' of undefined
2018-09-28T11:15:23.9183218Z Running tests using vstest.console.exe runner.

[Luchs2012]

Yes same here Issue is back.

[JonathanJacksonAO]

Mines all good.