Testfx: Passing variables and secrets from VSTS

[Adebeer]

Description

I have several secrets stored in KeyVault. Unfortunately I cannot find a way to pass parameters to my test run via VSTS

Documentation says that Keyvault secrets can only be supplied via VSTS variables - fair enough - but how do I actually do this? All the information I can find on web seems outdated or doesn't work.

For example - consider RunSettings file:

<RunSettings>
    <TestRunParameters>  
        <Parameter name="webAppUrl" value="http://localhost" /> 
        <Parameter name="webAppUserName" />
        <Parameter name="webAppPassword" />
  </TestRunParameters> 
</RunSettings>

I tried passing values for last 2 parameters via cmd line as follow:

vsts.console MyTest.dll /Settings:vsts.runsettings -- -webAppUserName foo
vsts.console MyTest.dll /Settings:vsts.runsettings -- webAppUserName=foo
dotnet test -s vsts.runsettings -- -webAppUserName foo
dotnet test -s vsts.runsettings -- webAppUserName=foo

but this has no effect - the webAppUserName value remains null (I can see a value for webAppUrl, so I know my code is right!)

I've also tried both the VSTS "dotnet test" task as well as the "VsTest" task from my VSTS 2017 build. The VsTest provides a Override test run parameters setting - and as per the tooltip, I tried:
-webAppUserName user -webAppPassword somethingSecret

Again - no effect!

Originally I used xUnit and had exactly the same issue - i.e. coudn't figure out a way to pass parameters via VSTS - so I tried MSTest, but same issue.

Going back to my original issue of injecting KeyVault secrets
Locally via VS Studio I was able to just do the following from my test:

var config = new ConfigurationBuilder()
                .AddJsonFile("appsettings.json", true, true)
                ...
                .AddAzureKeyVault(...)

Unfortunately, when run via VSTS, the test runner just hangs (i.e. I had to stop it after several minutes) with no log output for test step if I use either

.AddAzureKeyVault(...) or .AddEnvironmentVariables()

So I tried using VSTS Variable Groups - and linking that to KeyVault. However, keyvault secrets are not accessible via environment variables (using Enviroment.GetEnvironmentVariable(...) directly from C#) - so that's no good. They say you can only pass these to tasks via VSTS variables...hence my problem!

Aside:

Even if I could use environment variables, it's not optimal because when using .AddAzureKeyVault() I can supply a custom IKeyVaultSecretManager to, for example, replace a special delimiter with the ':' character - this means that I can nest my json config values - e.g. if I had this in my config file:
{ "A" : { "B" : "somevalue" } }

then using normal configuration builder I can access the above via config["A:B"]. Unfortunately KeyVault doesn't like the ":" character - so you have to replace it with something like "--" and then use a custom IKeyVaultSecretManager to replace "--" with ":"

Please help! All I wanted for Christmas was not to put my KeyVault secrets into Git... but still use this from within my tests... surely I'm missing something??

[abhishkk]

@Adebeer
Can you run a vstest task based run with debug mode as true (set system.debug variable to true).
Add override run parameters as you did in above question. Please share the build logs.

override run parameters is a supported scenario and thus we need logs to debug the issue further.

[abhishkk]

@Adebeer You can follow this link to know how to override test run parameters.

Closing this issue. Please let us know if you are not facing any issues.

[JohnGalt1717]

Closing for what reason? You didn't answer the issue.

Override what run parameters? Where? Do you mean in the "Override test run parameters" field of VSTS?

And how exactly will those get passed? As environment variables? Or something else?

This should be a sample that is right in the .NET core documentation AND in the vsts documentation. This is automated testing 101: How do you pass variables per run and use those to configure your tests? There is 0 documentation on this.

And the link says this only works for MSTest, thus no xunit etc.

This should be part of the ConfigurationBuilder in .NET Core and it should be able to pull this out.

[Adebeer]

I agree with @JohnGalt1717 . The link you provided is not relevant - it's dated 2015. The issue is trying to pass parameters to tests in VS2017 and .net core test projects.
We're trying to write end to end and integration tests - but without the ability to pass parameters this is not feasible. Can you please provide a solution to this fundamental issue

[Adebeer]

For now we resolved the issue by converting our test project to .Net Full framework *sigh*

Please don't close this issue as this needs to be addressed in .net Core test projects

[abhishkk]

@Adebeer @JohnGalt1717
Updating test run parameter in dotnet core targeting test projects is not supported from tasks because of following reasons:

1. dotnet core targeting test projects can be ran only with dotnet task. vstest task doesn't support dotnet core tests.
2. Updating test run parameters from dotnet task is not supported.

Test projects targeting .Net full framework can be ran with vstest task. vstest task supports overriding test run parameters. Following is the way to do this:

1. Sample runsettings file:

<?xml version="1.0" encoding="UTF-8"?>
<RunSettings>
  <TestRunParameters>
    <Parameter name="name" />
    <Parameter name="password" />
  </TestRunParameters>
</RunSettings>

1. Sample code:

        [TestMethod]
        public void TestMethod1()
        {
            Assert.AreEqual("sampleName", TestContext.Properties["name"]?.ToString());
            Assert.AreEqual("samplePassword", TestContext.Properties["password"]?.ToString());
        }

1. Snap of Override test run parameters field:
   

2. Snap of task variables:
   

Password field can be set secret in task variable.

Adding @PBoraMSFT to comment on the support for overriding test run parameters for dotnet core test projects.

[JohnGalt1717]

However you can pass environment variables in.

Ie. adding these process variables on your build definition will work and the unit tests will be run:

Azure:Auth:ClientId

ASPNETCORE_ENVIRONMENT

etc.

Further, if you're using docker you can pass these by running a command instead of just docker run and pass -e {variable above} etc.

In the build task you can pass build-args as =

which solves the problem but the documentation doesn't tell you any of this and should.

Sadly a separate issue is still open because you can't get the container id that was published to be able to pull out your test results from a build and instead have to use a run to get them by mapping a path. Hopefully that will be fixed soon.

[gabrieljoelc]

@JohnGalt1717 I can't get Variable Group variables to be visible inside the test context. How are you achieving this? How are you able to "pass environment variables in"?

[Adebeer]

This issue seems fixed to me now.

I've recently created a new xUnit .Net Core 2.1 app and can use environment variables just fine, as mentioned by @JohnGalt1717 .

E.g.
Create build variable - e.g. ASPNETCORE_ENVIRONMENT
and then in my code, I simply use: System.Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT")

In Azure DevOps - I simply use the Visual Studio Test task

For me, this is all I needed and much better than mucking around with test setting files etc.

From my perspective, happy to close this issue

[gabrieljoelc]

@Adebeer I could not get "Runsetting arguments" to work with or without a *.runsettings file using dotnet test (see the vstest-docs).

Use case 1:

dotnet test  -- MSTest.MapInconclusiveToFailed=True MSTest.DeploymentEnabled=False

Expected: MSTest.MapInconclusiveToFailed or MSTest.DeploymentEnabled should be set with their respective values in TestContext
Actual: Neither variables are in TestContext

Use case 2:

<!-- additionalargs.runsettings -->  
<?xml version="1.0" encoding="utf-8"?>  
<RunSettings>  
  <!-- MSTest adapter -->  
  <MSTest>  
    <MapInconclusiveToFailed>True</MapInconclusiveToFailed>
    <DeploymentEnabled>False</DeploymentEnabled>
  </MSTest>   
</RunSettings>

dotnet test --settings additionalargs.runsettings -- MSTest.MapInconclusiveToFailed=False

Expected: MSTest.MapInconclusiveToFailed=False and MSTest.DeploymentEnabled=False should be set in TestContext
Actual: MSTest.MapInconclusiveToFailed=True and MSTest.DeploymentEnabled=False are set in TestContext

[Adebeer]

@gabrieljoelc try this approach instead: https://stackoverflow.com/questions/50065991/how-to-configure-xunit-dotnet-core-project-to-initialize-from-different-configur/53220339?noredirect=1#comment93390698_53220339

Let me know if that worked

[gabrieljoelc]

@Adebeer thank you for the proposed solution. Would you consider this open issue still a problem since this solution just works around runsetting arguments being broken with dotnet test?

I believe the SO post seems to be an ASP.NET Core app. I am building Azure Functions and Azure Webjobs. My use case doesn't allow me to easily build up the configuration myself.

I ended up working around this problem in a similar way already and wrote it up in this gist. Here's a highlight from the gist:

// Azure Function
public static class Endpoint
{
    private static string EhConnectionString { get; } =
        Environment.GetEnvironmentVariable("EventHubConnectionString");

    [FunctionName("Endpoint")]
    public static async Task<IActionResult> RunAsync(
        [HttpTrigger(AuthorizationLevel.Function, "post")]HttpRequest request, ILogger log)
    { /* do cool stuff */ }
}

// test fixture
[TestClass]
public class EndpointTests : FunctionTest
{
    [TestInitialize]
    public void Setup()
    {
        var settings = new ConfigurationBuilder()
            .AddJsonFile("appsettings.test.json", false)
            .Build();
        foreach (var item in settings.AsEnumerable())
        {
            Environment.SetEnvironmentVariable(item.Key, item.Value);
        }
    }

    // test methods
}

// local appsettings.test.json that is also gitignored
{
  "EventHubConnectionString": "<ConnectionString>"
}

[Adebeer]

Hi @gabrieljoelc - you have a good solution too.

The SO post should work same within any .net core unit test project as the .net core MS configuration is isolated in seperate nugget packages and not limited to aspnet core projects.

Going forward, for me this is a better approach as I think the ms test runsetting approach is less framework agnostic and tied to their test platform. I like to be in a position where I can have more configuration reuse between my tests and non-test projects.