Runtime: `ConditionalFactAttribute` on the tests are ignored when ran on VS Test Explorer

[Gnbrkm41]

It appears that ConditionalTheory Attributes on the tests are ignored during discovery / run and the tests are allowed to run regardless of the value returned from the predicate.

For example, Various tests under System.Net.Security.Tests were failing with TypeInitializationException, PlatformNotSupportedException and ObjectDisposedExceptions; it appears that most of them have ConditionalFact attributes, which when evaluated returns false.

Tests that had failed, though not all of them have the attribute:

Test    Duration    Traits  Error Message
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowTwoOnBothSidesWithSingleOverlapNonTls13_Success Failed  < 1 ms      System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests Failed (31)   33 sec      
System.Net.Security.Tests Failed (31)   33 sec      
ClientAsyncAuthenticateTest Failed (3)  448 ms      
System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_AllServerVsIndividualClientSupportedProtocols_Success(clientProtocol: Ssl3) Failed    6 ms        System.ComponentModel.Win32Exception : 클라이언트 및 서버에 공용 알고리즘이 없기 때문에 통신할 수 없습니다.
System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_EachSupportedProtocol_Success(protocol: Ssl3) Failed  10 ms       System.ComponentModel.Win32Exception : 클라이언트 및 서버에 공용 알고리즘이 없기 때문에 통신할 수 없습니다.
System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success(serverProtocol: Ssl3) Failed    432 ms      System.IO.IOException : Authentication failed because the remote party has closed the transport stream.
NegotiateStreamKerberosTest Failed (3)  14 ms       
System.Net.Security.Tests.NegotiateStreamKerberosTest.NegotiateStream_ClientAuthenticationRemote_Fails Failed   1 ms    category [outerloop]    System.InvalidOperationException : The test method expected 1 parameter value, but 0 parameter values were provided.
System.Net.Security.Tests.NegotiateStreamKerberosTest.NegotiateStream_ClientAuthenticationRemote_Success Failed < 1 ms  category [outerloop]    System.InvalidOperationException : The test method expected 1 parameter value, but 0 parameter values were provided.
System.Net.Security.Tests.NegotiateStreamKerberosTest.NegotiateStream_ServerAuthenticationRemote_Success Failed 13 ms   category [outerloop]    System.NullReferenceException : Object reference not set to an instance of an object.
NegotiatedCipherSuiteTest Failed (18)   1 sec       
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowOneOnOneSideTls13_Success Failed    2 ms        System.PlatformNotSupportedException : CipherSuitesPolicy is not supported on this platform.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowSameTwoOnBothSidesLessPreferredIsTls13_Success Failed   < 1 ms      System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowSomeCipherSuitesWithNoEncryptionOption_Fails Failed < 1 ms      System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowTwoOnBothSidesWithNoOverlapNonTls13_Fails Failed    < 1 ms      System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowedCipherSuitesIncludesSubsetOfInput_Success Failed  2 ms        System.PlatformNotSupportedException : CipherSuitesPolicy is not supported on this platform.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_NothingAllowed_Fails Failed  < 1 ms      System.PlatformNotSupportedException : CipherSuitesPolicy is not supported on this platform.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_OnlyNonTls13CipherSuiteAllowedButOtherSideDoesNotAllowIt_Fails Failed    < 1 ms      System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_OnlyNonTls13CipherSuitesAllowedButChosenProtocolDoesNotAllowIt_Fails Failed  484 ms      System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_OnlyTls13CipherSuiteAllowedButChosenProtocolsDoesNotAllowIt_Fails Failed 1 ms        System.PlatformNotSupportedException : CipherSuitesPolicy is not supported on this platform.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_OnlyTls13CipherSuiteAllowedOtherSideDoesNotAllowTls13_Fails Failed   < 1 ms      System.PlatformNotSupportedException : CipherSuitesPolicy is not supported on this platform.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_ThreeCipherSuitesWithTwoOverlapping_Success Failed   < 1 ms      System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_TwoCipherSuitesWithAllOverlapping_Success Failed < 1 ms      System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.NegotiatedCipherSuite_SslProtocolIsLowerThanTls13_ShouldMatchTheProtocol Failed (3) 599 ms      
System.Net.Security.Tests.NegotiatedCipherSuiteTest.NegotiatedCipherSuite_SslProtocolIsLowerThanTls13_ShouldMatchTheProtocol(protocol: Tls) Failed  555 ms      System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.NegotiatedCipherSuite_SslProtocolIsLowerThanTls13_ShouldMatchTheProtocol(protocol: Tls11) Failed    21 ms       System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.NegotiatedCipherSuite_SslProtocolIsLowerThanTls13_ShouldMatchTheProtocol(protocol: Tls12) Failed    23 ms       System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.NegotiatedCipherSuite_SslProtocolIsTls13_ShouldBeTls13 Failed   464 ms      System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.Tls13IsSupported_GetValue_ReturnsTrue Failed    < 1 ms      System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
ServerAsyncAuthenticateTest Failed (2)  435 ms      
System.Net.Security.Tests.ServerAsyncAuthenticateTest.ServerAsyncAuthenticate_AllClientVsIndividualServerSupportedProtocols_Success(serverProtocol: Ssl3) Failed    6 ms        System.ComponentModel.Win32Exception : 클라이언트 및 서버에 공용 알고리즘이 없기 때문에 통신할 수 없습니다.
System.Net.Security.Tests.ServerAsyncAuthenticateTest.ServerAsyncAuthenticate_EachSupportedProtocol_Success(protocol: Ssl3) Failed  429 ms      System.ComponentModel.Win32Exception : 클라이언트 및 서버에 공용 알고리즘이 없기 때문에 통신할 수 없습니다.
SslStreamEKUTest Failed (4) 30 sec      
System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUClientAuth_Ok Failed  34 ms       System.Security.Authentication.AuthenticationException : The remote certificate is invalid according to the validation procedure.
System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUServerAuth_Ok Failed  15 sec      System.TimeoutException : Task timed out after 00:00:15
System.Net.Security.Tests.SslStreamEKUTest.SslStream_SelfSignedClientEKUClientAuth_Ok Failed    15 sec      System.TimeoutException : Task timed out after 00:00:15
System.Net.Security.Tests.SslStreamEKUTest.SslStream_ServerEKUClientAuth_Fails Failed   31 ms       Assert.Throws() Failure Expected: typeof(System.IO.IOException) Actual:   typeof(System.Security.Authentication.AuthenticationException): The remote certificate is invalid according to the validation procedure. ---- System.Security.Authentication.AuthenticationException : The remote certificate is invalid according to the validation procedure.
SslStreamNetworkStreamTest Failed (1)   436 ms      
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_SendReceiveOverNetworkStream_AuthenticationException Failed  436 ms  category [nonfreebsdtests], category [nonnetbsdtests], category [nonosxtests], category [nonwindowstests]   Assert.Contains() Failure Not found: SSL_ERROR_SSL In value:  클라이언트 및 서버에 공용 알고리즘이 없기 때문에 통신할 수 없습니다.

I'm running VS2019 16.5 P1. It's possible that there's other repositories this might be suitable for but I am not entirely sure which one it is, or if this is specific to something on this repo.

[wfurt]

any idea @safern or @ViktorHofer ???

[safern]

@Gnbrkm41 does this apply for ConditionalFact and ConditionalTheory or only for one of them?

[Gnbrkm41]

Both it seems. It also happens for other attributes like PlatformSpecific.

[ViktorHofer]

Yes, that's the expected behavior as VS Test Explorer by default doesn't filter on traits. I started a discussion with vstest some months ago to allow specifying default included/excluded traits via the runsettings file.

[safern]

@ViktorHofer do you want to close this issue then, or open is fine to have it under your radar to track the status of the discussion you're having with VS Test Explorer

[ViktorHofer]

Tracking vstest issue: https://github.com/microsoft/vstest/issues/2273

[ViktorHofer]

Let's keep this open on our side for tracking.

[safern]

Thanks @ViktorHofer -- I assigned to you since you're tracking that work.

[Gnbrkm41]

@ViktorHofer, Has this been addressed? Looks like the tracking issue on the vstest side has been fixed.

[ViktorHofer]

Unfortunately not as we require an updated xunit.runner.visualstudio package: https://github.com/xunit/visualstudio.xunit/pull/205#issuecomment-625886382

cc @clairernovotny

[clairernovotny]

Sorry about that, just released it:

https://www.nuget.org/packages/xunit.runner.visualstudio/2.4.2

not sure what templates need to be updated to use 2.4.2 instead of 2.4.1