Kudu: Running unit tests fails with error loading Microsoft.VisualStudio.QualityTools.UTFResources

[davidebbo]

This came from this StackOverflow question.

To repro:

• Clone https://github.com/smithkl42/Troubleshooting
• Push it to a git enabled Azure site

Result: It fails while running the tests, with this error:

Error: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.QualityTools.UTFResources, Version=10.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

After the initial push, it can be easily reproed again from Kudu console:

• Go to site\repository folder
• Run: vstest.console.exe D:\home\site\repository\Payboard.Common.Tests\bin\Release\Payboard.Common.Tests.dll

Result:

Microsoft (R) Test Execution Command Line Tool Version 11.0.60315.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Passed   ToSegmentedList_ShouldSegmentList
Passed   GetIntHash_ShouldReturnSameForSameValues
Passed   GetIntHash_ShouldBeGoodAndRandom
Passed   GetBucket_ShouldBeGoodAndRandom
Passed   GetBucket_ForGuid_ShouldBeGoodAndRandom
Passed   GetBucket_ForGuid_ShouldBeDeterministic
Passed   AddRange_ShouldWork
Passed   AddSingle_ShouldWork
Passed   AddWithMask_ShouldWork
Passed   Retries_ShouldOnlyExecuteOnce_IfNoErrors
Passed   Retries_ShouldExecuteMaxNumberOfTimes_IfErrors
Passed   Retries_ShouldThrowFinalException_IfErrors
Passed   ProtobufSerializer_ShouldRoundTrip
Passed   ProtobufSerializer_ShouldRoundTripCircularReferences
Passed   JsonSerializer_ShouldRoundTrip
Passed   JsonSerializer_ShouldRoundTripCircularReferences
Passed   SubstringSafe_NullString
Passed   SubstringSafe_EmptyString
Passed   SubstringSafe_StartOutOfRange
Passed   SubstringSafe_LengthOutOfRange
Passed   SubstringSafe_Normal
Passed   AddHtmlParagraphs
Passed   AddHtmlParagraphs_ShouldHandleNullString
Passed   FixUrls_MatchWww
Passed   FixUrls_MatchHttp
Passed   FixUrls_MatchHttps
Passed   FixUrls_MatchUpperCase
Passed   FixUrls_DontFixAnchors
Passed   FixUrls_DontFixAnchorsWithUrls
Passed   FixUrls_DontFixAnchorsWithSingleQuotes
Passed   FixUrls_DontFixAnchorsWithUrlsAndSpaces
Passed   FixUrls_MatchMultiple
Passed   FixUrls_MatchAfterElement
Passed   FixUrls_MultipleTimes
Passed   FixUrls_HandlesNull
Passed   ReplaceNonAlphaNumerics_ShouldReplacePunctuation
Passed   ReplaceNonAlphaNumerics_ShouldReplaceSpaces
Passed   TrimStartNonAlphaNumerics_ShouldTrimStartNonAlphanumerics
Passed   TrimEndNonAlphaNumerics_ShouldTrimEndNonAlphanumerics
Passed   TrimAlphaNumerics_ShouldTrimNonAlphanumerics
Passed   TrimAlphaNumerics_ShouldHandleAllAlphas
Passed   TrimAlphaNumerics_ShouldHandleAllNonAlphas
Passed   IsValidEmail_ShouldRecognizeValidEmails
Passed   IsValidEmail_ShouldNotRecognizeInvalidEmails
Passed   SmartBraceSplit_ShouldSplitOnNormalBraces
Passed   SmartBraceSplit_ShouldSplitOnMatchedBraces
Passed   SmartBraceSplit_ShouldIncludeTrailingData
Passed   SmartBraceSplit_ShouldIncludeUnclosedBraces
Passed   HtmlDecodeInBraces_ShouldDecodeInBraces
Passed   HtmlDecodeInBraces_ShouldDecodeHandleMultipleLevels
Passed   HtmlDecodeInBraces_ShouldNotDecodeUnclosedBraces
Passed   HtmlDecodeInBraces_ShouldDecodeMultipleInstances
Passed   FromPascalCase_ShouldSplitCapitalizedWords
Passed   FromPascalCase_ShouldGroupCamelCase
Passed   FromPascalCase_ShouldIgnoreLowerCaseWords
Passed   FromPascalCase_ShouldLowercaseWordsWithLessThanThreeCharacters
Passed   ToMd5_ShouldConvertToMd5
Passed   StripAfter_ShouldStripCharactersAfter
Passed   ToExpando_ShouldSetSimpleProperties_ToLowerCase
Passed   ToExpando_WithRetainCase_ShouldRetainCase
Passed   ToExpando_ShouldNotSetComplexProperties
Passed   ToFullExpando_ShouldSetComplexProperties
Passed   SimpleCachePassThrough_ShouldCacheSerializableObjects
Passed   SimpleCachePassThrough_ShouldNotCacheUnserializableObjects
Passed   SimpleCache_ShouldBeThreadsafe
Passed   GetKey_ShouldPrependTypeName
Passed   GetKey_ShouldPrependKeyPrefix
Passed   GetKey_ShouldPrependGenericArgs
Passed   ToUnixTime_ShouldHaveEpochTimeEqualsZero
Skipped  ToUnixTime_ShouldRespectTimeZone
Passed   ToUnixTime_ShouldBeConvertibleWithLocalTime
Passed   ToUnixTime_ShouldBeConvertibleWithUtc
Passed   ToShortIsoDateString_ShouldReturnShortIsoDateString
Passed   ToFirstDateOfWeek_ShouldUseMondayForStartOfWeek
Passed   ToFirstDateOfWeek_ShouldUsePassedInFirstDayOfWeek
Passed   FirstDateOfMonth_ShouldReturnFirstDateOfMonth
Passed   ConditionalFormatting_ShouldHandleSimpleConditions
Passed   ConditionalFormatting_ShouldHandleComplexNumericConditions
Passed   ConditionalFormatting_ShouldHandleNullStrings
Passed   ConditionalFormatting_ShouldHandleNullObjects
Passed   ShallowClone_ShouldCopyIntegers
Passed   ShallowClone_ShouldCopyNullableIntegers
Passed   ShallowClone_ShouldCopyStrings
Passed   ShallowClone_ShouldNotCopyReferences
Passed   ShallowClone_ShouldReturnNullIfSourceIsNull
Passed   ShallowCopyTo_ShouldCopyIntegers
Passed   ShallowCopyTo_ShouldCopyNullableIntegers
Passed   ShallowCopyTo_ShouldCopyStrings
Passed   ShallowCopyTo_ShouldNotCopyReferences
Passed   ShallowCopyTo_ShouldNotCopyExceptions
Passed   ShallowCopyTo_ShouldNotCopyExceptionExpressions
Passed   GetFriendlyTypeName_ShouldHandleSimpleTypes
Passed   GetFriendlyTypeName_ShouldHandleComplexTypes
Passed   GetFriendlyTypeName_ShouldHandleReallyComplexTypes
Passed   MultiDateTimeConverter_ShouldSupportMultipleFormats
Passed   OrIfGreater_ShouldSelectGreaterValue
Passed   OrIfGreater_ShouldSelectRealValue
Passed   OrIfLesser_ShouldSelectGreaterValue
Passed   OrIfLesser_ShouldSelectRealValue
Passed   GetKey_ShouldSupportStrings
Passed   GetKey_ShouldSupportInts
Passed   GetKey_ShouldSupportDates
Passed   ForEachParallel_ShouldExecuteAllActions
Passed   ForEachParallel_ShouldExecuteWithMaxParallelization
Passed   SelectParallel_ShouldSelectAllData
Passed   SelectParallel_ShouldSelectWithMaxParallelization
Passed   SortOfMatches_ShouldMatchSameUrls
Passed   SortOfMatches_ShouldIgnoreQueryStrings
Passed   SortOfMatches_ShouldIgnoreShortPrefixes
Passed   SortOfMatches_ShouldNotMatchOnPathMismatch
Passed   SortOfMatches_ShouldNotMatchOnHostMismatch
Passed   MainDomain_ShouldRemoveWww
Passed   MainDomain_ShouldRemoveApp
Passed   MainDomain_ShouldNotRemoveExample
Passed   MainDomain_ShouldNotRemoveMainDomain
Passed   MainDomain_ShouldHandleCountryDomains
Error: Exception thrown executing tests in D:\home\site\repository\Payboard.Common.Tests\bin\Release\Payboard.Common.Tests.dll

Error: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.QualityTools.UTFResources, Version=10.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'Microsoft.VisualStudio.QualityTools.UTFResources, Version=10.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

Server stack trace: 
   at Microsoft.VisualStudio.TestTools.UnitTesting.ExpectedExceptionAttribute..ctor(Type exceptionType, String noExceptionMessage)
   at Microsoft.VisualStudio.TestTools.UnitTesting.ExpectedExceptionAttribute..ctor(Type exceptionType)
   at System.Reflection.CustomAttribute._CreateCaObject(RuntimeModule pModule, IRuntimeMethodInfo pCtor, Byte** ppBlob, Byte* pEndBlob, Int32* pcNamedArgs)
   at System.Reflection.CustomAttribute.CreateCaObject(RuntimeModule module, IRuntimeMethodInfo ctor, IntPtr& blob, IntPtr blobEnd, Int32& namedArgs)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType, Boolean mustBeInheritable, IList derivedAttributes, Boolean isDecoratedTargetSecurityTransparent)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeMethodInfo method, RuntimeType caType, Boolean inherit)
   at System.Reflection.RuntimeMethodInfo.GetCustomAttributes(Boolean inherit)
   at NUnit.Core.Reflect.GetAttributes(ICustomAttributeProvider member, Boolean inherit)
   at NUnit.Core.Reflect.HasAttribute(ICustomAttributeProvider member, String attrName, Boolean inherit)
   at NUnit.Core.Reflect.HasMethodWithAttribute(Type fixtureType, String attributeName, Boolean inherit)
   at NUnit.Core.Builders.NUnitTestFixtureBuilder.CanBuildFrom(Type type)
   at NUnit.Core.Extensibility.SuiteBuilderCollection.CanBuildFrom(Type type)
   at NUnit.Core.TestFixtureBuilder.CanBuildFrom(Type type)
   at NUnit.Core.Builders.TestAssemblyBuilder.GetFixtures(Assembly assembly, String ns)
   at NUnit.Core.Builders.TestAssemblyBuilder.Build(String assemblyName, Boolean autoSuites)
   at NUnit.Core.Builders.TestAssemblyBuilder.Build(String assemblyName, String testName, Boolean autoSuites)
   at NUnit.Core.TestSuiteBuilder.BuildSingleAssembly(TestPackage package)
   at NUnit.Core.TestSuiteBuilder.Build(TestPackage package)
   at NUnit.Core.SimpleTestRunner.Load(TestPackage package)
   at NUnit.Core.ProxyTestRunner.Load(TestPackage package)
   at NUnit.Core.ProxyTestRunner.Load(TestPackage package)
   at NUnit.Core.RemoteTestRunner.Load(TestPackage package)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at NUnit.Core.TestRunner.Load(TestPackage package)
   at NUnit.Util.TestDomain.Load(TestPackage package)
   at NUnit.VisualStudio.TestAdapter.NUnitTestExecutor.RunAssembly(String assemblyName, ITestExecutionRecorder testLog, TestFilter filter, IRunContext runContext)

Test Run Failed.Total tests: 116. Passed: 115. Failed: 0. Skipped: 1.

[davidebbo]

What's interesting is that it's successfully running all the tests, but seems to die at the very end.

Other notes:

• it does not happen with a test environment running the previews Azure Websites build
• the assembly Microsoft.VisualStudio.QualityTools.UTFResources is not in the GAC with the old and the new build

[smithkl42]

Thanks for the help troubleshooting, David. I work just a quarter mile from the MS campus - if I can ever buy you a beer, lemme know.

[davidebbo]

I took me a while, but I now see what's going on. The problem is that we now have this in the GAC:

Microsoft.VisualStudio.QualityTools.UnitTestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL
Microsoft.VisualStudio.QualityTools.UnitTestFramework, Version=10.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL

While before we only had 10.0.0.0. But 10.1.0.0 needs Microsoft.VisualStudio.QualityTools.UTFResources in the GAC, and we don't have it!

I'm not exactly sure why yet, but the good news is that I have a workaround that should work for you.

In your Payboard.Common.Tests.csproj, change

    <Reference Include="Microsoft.VisualStudio.QualityTools.UnitTestFramework" />

to

    <Reference Include="Microsoft.VisualStudio.QualityTools.UnitTestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL" />

This will lock it to the one that doesn't have the missing dependency.

Ok, I could use that :beer: just about right now :)

[smithkl42]

That's great. I tested your suggested workaround, and it seems to work correctly.

Thanks, David.

Too late for tonight, but I'm meeting some friends at Mugshots tomorrow at 9:00 pm if you need a beer then :-).

[manibs]

Thanks a ton for giving this suggestion David. It really helped me in resolving my issue.

[MrYossu]

I took me a while, but I now see what's going on. The problem is that we now have this in the GAC:
...

I know this is an old one, but I just wanted to thank you. This fixed it for me too :)

[intellectuallyswole]

This helped me just now. @smithkl42 and @davidebbo did you ever grab that beer?

[smithkl42]

Ha! Nope, never did. Offer's still open, @davidebbo :-).

[davidebbo]

I'm a Googler now. But I still might claim that beer sometime! :)