Runtime: XmlDsigC14NTransformTest failure due to concurrent file access

[danmosemsft]

@krwq @tintoy @anthonylangsworth there are several tests potentially creating a file named doc.dtd in the same folder concurrently.

They should use a pattern like
``` c#
using (var directory = new TempDirectory())
using (var file = new TempFile(Path.Combine(directory.Path, "doc.dtd")))
{.. }

MESSAGE:
System.IO.IOException : The process cannot access the file 'D:\j\workspace\outerloop_win---bf7c4efa\bin\AnyOS.AnyCPU.Debug\System.Security.Cryptography.Xml.Tests\netcoreapp\doc.dtd' because it is being used by another process.
+++++++++++++++++++
STACK TRACE:
at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options) at System.IO.FileStreamHelpers.CreateFileStream(String path, Boolean write, Boolean append) in D:\j\workspace\outerloop_win---bf7c4efa\src\System.Runtime.Extensions\src\System\IO\FileStreamHelpers.CoreClr.cs:line 10 at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize) in D:\j\workspace\outerloop_win---bf7c4efa\src\System.Runtime.Extensions\src\System\IO\StreamWriter.cs:line 140 at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding) in D:\j\workspace\outerloop_win---bf7c4efa\src\System.Runtime.Extensions\src\System\IO\StreamWriter.cs:line 125 at System.Security.Cryptography.Xml.Tests.XmlDsigC14NTransformTest.C14NSpecExample1() in D:\j\workspace\outerloop_win---bf7c4efa\src\System.Security.Cryptography.Xml\tests\XmlDsigC14NTransformTest.cs:line 217
```

[tintoy]

Give me 5 mins and I'll get onto this.

[tintoy]

@danmosemsft I think these tests may have been ported from mono (their design is a little odd) - it may take a while to split out the file-access stuff, so would you like me to mark them for Skip until they've been modified?

[krwq]

@tintoy is it possible to simply replace TempFile with MemoryStream?

[tintoy]

I think the problem is that these tests are relying on the fact that resources (such as doc.dtd) referenced in the in the input XML will be resolved from the current directory by default. I'm a little rusty, but isn't there some sort of entity resolver we could hook instead?

[tintoy]

Oh yes - XmlUrlResolver. Perhaps an alternative implementation could do the trick?

[tintoy]

XmlPreloadedResolver, for example, sounds like an ideal candidate.

[tintoy]

Ugh, in corefx that's in System.Private.Xml. I take it that means it's off-limits, yes?

[krwq]

@tintoy can we quickly rename the temporary file names to like doc1, doc2 etc so that the build is unblocked and then after we finish fixing the tests we can get it properly fixed?

[tintoy]

Ok, I can do that I think.

[tintoy]

Where does TempFile live?

[krwq]

https://github.com/dotnet/corefx/blob/master/src/Common/tests/System/IO/TempFile.cs

[tintoy]

Ta.

[krwq]

@tintoy - press ? anywhere but text editor on github and see all the shortcuts. when you switch to code you can press 't' and type a class name

[danmosemsft]

Fixed my snippet above.

[tintoy]

@krwq thanks! I knew about the keyboard shortcuts, but didn't know the code-search was so useful :)

[tintoy]

Ugh, these tests are a nightmare. This may take a little longer. The suggested approach is insufficient because a lot of the tests depend specific file names (which are hard-coded in static strings used as input to the tests).

[krwq]

@tintoy would [assembly: CollectionBehavior(DisableTestParallelization = true)] help us for now? (Xunit namespace)

[tintoy]

Probably, but it would mean I've wasted the last 30 mins of work :-)

If you guys are in a hurry, go for it. Otherwise I'll only be another 10 mins or so I hope.

[krwq]

@tintoy go ahead and finish :+1: I just didn't want you to spend too much time on it

[tintoy]

No worries, I'm just fighting to do it without an IDE ;-P

[krwq]

@tintoy have you tried Code? Should work on most of the OSes. By the way, thank you for helping here!

[tintoy]

Oh yes it's what I use normally, but omnisharp can't cope with corefx project layout :-)

[tintoy]

@krwq could you please review and let me know if this approach looks workable (as an interim situation, at least)?

[tintoy]

If so, I'll open a PR or you can cherry-pick (up to you).

[tintoy]

I went for the dumbest approach possible on the basis that this is just a hack to keep things moving :)

[tintoy]

BTW, I do have a Windows laptop, too, just not with me.

[KristinXie1]

Test System.Security.Cryptography.Xml.Tests.XmlDsigC14NTransformTest.C14NSpecExample1 failed with System.IO.IOException.

Detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_windows_nt_debug/119/testReport/System.Security.Cryptography.Xml.Tests/XmlDsigC14NTransformTest/C14NSpecExample1/

Message:
~
System.Security.Cryptography.Xml.Tests.XmlDsigC14NTransformTest.C14NSpecExample1 [FAIL]
System.IO.IOException : The process cannot access the file 'D:\j\workspace\outerloop_win---bf7c4efa\bin\AnyOS.AnyCPU.Debug\System.Security.Cryptography.Xml.Tests\netcoreappdoc.dtd' because it is being used by another process.
~

Stack Trace:
~
at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
D:\j\workspace\outerloop_win---bf7c4efa\src\System.Runtime.Extensions\src\System\IO\FileStreamHelpers.CoreClr.cs(10,0): at System.IO.FileStreamHelpers.CreateFileStream(String path, Boolean write, Boolean append)
D:\j\workspace\outerloop_win---bf7c4efa\src\System.Runtime.Extensions\src\System\IO\StreamWriter.cs(140,0): at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize)
D:\j\workspace\outerloop_win---bf7c4efa\src\System.Runtime.Extensions\src\System\IO\StreamWriter.cs(125,0): at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
D:\j\workspace\outerloop_win---bf7c4efa\src\System.Security.Cryptography.Xml\tests\XmlDsigC14NTransformTest.cs(217,0): at System.Security.Cryptography.Xml.Tests.XmlDsigC14NTransformTest.C14NSpecExample1()
at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
D:\j\workspace\outerloop_win---bf7c4efa\src\System.Runtime.Extensions\src\System\IO\FileStreamHelpers.CoreClr.cs(10,0): at System.IO.FileStreamHelpers.CreateFileStream(String path, Boolean write, Boolean append)
D:\j\workspace\outerloop_win---bf7c4efa\src\System.Runtime.Extensions\src\System\IO\StreamWriter.cs(140,0): at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize)
D:\j\workspace\outerloop_win---bf7c4efa\src\System.Runtime.Extensions\src\System\IO\StreamWriter.cs(125,0): at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
D:\j\workspace\outerloop_win---bf7c4efa\src\System.Security.Cryptography.Xml\tests\XmlDsigC14NTransformTest.cs(217,0): at System.Security.Cryptography.Xml.Tests.XmlDsigC14NTransformTest.C14NSpecExample1()
~

[tintoy]

Hi @Jiayili1, will hopefully be fixed by dotnet/corefx#16657 (just awaiting approval from @danmosemsft and @krwq).

[danmosemsft]

I'll let @krwq sign off.. thanks for fixing so promptly

[anthonylangsworth]

Thanks for jumping on this so quickly, @tintoy .

[anthonylangsworth]

... and welcome to my world trying to get these tests working in the first place. People wonder why I want to clean them up. :-)

[bartonjs]

Maybe all these tests should be changed to not read off of disk at all? Embedded resources, string literals, etc.

Just thoughts.

[danmosemsft]

fixed by https://github.com/dotnet/corefx/pull/16657

[tintoy]

@bartonjs yeah, I was thinking perhaps a customised XmlResolver would handle most cases we care about in the current tests.

[anthonylangsworth]

Agreed. Ironically, I wrote the skeleton of one last night then removed it. Have a look at XmlPreloadedResolver in System.Xml.Resolvers. It might already do what we need.

[tintoy]

@anthonylangsworth great minds think alike. And so do ours, apparently ;-)

[krwq]

@tintoy @anthonylangsworth let's leave it as a future improvement - I'd rather have more issues fixed than this tiny annoyance with test 😄 Anyways, if you feel strongly we should do it early go ahead (take a look at my PR first so that we don't get merge conflicts)

[anthonylangsworth]

@krwq @tintoy I have (hopefully) resolved the concurrency issues with krwq#1. This appears to be causing some consternation, anyway.