Test System.Security.Cryptography.Algorithms.Tests.AesCcmTests/AesCcmNistTests(testCase: NIST CCM Test Vectors - DVPT128.rsp - 0 (128/56/0/32/0)) has failed.
Message :
System.TypeInitializationException : The type initializer for 'Crypto' threw an exception.
---- System.TypeInitializationException : The type initializer for 'CryptoInitializer' threw an exception.
-------- System.DllNotFoundException : Unable to load shared library 'System.Security.Cryptography.Native.OpenSsl' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libSystem.Security.Cryptography.Native.OpenSsl, 1): image not found
Stack Trace :
at Interop.Crypto.EvpAes128Ccm()
at System.Security.Cryptography.AesCcm.GetCipher(Int32 keySizeInBits) in /_/src/System.Security.Cryptography.Algorithms/src/System/Security/Cryptography/AesCcm.Unix.cs:line 130
at System.Security.Cryptography.AesCcm.EncryptInternal(ReadOnlySpan`1 nonce, ReadOnlySpan`1 plaintext, Span`1 ciphertext, Span`1 tag, ReadOnlySpan`1 associatedData) in /_/src/System.Security.Cryptography.Algorithms/src/System/Security/Cryptography/AesCcm.Unix.cs:line 37
at System.Security.Cryptography.AesCcm.Encrypt(ReadOnlySpan`1 nonce, ReadOnlySpan`1 plaintext, Span`1 ciphertext, Span`1 tag, ReadOnlySpan`1 associatedData) in /_/src/System.Security.Cryptography.Algorithms/src/System/Security/Cryptography/AesCcm.cs:line 44
at System.Security.Cryptography.AesCcm.Encrypt(Byte[] nonce, Byte[] plaintext, Byte[] ciphertext, Byte[] tag, Byte[] associatedData) in /_/src/System.Security.Cryptography.Algorithms/src/System/Security/Cryptography/AesCcm.cs:line 33
at System.Security.Cryptography.Algorithms.Tests.AesCcmTests.AesCcmNistTests(AEADTest testCase) in /_/src/System.Security.Cryptography.Algorithms/tests/AesCcmTests.cs:line 333
----- Inner Stack Trace -----
at Interop.CryptoInitializer.Initialize() in /_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs:line 42
at Interop.Crypto..cctor() in /_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs:line 20
----- Inner Stack Trace -----
at Interop.CryptoInitializer.EnsureOpenSslInitialized()
at Interop.CryptoInitializer..cctor() in /_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs:line 27
Build: -20190612.69(Master)
Failing configurations:
Details;
https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fheads~2Fmaster/test~2Ffunctional~2Fcli~2Finnerloop~2F/20190612.69/workItem/System.Security.Cryptography.Algorithms.Tests/analysis/xunit/System.Security.Cryptography.Algorithms.Tests.AesCcmTests~2FAesCcmNistTests(testCase:%20NIST%20CCM%20Test%20Vectors%20-%20DVPT128.rsp%20-%200%20(128~2F56~2F0~2F32~2F0))
Build or deployment problem?
cc: @bartonjs
dci-mac-build-011 ran the OpenSSL tests successfully, so at least one of the 10.14 machines has it; but the failing CCM and GCM tests in this run ran on dci-mac-build-107.
@MattGal What's the best way for me to ask if dci-mac-build-107 is missing the OpenSSL prereq? :)
What's the best way for me to ask if dci-mac-build-107 is missing the OpenSSL prereq? :)
It would be nice if the tests printed this diagnostic information in the event of this class of failure.
It would be nice if the tests printed this diagnostic information in the event of this class of failure.
As a bit of a hack, we print out a bunch of diagnostic data in:
https://github.com/dotnet/corefx/blob/5955ee2583d4509d37ecf55243e9e3c9af128487/src/System.Runtime.InteropServices.RuntimeInformation/tests/DescriptionNameTests.cs#L16
This could be added to the mix.
@bartonjs best way is to ask me and I'll look at the machine (looking)
@bartonjs looks like this isn't explicitly added as an artifact and 011 was an older machine (machines are now auto-configured and due to a strange set of circumstances I can't even log into one) https://helix.dot.net/api/2018-03-14/info/queues/osx.1014.amd64.open . I filed https://github.com/dotnet/core-eng/issues/6702 to track updating this, whoever takes it will want some details about .NET Core's requirements in this space.
Test System.Security.Cryptography.Dsa.Tests.DSAImportExport/ExportAfterDispose(importKey: False) failed with same exception.
Message :
System.TypeInitializationException : The type initializer for 'Crypto' threw an exception.
---- System.TypeInitializationException : The type initializer for 'CryptoInitializer' threw an exception.
-------- System.DllNotFoundException : Unable to load shared library 'System.Security.Cryptography.Native.OpenSsl' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libSystem.Security.Cryptography.Native.OpenSsl, 1): image not found
Stack Trace :
at Interop.Crypto.RsaCreate()
at System.Security.Cryptography.RSAOpenSsl.ImportParameters(RSAParameters parameters) in /_/src/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 393
at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt.DecryptSavedAnswer_OaepSHA256() in /_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.cs:line 156
at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor) in /_/src/System.Private.CoreLib/shared/System/Lazy.cs:line 344
at System.Lazy`1.CreateValue() in /_/src/System.Private.CoreLib/shared/System/Lazy.cs:line 432
at System.Lazy`1.get_Value() in /_/src/System.Private.CoreLib/shared/System/Lazy.cs:line 505
at System.Security.Cryptography.DSAOpenSsl.GetKey() in /_/src/Common/src/System/Security/Cryptography/DSAOpenSsl.cs:line 323
at System.Security.Cryptography.DSAOpenSsl.CreateSignature(Byte[] rgbHash) in /_/src/Common/src/System/Security/Cryptography/DSAOpenSsl.cs:line 210
at System.Security.Cryptography.Dsa.Tests.DSAImportExport.ExportAfterDispose(Boolean importKey) in /_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSAImportExport.cs:line 148
----- Inner Stack Trace -----
at Interop.CryptoInitializer.Initialize() in /_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs:line 42
at Interop.Crypto..cctor() in /_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs:line 20
----- Inner Stack Trace -----
at Interop.CryptoInitializer.EnsureOpenSslInitialized()
at Interop.CryptoInitializer..cctor() in /_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs:line 27
Build: -20190617.81(Master)
Failing configurations:
Note the machine involved has been re-imaged relatively recently; if machine configuration is suspect, please work with myself or @ilyas1974 to investigate this.
@MattGal This is still the same problem tracked by a still-active machine configuration bug (that OpenSSL isn't installed on the machine / isn't symlinked into /usr/local/lib/).
@bartonjs thanks, I also got on the machine and confirmed this. Apologies for the inconvenience.
As a bit of a hack, we print out a bunch of diagnostic data in:
Note that this diagnostic data is based on a single instance whereas tests run on multiple machines as they are scaled-out. If all machines in the pool are consistent then this information is useful, otherwise it's just a hint.
@ViktorHofer as of this morning at least (had to take a couple rogue machines offline) all the OSX.1014.Amd64 machines should be uniformly updated to latest OpenSSL; please ping me for any variances you see there and CC @parose1
Looks like these tests just failed again from https://github.com/dotnet/corefx/pull/40996 / https://github.com/dotnet/corefx/pull/40997
MacOS Build x64_Debug test failures are unrelated:
https://dev.azure.com/dnceng/public/_build/results?buildId=348664&view=ms.vss-test-web.build-test-results-tab&runId=10473744&resultId=168822&paneView=debug
System.Security.Cryptography.OpenSsl.Tests on netcoreapp-OSX-Debug-x64-OSX.1014.Amd64.Open
System.Security.Cryptography.OpenSsl.Tests Total: 649, Errors: 0, Failed: 565, Skipped: 14, Time: 1.083s
Here's a related issue: https://github.com/dotnet/corefx/issues/40262
Example test failure:
System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.ReadWriteDsa1024EncryptedPkcs8_PasswordBytes [FAIL]
System.TypeInitializationException : The type initializer for 'Crypto' threw an exception.
---- System.TypeInitializationException : The type initializer for 'CryptoInitializer' threw an exception.
-------- System.DllNotFoundException : Unable to load shared library 'System.Security.Cryptography.Native.OpenSsl' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libSystem.Security.Cryptography.Native.OpenSsl, 1): image not found
Stack Trace:
at Interop.Crypto.DsaKeyCreateByExplicitParameters(SafeDsaHandle& dsa, Byte[] p, Int32 pLength, Byte[] q, Int32 qLength, Byte[] g, Int32 gLength, Byte[] y, Int32 yLength, Byte[] x, Int32 xLength)
/_/src/Common/src/System/Security/Cryptography/DSAOpenSsl.cs(112,0): at System.Security.Cryptography.DSAOpenSsl.ImportParameters(DSAParameters parameters)
/_/src/System.Security.Cryptography.Algorithms/src/System/Security/Cryptography/DSA.cs(349,0): at System.Security.Cryptography.DSA.ImportEncryptedPkcs8PrivateKey(ReadOnlySpan`1 passwordBytes, ReadOnlySpan`1 source, Int32& bytesRead)
/_/src/Common/src/System/Security/Cryptography/DSAOpenSsl.cs(132,0): at System.Security.Cryptography.DSAOpenSsl.ImportEncryptedPkcs8PrivateKey(ReadOnlySpan`1 passwordBytes, ReadOnlySpan`1 source, Int32& bytesRead)
/_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSAKeyFileTests.cs(593,0): at System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.<>c__DisplayClass23_0.<ReadBase64EncryptedPkcs8>b__0(DSA dsa, ReadOnlySpan`1 source, Int32& read)
/_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSAKeyFileTests.cs(652,0): at System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.ReadWriteKey(String base64, DSAParameters& expected, ReadKeyAction readAction, Func`2 writeArrayFunc, WriteKeyToSpanFunc writeSpanFunc, Boolean isEncrypted)
/_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSAKeyFileTests.cs(589,0): at System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.ReadBase64EncryptedPkcs8(String base64EncPkcs8, Byte[] passwordBytes, PbeParameters pbeParameters, DSAParameters& expected)
/_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSAKeyFileTests.cs(174,0): at System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.ReadWriteDsa1024EncryptedPkcs8_PasswordBytes()
----- Inner Stack Trace -----
/_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(40,0): at Interop.CryptoInitializer.Initialize()
/_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(19,0): at Interop.Crypto..cctor()
----- Inner Stack Trace -----
at Interop.CryptoInitializer.EnsureOpenSslInitialized()
/_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(27,0): at Interop.CryptoInitializer..cctor()
Other failures:
System.Security.Cryptography.EcDiffieHellman.Tests.ECDiffieHellmanTests.HmacDerivationIsStable [FAIL]
System.TypeInitializationException : The type initializer for 'Crypto' threw an exception.
---- System.TypeInitializationException : The type initializer for 'CryptoInitializer' threw an exception.
-------- System.DllNotFoundException : Unable to load shared library 'System.Security.Cryptography.Native.OpenSsl' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libSystem.Security.Cryptography.Native.OpenSsl, 1): image not found
Stack Trace:
at Interop.Crypto.DsaKeyCreateByExplicitParameters(SafeDsaHandle& dsa, Byte[] p, Int32 pLength, Byte[] q, Int32 qLength, Byte[] g, Int32 gLength, Byte[] y, Int32 yLength, Byte[] x, Int32 xLength)
/_/src/Common/src/System/Security/Cryptography/DSAOpenSsl.cs(112,0): at System.Security.Cryptography.DSAOpenSsl.ImportParameters(DSAParameters parameters)
/_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSASignVerify.cs(381,0): at System.Security.Cryptography.Dsa.Tests.DSASignVerify.SignAndVerify(Byte[] data, String hashAlgorithmName, DSAParameters dsaParameters, Int32 expectedSignatureLength)
/_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSASignVerify.cs(238,0): at System.Security.Cryptography.Dsa.Tests.DSASignVerify.SignAndVerifyDataExplicit1024()
----- Inner Stack Trace -----
/_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(40,0): at Interop.CryptoInitializer.Initialize()
/_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(19,0): at Interop.Crypto..cctor()
----- Inner Stack Trace -----
at Interop.CryptoInitializer.EnsureOpenSslInitialized()
/_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(27,0): at Interop.CryptoInitializer..cctor()
System.Security.Cryptography.Dsa.Tests.DSASignVerify_Span.SignAndVerifyDataExplicit1024 [FAIL]
System.TypeInitializationException : The type initializer for 'Crypto' threw an exception.
---- System.TypeInitializationException : The type initializer for 'CryptoInitializer' threw an exception.
-------- System.DllNotFoundException : Unable to load shared library 'System.Security.Cryptography.Native.OpenSsl' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libSystem.Security.Cryptography.Native.OpenSsl, 1): image not found
Stack Trace:
at Interop.Crypto.DsaKeyCreateByExplicitParameters(SafeDsaHandle& dsa, Byte[] p, Int32 pLength, Byte[] q, Int32 qLength, Byte[] g, Int32 gLength, Byte[] y, Int32 yLength, Byte[] x, Int32 xLength)
/_/src/Common/src/System/Security/Cryptography/DSAOpenSsl.cs(112,0): at System.Security.Cryptography.DSAOpenSsl.ImportParameters(DSAParameters parameters)
/_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSASignVerify.cs(381,0): at System.Security.Cryptography.Dsa.Tests.DSASignVerify.SignAndVerify(Byte[] data, String hashAlgorithmName, DSAParameters dsaParameters, Int32 expectedSignatureLength)
/_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSASignVerify.cs(238,0): at System.Security.Cryptography.Dsa.Tests.DSASignVerify.SignAndVerifyDataExplicit1024()
----- Inner Stack Trace -----
/_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(40,0): at Interop.CryptoInitializer.Initialize()
/_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(19,0): at Interop.Crypto..cctor()
----- Inner Stack Trace -----
at Interop.CryptoInitializer.EnsureOpenSslInitialized()
/_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(27,0): at Interop.CryptoInitializer..cctor()
@MattGal can you please check if a machine regressed and isn't running with the latest openssl version?
Alas I am OOF right now, but @ilyas1974 I think can help you. Looks like the machine got reimaged and lost the updated OpenSSL version yesterday.
Thanks. I'm not sure how much control we have other these machines but we really should find a mechanism to not loose these hard dependencies. We just discussed that in our infra meeting. We try to not lock ourselves too much on specific versions of dependencies (for our test runs) but for various reasons that's not possible with openssl.
Spoke with @ilyas1974 offline. Two clients had a different openssl version installed. Those will be fixed and the onboarding script will be updated to point to the latest 1.0.2t release.
https://github.com/dotnet/core-eng/issues/7834 to update the Onboarding scripts
https://github.com/dotnet/core-eng/issues/7835 to update the existing systems.
Closing as the machines are now patched and the issue shouldn't re-occur.
Re-opening as it failed again here: https://github.com/dotnet/runtime/pull/436
On netcoreapp-OSX-Debug-x64-OSX.1013.Amd64.Open
https://dev.azure.com/dnceng/public/_build/results?buildId=449153&view=ms.vss-test-web.build-test-results-tab&runId=14332654&paneView=debug&resultId=100000
Starting: System.Security.Cryptography.OpenSsl.Tests (parallel test collections = on, max threads = 4)
System.Security.Cryptography.EcDiffieHellman.Tests.ECDiffieHellmanTests.HmacDerivationIsStable [FAIL]
System.TypeInitializationException : The type initializer for 'Crypto' threw an exception.
---- System.TypeInitializationException : The type initializer for 'CryptoInitializer' threw an exception.
-------- System.DllNotFoundException : Unable to load shared library 'System.Security.Cryptography.Native.OpenSsl' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libSystem.Security.Cryptography.Native.OpenSsl, 1): image not found
Stack Trace:
/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EcKey.cs(17,0): at Interop.Crypto.EcKeyCreateByOid(String oid)
/_/src/libraries/Common/src/System/Security/Cryptography/ECOpenSsl.ImportExport.cs(184,0): at System.Security.Cryptography.ECOpenSsl.GenerateKeyByKeySize(Int32 keySize)
/_/src/libraries/Common/src/System/Security/Cryptography/ECOpenSsl.cs(37,0): at System.Security.Cryptography.ECOpenSsl.GenerateKeyLazy(AsymmetricAlgorithm owner)
/_/src/libraries/Common/src/System/Security/Cryptography/ECOpenSsl.cs(21,0): at System.Security.Cryptography.ECOpenSsl.<>c__DisplayClass2_0.<.ctor>b__0()
/_/src/libraries/System.Private.CoreLib/src/System/Lazy.cs(322,0): at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
/_/src/libraries/System.Private.CoreLib/src/System/Lazy.cs(346,0): at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
/_/src/libraries/System.Private.CoreLib/src/System/Lazy.cs(426,0): at System.Lazy`1.CreateValue()
/_/src/libraries/System.Private.CoreLib/src/System/Lazy.cs(507,0): at System.Lazy`1.get_Value()
/_/src/libraries/Common/src/System/Security/Cryptography/ECOpenSsl.cs(48,0): at System.Security.Cryptography.ECOpenSsl.UpRefKeyHandle()
/_/src/libraries/Common/src/System/Security/Cryptography/ECDiffieHellmanOpenSsl.cs(84,0): at System.Security.Cryptography.ECDiffieHellmanOpenSsl.get_PublicKey()
/_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDiffieHellman/ECDiffieHellmanTests.Hmac.cs(185,0): at System.Security.Cryptography.EcDiffieHellman.Tests.ECDiffieHellmanTests.HmacDerivationIsStable()
----- Inner Stack Trace -----
/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(40,0): at Interop.CryptoInitializer.Initialize()
/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(19,0): at Interop.Crypto..cctor()
----- Inner Stack Trace -----
at Interop.CryptoInitializer.EnsureOpenSslInitialized()
/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(27,0): at Interop.CryptoInitializer..cctor()
...
...
System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.NoFuzzySubjectPublicKeyInfo [FAIL]
System.TypeInitializationException : The type initializer for 'Crypto' threw an exception.
---- System.TypeInitializationException : The type initializer for 'CryptoInitializer' threw an exception.
-------- System.DllNotFoundException : Unable to load shared library 'System.Security.Cryptography.Native.OpenSsl' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libSystem.Security.Cryptography.Native.OpenSsl, 1): image not found
Stack Trace:
at Interop.Crypto.DsaKeyCreateByExplicitParameters(SafeDsaHandle& dsa, Byte[] p, Int32 pLength, Byte[] q, Int32 qLength, Byte[] g, Int32 gLength, Byte[] y, Int32 yLength, Byte[] x, Int32 xLength)
/_/src/libraries/Common/src/System/Security/Cryptography/DSAOpenSsl.cs(112,0): at System.Security.Cryptography.DSAOpenSsl.ImportParameters(DSAParameters parameters)
/_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSAKeyFileTests.cs(309,0): at System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.NoFuzzySubjectPublicKeyInfo()
----- Inner Stack Trace -----
/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(40,0): at Interop.CryptoInitializer.Initialize()
/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(19,0): at Interop.Crypto..cctor()
----- Inner Stack Trace -----
at Interop.CryptoInitializer.EnsureOpenSslInitialized()
/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(27,0): at Interop.CryptoInitializer..cctor()
Finished: System.Security.Cryptography.OpenSsl.Tests
=== TEST EXECUTION SUMMARY ===
System.Security.Cryptography.OpenSsl.Tests Total: 687, Errors: 0, Failed: 605, Skipped: 14, Time: 0.758s
/private/tmp/helix/working/6443eafb-99d8-4a99-943d-d98e792afdc8/Work/9be86154-d23f-4386-8a63-bc7646272730/Exec
----- end Fri Dec 6 06:48:57 PST 2019 ----- exit code 1 ----------------------------------------------------------
@ilyas1974 can you please check the openssl version installed on dci-mac-build-064.local? I think this regressed again.
He went home, I'm looking now.
This seems to have been resolved via infrastructure.