Runtime: MacOS - Special hidden folders

[hgbloch]

Dear corefx team,
We have encountered a special case of hidden folders in MacOs based machines.

When using DirectoryInfo to map the users machine, we are interested in knowing which folders are hidden, therefore checking the FileAttributes.Hidden attribute, which is determined if the file\directory name starts with a dot - '.'
Sadly, we found out that in MacOS uses additional file system flags to mark folders as - hidden.

We can check it by using the Terminal, or iTerm in my case and navigating to the home folder of a user and using ls -l to output the file dir.

as we can see there are no UNIX hidden folders which starts with a dot.
But Library is a hidden folder.
If we would pass the O flag to show file flags (-O is a flag only in MacOS)

As we can see there are additional flags that mark a folder as hidden.
to set or unset the flag we use chflags with the hidden or nohidden arguments.

.NET Core info

.NET Command Line Tools (2.0.0)

Product Information:
 Version:            2.0.0
 Commit SHA-1 hash:  cdcd1928c9

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.13
 OS Platform: Darwin
 RID:         osx.10.12-x64
 Base Path:   /usr/local/share/dotnet/sdk/2.0.0/

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.0
  Build    : e8b8861ac7faf042c87a5c2f9f2d04c98b69f28d

Is there anything we can do to workaround this?
I've poked around corefx + coreclr and saw that the DirectoryEntry struct only uses iNodeType to determine multiple attributes.
Is there a plan to support flags in the future?

Thank you very much,
Hagai.

[danmosemsft]

Seems that on macOS FileSystemAttributes.Hidden should check both the leading dot and this special system flag. Sound right @JeremyKuhne ?

[JeremyKuhne]

We should consider surfacing that flag. How does the terminal experience compare to files that start with a period? (And how does it compare to Linux?) I presume the flag only impacts the UI?

[hgbloch]

The main difference is that dotted folder or files are hidden by default in the terminal as well.
While finder hides both flagged folders and dotted folders by default.

In Linux on the other hand, dotted files\folders are hidden by default in terminal and on most of the Desktop Environments.

Our main problem is in the use case of mapping users folders and drives, for example Volumes is also an hidden folder that contains a symlink to the drive itself which in the end resolves in a recursive loop.

I hope it explains the difference and behavior.

[JeremyKuhne]

I believe we should return the readonly flag on MacOS if the file begins with period or if the MacOS specific flag is set. I'm still undecided on what set should do. Set both? Set neither if one is set? Set one of the two?

@pjanotti, any thoughts?

[danmosemsft]

I'm still undecided on what set should do. Set both? Set neither if one is set? Set one of the two

We don't ever prefix a dot for you so the two options are fiddle with write permissions (what we do right now on Unix) and set the magic Mac flag? My vote is both.

[pjanotti]

I agree with @danmosemsft: for macOS seems natural to fiddle with write permissions and set the macOS flag (we got covered about the intention of the user).

[JeremyKuhne]

Derp, my bad. I remembered incorrectly. We don't set hidden. So the answer is:

• On get, check the flag on MacOS if the period check fails (doesn't begin with)
• On set, if MacOS, add the flag if it doesn't start with a period or have the flag

Thoughts?

[pjanotti]

On set, if MacOS, add the flag if it doesn't start with a period or have the flag

I understood it to be "add the flag if it starts with a period or Hidden was specified"

[nabeelomer]

Hello, I'd like to work on this.

[danmosemsft]

@nabeelomer go for it! @karelz can you please add so we can assign.

[karelz]

@nabeelomer I sent you collaborator invite, so that we can assign the work item to you (GH limitation). Ping me when you accept, assigning to myself temporarily.
Pro-tip: Becoming collaborator will automatically subscribe you to all repo notifications (500+ per day). We recommend to switch to "Not Watching" which will send you notifications only on your mentions and explicit issue subscriptions, assignments, etc.

[nabeelomer]

@karelz Done and thanks for the tip.

[nabeelomer]

I have a question, supposing I've built the corefx using the build.sh script, how do I run a binary that I built with the version of .NET core installed on my computer such that it uses the libraries that I built?

[karelz]

See dogfooding instructions: https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/dogfooding.md

[pjanotti]

@nabeelomer keep in mind that adding a test for macOS that covers the issue maybe easier than dogfooding (we are going to ask about tests anyway :smile:). Thanks for picking this up.

[nabeelomer]

@pjanotti Where will the test for this particular feature go? I was thinking https://github.com/dotnet/corefx/blob/master/src/System.IO.FileSystem/tests/Enumeration/AttributeTests.netcoreapp.cs#L117

Its my first time working on a major project, so figuring out what goes where is difficult.

[pjanotti]

@nabeelomer that test is focusing on enumeration but it will be good to cover it there too, but also take a look at the tests dealing directly with files and directories attributes.

[davidkaya]

@nabeelomer Are you working on this issue or can I pick it up?

[karelz]

@davidkaya it's been a long time, feel free to pick it up ...

[karelz]

I sent you collaborator invite. Once you accept, ping me and we will be able to assign it to you.
Pro-tip: Becoming collaborator will automatically subscribe you to all repo notifications (500+ per day), we recommend to set the Watching setting to Not Watching - which will send you notifications only for issues where you are mentioned, assigned or explicitly subscribed.

[davidkaya]

@karelz Thanks! Accepted.

[danmosemsft]

All yours!

[davidkaya]

@danmosemsft I've got a problem with tests. I got following SIGABRT

----- start 21:38:54 =============== To repro directly: =====================================================
  pushd /Users/kaya/workspace/personal/corefx/bin/tests/System.IO.FileSystem.Tests/netcoreapp-OSX-Debug-x64
  chmod +x /Users/kaya/workspace/personal/corefx/bin/testhost/netcoreapp-OSX-Debug-x64//dotnet
  /Users/kaya/workspace/personal/corefx/bin/testhost/netcoreapp-OSX-Debug-x64//dotnet xunit.console.netcore.exe System.IO.FileSystem.Tests.dll  -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapptests -notrait category=nonosxtests  -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
  ~/workspace/personal/corefx/bin/tests/System.IO.FileSystem.Tests/netcoreapp-OSX-Debug-x64 ~/workspace/personal/corefx/src/System.IO.FileSystem/tests
  xUnit.net console test runner (64-bit .NET Core)
  Copyright (C) 2014 Outercurve Foundation.

  Discovering: System.IO.FileSystem.Tests
  /Users/kaya/workspace/personal/corefx/bin/tests/System.IO.FileSystem.Tests/netcoreapp-OSX-Debug-x64/RunTests.sh: line 124: 55235 Abort trap: 6           $RUNTIME_PATH/dotnet xunit.console.netcore.exe System.IO.FileSystem.Tests.dll -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapptests -notrait category=nonosxtests -notrait category=OuterLoop -notrait category=failing
  ~/workspace/personal/corefx/src/System.IO.FileSystem/tests
  ----- end 21:38:54 ----- exit code 134 ----------------------------------------------------------
  exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
/Users/kaya/workspace/personal/corefx/Tools/tests.targets(514,5): warning MSB3073: The command "/Users/kaya/workspace/personal/corefx/bin/tests/System.IO.FileSystem.Tests/netcoreapp-OSX-Debug-x64/RunTests.sh /Users/kaya/workspace/personal/corefx/bin/testhost/netcoreapp-OSX-Debug-x64/" exited with code 134. [/Users/kaya/workspace/personal/corefx/src/System.IO.FileSystem/tests/System.IO.FileSystem.Tests.csproj]
/Users/kaya/workspace/personal/corefx/Tools/tests.targets(522,5): error : One or more tests failed while running tests from 'System.IO.FileSystem.Tests' please check /Users/kaya/workspace/personal/corefx/bin/tests/System.IO.FileSystem.Tests/netcoreapp-OSX-Debug-x64/testResults.xml for details! [/Users/kaya/workspace/personal/corefx/src/System.IO.FileSystem/tests/System.IO.FileSystem.Tests.csproj]

I had this problem in the beginning however after ./clean.sh it disappeared. Now I encountered it again but now ./clean.sh does not help. This happens with all of the tests (even in other projects - non FileSystem related). Do you happen to know what could be the problem?

[danmosemsft]

No. Can you use a debugger to get acallstack? Try lldb: there are docs in this repo.

[davidkaya]

@danmosemsft well, I got this in VS Code, last thing logged is "Discovering: System.IO.FileSystem.Tests"

I've opened the dumped core in lldb and output for bt all is this:

* thread dotnet/corefx#1, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff58eeba16 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame dotnet/corefx#1: 0x00007fff590b4589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame dotnet/corefx#2: 0x0000000108f4b112 libcoreclr.dylib`CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 338
    frame dotnet/corefx#3: 0x0000000108f4ad27 libcoreclr.dylib`CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) + 375
    frame dotnet/corefx#4: 0x0000000108f4f6a3 libcoreclr.dylib`CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) + 1891
    frame dotnet/corefx#5: 0x0000000108f4f932 libcoreclr.dylib`WaitForMultipleObjectsEx + 82
    frame dotnet/corefx#6: 0x00000001090d6440 libcoreclr.dylib`Thread::DoAppropriateWaitWorker(int, void**, int, unsigned int, WaitMode) + 912
    frame dotnet/corefx#7: 0x00000001090d10d0 libcoreclr.dylib`Thread::DoAppropriateWait(int, void**, int, unsigned int, WaitMode, PendingSync*) + 48
    frame dotnet/corefx#8: 0x000000010913ba0a libcoreclr.dylib`WaitHandleNative::CorWaitOneNative(SafeHandle*, int, bool, bool) + 266
    frame dotnet/corefx#9: 0x0000000117b3523c
    frame dotnet/corefx#10: 0x0000000117fe8296
    frame dotnet/corefx#11: 0x0000000117fde330
    frame dotnet/corefx#12: 0x0000000117fd1d8d
    frame dotnet/corefx#13: 0x00000001092afcf7 libcoreclr.dylib`CallDescrWorkerInternal + 124
    frame dotnet/corefx#14: 0x0000000109117f11 libcoreclr.dylib`MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 945
    frame dotnet/corefx#15: 0x0000000108fe9e94 libcoreclr.dylib`RunMain(MethodDesc*, short, int*, PtrArray**) + 724
    frame dotnet/corefx#16: 0x0000000108fea12f libcoreclr.dylib`Assembly::ExecuteMainMethod(PtrArray**, int) + 223
    frame dotnet/runtime#13850: 0x000000010902ab64 libcoreclr.dylib`CorHost2::ExecuteAssembly(unsigned int, char16_t const*, int, char16_t const**, unsigned int*) + 436
    frame dotnet/runtime#13851: 0x0000000108f5c516 libcoreclr.dylib`coreclr_execute_assembly + 230
    frame dotnet/corefx#19: 0x0000000108ea6e08 libhostpolicy.dylib`___lldb_unnamed_symbol948$$libhostpolicy.dylib + 152
    frame dotnet/runtime#13852: 0x0000000108e90299 libhostpolicy.dylib`___lldb_unnamed_symbol752$$libhostpolicy.dylib + 35929
    frame dotnet/corefx#21: 0x0000000108e96b09 libhostpolicy.dylib`___lldb_unnamed_symbol774$$libhostpolicy.dylib + 297
    frame dotnet/runtime#13853: 0x0000000108dbb2e3 libhostfxr.dylib`___lldb_unnamed_symbol931$$libhostfxr.dylib + 451
    frame dotnet/corefx#23: 0x0000000108dd8b2e libhostfxr.dylib`___lldb_unnamed_symbol995$$libhostfxr.dylib + 18606
    frame dotnet/corefx#24: 0x0000000108ddb702 libhostfxr.dylib`___lldb_unnamed_symbol1004$$libhostfxr.dylib + 1602
    frame dotnet/corefx#25: 0x0000000108dd9bb8 libhostfxr.dylib`___lldb_unnamed_symbol1002$$libhostfxr.dylib + 1976
    frame dotnet/runtime#13854: 0x0000000108dbc359 libhostfxr.dylib`___lldb_unnamed_symbol934$$libhostfxr.dylib + 297
    frame dotnet/corefx#27: 0x0000000108d04042 dotnet`___lldb_unnamed_symbol58$$dotnet + 7842
    frame dotnet/corefx#28: 0x0000000108d04635 dotnet`___lldb_unnamed_symbol60$$dotnet + 165
    frame dotnet/corefx#29: 0x00007fff58d9b015 libdyld.dylib`start + 1
  thread dotnet/corefx#2, stop reason = signal SIGSTOP
    frame #0: 0x00007fff58ee220a libsystem_kernel.dylib`mach_msg_trap + 10
    frame dotnet/corefx#1: 0x00007fff58ee1724 libsystem_kernel.dylib`mach_msg + 60
    frame dotnet/corefx#2: 0x0000000108f58d98 libcoreclr.dylib`MachMessage::Receive(unsigned int) + 72
    frame dotnet/corefx#3: 0x0000000108f57cee libcoreclr.dylib`SEHExceptionThread(void*) + 94
    frame dotnet/corefx#4: 0x00007fff590b3661 libsystem_pthread.dylib`_pthread_body + 340
    frame dotnet/corefx#5: 0x00007fff590b350d libsystem_pthread.dylib`_pthread_start + 377
    frame dotnet/corefx#6: 0x00007fff590b2bf9 libsystem_pthread.dylib`thread_start + 13
  thread dotnet/corefx#3, stop reason = signal SIGSTOP
    frame #0: 0x00007fff58eed09a libsystem_kernel.dylib`poll + 10
    frame dotnet/corefx#1: 0x0000000108f4b41e libcoreclr.dylib`CorUnix::CPalSynchronizationManager::ThreadPrepareForShutdown() + 30
    frame dotnet/corefx#2: 0x0000000108f4cf50 libcoreclr.dylib`CorUnix::CPalSynchronizationManager::WorkerThread(void*) + 944
    frame dotnet/corefx#3: 0x0000000108f55748 libcoreclr.dylib`CorUnix::CPalThread::ThreadEntry(void*) + 328
    frame dotnet/corefx#4: 0x00007fff590b3661 libsystem_pthread.dylib`_pthread_body + 340
    frame dotnet/corefx#5: 0x00007fff590b350d libsystem_pthread.dylib`_pthread_start + 377
    frame dotnet/corefx#6: 0x00007fff590b2bf9 libsystem_pthread.dylib`thread_start + 13
  thread dotnet/corefx#4, stop reason = signal SIGSTOP
    frame #0: 0x00007fff58eeb862 libsystem_kernel.dylib`__open + 10
    frame dotnet/corefx#1: 0x0000000108fc820f libcoreclr.dylib`TwoWayPipe::WaitForConnection() + 31
    frame dotnet/corefx#2: 0x0000000108fbfc2f libcoreclr.dylib`DbgTransportSession::TransportWorker() + 159
    frame dotnet/corefx#3: 0x0000000108fbe589 libcoreclr.dylib`DbgTransportSession::TransportWorkerStatic(void*) + 9
    frame dotnet/corefx#4: 0x0000000108f55748 libcoreclr.dylib`CorUnix::CPalThread::ThreadEntry(void*) + 328
    frame dotnet/corefx#5: 0x00007fff590b3661 libsystem_pthread.dylib`_pthread_body + 340
    frame dotnet/corefx#6: 0x00007fff590b350d libsystem_pthread.dylib`_pthread_start + 377
    frame dotnet/corefx#7: 0x00007fff590b2bf9 libsystem_pthread.dylib`thread_start + 13
  thread dotnet/corefx#5, stop reason = signal SIGSTOP
    frame #0: 0x00007fff58eeba16 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame dotnet/corefx#1: 0x00007fff590b4589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame dotnet/corefx#2: 0x0000000108f4b112 libcoreclr.dylib`CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 338
    frame dotnet/corefx#3: 0x0000000108f4ad27 libcoreclr.dylib`CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) + 375
    frame dotnet/corefx#4: 0x0000000108f4f6a3 libcoreclr.dylib`CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) + 1891
    frame dotnet/corefx#5: 0x0000000108f4f932 libcoreclr.dylib`WaitForMultipleObjectsEx + 82
    frame dotnet/corefx#6: 0x0000000108fbc9d8 libcoreclr.dylib`DebuggerRCThread::MainLoop() + 264
    frame dotnet/corefx#7: 0x0000000108fbc86b libcoreclr.dylib`DebuggerRCThread::ThreadProc() + 251
    frame dotnet/corefx#8: 0x0000000108fbc574 libcoreclr.dylib`DebuggerRCThread::ThreadProcStatic(void*) + 132
    frame dotnet/corefx#9: 0x0000000108f55748 libcoreclr.dylib`CorUnix::CPalThread::ThreadEntry(void*) + 328
    frame dotnet/corefx#10: 0x00007fff590b3661 libsystem_pthread.dylib`_pthread_body + 340
    frame dotnet/corefx#11: 0x00007fff590b350d libsystem_pthread.dylib`_pthread_start + 377
    frame dotnet/corefx#12: 0x00007fff590b2bf9 libsystem_pthread.dylib`thread_start + 13
  thread dotnet/corefx#6, stop reason = signal SIGSTOP
    frame #0: 0x00007fff58eeba16 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame dotnet/corefx#1: 0x00007fff590b4589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame dotnet/corefx#2: 0x0000000108f4b0d5 libcoreclr.dylib`CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 277
    frame dotnet/corefx#3: 0x0000000108f4ad27 libcoreclr.dylib`CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) + 375
    frame dotnet/corefx#4: 0x0000000108f4f6a3 libcoreclr.dylib`CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) + 1891
    frame dotnet/corefx#5: 0x0000000108f4f86d libcoreclr.dylib`WaitForSingleObjectEx + 77
    frame dotnet/corefx#6: 0x00000001091f6fee libcoreclr.dylib`CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync*) + 206
    frame dotnet/corefx#7: 0x000000010916132f libcoreclr.dylib`FinalizerThread::WaitForFinalizerEvent(CLREvent*) + 31
    frame dotnet/corefx#8: 0x00000001091614a2 libcoreclr.dylib`FinalizerThread::FinalizerThreadWorker(void*) + 114
    frame dotnet/corefx#9: 0x00000001090dbb73 libcoreclr.dylib`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 419
    frame dotnet/corefx#10: 0x00000001090dc219 libcoreclr.dylib`ManagedThreadBase::FinalizerBase(void (*)(void*)) + 73
    frame dotnet/corefx#11: 0x00000001091617cc libcoreclr.dylib`FinalizerThread::FinalizerThreadStart(void*) + 204
    frame dotnet/corefx#12: 0x0000000108f55748 libcoreclr.dylib`CorUnix::CPalThread::ThreadEntry(void*) + 328
    frame dotnet/corefx#13: 0x00007fff590b3661 libsystem_pthread.dylib`_pthread_body + 340
    frame dotnet/corefx#14: 0x00007fff590b350d libsystem_pthread.dylib`_pthread_start + 377
    frame dotnet/corefx#15: 0x00007fff590b2bf9 libsystem_pthread.dylib`thread_start + 13
  thread dotnet/corefx#7, stop reason = signal SIGSTOP
    frame #0: 0x00007fff58ee220a libsystem_kernel.dylib`mach_msg_trap + 10
    frame dotnet/corefx#1: 0x00007fff58ee1724 libsystem_kernel.dylib`mach_msg + 60
    frame dotnet/corefx#2: 0x0000000109ccd8a8 libclrjit.dylib`MachMessage::Receive(unsigned int) + 72
    frame dotnet/corefx#3: 0x0000000109ccc7fe libclrjit.dylib`SEHExceptionThread(void*) + 94
    frame dotnet/corefx#4: 0x00007fff590b3661 libsystem_pthread.dylib`_pthread_body + 340
    frame dotnet/corefx#5: 0x00007fff590b350d libsystem_pthread.dylib`_pthread_start + 377
    frame dotnet/corefx#6: 0x00007fff590b2bf9 libsystem_pthread.dylib`thread_start + 13
  thread dotnet/corefx#8, stop reason = signal SIGSTOP
    frame #0: 0x00007fff58eebb66 libsystem_kernel.dylib`__pthread_kill + 10
    frame dotnet/corefx#1: 0x00007fff590b6080 libsystem_pthread.dylib`pthread_kill + 333
    frame dotnet/corefx#2: 0x00007fff58e471ae libsystem_c.dylib`abort + 127
    frame dotnet/corefx#3: 0x0000000108f53cde libcoreclr.dylib`PROCAbort + 14
    frame dotnet/corefx#4: 0x0000000108f52852 libcoreclr.dylib`PROCEndProcess(void*, unsigned int, int) + 226
    frame dotnet/corefx#5: 0x0000000109147f79 libcoreclr.dylib`SafeExitProcess(unsigned int, int, ShutdownCompleteAction) + 489
    frame dotnet/corefx#6: 0x0000000109149837 libcoreclr.dylib`EEPolicy::HandleFatalError(unsigned int, unsigned long, char16_t const*, _EXCEPTION_POINTERS*, char16_t const*, char16_t const*) + 663
    frame dotnet/corefx#7: 0x00000001092243bf libcoreclr.dylib`ProcessCLRException + 1535
    frame dotnet/corefx#8: 0x0000000109228a56 libcoreclr.dylib`UnwindManagedExceptionPass1(PAL_SEHException&, _CONTEXT*) + 358
    frame dotnet/corefx#9: 0x0000000109228d70 libcoreclr.dylib`DispatchManagedException(PAL_SEHException&, bool) + 304
    frame dotnet/corefx#10: 0x0000000109222fed libcoreclr.dylib`HandleHardwareException(PAL_SEHException*) + 669
    frame dotnet/corefx#11: 0x0000000108f1bfe1 libcoreclr.dylib`SEHProcessException(PAL_SEHException*) + 353
    frame dotnet/corefx#12: 0x0000000108f57b75 libcoreclr.dylib`PAL_DispatchException + 181
    frame dotnet/corefx#13: 0x0000000108f57717 libcoreclr.dylib`PAL_DispatchExceptionWrapper + 10
    frame dotnet/corefx#14: 0x0000000117a7dd3c
    frame dotnet/corefx#15: 0x0000000117a7dc0c
    frame dotnet/corefx#16: 0x0000000117a7e3e1
    frame dotnet/runtime#13850: 0x0000000117a7876d
    frame dotnet/runtime#13851: 0x0000000117a1f0ea
    frame dotnet/corefx#19: 0x0000000118017640
    frame dotnet/runtime#13852: 0x0000000118017356
    frame dotnet/corefx#21: 0x0000000118016c83
    frame dotnet/runtime#13853: 0x0000000118016b38
    frame dotnet/corefx#23: 0x0000000118016ad0
    frame dotnet/corefx#24: 0x00000001092b0d8b libcoreclr.dylib`UMThunkStub + 273
    frame dotnet/corefx#25: 0x000000010b7ee8d4 System.Native.dylib`SignalHandlerLoop(arg=0x00007ff75a5482f0) at pal_signal.c:154
    frame dotnet/runtime#13854: 0x00007fff590b3661 libsystem_pthread.dylib`_pthread_body + 340
    frame dotnet/corefx#27: 0x00007fff590b350d libsystem_pthread.dylib`_pthread_start + 377
    frame dotnet/corefx#28: 0x00007fff590b2bf9 libsystem_pthread.dylib`thread_start + 13
  thread dotnet/corefx#9, stop reason = signal SIGSTOP
    frame #0: 0x00007fff58eeba16 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame dotnet/corefx#1: 0x00007fff590b4589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame dotnet/corefx#2: 0x0000000108f4b112 libcoreclr.dylib`CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 338
    frame dotnet/corefx#3: 0x0000000108f4ad27 libcoreclr.dylib`CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) + 375
    frame dotnet/corefx#4: 0x0000000108f4f6a3 libcoreclr.dylib`CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) + 1891
    frame dotnet/corefx#5: 0x0000000108f4f932 libcoreclr.dylib`WaitForMultipleObjectsEx + 82
    frame dotnet/corefx#6: 0x00000001090d6440 libcoreclr.dylib`Thread::DoAppropriateWaitWorker(int, void**, int, unsigned int, WaitMode) + 912
    frame dotnet/corefx#7: 0x00000001090d10d0 libcoreclr.dylib`Thread::DoAppropriateWait(int, void**, int, unsigned int, WaitMode, PendingSync*) + 48
    frame dotnet/corefx#8: 0x000000010913ba0a libcoreclr.dylib`WaitHandleNative::CorWaitOneNative(SafeHandle*, int, bool, bool) + 266
    frame dotnet/corefx#9: 0x0000000117b3523c
    frame dotnet/corefx#10: 0x0000000118016354
    frame dotnet/corefx#11: 0x0000000118016146
    frame dotnet/corefx#12: 0x0000000118016077
    frame dotnet/corefx#13: 0x0000000118016011
    frame dotnet/corefx#14: 0x00000001180101c4
    frame dotnet/corefx#15: 0x000000011800f7d7
    frame dotnet/corefx#16: 0x000000011800f66e
    frame dotnet/runtime#13850: 0x000000011800f5b5
    frame dotnet/runtime#13851: 0x000000011800f4e4
    frame dotnet/corefx#19: 0x000000011800cc16
    frame dotnet/runtime#13852: 0x00000001092afcf7 libcoreclr.dylib`CallDescrWorkerInternal + 124
    frame dotnet/corefx#21: 0x000000010911742e libcoreclr.dylib`CallDescrWorkerWithHandler(CallDescrData*, int) + 110
    frame dotnet/runtime#13853: 0x00000001091d0dcf libcoreclr.dylib`CallDescrWorkerReflectionWrapper(CallDescrData*, Frame*) + 127
    frame dotnet/corefx#23: 0x00000001091d1d3f libcoreclr.dylib`RuntimeMethodHandle::InvokeMethod(Object*, PtrArray*, SignatureNative*, bool, bool) + 3039
    frame dotnet/corefx#24: 0x0000000117a7427c
    frame dotnet/corefx#25: 0x000000011800c21b
    frame dotnet/runtime#13854: 0x0000000118003a41
    frame dotnet/corefx#27: 0x0000000118002cd4
    frame dotnet/corefx#28: 0x00000001180026ac
    frame dotnet/corefx#29: 0x0000000118000506
    frame dotnet/corefx#30: 0x0000000118000341
    frame dotnet/corefx#31: 0x0000000117a0ef8a
    frame dotnet/corefx#32: 0x0000000117ad627f
    frame dotnet/corefx#33: 0x0000000117a0ef8a
    frame dotnet/corefx#34: 0x00000001092afcf7 libcoreclr.dylib`CallDescrWorkerInternal + 124
    frame dotnet/runtime#13855: 0x0000000109117f11 libcoreclr.dylib`MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 945
    frame dotnet/runtime#13856: 0x000000010912c04f libcoreclr.dylib`ThreadNative::KickOffThread_Worker(void*) + 431
    frame dotnet/corefx#37: 0x00000001090dbb73 libcoreclr.dylib`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 419
    frame dotnet/corefx#38: 0x00000001090dc183 libcoreclr.dylib`ManagedThreadBase::KickOff(ADID, void (*)(void*), void*) + 51
    frame dotnet/corefx#39: 0x000000010912c26f libcoreclr.dylib`ThreadNative::KickOffThread(void*) + 351
    frame dotnet/corefx#40: 0x0000000108f55748 libcoreclr.dylib`CorUnix::CPalThread::ThreadEntry(void*) + 328
    frame dotnet/runtime#13857: 0x00007fff590b3661 libsystem_pthread.dylib`_pthread_body + 340
    frame dotnet/corefx#42: 0x00007fff590b350d libsystem_pthread.dylib`_pthread_start + 377
    frame dotnet/runtime#13858: 0x00007fff590b2bf9 libsystem_pthread.dylib`thread_start + 13
  thread dotnet/corefx#10, stop reason = signal SIGSTOP
    frame #0: 0x00007fff58eeba16 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame dotnet/corefx#1: 0x00007fff590b4589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame dotnet/corefx#2: 0x0000000108f4b112 libcoreclr.dylib`CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 338
    frame dotnet/corefx#3: 0x0000000108f4ad27 libcoreclr.dylib`CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) + 375
    frame dotnet/corefx#4: 0x0000000108f4f6a3 libcoreclr.dylib`CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) + 1891
    frame dotnet/corefx#5: 0x0000000108f4f932 libcoreclr.dylib`WaitForMultipleObjectsEx + 82
    frame dotnet/corefx#6: 0x00000001090d6440 libcoreclr.dylib`Thread::DoAppropriateWaitWorker(int, void**, int, unsigned int, WaitMode) + 912
    frame dotnet/corefx#7: 0x00000001090d10d0 libcoreclr.dylib`Thread::DoAppropriateWait(int, void**, int, unsigned int, WaitMode, PendingSync*) + 48
    frame dotnet/corefx#8: 0x000000010913ba0a libcoreclr.dylib`WaitHandleNative::CorWaitOneNative(SafeHandle*, int, bool, bool) + 266
    frame dotnet/corefx#9: 0x0000000118001047
    frame dotnet/corefx#10: 0x0000000118000341
    frame dotnet/corefx#11: 0x0000000117a0ef8a
    frame dotnet/corefx#12: 0x0000000117ad627f
    frame dotnet/corefx#13: 0x0000000117a0ef8a
    frame dotnet/corefx#14: 0x00000001092afcf7 libcoreclr.dylib`CallDescrWorkerInternal + 124
    frame dotnet/corefx#15: 0x0000000109117f11 libcoreclr.dylib`MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 945
    frame dotnet/corefx#16: 0x000000010912c04f libcoreclr.dylib`ThreadNative::KickOffThread_Worker(void*) + 431
    frame dotnet/runtime#13850: 0x00000001090dbb73 libcoreclr.dylib`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 419
    frame dotnet/runtime#13851: 0x00000001090dc183 libcoreclr.dylib`ManagedThreadBase::KickOff(ADID, void (*)(void*), void*) + 51
    frame dotnet/corefx#19: 0x000000010912c26f libcoreclr.dylib`ThreadNative::KickOffThread(void*) + 351
    frame dotnet/runtime#13852: 0x0000000108f55748 libcoreclr.dylib`CorUnix::CPalThread::ThreadEntry(void*) + 328
    frame dotnet/corefx#21: 0x00007fff590b3661 libsystem_pthread.dylib`_pthread_body + 340
    frame dotnet/runtime#13853: 0x00007fff590b350d libsystem_pthread.dylib`_pthread_start + 377
    frame dotnet/corefx#23: 0x00007fff590b2bf9 libsystem_pthread.dylib`thread_start + 13

[danmosemsft]

@tarekgH maybe could help

[tarekgh]

I believe this either a runtime issue or there is a real memory corruption. It just happened to perform corelcr internal call for DateTime.Now. the PAL call for DateTime.Now is very simple and not doing anything with the memory
https://github.com/dotnet/coreclr/blob/dd651509b9082f3117000e168007d5e1e1e72d32/src/pal/src/file/filetime.cpp#L159

@danmosemsft who from the runtime can help with this? CC @jkotas

  thread dotnet/corefx#8, stop reason = signal SIGSTOP
    frame #0: 0x00007fff58eebb66 libsystem_kernel.dylib`__pthread_kill + 10
    frame dotnet/corefx#1: 0x00007fff590b6080 libsystem_pthread.dylib`pthread_kill + 333
    frame dotnet/corefx#2: 0x00007fff58e471ae libsystem_c.dylib`abort + 127
    frame dotnet/corefx#3: 0x0000000108f53cde libcoreclr.dylib`PROCAbort + 14
    frame dotnet/corefx#4: 0x0000000108f52852 libcoreclr.dylib`PROCEndProcess(void*, unsigned int, int) + 226
    frame dotnet/corefx#5: 0x0000000109147f79 libcoreclr.dylib`SafeExitProcess(unsigned int, int, ShutdownCompleteAction) + 489
    frame dotnet/corefx#6: 0x0000000109149837 libcoreclr.dylib`EEPolicy::HandleFatalError(unsigned int, unsigned long, char16_t const*, _EXCEPTION_POINTERS*, char16_t const*, char16_t const*) + 663
    frame dotnet/corefx#7: 0x00000001092243bf libcoreclr.dylib`ProcessCLRException + 1535
    frame dotnet/corefx#8: 0x0000000109228a56 libcoreclr.dylib`UnwindManagedExceptionPass1(PAL_SEHException&, _CONTEXT*) + 358
    frame dotnet/corefx#9: 0x0000000109228d70 libcoreclr.dylib`DispatchManagedException(PAL_SEHException&, bool) + 304
    frame dotnet/corefx#10: 0x0000000109222fed libcoreclr.dylib`HandleHardwareException(PAL_SEHException*) + 669
    frame dotnet/corefx#11: 0x0000000108f1bfe1 libcoreclr.dylib`SEHProcessException(PAL_SEHException*) + 353
    frame dotnet/corefx#12: 0x0000000108f57b75 libcoreclr.dylib`PAL_DispatchException + 181
    frame dotnet/corefx#13: 0x0000000108f57717 libcoreclr.dylib`PAL_DispatchExceptionWrapper + 10
    frame dotnet/corefx#14: 0x0000000117a7dd3c
    frame dotnet/corefx#15: 0x0000000117a7dc0c
    frame dotnet/corefx#16: 0x0000000117a7e3e1
    frame dotnet/runtime#13850: 0x0000000117a7876d
    frame dotnet/runtime#13851: 0x0000000117a1f0ea
    frame dotnet/corefx#19: 0x0000000118017640
    frame dotnet/runtime#13852: 0x0000000118017356
    frame dotnet/corefx#21: 0x0000000118016c83
    frame dotnet/runtime#13853: 0x0000000118016b38
    frame dotnet/corefx#23: 0x0000000118016ad0
    frame dotnet/corefx#24: 0x00000001092b0d8b libcoreclr.dylib`UMThunkStub + 273
    frame dotnet/corefx#25: 0x000000010b7ee8d4 System.Native.dylib`SignalHandlerLoop(arg=0x00007ff75a5482f0) at pal_signal.c:154
    frame dotnet/runtime#13854: 0x00007fff590b3661 libsystem_pthread.dylib`_pthread_body + 340
    frame dotnet/corefx#27: 0x00007fff590b350d libsystem_pthread.dylib`_pthread_start + 377
    frame dotnet/corefx#28: 0x00007fff590b2bf9 libsystem_pthread.dylib`thread_start + 13

[jkotas]

@tarekgh What are the repro steps?

[tarekgh]

What are the repro steps?

@davidkaya mentioned that in the comment https://github.com/dotnet/corefx/issues/29323#issuecomment-408700828 he may send the debugger dump file too if cannot repro it locally.

[davidkaya]

Basically what happened is that everything was working fine, I made changes in FileSystem (C# part). Then I made changes to Native code (pal_io.c) and thus rebuilt the whole corefx (I do not know whether I called ./build-native.sh or ./build.sh). After that the discovery part started failing. I do not know whether the changes could've caused that.

[davidkaya]

After a very long pause I was finally able to get back to the issue. I have the PR somehow ready however there is one problem that I encountered and do not know how to correctly do it.

UF_HIDDEN flag is available (as far as I found, I might be mistaken) only on OSX and FreeBSD. FileStatus.Unix.cs is for all Unix targets. So should there be new FileStatus.OSX.cs (+ FreeBSD), should I ignore FreeBSD? It should _not_ be a problem if we get attributes (since it does not send the UF_HIDDEN flag to the system, just ANDs it). However if the user add/removes the attributes it calls

if ((attributes & FileAttributes.Hidden) != 0)
    Interop.Sys.LChflags(path, (uint)Interop.Sys.UserFlags.UF_HIDDEN);
else
    Interop.Sys.LChflags(path, (_fileStatus.UserFlags & ~(uint)Interop.Sys.UserFlags.UF_HIDDEN));

@JeremyKuhne any suggestions?

[danmosemsft]

You can use something like this in your csproj:

<ItemGroup Condition=" '$(TargetsOsx)' == 'true' OR '$(TargetsFreeBSD)' == 'true'">

[JeremyKuhne]

You can use something like this in your csproj:

We're not building specifically for OSX for this project. I don't think that we should.

System.Runtime.InteropServices.IsOSPlatform() will tell you if you are OSX or FreeBSD. That said, we typically abstract this sort of thing through the PAL (native abstraction layer) for Unix and put it behind an API check, something like Interop.Sys.HasChFlags(). A similar functionality check: https://github.com/dotnet/corefx/blob/master/src/Native/Unix/System.Native/pal_io.c#L570.

https://github.com/dotnet/corefx/blob/master/src/Native/Unix/configure.cmake is where we set up defines for API availability.

@stephentoub Please correct me / elaborate if I'm off track or missing something here.

[stephentoub]

Please correct me / elaborate if I'm off track or missing something here.

That sounds about right.

It seems like the get case should be fairly easy, since it's returned in st_flags from a stat call... we should probably just update https://github.com/dotnet/corefx/blob/775347f642ae10badf82bcd0352c9f316093cd6f/src/Common/src/CoreLib/Interop/Unix/System.Native/Interop.Stat.cs#L50 to include an entry for that, and set that if UF_HIDDEN is set (guarded behind a cmake define as Jeremy suggests.

For set, we could then just expose a LChFlags, which would be a nop if that define wasn't implemented. Normally I'd be a little concerned about the unnecessary P/Invoke overhead, but in this case I wouldn't expect the overhead to be particularly impactful, especially since we'd only be invoking it when hidden was set during a call to set attributes.

[davidkaya]

@stephentoub Basically thats very similar to what I already did (https://github.com/davidkaya/corefx/pull/1/files) however I am also invoking the LChFlags when it is not set so it actually removes the flag from the file when the user wants to remove it so there might be a overhead since it would be called always when the flag is not there however it might be solved by the proposal below.

@JeremyKuhne Well but we cannot put whole LChFlags behind the check just because UF_HIDDEN might have not been defined. Calling chflags("testfile", 0x4000); sets the st_flags to 0x4000 even though the 0x4000 is not defined anywhere (in stat.h), it does not check whether the flag makes sense. Wouldn't it be better to just use System.Runtime.InteropServices.IsOSPlatform() in FileStatus.Unix.cs to check the platform and if it is OSX || FreeBSD then call Interop.LChflags with UF_HIDDEN? Because Interop.LChflags might still be used by someone else who might want to change different flags.

I have one more question. Since I have extended Interop.Stat.cs in CoreLib, should this be in the PR to CoreFX or in separate PR to CoreClr?

[JeremyKuhne]

Well but we cannot put whole LChFlags behind the check just because UF_HIDDEN might have not been defined.

I'm not sure I understand why that is blocking. From what I can see the API shouldn't exist on platforms where this doesn't make sense, and even if it did you can also check for the flag being defined as your gate.

Worst comes to worst we can check the platform at runtime. That is something we want to avoid unless we really know we have to and would likely do in the PAL anyway. The whole point of the PAL is to contain this logic as much as possible in one distribution-specific binary. If we need to check for the hidden capability in multiple places one update will make everything work correctly.

[davidkaya]

I don’t know whether I understood you correctly. You would make LChflags noop if UF_HIDDEN is not defined (check_symbol_exists in cmake) or if lchflags is not defined (check_function_exists in cmake)? Because only UF_HIDDEN (0x8000) is defined only on OSX and FreeBSD.

[JeremyKuhne]

You would make LChflags noop if UF_HIDDEN is not defined (check_symbol_exists in cmake) or if lchflags is not defined (check_function_exists in cmake)? Because only UF_HIDDEN (0x8000) is defined only on OSX and FreeBSD.

I would make a method in the PAL (int SupportsHiddenFlag() perhaps) that returns true if the define & API are available. Use that method in C# to determine if the flag is available- on set throw PlatformNotSupported if it isn't (as we are for other flags). On get, try getting the flag if it's available and it isn't already starting with a period.

If code wants to call the LChflags wrapper when it isn't available you can just return EOPNOTSUPP.

[davidkaya]

I finally got back to it again.

@JeremyKuhne I have the implementation somehow ready, I can't get the tests running though, so there might be changes in tests if I get it running. I tried 2 things:

1. ./build.sh in CoreFX succeeds. ./build.sh -test in CoreFX fails on sigabrts on all tests (maybe related to the fact that I had to make change in corelib? In the second step I tried to override the coreclr path).
2. ./build.sh in CoreCLR succeeds. ./build.sh /p:CoreCLROverridePath="/Users/kaya/workspace/personal/coreclr/bin/Product/OSX.x64.Debug" in CoreFX fails on

/Users/kaya/.nuget/packages/microsoft.dotnet.genfacades/1.0.0-beta.19058.5/build/Microsoft.DotNet.GenFacades.targets(93,5): warning : Did not find type 'T:System.Runtime.CompilerServices.ConfiguredAsyncEnumerable`1' in any of the seed assemblies. [/Users/kaya/workspace/personal/corefx/src/System.Threading.Tasks/src/System.Threading.Tasks.csproj]
/Users/kaya/.nuget/packages/microsoft.dotnet.genfacades/1.0.0-beta.19058.5/build/Microsoft.DotNet.GenFacades.targets(93,5): warning : Did not find type 'T:System.Runtime.CompilerServices.ConfiguredAsyncEnumerable`1.Enumerator' in any of the seed assemblies. [/Users/kaya/workspace/personal/corefx/src/System.Threading.Tasks/src/System.Threading.Tasks.csproj]
/Users/kaya/.nuget/packages/microsoft.dotnet.genfacades/1.0.0-beta.19058.5/build/Microsoft.DotNet.GenFacades.targets(93,5): warning : Errors were encountered while generating the facade. [/Users/kaya/workspace/personal/corefx/src/System.Threading.Tasks/src/System.Threading.Tasks.csproj]
/Users/kaya/.nuget/packages/microsoft.dotnet.genfacades/1.0.0-beta.19058.5/build/Microsoft.DotNet.GenFacades.targets(93,5): error : Errors were encountered when generating facade(s). [/Users/kaya/workspace/personal/corefx/src/System.Threading.Tasks/src/System.Threading.Tasks.csproj]

My changes are visible at https://github.com/davidkaya/corefx/pull/1/files.

I have up-to-date masters (tried yesterday and today). Yesterday it failed on different types, however still in GenFacades.

Am I doing something wrong? In other issues that I contributed to, I had no issues with this before.

[danmosemsft]

@davidkaya when you patch corefx with coreclr it can happen that they are not quite in sync. You could try to find the relevant change and avoid it, or if these test failures are just in threading tests, ignore them for the purposes of your work.

[hgbloch]

@davidkaya thank you very much for solving this :)