Osquery: "Too many open files" from clients running OS X

Created on 4 Dec 2017 · 19Comments · Source: osquery/osquery

I'm seeing this happening widely across our fleet of OS X machines with OSQuery deployed:

libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >: kqueue: Too many open files
E1204 15:20:39.108371 2711286592 registry.cpp:423] enroll registry tls plugin caused exception: kqueue: Too many open files
E1204 15:20:39.109400 2711286592 registry.cpp:423] enroll registry tls plugin caused exception: kqueue: Too many open files
I1204 15:20:39.109446 2711286592 events.cpp:825] Event publisher not enabled: event_tapping: Publisher disabled via configuration
I1204 15:20:39.109468 2711286592 events.cpp:825] Event publisher not enabled: openbsm: Publisher disabled via configuration
I1204 15:20:39.109488 2711286592 events.cpp:825] Event publisher not enabled: scnetwork: Publisher not used
E1204 15:20:39.119313 2711286592 registry.cpp:423] sql registry sql plugin caused exception: map::at:  key not found
2017-12-04 15:20 r[31417] (FSEvents.framework) FSEventStreamCreate: _FSEventStreamCreate: ERROR: could not open kqueue
E1204 15:20:39.137084 29614080 fsevents.cpp:114] Cannot create FSEvent stream: FSEventStreamCreate failed
E1204 15:20:39.138051 30687232 registry.cpp:423] enroll registry tls plugin caused exception: kqueue: Too many open files
E1204 15:20:39.138152 30687232 registry.cpp:423] distributed registry tls plugin caused exception: kqueue: Too many open files
libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >: kqueue: Too many open files
E1204 15:21:11.088346 2711286592 registry.cpp:423] enroll registry tls plugin caused exception: kqueue: Too many open files
E1204 15:21:11.089359 2711286592 registry.cpp:423] enroll registry tls plugin caused exception: kqueue: Too many open files
I1204 15:21:11.089404 2711286592 events.cpp:825] Event publisher not enabled: event_tapping: Publisher disabled via configuration
I1204 15:21:11.089432 2711286592 events.cpp:825] Event publisher not enabled: openbsm: Publisher disabled via configuration
I1204 15:21:11.089452 2711286592 events.cpp:825] Event publisher not enabled: scnetwork: Publisher not used
E1204 15:21:11.099494 2711286592 registry.cpp:423] sql registry sql plugin caused exception: map::at:  key not found

Invocation of *osqueryd:*

/usr/local/bin/osqueryd --database_path=/usr/local/zentral/osquery/db --tls_hostname=censored --enroll_tls_endpoint=/osquery/enroll --enroll_secret_path=/usr/local/zentral/osquery/enroll_secret.txt --config_plugin=tls --config_tls_endpoint=/osquery/config --config_tls_refresh 120 --logger_plugin=aws_kinesis,aws_firehose --logger_tls_endpoint=/osquery/log --logger_tls_period 60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_read_endpoint=/osquery/distributed/read --distributed_tls_write_endpoint=/osquery/distributed/write --distributed_interval 60 --tls_server_certs=/usr/local/zentral/tls_server_certs.crt --aws_kinesis_stream=censored-endpoints_stream_alert_kinesis --aws_firehose_stream=censored-endpoints_stream_alert_firehose --aws_access_key_id=censored --aws_secret_access_key=censored --aws_region=censored

Versions:

$ /usr/local/bin/osqueryd --version
osqueryd version 2.10.2

OS X 10.13.1

Please let me know if there's any other information I can include to assist in debugging.

bug database macOS

Source

zbuc

Most helpful comment

On the RADAR now - http://www.openradar.me/36148377

uptycs-nishant on 23 Dec 2017

👍2

All 19 comments

OSX: 10.12.6
osquery v2.10.2

+1 on this error - seeing this here as well

This is really not good because the TLS server shows these hosts as online, but they're not actually logging any data to it.

2017-12-12 14:48 osqueryd[13156] (FSEvents.framework) FSEventStreamCreate: _FSEventStreamCreate: ERROR: could not open kqueue
E1212 14:48:02.556901 114851840 fsevents.cpp:114] Cannot create FSEvent stream: FSEventStreamCreate failed
E1212 14:48:02.556977 115924992 registry.cpp:423] distributed registry tls plugin caused exception: kqueue: Too many open files
libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >: kqueue: Too many open files

clong on 12 Dec 2017

It's interesting, if you Google around there's a handful of projects having issues with FSEvents somewhat recently.

Examples:
http://livereload.com/troubleshooting/os-x-fsevents-bug-may-prevent-monitoring-of-certain-folders/
https://github.com/gohugoio/hugo/issues/3582
https://blog.dekstroza.io/ulimit-shenanigans-on-osx-el-capitan/

I just checked with my co-worker who is experiencing this issue and our ulimit settings are somehow different:

His (broken osquery):

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited

Mine (Working):

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 7168
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited

Not sure how that discrepancy happened since we're on the same OS version :-/
@zbuc I would be interested to see your ulimit -a output!

clong on 13 Dec 2017

Full stack trace:

```Process: Path: Identifier: osqueryd
Version: 0
Code Type: Parent Process: Responsible: User ID: 0
Date/Time: OS Version: Report Version: 12
Anonymous UUID: Time Awake System Integrity Crashed Thread: 5
Exception Type: Exception Codes: Exception Note: Application Specific Information:
terminating abort() called
Thread 0:: 0 libsystem_kernel.dylib 1 libsystem_pthread.dylib 2 libc++.1.dylib 3 osqueryd 4 osqueryd 5 osqueryd 6 osqueryd 7 libdyld.dylib Thread 1:
0 libsystem_kernel.dylib 1 libsystem_pthread.dylib 2 libc++.1.dylib 3 osqueryd 4 osqueryd 5 osqueryd 6 libsystem_pthread.dylib 7 libsystem_pthread.dylib 8 libsystem_pthread.dylib Thread 2:
0 libsystem_kernel.dylib 1 libsystem_pthread.dylib 2 libc++.1.dylib 3 osqueryd 4 osqueryd 5 osqueryd 6 libsystem_pthread.dylib 7 libsystem_pthread.dylib 8 libsystem_pthread.dylib Thread 3:
0 libsystem_kernel.dylib 1 libsystem_pthread.dylib 2 libc++.1.dylib 3 osqueryd 4 osqueryd 5 osqueryd 6 libsystem_pthread.dylib 7 libsystem_pthread.dylib 8 libsystem_pthread.dylib Thread 4:
0 libsystem_kernel.dylib 1 libsystem_pthread.dylib 2 libc++.1.dylib 3 osqueryd 4 osqueryd 5 osqueryd 6 libsystem_pthread.dylib 7 libsystem_pthread.dylib 8 libsystem_pthread.dylib Thread 5 Crashed:
0 libsystem_kernel.dylib 1 libsystem_pthread.dylib 2 libsystem_c.dylib 3 libsystem_c.dylib 4 libc++abi.dylib 5 libc++abi.dylib 6 libobjc.A.dylib 7 libc++abi.dylib 8 libc++abi.dylib 9 osqueryd 10 osqueryd 11 osqueryd 12 osqueryd 13 osqueryd 14 osqueryd 15 osqueryd 16 osqueryd 17 osqueryd 18 osqueryd 19 osqueryd 20 osqueryd 21 osqueryd 22 osqueryd 23 osqueryd 24 libsystem_pthread.dylib 25 libsystem_pthread.dylib 26 libsystem_pthread.dylib Thread 6:
0 libsystem_kernel.dylib 1 libsystem_kernel.dylib 2 com.apple.CoreFoundation 3 com.apple.CoreFoundation 4 com.apple.CoreFoundation 5 com.apple.CoreFoundation 6 osqueryd 7 osqueryd 8 osqueryd 9 libsystem_pthread.dylib 10 libsystem_pthread.dylib 11 libsystem_pthread.dylib Thread 7:
0 libsystem_kernel.dylib 1 libsystem_kernel.dylib 2 com.apple.CoreFoundation 3 com.apple.CoreFoundation 4 com.apple.CoreFoundation 5 com.apple.CoreFoundation 6 osqueryd 7 osqueryd 8 osqueryd 9 libsystem_pthread.dylib 10 libsystem_pthread.dylib 11 libsystem_pthread.dylib Thread 8:
0 libsystem_kernel.dylib 1 libsystem_kernel.dylib 2 com.apple.CoreFoundation 3 com.apple.CoreFoundation 4 com.apple.CoreFoundation 5 com.apple.CoreFoundation 6 osqueryd 7 osqueryd 8 osqueryd 9 libsystem_pthread.dylib 10 libsystem_pthread.dylib 11 libsystem_pthread.dylib Thread 9:
0 libsystem_pthread.dylib Thread 10:
0 libsystem_pthread.dylib 1 ??? Thread 11:
0 libsystem_pthread.dylib 1 ??? Thread 5 crashed rax: 0x0000000000000000 rdi: 0x0000000000001d0b r8: 0x0000000000000000 r12: 0x000070000dfccbe0 rip: 0x00007fff91d27d42 Logical CPU: 0
Error Code: 0x02000148
Trap Number: 133
Binary Images:
0x102815000 - 0x10d4ad000 - 0x7fff78f46000 - 0x7fff78f5f000 - 0x7fff79479000 - 0x7fff795eb000 - 0x7fff79600000 - 0x7fff799f7000 - /System/Library/Fram 0x7fff79a0e000 - 0x7fff79a15000 - 0x7fff79a2a000 - 0x7fff79bb2000 - 0x7fff79c65000 - 0x7fff7ad0f000 - 0x7fff7ad10000 - /System/Library/Fram 0x7fff7ae18000 - /System/Library/Fram 0x7fff7af48000 - /System/Library/Fram 0x7fff7b08f000 - /System/Library/Fram 0x7fff7b13a000 - /System/Library/Fram 0x7fff7b18c000 - /System/Library/Fram 0x7fff7b19c000 - /System/Library/Fram 0x7fff7b1ea000 - 0x7fff7b226000 - /System/Library/Fram 0x7fff7b232000 - 0x7fff7b5a8000 - 0x7fff7bfa0000 - 0x7fff7c042000 - 0x7fff7c3ec000 - 0x7fff7c886000 - 0x7fff7d2b1000 - 0x7fff7d2b2000 - 0x7fff7d304000 - /System/Library/Fram 0x7fff7d5e0000 - /System/Library/Fram 0x7fff7d614000 - /System/Library/Fram 0x7fff7d61d000 - /System/Library/Fram 0x7fff7d78a000 - 0x7fff7d83b000 - /System/Library/Fram 0x7fff7d89b000 - 0x7fff7d90c000 - /System/Library/Fram 0x7fff7d9da000 - 0x7fff7db5e000 - 0x7fff7dccd000 - 0x7fff7de64000 - 0x7fff7e236000 - 0x7fff7e3cb000 - 0x7fff7e462000 - 0x7fff7e4bb000 - 0x7fff7e61c000 - 0x7fff7e621000 - 0x7fff7e712000 - 0x7fff7e736000 - 0x7fff7e75e000 - 0x7fff7e761000 - 0x7fff7f518000 - 0x7fff80655000 - 0x7fff8088c000 - /System/Library/Fram 0x7fff808a6000 - 0x7fff826f9000 - 0x7fff829fc000 - 0x7fff82a9c000 - 0x7fff82e26000 - 0x7fff85833000 - 0x7fff87346000 - 0x7fff87a98000 - 0x7fff88394000 - 0x7fff8ab2f000 - 0x7fff8b2e7000 - 0x7fff8bfbc000 - 0x7fff8c096000 - 0x7fff8eef2000 - 0x7fff90301000 - 0x7fff90358000 - 0x7fff90374000 - 0x7fff90411000 - 0x7fff9064a000 - 0x7fff90650000 - 0x7fff906bd000 - 0x7fff90769000 - 0x7fff9076a000 - 0x7fff9077b000 - 0x7fff9078a000 - 0x7fff907e1000 - 0x7fff9080b000 - 0x7fff9081c000 - 0x7fff90833000 - 0x7fff90834000 - 0x7fff90b71000 - 0x7fff90c41000 - 0x7fff90c51000 - 0x7fff90c57000 - 0x7fff90d4a000 - 0x7fff90f75000 - 0x7fff90f77000 - 0x7fff90f91000 - 0x7fff90fa8000 - 0x7fff91283000 - 0x7fff912fe000 - 0x7fff916d3000 - 0x7fff916d8000 - 0x7fff91726000 - 0x7fff91792000 - 0x7fff919d5000 - 0x7fff919e3000 - 0x7fff91ad3000 - 0x7fff91afd000 - 0x7fff91b1d000 - 0x7fff91b22000 - 0x7fff91b2d000 - 0x7fff91b35000 - 0x7fff91b3e000 - 0x7fff91bc2000 - 0x7fff91bf4000 - 0x7fff91bfa000 - 0x7fff91bfb000 - 0x7fff91c08000 - 0x7fff91c09000 - 0x7fff91c0f000 - 0x7fff91c12000 - 0x7fff91c14000 - 0x7fff91c2d000 - 0x7fff91c2e000 - 0x7fff91cbc000 - 0x7fff91cc0000 - 0x7fff91cc4000 - 0x7fff91cdd000 - 0x7fff91ce4000 - 0x7fff91d0e000 - 0x7fff91d31000 - 0x7fff91d79000 - 0x7fff91d98000 - 0x7fff91df2000 - 0x7fff91dfc000 - 0x7fff91e06000 - 0x7fff91e0f000 - 0x7fff91e1a000 - 0x7fff91e1e000 - 0x7fff91e20000 - 0x7fff91e28000 - 0x7fff91e3c000 - 0x7fff91e42000 - External Modification Summary:
Calls made task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made task_for_pid: 969
thread_create: 0
thread_set_state: 0
VM Region Summary:
ReadOnly portion Writable regions: REGION TYPE =========== Activity Tracing Dispatch continuations Kernel Alloc Once MALLOC MALLOC guard page STACK GUARD Stack __DATA __LINKEDIT __TEXT __UNICODE shared memory =========== TOTAL osqueryd [2735]
/usr/local/bin/osqueryd
X86-64 (Native)
osqueryd [141]
osqueryd [2735]
2017-12-12 16:31:04.662 -0800
Mac OS X 10.12.6 (16G1114)
E0161CD2-579A-698D-4F26-4E891C9ABEFE
Since Boot: 230 seconds
Protection: enabled
EXC_CRASH (SIGABRT)
0x0000000000000000, 0x0000000000000000
EXC_CORPSE_NOTIFY
with uncaught exception of type boost::exception_detail::clone_impl >: kqueue: Too many open files
Dispatch queue: com.apple.main-thread
0x00007fff91d27f46 __semwait_signal + 10
0x00007fff91e146e5 pthread_join + 425
0x00007fff907d24b7 std::__1::thread::join() + 23
0x0000000102e46f95 0x102815000 + 6496149
0x0000000102e27776 0x102815000 + 6367094
0x0000000102d6a174 0x102815000 + 5591412
0x0000000102d6b5ac 0x102815000 + 5596588
0x00007fff91bf9235 start + 1
0x00007fff91d27bf2 __psynch_cvwait + 10
0x00007fff91e137fa _pthread_cond_wait + 712
0x00007fff907914cd std::__1::condition_variable::wait(std::__1::unique_lock&) + 47
0x0000000102be8e0b 0x102815000 + 4013579
0x0000000102be91a3 0x102815000 + 4014499
0x0000000102bea7e1 0x102815000 + 4020193
0x00007fff91e1293b _pthread_body + 180
0x00007fff91e12887 _pthread_start + 286
0x00007fff91e1208d thread_start + 13
0x00007fff91d27bf2 __psynch_cvwait + 10
0x00007fff91e137fa _pthread_cond_wait + 712
0x00007fff907914cd std::__1::condition_variable::wait(std::__1::unique_lock&) + 47
0x0000000102be8e0b 0x102815000 + 4013579
0x0000000102be91a3 0x102815000 + 4014499
0x0000000102bea7e1 0x102815000 + 4020193
0x00007fff91e1293b _pthread_body + 180
0x00007fff91e12887 _pthread_start + 286
0x00007fff91e1208d thread_start + 13
0x00007fff91d27bf2 __psynch_cvwait + 10
0x00007fff91e137fa _pthread_cond_wait + 712
0x00007fff907914cd std::__1::condition_variable::wait(std::__1::unique_lock&) + 47
0x0000000102be8e0b 0x102815000 + 4013579
0x0000000102be91a3 0x102815000 + 4014499
0x0000000102bea7e1 0x102815000 + 4020193
0x00007fff91e1293b _pthread_body + 180
0x00007fff91e12887 _pthread_start + 286
0x00007fff91e1208d thread_start + 13
0x00007fff91d27bf2 __psynch_cvwait + 10
0x00007fff91e137fa _pthread_cond_wait + 712
0x00007fff907914cd std::__1::condition_variable::wait(std::__1::unique_lock&) + 47
0x0000000102be8e0b 0x102815000 + 4013579
0x0000000102be91a3 0x102815000 + 4014499
0x0000000102bea7e1 0x102815000 + 4020193
0x00007fff91e1293b _pthread_body + 180
0x00007fff91e12887 _pthread_start + 286
0x00007fff91e1208d thread_start + 13
0x00007fff91d27d42 __pthread_kill + 10
0x00007fff91e15457 pthread_kill + 90
0x00007fff91c8d4bb __abort + 140
0x00007fff91c8d42f abort + 144
0x00007fff907e194a abort_message + 266
0x00007fff90806c17 default_terminate_handler() + 243
0x00007fff91315713 _objc_terminate() + 124
0x00007fff90803d49 std::__terminate(void ()()) + 8
0x00007fff908037be __cxa_throw + 121
0x0000000102e9cceb 0x102815000 + 6847723
0x0000000102f12371 0x102815000 + 7328625
0x0000000102f12100 boost::asio::io_service::service boost::asio::detail::service_registry::create(boost::asio::io_service&) + 52
0x0000000102f103cf 0x102815000 + 7320527
0x0000000102f11ff1 boost::asio::io_service::service* boost::asio::detail::service_registry::create >(boost::asio::io_service&) + 105
0x0000000102f103cf 0x102815000 + 7320527
0x0000000102f0f6e3 0x102815000 + 7317219
0x0000000102f0e97d 0x102815000 + 7313789
0x0000000102e9fd59 0x102815000 + 6860121
0x0000000102e9f1e9 0x102815000 + 6857193
0x0000000102e9ec41 0x102815000 + 6855745
0x0000000102ee6647 0x102815000 + 7149127
0x0000000102ee1765 0x102815000 + 7128933
0x0000000102e4699d 0x102815000 + 6494621
0x0000000102e4765d 0x102815000 + 6497885
0x00007fff91e1293b _pthread_body + 180
0x00007fff91e12887 _pthread_start + 286
0x00007fff91e1208d thread_start + 13
0x00007fff91d2034a mach_msg_trap + 10
0x00007fff91d1f797 mach_msg + 55
0x00007fff7c474434 __CFRunLoopServiceMachPort + 212
0x00007fff7c4738c1 __CFRunLoopRun + 1361
0x00007fff7c473114 CFRunLoopRunSpecific + 420
0x00007fff7c4b2b91 CFRunLoopRun + 97
0x0000000102eb07ae 0x102815000 + 6928302
0x0000000102e518e0 0x102815000 + 6539488
0x0000000102e57686 0x102815000 + 6563462
0x00007fff91e1293b _pthread_body + 180
0x00007fff91e12887 _pthread_start + 286
0x00007fff91e1208d thread_start + 13
0x00007fff91d2034a mach_msg_trap + 10
0x00007fff91d1f797 mach_msg + 55
0x00007fff7c474434 __CFRunLoopServiceMachPort + 212
0x00007fff7c4738c1 __CFRunLoopRun + 1361
0x00007fff7c473114 CFRunLoopRunSpecific + 420
0x00007fff7c4b2b91 CFRunLoopRun + 97
0x0000000102eb502f 0x102815000 + 6946863
0x0000000102e518e0 0x102815000 + 6539488
0x0000000102e57686 0x102815000 + 6563462
0x00007fff91e1293b _pthread_body + 180
0x00007fff91e12887 _pthread_start + 286
0x00007fff91e1208d thread_start + 13
0x00007fff91d2034a mach_msg_trap + 10
0x00007fff91d1f797 mach_msg + 55
0x00007fff7c474434 __CFRunLoopServiceMachPort + 212
0x00007fff7c4738c1 __CFRunLoopRun + 1361
0x00007fff7c473114 CFRunLoopRunSpecific + 420
0x00007fff7c4b2b91 CFRunLoopRun + 97
0x0000000102eb9e65 0x102815000 + 6966885
0x0000000102e518e0 0x102815000 + 6539488
0x0000000102e57686 0x102815000 + 6563462
0x00007fff91e1293b _pthread_body + 180
0x00007fff91e12887 _pthread_start + 286
0x00007fff91e1208d thread_start + 13
0x00007fff91e12070 start_wqthread + 0
0x00007fff91e12070 start_wqthread + 0
0x00007fbe05e2e8f0 0 + 140454119270640
0x00007fff91e12070 start_wqthread + 0
0x000070000dec7b50 0 + 123145535912784
with X86 Thread State (64-bit):
rbx: 0x0000000000000006 rcx: 0x000070000dfcca28 rdx: 0x0000000000000000
rsi: 0x0000000000000006 rbp: 0x000070000dfcca50 rsp: 0x000070000dfcca28
r9: 0x0000000000989680 r10: 0x0000000008000000 r11: 0x0000000000000206
r13: 0x0000000000000030 r14: 0x000070000dfce000 r15: 0x0000000000000008
rfl: 0x0000000000000206 cr2: 0x000055a00ae7cb70
0x10320cfff +osqueryd (0) /usr/local/bin/osqueryd
0x10d4eadc7 dyld (433.5) <322C06B7-8878-311D-888C-C8FD2CA96FF3> /usr/lib/dyld
0x7fff78f46fff com.apple.Accelerate (1.11 - Accelerate 1.11) <916E360F-323C-3AE1-AB3D-D1F3B284AEE9> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
0x7fff79478feb com.apple.vImage (8.1 - ???) /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage
0x7fff795eaff3 libBLAS.dylib (1185.50.4) <4087FFE0-627E-3623-96B4-F0A9A1991E09> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
0x7fff795ffffb libBNNS.dylib (15) <254698C7-7D36-3FFF-864E-ADEEEE543076> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBNNS.dylib
0x7fff799f6fef libLAPACK.dylib (1185.50.4) /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
0x7fff79a0dfff libLinearAlgebra.dylib (1185.50.4) <345CAACF-7263-36EF-B69B-793EA8B390AF>
eworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLinearAlgebra.dylib
0x7fff79a14fff libQuadrature.dylib (3) /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libQuadrature.dylib
0x7fff79a29ff7 libSparseBLAS.dylib (1185.50.4) <67BA432E-FB59-3C78-A8BE-ED4274CBC359> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparseBLAS.dylib
0x7fff79bb1fe7 libvDSP.dylib (600.60.1) <4155F45B-41CD-3782-AE8F-7AE740FD83C3> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib
0x7fff79c64fff libvMisc.dylib (600.60.1) /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib
0x7fff79c65fff com.apple.Accelerate.vecLib (3.11 - vecLib 3.11) <7C5733E7-0568-3E7D-AF61-160F19FED544> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
0x7fff7ad0ffff com.apple.ApplicationServices (48 - 48) /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
0x7fff7ad7eff7 com.apple.ApplicationServices.ATS (377 - 422.3) <73C230CF-3213-3DCA-B528-664FEC76A90C>
eworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS
0x7fff7af47fff libFontParser.dylib (194.13)
eworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.dylib
0x7fff7af92fff libFontRegistry.dylib (196.4)
eworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib
0x7fff7b139ff7 com.apple.ColorSync (4.12.0 - 502.2)
eworks/ApplicationServices.framework/Versions/A/Frameworks/ColorSync.framework/Versions/A/ColorSync
0x7fff7b18bfff com.apple.HIServices (1.22 - 593) <045870A9-1D92-3FEF-8722-671E99A35E71>
eworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices
0x7fff7b19bff3 com.apple.LangAnalysis (1.7.0 - 1.7.0) <2CBE7F61-2056-3F96-99A1-0D527796AFA6>
eworks/ApplicationServices.framework/Versions/A/Frameworks/LangAnalysis.framework/Versions/A/LangAnalysis
0x7fff7b1e9fff com.apple.print.framework.PrintCore (12 - 491) <5027FD58-F0EE-33E4-8577-934CA06CD2AF>
eworks/ApplicationServices.framework/Versions/A/Frameworks/PrintCore.framework/Versions/A/PrintCore
0x7fff7b225fff com.apple.QD (3.12 - 313) /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/QD.framework/Versions/A/QD
0x7fff7b231fff com.apple.speech.synthesis.framework (6.6.2 - 6.6.2) <7853EFF4-62B9-394E-B7B8-41A645656820>
eworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis
0x7fff7b43eff7 com.apple.audio.toolbox.AudioToolbox (1.14 - 1.14) /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
0x7fff7b984fff com.apple.CFNetwork (811.8 - 811.8) <3FB812CB-F4BF-3C47-B772-042FC4D59A33> /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
0x7fff7c02dfff com.apple.audio.CoreAudio (4.3.0 - 4.3.0) <78767F88-91D4-31CE-AAC6-1F9407F479BB> /System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
0x7fff7c33dfff com.apple.CoreData (120 - 754.2) <4C9CAB2C-60D4-3694-A0A0-5B04B14BD14E> /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
0x7fff7c885ff7 com.apple.CoreFoundation (6.9 - 1349.90) /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
0x7fff7cf08fff com.apple.CoreGraphics (2.0 - 1070.22) /System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
0x7fff7d2b1fff com.apple.CoreServices (775.19 - 775.19) <5046FF69-034E-3570-AC30-D14AA469F669> /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
0x7fff7d303fff com.apple.AE (712.5 - 712.5) <61F2AE2C-E04E-3FDF-9E88-201325136C83> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
0x7fff7d5dfff7 com.apple.CoreServices.CarbonCore (1159.6 - 1159.6) <08AC074C-965B-3EDF-8E45-0707C8DE9EAD>
eworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0x7fff7d613fff com.apple.DictionaryServices (1.2 - 274)
eworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
0x7fff7d61cff3 com.apple.CoreServices.FSEvents (1230.50.1 - 1230.50.1) <2AD1B0E5-7214-37C4-8D11-A27C9CAC0F74>
eworks/CoreServices.framework/Versions/A/Frameworks/FSEvents.framework/Versions/A/FSEvents
0x7fff7d789ff7 com.apple.LaunchServices (775.19 - 775.19) <76F27409-F3E0-31A7-BA40-060FBD10CB8B>
eworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
0x7fff7d83affb com.apple.Metadata (10.7.0 - 1075.40) /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
0x7fff7d89afff com.apple.CoreServices.OSServices (775.19 - 775.19)
eworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
0x7fff7d90bfff com.apple.SearchKit (1.4.0 - 1.4.0) <7A6DDA2B-03F1-3137-BA9E-1CC211973E26> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
0x7fff7d951ff7 com.apple.coreservices.SharedFileList (38 - 38)
eworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList
0x7fff7db27ffb com.apple.CoreText (352.0 - 544.16) <543AFBA5-4E6B-3F81-BC45-81ED9707BD9C> /System/Library/Frameworks/CoreText.framework/Versions/A/CoreText
0x7fff7dbcfffb com.apple.framework.CoreWLAN (11.0 - 1200.31) <64338DCB-8204-3B62-A9C5-0CDDC89F60D6> /System/Library/Frameworks/CoreWLAN.framework/Versions/A/CoreWLAN
0x7fff7dcd2fff com.apple.DiskArbitration (2.7 - 2.7) <70F56042-EF04-3BAF-9E04-90F423FB2BCC> /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
0x7fff7e20afff com.apple.Foundation (6.9 - 1349.91) /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
0x7fff7e267ff7 com.apple.GSS (4.0 - 2.0) <30A77A95-CBD4-3BD7-9BFF-8426071F5D9D> /System/Library/Frameworks/GSS.framework/Versions/A/GSS
0x7fff7e461ff7 com.apple.framework.IOKit (2.0.2 - 1324.60.4) <454036AD-86AA-3ABB-8C53-B5F654A6824D> /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
0x7fff7e468ffb com.apple.IOSurface (159.12 - 159.12) /System/Library/Frameworks/IOSurface.framework/Versions/A/IOSurface
0x7fff7e61bfef com.apple.ImageIO.framework (3.3.0 - 1599.13) <443879A7-48C6-37ED-AC0B-166EEEABA662> /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
0x7fff7e620fff libGIF.dylib (1599.13) <11C5D9D7-B8A1-3D53-BA50-B25976FE6379> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib
0x7fff7e711ff7 libJP2.dylib (1599.13) /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib
0x7fff7e735ffb libJPEG.dylib (1599.13) /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib
0x7fff7e75dff7 libPng.dylib (1599.13) <561865FA-A51B-3EFF-ABFF-B4566B90B1A2> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
0x7fff7e760ff3 libRadiance.dylib (1599.13) <6559253B-8FE2-3884-BC93-95BB7705A6D1> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib
0x7fff7e7afffb libTIFF.dylib (1599.13) <8FC594B3-6BBF-3A70-9994-203E9B330B5F> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
0x7fff7f531ff7 com.apple.Kerberos (3.0 - 1) /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
0x7fff8065dfff com.apple.NetFS (6.0 - 4.0) <14A24D00-5673-330A-959D-87F72040DEFF> /System/Library/Frameworks/NetFS.framework/Versions/A/NetFS
0x7fff808a5ffb com.apple.CFOpenDirectory (10.12 - 194)
eworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory
0x7fff808b1ff7 com.apple.OpenDirectory (10.12 - 194) <4298FFD0-B1A7-3064-AF5B-708B3FA38671> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/OpenDirectory
0x7fff829fbff7 com.apple.security (7.0 - 57740.60.20) /System/Library/Frameworks/Security.framework/Versions/A/Security
0x7fff82a71fff com.apple.securityfoundation (6.0 - 55132.50.7) <1276FFB8-05B8-3760-9EA7-2037B7FE2FBC> /System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
0x7fff82a9fff3 com.apple.xpc.ServiceManagement (1.0 - 1) /System/Library/Frameworks/ServiceManagement.framework/Versions/A/ServiceManagement
0x7fff82e95ff7 com.apple.SystemConfiguration (1.14 - 1.14) /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
0x7fff858be97f com.apple.AppleJPEG (1.0 - 1) /System/Library/PrivateFrameworks/AppleJPEG.framework/Versions/A/AppleJPEG
0x7fff8734fffb com.apple.CommonAuth (4.0 - 2.0) <44E4467E-90BB-3CC4-9E57-9B09D7E90081> /System/Library/PrivateFrameworks/CommonAuth.framework/Versions/A/CommonAuth
0x7fff87aa8fff com.apple.CoreEmoji (1.0 - 40.3.3) /System/Library/PrivateFrameworks/CoreEmoji.framework/Versions/A/CoreEmoji
0x7fff883f9ff3 com.apple.framework.CoreWiFi (12.0 - 1200.31) <926A9CE7-6F3F-3EB6-A392-81FF485F0B7A> /System/Library/PrivateFrameworks/CoreWiFi.framework/Versions/A/CoreWiFi
0x7fff8aba5ff3 com.apple.Heimdal (4.0 - 2.0) <9541C717-96FF-32D3-8F0B-5048525DAA70> /System/Library/PrivateFrameworks/Heimdal.framework/Versions/A/Heimdal
0x7fff8b49efff com.apple.LanguageModeling (1.0 - 123.2.5) /System/Library/PrivateFrameworks/LanguageModeling.framework/Versions/A/LanguageModeling
0x7fff8bfe4ff7 com.apple.MultitouchSupport.framework (368.16 - 368.16) /System/Library/PrivateFrameworks/MultitouchSupport.framework/Versions/A/MultitouchSupport
0x7fff8c0a1fff com.apple.NetAuth (6.2 - 6.2) <97F487D6-8089-31A8-B68C-6C1EAC6ED1B5> /System/Library/PrivateFrameworks/NetAuth.framework/Versions/A/NetAuth
0x7fff8eef8ff7 com.apple.TCC (1.0 - 1) <911B534B-4AC7-34E4-935E-E42ECD008CBC> /System/Library/PrivateFrameworks/TCC.framework/Versions/A/TCC
0x7fff90303ffb com.apple.loginsupport (1.0 - 1) /System/Library/PrivateFrameworks/login.framework/Versions/A/Frameworks/loginsupport.framework/Versions/A/loginsupport
0x7fff90373ff7 libCRFSuite.dylib (34) /usr/lib/libCRFSuite.dylib
0x7fff9037ffff libChineseTokenizer.dylib (21) <0886E908-A825-36AF-B94B-2361FD8BC2A1> /usr/lib/libChineseTokenizer.dylib
0x7fff90412ff3 libDiagnosticMessagesClient.dylib (102) <84A04D24-0E60-3810-A8C0-90A65E2DF61A> /usr/lib/libDiagnosticMessagesClient.dylib
0x7fff9064afff libOpenScriptingUtil.dylib (172.1) <0F1BA407-97D1-36F6-882D-A355EAAD5E00> /usr/lib/libOpenScriptingUtil.dylib
0x7fff90651ffb libSystem.B.dylib (1238.60.2) <03D8D904-05E2-3A15-B14E-3472703550DC> /usr/lib/libSystem.B.dylib
0x7fff906e8ff3 libarchive.2.dylib (41.70.2) <907D1FB1-9A65-33F5-AFC8-0B6E5AE9D83A> /usr/lib/libarchive.2.dylib
0x7fff90769ff3 libauto.dylib (187) <34388D0B-C539-3C1B-9408-2BC152162E43> /usr/lib/libauto.dylib
0x7fff9077aff3 libbsm.0.dylib (34) <20084796-B04D-3B35-A003-EA11459557A9> /usr/lib/libbsm.0.dylib
0x7fff90789ff7 libbz2.1.0.dylib (38) /usr/lib/libbz2.1.0.dylib
0x7fff907e0ff7 libc++.1.dylib (307.5) <0B43BB5D-E6EB-3464-8DE9-B41AC8ED9D1C> /usr/lib/libc++.1.dylib
0x7fff9080aff7 libc++abi.dylib (307.4) /usr/lib/libc++abi.dylib
0x7fff9081bffb libcmph.dylib (6) <2B5D405E-2D0B-3320-ABD6-622934C86ABE> /usr/lib/libcmph.dylib
0x7fff90832fcf libcompression.dylib (39) /usr/lib/libcompression.dylib
0x7fff90833ff7 libcoretls.dylib (121.50.4) <64B1001E-10F6-3542-A3B2-C4B49F51817F> /usr/lib/libcoretls.dylib
0x7fff90835ff3 libcoretls_cfhelpers.dylib (121.50.4) <1A10303E-5EB0-3C7C-9165-021FCDFD934D> /usr/lib/libcoretls_cfhelpers.dylib
0x7fff90bc4ff7 libcups.2.dylib (450) <9950BFCB-7882-33C9-9ECF-CE66773C5657> /usr/lib/libcups.2.dylib
0x7fff90c41fff libenergytrace.dylib (15) /usr/lib/libenergytrace.dylib
0x7fff90c56ff7 libheimdal-asn1.dylib (498.50.9) <92C953A2-5E1D-39E9-875E-92F29A12D6FB> /usr/lib/libheimdal-asn1.dylib
0x7fff90d49ff7 libiconv.2.dylib (50) <42125B35-81D7-3FC4-9475-A26DBE10884D> /usr/lib/libiconv.2.dylib
0x7fff90f6fffb libicucore.A.dylib (57167.0.1) /usr/lib/libicucore.A.dylib
0x7fff90f76fff liblangid.dylib (126) <2085E7A7-9A34-3735-87F4-F174EF8EABF0> /usr/lib/liblangid.dylib
0x7fff90f90ffb liblzma.5.dylib (10) <44BD0279-99DD-36B5-8A6E-C11432E2098D> /usr/lib/liblzma.5.dylib
0x7fff90fa7ff7 libmarisa.dylib (5) <9030D214-5D0F-30CB-AC03-902C63909362> /usr/lib/libmarisa.dylib
0x7fff91250ff7 libmecabra.dylib (744.8) /usr/lib/libmecabra.dylib
0x7fff912fdff3 libnetwork.dylib (856.60.1) <191E99F5-4723-3180-8013-02AF2F9AE4B8> /usr/lib/libnetwork.dylib
0x7fff916d0047 libobjc.A.dylib (709.1) <70614861-0340-32E2-85ED-FE65759CDFFA> /usr/lib/libobjc.A.dylib
0x7fff916d7fff libpam.2.dylib (21.30.1) <71EB0D88-DE84-3C8D-A2C5-58AA282BC5BC> /usr/lib/libpam.2.dylib
0x7fff91709fff libpcap.A.dylib (67.60.2) /usr/lib/libpcap.A.dylib
0x7fff91742ffb libresolv.9.dylib (64) /usr/lib/libresolv.9.dylib
0x7fff918e0ff7 libsqlite3.dylib (254.8) <1ECF7DF7-7A07-3B4B-A63B-F4EFF6BC7ACF> /usr/lib/libsqlite3.dylib
0x7fff919e2fff libxar.1.dylib (357) <69547C64-E811-326F-BBED-490C6361BDCB> /usr/lib/libxar.1.dylib
0x7fff91ad2ff3 libxml2.2.dylib (30.18) <75135C3D-47B1-3217-AC2B-B566E2E1A4A3> /usr/lib/libxml2.2.dylib
0x7fff91afcfff libxslt.1.dylib (15.9.1) /usr/lib/libxslt.1.dylib
0x7fff91b0eff3 libz.1.dylib (67) <46E3FFA2-4328-327A-8D34-A03E20BFFB8E> /usr/lib/libz.1.dylib
0x7fff91b21ff7 libcache.dylib (79) <093A4DAB-8385-3D47-A350-E20CB7CCF7BF> /usr/lib/system/libcache.dylib
0x7fff91b2cfff libcommonCrypto.dylib (60092.50.5) <8A64D1B0-C70E-385C-92F0-E669079FDA90> /usr/lib/system/libcommonCrypto.dylib
0x7fff91b34fff libcompiler_rt.dylib (62) <55D47421-772A-32AB-B529-1A46C2F43B4D> /usr/lib/system/libcompiler_rt.dylib
0x7fff91b3dfff libcopyfile.dylib (138) <819BEA3C-DF11-3E3D-A1A1-5A51C5BF1961> /usr/lib/system/libcopyfile.dylib
0x7fff91bc1fdf libcorecrypto.dylib (442.50.19) <65D7165E-2E71-335D-A2D6-33F78E2DF0C1> /usr/lib/system/libcorecrypto.dylib
0x7fff91bf3fff libdispatch.dylib (703.50.37) <6582BAD6-ED27-3B30-B620-90B1C5A4AE3C> /usr/lib/system/libdispatch.dylib
0x7fff91bf9ffb libdyld.dylib (433.5) <990CF9D7-4AD1-310D-A4B4-010F6D84FFD8> /usr/lib/system/libdyld.dylib
0x7fff91bfaffb libkeymgr.dylib (28) <7AA011A9-DC21-3488-BF73-3B5B14D1FDD6> /usr/lib/system/libkeymgr.dylib
0x7fff91c07fff libkxld.dylib (3789.72.11) /usr/lib/system/libkxld.dylib
0x7fff91c08fff liblaunch.dylib (972.70.1) /usr/lib/system/liblaunch.dylib
0x7fff91c0eff3 libmacho.dylib (898) <17D5D855-F6C3-3B04-B680-E9BF02EF8AED> /usr/lib/system/libmacho.dylib
0x7fff91c11ff3 libquarantine.dylib (85.50.1) <12448CC2-378E-35F3-BE33-9DC395A5B970> /usr/lib/system/libquarantine.dylib
0x7fff91c13ffb libremovefile.dylib (45) <38D4CB9C-10CD-30D3-8B7B-A515EC75FE85> /usr/lib/system/libremovefile.dylib
0x7fff91c2cff7 libsystem_asl.dylib (349.50.5) <096E4228-3B7C-30A6-8B13-EC909A64499A> /usr/lib/system/libsystem_asl.dylib
0x7fff91c2dff7 libsystem_blocks.dylib (67) <10DC5404-73AB-35B3-A277-A8AFECB476EB> /usr/lib/system/libsystem_blocks.dylib
0x7fff91cbbfef libsystem_c.dylib (1158.50.2) /usr/lib/system/libsystem_c.dylib
0x7fff91cbfffb libsystem_configuration.dylib (888.60.2) /usr/lib/system/libsystem_configuration.dylib
0x7fff91cc3fff libsystem_coreservices.dylib (41.4) <7D26DE79-B424-3450-85E1-F7FAB32714AB> /usr/lib/system/libsystem_coreservices.dylib
0x7fff91cdcfff libsystem_coretls.dylib (121.50.4) /usr/lib/system/libsystem_coretls.dylib
0x7fff91ce3fff libsystem_dnssd.dylib (765.50.9) /usr/lib/system/libsystem_dnssd.dylib
0x7fff91d0dff7 libsystem_info.dylib (503.50.4) <611DB84C-BF70-3F92-8702-B9F28A900920> /usr/lib/system/libsystem_info.dylib
0x7fff91d30ff7 libsystem_kernel.dylib (3789.72.11) /usr/lib/system/libsystem_kernel.dylib
0x7fff91d78fe7 libsystem_m.dylib (3121.6) <86D499B5-BBDC-3D3B-8A4E-97AE8E6672A4> /usr/lib/system/libsystem_m.dylib
0x7fff91d97ff7 libsystem_malloc.dylib (116.50.8) /usr/lib/system/libsystem_malloc.dylib
0x7fff91df1ffb libsystem_network.dylib (856.60.1) <369D0221-56CA-3C3E-9EDE-94B41CAE77B7> /usr/lib/system/libsystem_network.dylib
0x7fff91dfbff3 libsystem_networkextension.dylib (563.60.2) /usr/lib/system/libsystem_networkextension.dylib
0x7fff91e05ff3 libsystem_notify.dylib (165.20.1) /usr/lib/system/libsystem_notify.dylib
0x7fff91e0efe7 libsystem_platform.dylib (126.50.8) <897462FD-B318-321B-A554-E61982630F7E> /usr/lib/system/libsystem_platform.dylib
0x7fff91e19ff7 libsystem_pthread.dylib (218.60.3) /usr/lib/system/libsystem_pthread.dylib
0x7fff91e1dff7 libsystem_sandbox.dylib (592.70.2) <19320A42-2E3B-361B-BBDA-2F5F2E87B100> /usr/lib/system/libsystem_sandbox.dylib
0x7fff91e1fff3 libsystem_secinit.dylib (24.50.4) /usr/lib/system/libsystem_secinit.dylib
0x7fff91e27ffb libsystem_symptoms.dylib (532.50.47) <3390E07C-C1CE-348F-ADBD-2C5440B45EAA> /usr/lib/system/libsystem_symptoms.dylib
0x7fff91e3bff7 libsystem_trace.dylib (518.70.1) /usr/lib/system/libsystem_trace.dylib
0x7fff91e41ffb libunwind.dylib (35.3) <3D50D8A8-C460-334D-A519-2DA841102C6B> /usr/lib/system/libunwind.dylib
0x7fff91e6bff7 libxpc.dylib (972.70.1) /usr/lib/system/libxpc.dylib
by other processes targeting this process:
by all processes on this machine:
of Libraries: Total=203.4M resident=0K(0%) swapped_out_or_unallocated=203.4M(100%)
Total=59.7M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=59.7M(100%)
VIRTUAL REGION
SIZE COUNT (non-coalesced)
======= =======
256K 2
8192K 2
8K 2
37.5M 14
16K 4
56.0M 13
13.6M 13
10.0M 138
114.6M 4
88.8M 141
556K 2
12K 4
======= =======
329.3M 327

clong on 13 Dec 2017

@clong Perhaps your coworker did an upgrade and you a clean install (to explain the different ulimit).

Ulimit on my machine for FDs is 10240.

obelisk on 13 Dec 2017

@clong:

$ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       1418
-n: file descriptors                4864

zbuc on 13 Dec 2017

@zbuc

Can you also provide the output of the following command?
sudo lsof | grep '^osqueryd'

uptycs-nishant on 13 Dec 2017

$ sudo lsof | grep '^osqueryd'
Password:
osqueryd  62841               root  cwd       DIR                1,4        1760          2 /
osqueryd  62841               root  txt       REG                1,4    11392016 8595254778 /usr/local/bin/osqueryd
osqueryd  62841               root  txt       REG                1,4      837248 8594921403 /usr/lib/dyld
osqueryd  62841               root  txt       REG                1,4  1147727872 8595105660 /private/var/db/dyld/dyld_shared_cache_x86_64h
osqueryd  62841               root    0r      CHR                3,2         0t0        311 /dev/null
osqueryd  62841               root    1u      REG                1,4           0 8591142029 /private/var/log/zentral_osqueryd_stdout.log
osqueryd  62841               root    2u      REG                1,4   105832253 8590699301 /private/var/log/zentral_osqueryd_stderr.log
osqueryd  62841               root    3u      REG                1,4       32768 8595491943 /private/var/tmp/etilqs_9bef2a7a750bcc51
osqueryd  81001               root  cwd       DIR                1,4        1760          2 /
osqueryd  81001               root  txt       REG                1,4    11392016 8595254778 /usr/local/bin/osqueryd
osqueryd  81001               root  txt       REG                1,4    26752912 8594922333 /usr/share/icu/icudt59l.dat
osqueryd  81001               root  txt       REG                1,4      837248 8594921403 /usr/lib/dyld
osqueryd  81001               root  txt       REG                1,4  1147727872 8595105660 /private/var/db/dyld/dyld_shared_cache_x86_64h
osqueryd  81001               root    0r      CHR                3,2         0t0        311 /dev/null
osqueryd  81001               root    1u      REG                1,4           0 8591142029 /private/var/log/zentral_osqueryd_stdout.log
osqueryd  81001               root    2u      REG                1,4   105832253 8590699301 /private/var/log/zentral_osqueryd_stderr.log
osqueryd  81001               root    3r      DIR                1,4       23520   35145990 /usr/local/zentral/osquery/db
osqueryd  81001               root    4u      REG                1,4           0   35145996 /usr/local/zentral/osquery/db/LOCK
osqueryd  81001               root    6   NPOLICY
osqueryd  81001               root    7w      REG                1,4       59086 8595749693 /usr/local/zentral/osquery/db/MANIFEST-479184
osqueryd  81001               root    9w      REG                1,4         980 8595749697 /usr/local/zentral/osquery/db/479186.log
osqueryd  81001               root   12u     unix 0xfab7ceed546f23cf         0t0            ->0xfab7ceed546f2f87
osqueryd  81001               root   13u      REG                1,4       32768 8595544965 /private/var/tmp/etilqs_44787e62d1fe2d91
osqueryd  81001               root   14u     unix 0xfab7ceed546f2f87         0t0            ->0xfab7ceed546f23cf
osqueryd  81001               root   15u     unix 0xfab7ceed546f1cc7         0t0            ->0xfab7ceed546f27b7
osqueryd  81001               root   16u     unix 0xfab7ceed546f27b7         0t0            ->0xfab7ceed546f1cc7
osqueryd  81001               root   18r      CHR               14,0         0t0        586 /dev/random
osqueryd  81001               root   19r      REG                1,4       25675 8595749699 /usr/local/zentral/osquery/db/479188.sst
osqueryd  81001               root   20u    systm 0xfab7ceed5dbcf7c7         0t0            [ctl com.apple.netsrc id 9 unit 49]
osqueryd  81001               root   21r      REG                1,4       11117 8595749698 /usr/local/zentral/osquery/db/479187.sst
osqueryd  81001               root   23u     unix 0xfab7ceed546f40b7         0t0            ->0xfab7ceed546f39af
osqueryd  81001               root   26u     unix 0xfab7ceed546f1a6f         0t0            /var/osquery/osquery.em

zbuc on 13 Dec 2017

Thanks @zbuc

Are you able to see message "kqueue: Too many open files" with the same instance of osqueryd for which you executed "lsof" ?

uptycs-nishant on 13 Dec 2017

Yes

On Wed, Dec 13, 2017 at 12:06 PM, uptycs-nishant notifications@github.com
wrote:

Thanks @zbuc https://github.com/zbuc

Are you able to see message "kqueue: Too many open files" with the same
instance of osqueryd for which you executed "lsof" ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/facebook/osquery/issues/3984#issuecomment-351456570,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAiyuBIXMOX0uEa-e2QCFrUlXwkauISoks5tAAQygaJpZM4Q1OL3
.

zbuc on 13 Dec 2017

Figured out a workaround for this. Ultimately, we were seeing this happen on hosts with a low value for open files according to ulimit -a:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited

The value of sysctl kern.maxfiles defines the upper bounds of what that number can be set to, and that value can be changed via launchctl limit maxfiles <soft limit> <hard limit>. Unfortunately, changes made from the command line don't persist past a reboot, so the two options here are:

Install the maxfiles plist listed here: https://blog.dekstroza.io/ulimit-shenanigans-on-osx-el-capitan. I recommend using the value of sysctl kern.maxfiles as the number to set it to. Changes will be applied after a reboot.
Untested (but should theoretically work): Modify the osquery plist arguments to run uname -n <large_number> before osqueryd in the program arguments to change the maxfile value for the session (not OS-wide)

clong on 13 Dec 2017

Based upon the evidence provided by @zbuc , I have come up with following analysis -

osqueryd limit for can be opened files provided by @zbuc
"-n: file descriptors 4864"
It is highly unlikely that in his case problem arose because of osqueryd ran out of file descriptors.
I was able to reproduce the problem via stressing the system with the following program -
With the following program I hit "Too many open files in system" which caused osquery to fail.

300_spartans.c

#include<unistd.h>
#include<fcntl.h>

int main()
{
    int max = 400;
    int counter;
    int fd;
    for(counter=1;counter < 300; counter++)
    {
        pid_t cpid = fork();
        if (cpid == 0) {
            break;
        }
    }

    for (counter = 1; counter <= 40; counter++)
    {
        fd = open("./300_spartans.c",O_RDONLY);
    }
    for(;;)
    sleep(100);
}

At First I was getting following message -
"03:21:55.192575 109350912 registry.cpp:423] distributed registry tls plugin caused exception: kqueue: Too many open files in system"
But after some time l the same error converted to the following message -
"03:21:55.192909 109350912 registry.cpp:423] distributed registry tls plugin caused exception: kqueue: Too many open files"

I believe in case of @zbuc system was under stress and that caused the problem. Need to increase system-wide limit as well.

Further finding

The above message "kqueue:Too many open files in system" converted to "kqueue:Too many open files" got me curious and I believe this is bug. It led me to -
https://opensource.apple.com/source/xnu/xnu-4570.1.46/bsd/kern/kern_descrip.c.auto.html
function falloc_withalloc_locked()

static int
falloc_withalloc_locked(proc_t p, struct fileproc **resultfp, int *resultfd,
    vfs_context_t ctx, fp_allocfn_t fp_zalloc, void *crarg,
    int locked) 
{
...
...
    if ( (error = fdalloc(p, 0, &nfd)) ) {
        if (!locked)
            proc_fdunlock(p);
        return (error);
    }

    if (nfiles >= maxfiles) {
        if (!locked)
            proc_fdunlock(p);
        tablefull("file");
        return (ENFILE);
    }
...
}

I believe these two if statements 'if ( (error = fdalloc(p, 0, &nfd)) )' and 'if (nfiles >= maxfiles)' needs to be swapped.

uptycs-nishant on 14 Dec 2017

Getting hit with this as well, let me know if there is anything I can provide.

johnnyramos on 16 Dec 2017

Change of message from "kqueue:Too many open files in system" to "kqueue:Too many open files" is manifestation of a bug in the form of resource leak ( process's file descriptor table slot is being leaked here) -
Call to fdalloc(p, 0, &nfd) checks that if the process is opening more than the number of files it can open and reserve a slot in the file-descriptor table which is returned via nfd.

If "(nfiles >= maxfiles) " fails then is not un-reserving that slot. That way here is the resource leak.
If these two if conditions are swapped then it solves this problem.

uptycs-nishant on 16 Dec 2017

@uptycs-nishant, if I understand correctly then you should report that bug to Apple's RADR bug tracker.

theopolis on 17 Dec 2017

We have seen an error similar to this on at least one of our hosts.

This environment: macOS 10.12.6

osquery version 2.10.0

E1219 12:13:30.443387 35753984 registry.cpp:423] distributed registry tls plugin caused exception: pipe_select_interrupter: Too many open files

Output of ulimit -a:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 7168
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited

And another mac, 10.12.6, osquery 2.10.0:

E1219 22:38:42.418575 264404992 fsevents.cpp:114] Cannot create FSEvent stream: FSEventStreamCreate failed
E1219 22:38:42.418942 265478144 registry.cpp:423] distributed registry tls plugin caused exception: kqueue: Too many open files

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       709-n: file descriptors                256

mcburrlin on 19 Dec 2017

opened a bug with apple -
https://bugreport.apple.com/web/?problemID=36148377

uptycs-nishant on 21 Dec 2017

I see a lot of folks with 256 as the soft limit for open descriptors. We can change this limit to be the max of about 10k when the program starts. This should handle the RocksDB cases. For those having issues I can provide some test binaries if you’d like. Just +1 this comment or DM me in Slack.

There’s another potential issue, alongside the FSEvents bug, where we could potentially subscribe to 10k+ locations for FIM. We should add some protection and alarming logic around this.

theopolis on 22 Dec 2017

On the RADAR now - http://www.openradar.me/36148377

uptycs-nishant on 23 Dec 2017

👍2

2.11.2 Completely resolved these issues for us

clong on 16 Jan 2018

👍1

Was this page helpful?

0 / 5 - 0 ratings