osqueryi doesn't create extension manager socket under certain conditions

[marpaia]

Consider the following minimum viable extension that you can create at ./extension.go and build with go build -o extension.ext ./extension.go.

package main

import (
    "context"
    "flag"
    "log"

    "github.com/kolide/osquery-go"
    "github.com/kolide/osquery-go/plugin/table"
)

func main() {
    flSocket := flag.String("socket", "", "")
    flag.Int("timeout", 0, "")
    flag.Int("interval", 0, "")
    flag.Bool("verbose", false, "")
    flag.Parse()

    if *flSocket == "" {
        log.Fatalln("--socket flag cannot be empty")
    }

    server, err := osquery.NewExtensionManagerServer("dev_extension", *flSocket)
    if err != nil {
        log.Fatalf("Error creating osquery extension server: %s\n", err)
    }

    server.RegisterPlugin(
        table.NewPlugin(
            "test_table",
            []table.ColumnDefinition{
                table.TextColumn("foo"),
            },
            func(ctx context.Context, queryContext table.QueryContext) ([]map[string]string, error) {
                return []map[string]string{
                    map[string]string{
                        "foo": "bar",
                    },
                }, nil
            },
        ),
    )

    if err := server.Run(); err != nil {
        log.Fatal(err)
    }
}

If you launch osqueryi with the --extension flag, consider the output working correctly:

$ osqueryi --extension=./extension.ext
Using a virtual database. Need help, type '.help'
osquery> .all test_table;
+-----+
| foo |
+-----+
| bar |
+-----+
osquery> .exit

Now, let's launch osqueryi with --verbose as well:

$ osqueryi --extension=./extension.ext --verbose
I0830 12:38:18.495394 3674387392 init.cpp:392] osquery initialized [version=2.7.0]
I0830 12:38:18.495800 3674387392 extensions.cpp:288] Could not autoload extensions: Failed reading: /var/osquery/extensions.load
I0830 12:38:18.496183 3674387392 init.cpp:651] Error reading config: config file does not exist: /var/osquery/osquery.conf
I0830 12:38:18.496206 195674112 interface.cpp:327] Extension manager service starting: /Users/marpaia/.osquery/shell.em
I0830 12:38:18.497160 194600960 watcher.cpp:556] Created and monitoring extension child (86353): ./extension.ext
2017/08/30 12:38:18 Error creating osquery extension server: opening socket transport: dial unix /Users/marpaia/.osquery/shell.em: connect: no such file or directory
Using a virtual database. Need help, type '.help'
osquery> W0830 12:38:21.504050 194600960 watcher.cpp:525] Extension respawning too quickly: ./extension.ext
I0830 12:38:21.504719 194600960 watcher.cpp:556] Created and monitoring extension child (86354): ./extension.ext
2017/08/30 12:38:21 Error creating osquery extension server: opening socket transport: dial unix /Users/marpaia/.osquery/shell.em: connect: no such file or directory
W0830 12:38:24.510625 194600960 watcher.cpp:525] Extension respawning too quickly: ./extension.ext
I0830 12:38:24.511242 194600960 watcher.cpp:556] Created and monitoring extension child (86355): ./extension.ext
2017/08/30 12:38:24 Error creating osquery extension server: opening socket transport: dial unix /Users/marpaia/.osquery/shell.em: connect: no such file or directory
W0830 12:38:27.514295 194600960 watcher.cpp:525] Extension respawning too quickly: ./extension.ext
I0830 12:38:27.515103 194600960 watcher.cpp:556] Created and monitoring extension child (86356): ./extension.ext
2017/08/30 12:38:27 Error creating osquery extension server: opening socket transport: dial unix /Users/marpaia/.osquery/shell.em: connect: no such file or directory

osquery> .exit
^C^D

As you can see, the extension can't connect to ~/.osqueryi/shell.em because it doesn't exist. The process has to be manually killed after this.

# ran while the above was running
$ file /Users/marpaia/.osquery/shell.em
/Users/marpaia/.osquery/shell.em: cannot open `/Users/marpaia/.osquery/shell.em' (No such file or directory)
$ ls -la /Users/marpaia/.osquery
total 8
drwxr-xr-x   3 marpaia  staff   102 Aug 30 12:38 .
drwxr-xr-x+ 69 marpaia  staff  2346 Aug 30 12:39 ..
-rw-------   1 marpaia  staff    23 Aug 30 12:38 .history
$

So for some reason, when osqueryi is launched with --verose AND an extension is specified, the shell.em file doesn't get created. However, if you just launch osqueryi --verbose, the socket IS created. I can then launch ./extension.ext --socket=/Users/marpaia/.osquery/shell.em and it will connect fine:

$ osqueryi --verbose
I0830 12:47:10.494910 3674387392 init.cpp:392] osquery initialized [version=2.7.0]
I0830 12:47:10.496155 3674387392 extensions.cpp:288] Could not autoload extensions: Failed reading: /var/osquery/extensions.load
I0830 12:47:10.496901 3674387392 init.cpp:651] Error reading config: config file does not exist: /var/osquery/osquery.conf
I0830 12:47:10.496923 242429952 interface.cpp:327] Extension manager service starting: /Users/marpaia/.osquery/shell.em
Using a virtual database. Need help, type '.help'
osquery> I0830 12:48:16.466702 243490816 interface.cpp:141] Registering extension (dev_extension, 41711, version=, sdk=)
I0830 12:48:16.467258 243490816 registry.cpp:348] Extension 41711 registered table plugin test_table

I've tried to make this as reproducible as possible, let me know if there's anything else I can explain further.

[theopolis]

Trying to reproduce:

 osqueryi --verbose --extensions_autoload /dev/null
I0901 13:05:46.939107 3468268480 init.cpp:392] osquery initialized [version=2.7.0]
I0901 13:05:46.943255 157679616 interface.cpp:327] Extension manager service starting: /Users/reed/.osquery/shell.em
I0901 13:05:46.943670 3468268480 init.cpp:651] Error reading config: config file does not exist: /var/osquery/osquery.conf
Using a virtual database. Need help, type '.help'
osquery> select path, type from file where path in (select value from osquery_flags where name = 'extensions_socket');
+-------------------------------+--------+
| path                          | type   |
+-------------------------------+--------+
| /Users/reed/.osquery/shell.em | socket |
+-------------------------------+--------+
osquery> 

Now with the inline-built extension:

osqueryi --verbose --extensions_autoload /dev/null --extension ./build/darwin/osquery/example_extension.ext
I0901 13:37:01.357025 3468268480 init.cpp:392] osquery initialized [version=2.7.0]
I0901 13:37:01.357656 170713088 interface.cpp:327] Extension manager service starting: /Users/reed/.osquery/shell.em
I0901 13:37:01.357759 3468268480 init.cpp:651] Error reading config: config file does not exist: /var/osquery/osquery.conf
I0901 13:37:01.358542 169639936 watcher.cpp:556] Created and monitoring extension child (93347): ./build/darwin/osquery/example_extension.ext
I0901 13:37:01.381961 3468268480 init.cpp:383] osquery extension initialized [sdk=2.7.0]
I0901 13:37:01.384196 171773952 interface.cpp:141] Registering extension (example, 16077, version=0.0.1, sdk=2.7.0)
I0901 13:37:01.384232 171773952 registry.cpp:348] Extension 16077 registered config plugin example
Using a virtual database. Need help, type '.help'
I0901 13:37:01.404259 171773952 registry.cpp:348] Extension 16077 registered table plugin complex_example
I0901 13:37:01.404289 171773952 registry.cpp:348] Extension 16077 registered table plugin example
I0901 13:37:01.406497 3468268480 extensions.cpp:462] Extension (example, 16077, 0.0.1, 2.7.0) registered
I0901 13:37:01.406517 187441152 interface.cpp:305] Extension service starting: /Users/reed/.osquery/shell.em.16077
osquery> select * from example;
+--------------+-----------------+
| example_text | example_integer |
+--------------+-----------------+
| example      | 1               |
+--------------+-----------------+
osquery> 

I0901 13:37:12.210743 189562880 interface.cpp:75] Extension 16077 requested shutdown

[arirubinstein]

I'm able to reproduce this with the brew version 2.11.2 and a simple extension that registers a few tables, and at bootup attempts to stat the socket location, sleeping for 1 second before retry if the file doesn't exist. The homebrew version succeeds once every 5-ish times, but mostly hangs. The brew version is from this bottle with hash 9512eff161f1d323c8cce485d3c1ad297f87e1bd590edf93f9aedc08cf130e60
I am unable to reproduce this with a binary from the osquery.io though, and the extension loads perfectly every time.

osqueryi 2.11.2 from osquery.io succeeding

aricomp:osqe ari$ osqueryi --verbose --extension ./osq.ext 
I0104 13:43:16.652858 3030364992 init.cpp:380] osquery initialized [version=2.11.2]
I0104 13:43:16.654491 3030364992 extensions.cpp:300] Could not autoload extensions: Failed reading: /var/osquery/extensions.load
I0104 13:43:16.656662 235294720 interface.cpp:338] Extension manager service starting: /var/folders/mt/ntcdcmd443g0_mzt1j9d2dsm0040gn/T/osquery-23001/shell.em
I0104 13:43:16.664507 234221568 watcher.cpp:591] Created and monitoring extension child (27181): ./osq.ext
2018/01/04 13:43:16 Interval: 3, Timeout: 3, Socket: /var/folders/mt/ntcdcmd443g0_mzt1j9d2dsm0040gn/T/osquery-23001/shell.em, Retry: 3
I0104 13:43:16.732692 3030364992 init.cpp:645] Error reading config: config file does not exist: /var/osquery/osquery.conf
I0104 13:43:16.732812 236355584 interface.cpp:141] Registering extension (osqe, 24433, version=, sdk=)
Using a virtual database. Need help, type '.help'
I0104 13:43:16.836000 236355584 registry.cpp:351] Extension 24433 registered table plugin ext_hostname
I0104 13:43:16.836031 236355584 registry.cpp:351] Extension 24433 registered table plugin ext_time

homebrew osqueryi @ 2.11.2, succeeding

This output happens every 5-ish times:

aricomp:osqe ari$ /usr/local/Cellar/osquery/2.11.2/bin/osqueryi --verbose --extension ./osq.ext 
I0104 13:43:41.890681 3030364992 init.cpp:380] osquery initialized [version=2.11.2]
I0104 13:43:41.891285 3030364992 extensions.cpp:300] Could not autoload extensions: Failed reading: /var/osquery/extensions.load
I0104 13:43:41.891912 20725760 interface.cpp:338] Extension manager service starting: /var/folders/mt/ntcdcmd443g0_mzt1j9d2dsm0040gn/T/osquery-27911/shell.em
I0104 13:43:41.893772 3030364992 init.cpp:645] Error reading config: config file does not exist: /var/osquery/osquery.conf
I0104 13:43:41.907867 19652608 watcher.cpp:591] Created and monitoring extension child (27185): ./osq.ext
Using a virtual database. Need help, type '.help'
2018/01/04 13:43:41 Interval: 3, Timeout: 3, Socket: /var/folders/mt/ntcdcmd443g0_mzt1j9d2dsm0040gn/T/osquery-27911/shell.em, Retry: 3
I0104 13:43:41.934353 21786624 interface.cpp:141] Registering extension (osqe, 21607, version=, sdk=)
I0104 13:43:41.935645 21786624 registry.cpp:351] Extension 21607 registered table plugin ext_hostname
I0104 13:43:41.935675 21786624 registry.cpp:351] Extension 21607 registered table plugin ext_time
osquery> .q

homebrew osqueryi @ 2.11.2, failing

The output "Socket x, doesn't exist yet" is generated by the extension attempting to stat the socket path.
While osqueryi is trying to launch the extension over and over again, and even at startup, there is no socket file created on disk.

aricomp:osqe ari$ /usr/local/Cellar/osquery/2.11.2/bin/osqueryi --verbose --extension ./osq.ext 
I0104 13:44:14.815134 3030364992 init.cpp:380] osquery initialized [version=2.11.2]
I0104 13:44:14.815564 3030364992 extensions.cpp:300] Could not autoload extensions: Failed reading: /var/osquery/extensions.load
I0104 13:44:14.816016 3030364992 init.cpp:645] Error reading config: config file does not exist: /var/osquery/osquery.conf
I0104 13:44:14.817302 176832512 watcher.cpp:591] Created and monitoring extension child (27193): ./osq.ext
2018/01/04 13:44:14 Interval: 3, Timeout: 3, Socket: /var/folders/mt/ntcdcmd443g0_mzt1j9d2dsm0040gn/T/osquery-22413/shell.em, Retry: 3
2018/01/04 13:44:14 Socket /var/folders/mt/ntcdcmd443g0_mzt1j9d2dsm0040gn/T/osquery-22413/shell.em, doesn't exist yet. Cur 0, Limit 3
2018/01/04 13:44:15 Socket /var/folders/mt/ntcdcmd443g0_mzt1j9d2dsm0040gn/T/osquery-22413/shell.em, doesn't exist yet. Cur 1, Limit 3
2018/01/04 13:44:16 Socket /var/folders/mt/ntcdcmd443g0_mzt1j9d2dsm0040gn/T/osquery-22413/shell.em, doesn't exist yet. Cur 2, Limit 3
panic: Error creating extension: opening socket transport: dial unix /var/folders/mt/ntcdcmd443g0_mzt1j9d2dsm0040gn/T/osquery-22413/shell.em: connect: no such file or directory

^CUsing a virtual database. Need help, type '.help'
osquery> I0104 13:44:22.578249 3030364992 init.cpp:151] Cannot stop event publisher threads or services
c        c^C

osqueryi hangs at this point, and needs to be force-killed.

[theopolis]

Thanks @arirubinstein, when you try to close osquery and it hangs, can you attack using lldb -p PID and then bt all and paste me the stacktrace for all of the threads?

[theopolis]

It seems likely that the threads spawned by Thrift are having some issue. It may be that control never returns from the Thrift server that is pausing while trying to open the socket, just total guesses here but stacktraces may provide more input.

[arirubinstein]

Here's the lldb output from a hung osqueryi session like above, version=2.11.2

(lldb) bt all
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff7bae4cee libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff7bc21662 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x00000001008e728f osqueryi`boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) + 79
    frame #3: 0x00000001008e68e4 osqueryi`boost::shared_mutex::lock() + 96
    frame #4: 0x00000001008e66f1 osqueryi`boost::unique_lock<boost::shared_mutex>::lock() + 47
    frame #5: 0x00000001009f528d osqueryi`osquery::ExtensionRunnerCore::stop() + 55
    frame #6: 0x00000001009c9c8f osqueryi`osquery::InterruptableRunnable::interrupt() + 65
    frame #7: 0x00000001009ca6d4 osqueryi`osquery::Dispatcher::stopServices() + 120
    frame #8: 0x00000001009a812d osqueryi`osquery::Initializer::requestShutdown(int) + 97
    frame #9: 0x00000001008e5f5f osqueryi`osquery::startOsquery(int, char**, std::__1::function<void ()>) + 1107
    frame #10: 0x00000001008e8fda osqueryi`main + 45
    frame #11: 0x00007fff7b995115 libdyld.dylib`start + 1
    frame #12: 0x00007fff7b995115 libdyld.dylib`start + 1
  thread #2
    frame #0: 0x00007fff7bae4cee libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff7bc21662 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x00000001008e728f osqueryi`boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) + 79
    frame #3: 0x00000001008e68e4 osqueryi`boost::shared_mutex::lock() + 96
    frame #4: 0x00000001008e66f1 osqueryi`boost::unique_lock<boost::shared_mutex>::lock() + 47
    frame #5: 0x00000001009c9f47 osqueryi`osquery::InternalRunnable::run() + 77
    frame #6: 0x00000001009cac61 osqueryi`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (osquery::InternalRunnable::*)(), osquery::InternalRunnable*> > >(void*) + 76
    frame #7: 0x00007fff7bc206c1 libsystem_pthread.dylib`_pthread_body + 340
    frame #8: 0x00007fff7bc2056d libsystem_pthread.dylib`_pthread_start + 377
    frame #9: 0x00007fff7bc1fc5d libsystem_pthread.dylib`thread_start + 13
  thread #3
    frame #0: 0x00007fff7bae4cee libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff7bc21662 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x00000001008e728f osqueryi`boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) + 79
    frame #3: 0x00000001008e68e4 osqueryi`boost::shared_mutex::lock() + 96
    frame #4: 0x00000001008e66f1 osqueryi`boost::unique_lock<boost::shared_mutex>::lock() + 47
    frame #5: 0x00000001009c9f47 osqueryi`osquery::InternalRunnable::run() + 77
    frame #6: 0x00000001009cac61 osqueryi`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (osquery::InternalRunnable::*)(), osquery::InternalRunnable*> > >(void*) + 76
    frame #7: 0x00007fff7bc206c1 libsystem_pthread.dylib`_pthread_body + 340
    frame #8: 0x00007fff7bc2056d libsystem_pthread.dylib`_pthread_start + 377
    frame #9: 0x00007fff7bc1fc5d libsystem_pthread.dylib`thread_start + 13
  thread #4
    frame #0: 0x00007fff7bae5432 libsystem_kernel.dylib`__ulock_wait + 10
    frame #1: 0x00007fff7bc196ba libsystem_platform.dylib`_os_unfair_lock_lock_slow + 140
    frame #2: 0x0000000101d152a9 libjemalloc.2.dylib`je_malloc_mutex_lock_slow + 153
    frame #3: 0x0000000101d0ebe2 libjemalloc.2.dylib`extent_recycle + 135
    frame #4: 0x0000000101d0eb55 libjemalloc.2.dylib`je_extents_alloc + 40
    frame #5: 0x0000000101cfaf72 libjemalloc.2.dylib`arena_bin_malloc_hard + 320
    frame #6: 0x0000000101cfaca3 libjemalloc.2.dylib`je_arena_tcache_fill_small + 286
    frame #7: 0x0000000101d1e49f libjemalloc.2.dylib`je_tcache_alloc_small_hard + 21
    frame #8: 0x0000000101ceede4 libjemalloc.2.dylib`malloc + 972
    frame #9: 0x00007fff7bb3d201 libsystem_malloc.dylib`malloc_zone_malloc + 103
    frame #10: 0x00007fff7bb3c50b libsystem_malloc.dylib`malloc + 24
    frame #11: 0x00007fff7ba0fd99 libsystem_c.dylib`__opendir_common + 196
    frame #12: 0x00007fff7ba0fc33 libsystem_c.dylib`__opendir2$INODE64 + 71
    frame #13: 0x00007fff7ba0f414 libsystem_c.dylib`glob2 + 737
    frame #14: 0x00007fff7ba0eda4 libsystem_c.dylib`glob0 + 514
    frame #15: 0x00007fff7ba0e27f libsystem_c.dylib`__glob + 685
    frame #16: 0x0000000100a00e1d osqueryi`osquery::platformGlob(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 88
    frame #17: 0x00000001009fdb45 osqueryi`osquery::genGlobs(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >&, osquery::GlobLimits) (.llvm.E5254C58) + 81
    frame #18: 0x00000001009fda88 osqueryi`osquery::resolveFilePattern(boost::filesystem::path const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >&, osquery::GlobLimits) + 64
    frame #19: 0x00000001009f5670 osqueryi`osquery::removeStalePaths(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 110
    frame #20: 0x00000001009f5398 osqueryi`osquery::ExtensionRunnerCore::startServer(std::__1::shared_ptr<apache::thrift::TProcessor>) + 204
    frame #21: 0x00000001009f5a67 osqueryi`osquery::ExtensionManagerRunner::start() + 465
    frame #22: 0x00000001009c9f2b osqueryi`osquery::InternalRunnable::run() + 49
    frame #23: 0x00000001009cac61 osqueryi`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (osquery::InternalRunnable::*)(), osquery::InternalRunnable*> > >(void*) + 76
    frame #24: 0x00007fff7bc206c1 libsystem_pthread.dylib`_pthread_body + 340
    frame #25: 0x00007fff7bc2056d libsystem_pthread.dylib`_pthread_start + 377
    frame #26: 0x00007fff7bc1fc5d libsystem_pthread.dylib`thread_start + 13

[theopolis]

Thanks! I think this is related to: https://github.com/jemalloc/jemalloc/issues/895

We do not use jemalloc in our releases.

I was able to replicate with an install from Homebrew.

Executable module set to "/usr/local/Cellar/osquery/2.11.2/bin/osqueryi".
Architecture set to: x86_64h-apple-macosx.
(lldb) bt all
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fffb498d31e libsystem_kernel.dylib`__ulock_wait + 10
    frame #1: 0x00007fffb4a70aff libsystem_platform.dylib`_os_ulock_wait + 25
    frame #2: 0x00007fffb4a703d2 libsystem_platform.dylib`_os_unfair_lock_lock_slow + 130
    frame #3: 0x000000010adb22a9 libjemalloc.2.dylib`je_malloc_mutex_lock_slow + 153
    frame #4: 0x000000010adabbe2 libjemalloc.2.dylib`extent_recycle + 135
    frame #5: 0x000000010adabb55 libjemalloc.2.dylib`je_extents_alloc + 40
    frame #6: 0x000000010ad97172 libjemalloc.2.dylib`je_arena_extent_alloc_large + 128
    frame #7: 0x000000010adaf9c9 libjemalloc.2.dylib`je_large_palloc + 268
    frame #8: 0x000000010ad8c19c libjemalloc.2.dylib`malloc + 1924
    frame #9: 0x00007fffb49e0282 libsystem_malloc.dylib`malloc_zone_malloc + 107
    frame #10: 0x00000001099e36c4 osqueryi`sqlite3MemMalloc + 43
    frame #11: 0x0000000109961df1 osqueryi`sqlite3Malloc.llvm.AB39680A + 206
    frame #12: 0x000000010997dcdc osqueryi`memjrnlWrite + 499
    frame #13: 0x000000010997fb98 osqueryi`writeJournalHdr + 420
    frame #14: 0x0000000109985516 osqueryi`pager_write + 504
    frame #15: 0x000000010999a48a osqueryi`insertCell + 204
    frame #16: 0x00000001099975df osqueryi`sqlite3BtreeInsert + 1619
    frame #17: 0x0000000109990881 osqueryi`sqlite3VdbeExec + 22201
    frame #18: 0x0000000109968fd1 osqueryi`sqlite3_step + 489
    frame #19: 0x000000010996d88b osqueryi`sqlite3_exec + 325
    frame #20: 0x0000000109b08d4d osqueryi`osquery::attachTableInternal(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<osquery::SQLiteDBInstance> const&) + 721
    frame #21: 0x0000000109b0b12e osqueryi`osquery::attachVirtualTables(std::__1::shared_ptr<osquery::SQLiteDBInstance> const&) + 489
    frame #22: 0x0000000109afe306 osqueryi`osquery::SQLiteDBManager::getConnection(bool) + 244
    frame #23: 0x0000000109b0ae74 osqueryi`osquery::attachFunctionInternal(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::function<void (sqlite3_context*, int, sqlite3_value**)>) + 54
    frame #24: 0x0000000109954150 osqueryi`osquery::launchIntoShell(int, char**) + 622
    frame #25: 0x000000010995cf55 osqueryi`osquery::startOsquery(int, char**, std::__1::function<void ()>) + 1097
    frame #26: 0x000000010995ffda osqueryi`main + 45
    frame #27: 0x00007fffb485e235 libdyld.dylib`start + 1
    frame #28: 0x00007fffb485e235 libdyld.dylib`start + 1
  thread #2
    frame #0: 0x00007fffb498cbf2 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fffb4a787fa libsystem_pthread.dylib`_pthread_cond_wait + 712
    frame #2: 0x00007fffb33f5572 libc++.1.dylib`std::__1::condition_variable::__do_timed_wait(std::__1::unique_lock<std::__1::mutex>&, std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >) + 126
    frame #3: 0x0000000109a40e22 osqueryi`osquery::InterruptableRunnable::pauseMilli(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >) + 174
    frame #4: 0x0000000109a37028 osqueryi`osquery::WatcherRunner::start() + 340
    frame #5: 0x0000000109a40f2b osqueryi`osquery::InternalRunnable::run() + 49
    frame #6: 0x0000000109a41c61 osqueryi`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (osquery::InternalRunnable::*)(), osquery::InternalRunnable*> > >(void*) + 76
    frame #7: 0x00007fffb4a7793b libsystem_pthread.dylib`_pthread_body + 180
    frame #8: 0x00007fffb4a77887 libsystem_pthread.dylib`_pthread_start + 286
    frame #9: 0x00007fffb4a7708d libsystem_pthread.dylib`thread_start + 13
  thread #3
    frame #0: 0x00007fffb498cbf2 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fffb4a787fa libsystem_pthread.dylib`_pthread_cond_wait + 712
    frame #2: 0x00007fffb33f5572 libc++.1.dylib`std::__1::condition_variable::__do_timed_wait(std::__1::unique_lock<std::__1::mutex>&, std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >) + 126
    frame #3: 0x0000000109a40e22 osqueryi`osquery::InterruptableRunnable::pauseMilli(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >) + 174
    frame #4: 0x0000000109a64122 osqueryi`osquery::ExtensionManagerWatcher::start() + 64
    frame #5: 0x0000000109a40f2b osqueryi`osquery::InternalRunnable::run() + 49
    frame #6: 0x0000000109a41c61 osqueryi`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (osquery::InternalRunnable::*)(), osquery::InternalRunnable*> > >(void*) + 76
    frame #7: 0x00007fffb4a7793b libsystem_pthread.dylib`_pthread_body + 180
    frame #8: 0x00007fffb4a77887 libsystem_pthread.dylib`_pthread_start + 286
    frame #9: 0x00007fffb4a7708d libsystem_pthread.dylib`thread_start + 13
  thread #4
    frame #0: 0x00007fffb498d31e libsystem_kernel.dylib`__ulock_wait + 10
    frame #1: 0x00007fffb4a70aff libsystem_platform.dylib`_os_ulock_wait + 25
    frame #2: 0x00007fffb4a703d2 libsystem_platform.dylib`_os_unfair_lock_lock_slow + 130
    frame #3: 0x000000010adb22a9 libjemalloc.2.dylib`je_malloc_mutex_lock_slow + 153
    frame #4: 0x000000010ad982c3 libjemalloc.2.dylib`je_arena_malloc_hard + 140
    frame #5: 0x000000010ad8ad50 libjemalloc.2.dylib`a0ialloc + 111
    frame #6: 0x000000010ad8be39 libjemalloc.2.dylib`malloc + 1057
    frame #7: 0x000000010adbe230 libjemalloc.2.dylib`void* newImpl<false>(unsigned long) + 22
    frame #8: 0x0000000109a6c8c2 osqueryi`osquery::ExtensionManagerRunner::start() + 44
    frame #9: 0x0000000109a40f2b osqueryi`osquery::InternalRunnable::run() + 49
    frame #10: 0x0000000109a41c61 osqueryi`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (osquery::InternalRunnable::*)(), osquery::InternalRunnable*> > >(void*) + 76
    frame #11: 0x00007fffb4a7793b libsystem_pthread.dylib`_pthread_body + 180
    frame #12: 0x00007fffb4a77887 libsystem_pthread.dylib`_pthread_start + 286
    frame #13: 0x00007fffb4a7708d libsystem_pthread.dylib`thread_start + 13
(lldb) ^D

[theopolis]

I wonder if there's a way to disable the jemalloc linking? Maybe it's pulled in from RocksDB? https://github.com/Homebrew/homebrew-core/blob/master/Formula/rocksdb.rb#L18

@ilovezfs usually knows everything :P

[ilovezfs]

The linkage is a consequence of https://github.com/Homebrew/homebrew-core/pull/15692. I'll see if we can make it go away by passing -DWITH_JEMALLOC=OFF to cmake.

[ilovezfs]

@theopolis this should take care of it: https://github.com/Homebrew/homebrew-core/pull/22485

[theopolis]

Wow, that was fast, I can re-run tests against that.

[ilovezfs]

It looks like ENV["DISABLE_JEMALLOC"] = "1" should work too.

[ilovezfs]

OK, I have shipped https://github.com/Homebrew/homebrew-core/pull/22485.

[theopolis]

@ilovezfs after a quick few tests it seems to be working fine! @arirubinstein, if you have a chance, do you mind re-testing the brew provided osqueryi?

[ilovezfs]

@theopolis excellent! @arirubinstein you'll need to brew update && brew upgrade since the changed formula is actually rocksdb.

[arirubinstein]

@theopolis @ilovezfs looks like that worked! I can't reproduce it in my environment now with the updated rocksdb formula, and the extension framework is working as expected. Thanks for your help and looking into this 🙇

[ilovezfs]

@arirubinstein hurray! You're welcome and thanks for testing. Hopefully the next jemalloc release will be less temperamental.