Gunicorn: Gunicorn doesn't work on macos high sierra

[SG5]

I run gunicorn with command python .venv/bin/gunicorn app.wsgi -w 2

After this I've got error objc[41382]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug. and gunicorn processes starts consume 100% of my cores

I found article about Unicorn and temporary workaround with export OBJC_DISABLE_INITIALIZE_FORK_SAFETY works for me, but maybe you can fix it without environment variable

macos 10.13.3 (17D47)
gunicorn (19.7.1)
Python 3.6.4
Django (1.11.4)

[berkerpeksag]

Thank you for the report. I think we should be user friendly and set the environment variable in Gunicorn. @benoitc @tilgovi what do you think?

[benoitc]

Well I'm using macosx latest version and don't reproduce the issue:

± |master ✓| → uname -a
Darwin Benoits-iPro.lan 17.4.0 Darwin Kernel Version 17.4.0: Tue Dec 19 10:52:05 PST 2017; root:xnu-4570.47.16~1/RELEASE_X86_64 x86_64

 2018-02-08 20:38:56 ⌚  Benoits-iPro in ~/MiscProjects/test_env/gunicorn/examples
± |master ✓| → gunicorn -w 3 test:app
[2018-02-08 20:39:01 +0100] [48884] [INFO] Starting gunicorn 19.7.1
[2018-02-08 20:39:01 +0100] [48884] [INFO] Listening at: http://127.0.0.1:8000 (48884)
[2018-02-08 20:39:01 +0100] [48884] [INFO] Using worker: sync
[2018-02-08 20:39:01 +0100] [48887] [INFO] Booting worker with pid: 48887
[2018-02-08 20:39:01 +0100] [48888] [INFO] Booting worker with pid: 48888
[2018-02-08 20:39:01 +0100] [48889] [INFO] Booting worker with pid: 48889

What does your application? Is .venv related to that pipenv thing?

[benoitc]

@berkerpeksag I agree on the general principle, but on that particular environment setting I don't know. I would rather try to understand what trigger it at first.

@SG5 does your application trigger a fork or start a thread in app.wsgi ?

[benoitc]

This resources is also interesting:
http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html

also:
https://bugs.ruby-lang.org/issues/14009

[SG5]

@benoitc yes, .venv is my local virtualenv folder.
app.wsgi is django-generated wsgi file from this template

[tilgovi]

I am also not sure about this one. I am on High Sierra and run Gunicorn every day. I have yet to encounter this issue.

On the other hand, I use latest Python from Homebrew. Is this using MacOS Python?

[berkerpeksag]

See also https://bugs.python.org/issue30837 for a similar report with a system Python 2.7 (Python version is not important)

[berkerpeksag]

Quoting from https://github.com/ansible/ansible/issues/31869#issuecomment-355994649:

Taking the latest python distributed from python.org seems to fix this error without needing to set the environment variable (tested on python 3.6).

[benoitc]

I would close it since the issue is not ours. DO you think we should still add this environment variable handling though?

[berkerpeksag]

Based on my last two comments, I'm inclined to close this as 'wontfix'. Perhaps we can warn users with system Python installed on macOS in release notes?

[benoitc]

closing the issue. nothing we should do.

[robertknight]

Several developers at Hypothesis running macOS 10.15 encountered this error after a routine brew upgrade upgraded libpq / postgres from v11 to v12.

Putting a breakpoint at the objc_initializeAfterForkError function in the worker gives this stack trace:

 thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x00007fff71f1da54 libobjc.A.dylib`objc_initializeAfterForkError
    frame #1: 0x00007fff71f1dc53 libobjc.A.dylib`performForkChildInitialize(objc_class*, objc_class*) + 293
    frame #2: 0x00007fff71f1e245 libobjc.A.dylib`initializeNonMetaClass + 519
    frame #3: 0x00007fff71f1e991 libobjc.A.dylib`initializeAndMaybeRelock(objc_class*, objc_object*, mutex_tt<false>&, bool) + 214
    frame #4: 0x00007fff71f103db libobjc.A.dylib`lookUpImpOrForward + 969
    frame #5: 0x00007fff71f0fb99 libobjc.A.dylib`_objc_msgSend_uncached + 73
    frame #6: 0x00007fff3bbcaac5 CoreFoundation`-[__NSDictionaryM __setObject:forKey:] + 507
    frame #7: 0x00007fff3bbdcdbf CoreFoundation`__85-[_CFXPreferences(PlistSourceAdditions) withManagedSourceForIdentifier:user:perform:]_block_invoke + 144
    frame #8: 0x00007fff3bbdccf7 CoreFoundation`-[_CFXPreferences(SourceAdditions) withSources:] + 56
    frame #9: 0x00007fff3bbd7dd7 CoreFoundation`-[_CFXPreferences(PlistSourceAdditions) withManagedSourceForIdentifier:user:perform:] + 253
    frame #10: 0x00007fff3bbd7cbf CoreFoundation`-[CFPrefsSearchListSource addManagedSourceForIdentifier:user:] + 101
    frame #11: 0x00007fff3bd32cd7 CoreFoundation`__108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke.249 + 298
    frame #12: 0x00007fff3bbd7b26 CoreFoundation`-[_CFXPreferences(SearchListAdditions) withSearchLists:] + 56
    frame #13: 0x00007fff3bbd79ca CoreFoundation`__108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke + 299
    frame #14: 0x00007fff3bbd751f CoreFoundation`-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:] + 364
    frame #15: 0x00007fff3bbd72ae CoreFoundation`-[_CFXPreferences copyAppValueForKey:identifier:container:configurationURL:] + 145
    frame #16: 0x00007fff3bbd71e3 CoreFoundation`_CFPreferencesCopyAppValueWithContainerAndConfiguration + 101
    frame #17: 0x00007fff5dd00601 Heimdal`init_context_from_config_file + 2961
    frame #18: 0x00007fff5dce7659 Heimdal`krb5_set_config_files + 345
    frame #19: 0x00007fff5dce7038 Heimdal`krb5_init_context_flags + 296
    frame #20: 0x00007fff5dce6f09 Heimdal`krb5_init_context + 25
    frame #21: 0x00007fff3e6a5f84 GSS`__ApplePrivate__gsskrb5_init + 84
    frame #22: 0x00007fff3e6beee0 GSS`_gssiakerb_acquire_cred + 80
    frame #23: 0x00007fff3e6a6c1c GSS`gss_acquire_cred + 940
    frame #24: 0x000000010f7a06e3 libpq.5.dylib`pg_GSS_have_cred_cache + 54
    frame #25: 0x000000010f78eeb0 libpq.5.dylib`PQconnectPoll + 6356
    frame #26: 0x000000010f78c026 libpq.5.dylib`connectDBComplete + 231
    frame #27: 0x000000010f78c1ad libpq.5.dylib`PQconnectdb + 36
    frame #28: 0x000000010f74c70f _psycopg.cpython-36m-darwin.so`conn_connect + 143
    frame #29: 0x000000010f74dd6a _psycopg.cpython-36m-darwin.so`connection_init + 330
    frame #30: 0x000000010d663c54 python`type_call + 292
    frame #31: 0x000000010d600f41 python`_PyObject_FastCallDict + 321
    frame #32: 0x000000010d601796 python`_PyObject_CallFunction_SizeT + 246
    frame #33: 0x000000010f747b2e _psycopg.cpython-36m-darwin.so`psyco_connect + 190
    frame #34: 0x000000010d64e4b7 python`PyCFunction_Call + 71
    frame #35: 0x000000010d6dc002 python`_PyEval_EvalFrameDefault + 25954
    frame #36: 0x000000010d6df973 python`_PyEval_EvalCodeWithName + 2867
    frame #37: 0x000000010d6d5a57 python`PyEval_EvalCodeEx + 55
    frame #38: 0x000000010d62c54f python`function_call + 351

This blog post about Phusion seems relevant here. The workaround mentioned is to just disable the warning by setting an environment variable and hoping that nothing breaks:

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

The section headed "Towards a real solution" proposes that packages should agree not to launch any additional threads at import time so they don't trigger this issue. I'm not sure at the moment who/what is launching the threads that ultimately trigger this error.

[boxed]

I'm using a brew supplied 3.8.2 and I get this. So I don't think it's because of the system installed python!

I would also note that if you look at the linked ansible issue, the latest comment is also of someone who says the problem is still there and there's something fishy going on.

[jamadden]

I'm using a brew supplied 3.8.2 and I get this. So I don't think it's because of the system installed python!

Right, this has nothing to do with a system supplied Python or one built from scratch. This has everything to do with the fact that the macOS system libraries themselves are not fork safe anymore. Setting the environment variable before starting gunicorn is the only way to work around the issue (and even then there are potentially other issues it may expose, but those are much more rare in my experience).

[boxed]

I might be confused, but isn't it that they aren't safe for a very specific scenario: doing some type of operations that touch system libraries between fork() and exec()? Can't this be avoided? (Maybe not by gunicorn, but...)

[jamadden]

Technically, at the POSIX level, there's actually very few things that are safe to do between fork and exec especially once you get threads involved. Here's an overview. In practice, most systems are pretty lax and you can get away with a lot — but threads remain a problem, and not just in system libraries. https://github.com/benoitc/gunicorn/issues/2478 is an example where starting a thread before fork results in the child process (the worker) simply hanging in a Python library.

Apple's system libraries can be unsafe to use after a fork if they've been used at all in the parent process, whether or not threads have been used, though again, threads make problems more likely. This is seen with the NSDatePlaceholder example, and it can also be seen in certain types of name (DNS) resolution, IIRC.

gunicorn never execs after a fork — it's written in the classic style, where a forked process just continues running. gunicorn can try to minimize this problem by making the fewest possible system calls in the parent process (but sometimes they are invisible, hidden in the depths of the runtime). You as an application designer can minimize this problem by not using preload (or if you use it, minimizing import-time side effects: especially don't start threads!), and if you use any hooks that run in the parent process, minimize how they interact with the system.

Other than that, there's the environment variable.

[jamadden]

Ah, I found the article I was looking for by one of the Objective-C runtime developers at Apple that explains this particular problem in a little more depth.

[boxed]

Thanks for the explanation! Hopefully that helps others who end up here from google in the future.