Ponyc: Ubuntu 18 x86_64's libatomic's 16 byte CAS isn't atomic

[slfritchie]

Last week, I'd mentioned in one of the Pony public Slack channels that I'd discovered that a 16 byte CAS operation in the Ubuntu 18.04 x86_64's libatomic library isn't what we thought/hope it is: it falls back to an implementation that uses pthread_mutex_lock, memcmp, memcpy, and pthread_mutex_unlock.

When using a Wallaroo application on an Ubu 18 x86_64 platform, I noticed a strange system call pattern where ponyint_mpmcq_pop was calling pthread_mutex_lock and pthread_mutex_unlock thousands of times per second. The multi-producer multi-consumer queue code in the Pony runtime is supposed to be using lock-free data structures, so the mutex-related system calls were a big surprise.

The discussions and related online materials include:

• Slack discussion in the #runtime channel at https://ponylang.slack.com/archives/CFFR9SG13/p1548993997099100 and during the following days
• https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80878
• https://gcc.gnu.org/ml/gcc/2017-01/msg00167.html
• https://patchwork.ozlabs.org/patch/721686/

When I built libatomic for myself on a VirtualBox VM on a MacBook Pro for gcc, here's what I found for configure's discovery process. I have no idea if that environment matches what the Ubuntu package management does for Ubuntu 18.04 x86_64's GCC 7.3.0 packaging, but the results here match my observations. At tag gcc-7_3_0-release:

libatomic/auto-config.h:#define HAVE_ATOMIC_CAS_1 1  
libatomic/auto-config.h:#define HAVE_ATOMIC_CAS_16 0 
libatomic/auto-config.h:#define HAVE_ATOMIC_CAS_2 1  
libatomic/auto-config.h:#define HAVE_ATOMIC_CAS_4 1  
libatomic/auto-config.h:#define HAVE_ATOMIC_CAS_8 1  
libatomic/auto-config.h:#define MAYBE_HAVE_ATOMIC_CAS_1 HAVE_ATOMIC_CAS_1
libatomic/auto-config.h:#define MAYBE_HAVE_ATOMIC_CAS_2 HAVE_ATOMIC_CAS_2
libatomic/auto-config.h:#define MAYBE_HAVE_ATOMIC_CAS_4 HAVE_ATOMIC_CAS_4
libatomic/auto-config.h:#define MAYBE_HAVE_ATOMIC_CAS_8 HAVE_ATOMIC_CAS_8
libatomic/auto-config.h:#define MAYBE_HAVE_ATOMIC_CAS_16 HAVE_ATOMIC_CAS_16

One workaround this problem is to compile ponyc with Clang. I used the default clang package available with Ubuntu 18.04. I've added the needs discussion during sync tag to suggest that perhaps a mix of documentation and code change are probably necessary, but I don't have good advice yet for either.

[jemc]

I haven't deeply familiarized myself with this yet, but is there a way for us to detect this problem in our Makefile when compiling libponyrt? If so, I'd suggest that we issue a warning or error from the Makefile when compiling in such an environment.

[slfritchie]

My memory from last week is that:

• mpmcq.c uses the atomic_compare_exchange_weak_explicit macro which substitutes a call to __atomic_compare_exchange_n
• libatomic compilation magic + linking ends up with a call to libat_compare_exchange (in https://github.com/gcc-mirror/gcc/blob/gcc-7_3_0-release/libatomic/gcas.c#L79 where n=16)
• The EXACT macro in gcas.c does nothing because the EXACT_INLINE macro says that HAVE_ATOMIC_CAS_16 isn't true.
• So the top half of libat_compare_exchange does nothing, and the bottom half does the mutex lock, memcmp, memcpy, and unlock.

But the libat_compare_exchange function is defined & that symbol is resolvable by the linker. My best guess would be to create a small test application that defines a pthread_mutex_lock function & symbol that overrides the one normally found by the linker + libpthread. When the test program runs, and if it calls our overriding function, then we don't have atomic CAS.

[SeanTAllen]

Sounds like a job for ./configure @slfritchie

[aturley]

Leaving as "needs discussion during sync" because we need input from @sylvanc.

[SeanTAllen]

At this point, this is a "well, it's an issue but we don't know what to do about it".