Libzmq: osx make check fails on tests/test_system

[colmsjo]

Got this error when running: make check

/Applications/Xcode.app/Contents/Developer/usr/bin/make  check-TESTS
FAIL: tests/test_system
PASS: tests/test_pair_inproc
PASS: tests/test_pair_tcp
PASS: tests/test_reqrep_inproc
PASS: tests/test_reqrep_tcp
PASS: tests/test_hwm
PASS: tests/test_hwm_pubsub
PASS: tests/test_reqrep_device
PASS: tests/test_sub_forward
PASS: tests/test_invalid_rep
PASS: tests/test_msg_flags
PASS: tests/test_msg_ffn
PASS: tests/test_connect_resolve
PASS: tests/test_immediate
PASS: tests/test_last_endpoint
PASS: tests/test_term_endpoint
PASS: tests/test_srcfd
PASS: tests/test_monitor
PASS: tests/test_router_mandatory
PASS: tests/test_router_mandatory_hwm
PASS: tests/test_router_handover
PASS: tests/test_probe_router
PASS: tests/test_stream
PASS: tests/test_stream_empty
PASS: tests/test_stream_disconnect
PASS: tests/test_stream_timeout
PASS: tests/test_disconnect_inproc
PASS: tests/test_unbind_inproc
PASS: tests/test_unbind_wildcard
PASS: tests/test_ctx_options
PASS: tests/test_ctx_destroy
PASS: tests/test_security_null
PASS: tests/test_security_plain
PASS: tests/test_security_curve
PASS: tests/test_iov
PASS: tests/test_spec_req
PASS: tests/test_spec_rep
PASS: tests/test_spec_dealer
PASS: tests/test_spec_router
./config/test-driver: line 107: 74698 Abort trap: 6           "$@" > $log_file 2>&1
FAIL: tests/test_spec_pushpull
./config/test-driver: line 107: 74721 Abort trap: 6           "$@" > $log_file 2>&1
XFAIL: tests/test_req_correlate
./config/test-driver: line 107: 74743 Abort trap: 6           "$@" > $log_file 2>&1
XFAIL: tests/test_req_relaxed
PASS: tests/test_conflate
PASS: tests/test_inproc_connect
PASS: tests/test_issue_566
PASS: tests/test_proxy
PASS: tests/test_proxy_single_socket
PASS: tests/test_proxy_terminate
PASS: tests/test_getsockopt_memset
PASS: tests/test_setsockopt
PASS: tests/test_many_sockets
PASS: tests/test_ipc_wildcard
PASS: tests/test_diffserv
PASS: tests/test_connect_rid
PASS: tests/test_bind_src_address
PASS: tests/test_metadata
PASS: tests/test_capabilities
PASS: tests/test_xpub_nodrop
PASS: tests/test_xpub_manual
PASS: tests/test_xpub_welcome_msg
PASS: tests/test_atomics
PASS: tests/test_client_server
PASS: tests/test_thread_safe
PASS: tests/test_sockopt_hwm
PASS: tests/test_heartbeats
PASS: tests/test_stream_exceeds_buffer
PASS: tests/test_poller
PASS: tests/test_timers
PASS: tests/test_radio_dish
PASS: tests/test_udp
PASS: tests/test_shutdown_stress
PASS: tests/test_pair_ipc
PASS: tests/test_reqrep_ipc
PASS: tests/test_use_fd_ipc
PASS: tests/test_use_fd_tcp
PASS: tests/test_timeo
PASS: tests/test_filter_ipc
PASS: tests/test_fork
============================================================================
Testsuite summary for zeromq 4.2.0
============================================================================
# TOTAL: 78
# PASS:  74
# SKIP:  0
# XFAIL: 2
# FAIL:  2
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to [email protected]
============================================================================
make[3]: *** [test-suite.log] Error 1
make[2]: *** [check-TESTS] Error 2
make[1]: *** [check-am] Error 2
make: *** [check-recursive] Error 1

[bluca]

Latest version? Environment? Did you increase the OS limits for file descriptors etc (via sudo sysctl -w kern.maxfiles=64000 ; sudo sysctl -w kern.maxfilesperproc=64000 ; sudo launchctl limit maxfiles 64000 64000 ; ulimit -n 64000)?

Works fine in the OSX CI: https://travis-ci.org/zeromq/libzmq/jobs/119515358

[colmsjo]

Yes, latest version but I did not increase the OS limits. It works after the limits were increased. Perhaps you should mention this in the installation page for OSX?

Thank's for the quick reply!

31 mars 2016 kl. 14:41 skrev Luca Boccassi [email protected]:

Latest version? Environment? Did you increase the OS limits for file descriptors etc (via sudo sysctl -w kern.maxfiles=64000 ; sudo sysctl -w kern.maxfilesperproc=64000 ; sudo launchctl limit maxfiles 64000 64000 ; ulimit -n 64000)?

Works fine in the OSX CI: https://travis-ci.org/zeromq/libzmq/jobs/119515358 https://travis-ci.org/zeromq/libzmq/jobs/119515358
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub https://github.com/zeromq/libzmq/issues/1878#issuecomment-203915116

[jakirkham]

Had the same issue. Both locally and on Travis CI.

Is it really advisable that we change system settings in this way? For that matter, is it reasonable that a test expects this sort of low level system change from users? How will this affect users who have no clue they are using zmq and then see some strange problem due to the fact they have not made this system change?

[bluca]

Having tests that exercise the library in hard conditions is not only reasonable but very desirable, in my opinion. One of the main selling points of ZMQ is its extreme scalability. We must ensure that we don't accidentally break that, and there's no better way than automated tests and CI.

Of course, tests should have reasonable requirements. If a test can work only in a very specific, narrow and hard to setup environment then it's not a reasonable test. For example, I recently removed a test that needed 2+ GB of ram (test_large_msg), as that was definitely an unreasonable requirement.

But in this case all that's needed is to increase the file limit which is a very simple operation, and then again it's not necessary on all sensible platforms. Sadly OSX is not sensible, and has a stupidly low default setting. That's Apple's fault.

We should of course document this (is there an OSX-page on the wiki? Couldn't find it by googling), but in the end make check is not a mandatory step. It's for developers, not for users, and I don't think we should tell the latter to run it at all.

[jakirkham]

As we both know, another user has demonstrated that this is not limited to OS X. So, I think blaming Apple is simply avoiding the severity and nature of the problem.

[bluca]

Well, we can blame both Apple and RedHat then :-)

But my main point still stands regardless. We should document, sure, but make check is for developers. Having to run a single command once when developing the library is not unreasonable if the gain is much higher coverage IMHO. Users don't have to run make check at all.

And it serves to highlight something else which is important - if you want your ZMQ application to scale above some arbitrary pre-defined limits, you must tweak your system, and this shows you how. So it should be documented regardless (I'm pretty sure it already was, although can't remember where!).

[mingwandroid]

Just to chime in here, make check is not just for developers, it is also for packagers, at least for any that care that they are packaging non-broken software.

[mingwandroid]

I'm not saying zeromq is broken, but if we don't run make check we cannot have very much confidence that our builds of it are not broken.

[bluca]

make check works fine and is ran by the CI on every commit on multiple platforms, including osx

[mingwandroid]

But only if you do things that we are unable to do as packagers (we make use of systems we do not have administrative control over), namely increasing various system limits. Would you accept a patch that checks these limits are satisfied and skips the checks when this is not the case? I cannot abide the fact that we must apply the following patch to our builds of zeromq:

https://github.com/conda-forge/zeromq-feedstock/blob/master/recipe/0005-osx-test.patch

[bluca]

sure, pull requests are always welcome

[mingwandroid]

Great! Before I start, would you rather this was done in the CMake files or in the test source code?

[bluca]

it's probably better as a runtime check, so that it doesn't have to be done twice for autoconf too