ompi 🚀 - OS-X: need a workaround for long session dir name

Do you have a good reproducer?

I'm tinkering in that area at the moment, and might be able to address this while I'm in the code.

jjhursey on 10 Feb 2017

ring_c.c?

hppritcha on 10 Feb 2017

The obscure error at startup typically is

[my_mac_laptop:68818] [[53415,0],0] ORTE_ERROR_LOG: Bad
parameter in file ../../orte/orted/pmix/pmix_server.c at line 264

[my_mac_laptopl:68818] [[53415,0],0] ORTE_ERROR_LOG: Bad
parameter in file ../../../../../orte/mca/ess/hnp/ess_hnp_module.c at line
666

I see this on my El Capitan OS. One of the users seeing this is running Sierra.

hppritcha on 10 Feb 2017

I'm pretty sure you have already noticed that despite the fact that the default TMPDIR is very long, the /tmp link to it. So, [for the last few months] as a workaround I just export TMPDIR to /tmp in my .bashrc.

bosilca on 10 Feb 2017

@bgoglin Yeah, but out of the box, it doesn't work for users. That's the main issue. 😦

jsquyres on 10 Feb 2017

Just to be clear: this is happening on both v2.0.x and v2.x (to me on my Mac OS Sierra laptop). It is not happening on master.

jsquyres on 10 Feb 2017

Yeah there look to be a number of session directory commits on master that are not on the release branches. https://github.com/open-mpi/ompi/commit/ee56d9dc1a7a3d24cac3aa5a6725370f76542604e in particular might be good to review.

jjhursey on 10 Feb 2017

note that in the v2 branches, PMIx communications only use unix socket.
master uses unix socket by default, but can use tcp sockets too.
i did not check that yet, but there is a possibility master fails using unix sockets on OS X
(bind failure because the unix path that contains $TMPDIR is too long, so fallback to tcp sockets)

on the v2 branches, what if we create a symlink in hardcoded /tmp (if the path is too long) and bind to the symlink ? could this be good enough ?

ggouaillardet on 11 Feb 2017

my bad, both master and v2 branches have the same issue.

the differences are

master implements pmix_show_help so an understandable error message is displayed when a failure occurs
in master, the rendezvous file name is $TMPDIR/pmix-<pid> whereas it is $TMPDIR/openmpi-sessions-<uid>@<hostname>/<jobfam>/pmix-<pid>, which is obviously longer.
i suspect that explains why master works out-of-the-box on @jsquyres laptop whereas v2 fails.

ggouaillardet on 11 Feb 2017

here is a proof of concept for the v2.0.x branch and based on my previous idea (use a temporary and hardcoded /tmp/pmixlink-<pid> when the rendezvous file name is too long.

it works fine on my laptop, even when i force an insanely long $TMPDIR

if we want to move forward with this, i will revamp the client part so we never leak the symbolic link if the connection fails.

diff --git a/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c b/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
index 226b84c..694ea5e 100644
--- a/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
+++ b/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
@@ -245,6 +245,7 @@ PMIX_EXPORT pmix_status_t PMIx_Init(pmix_proc_t *proc)
     pmix_nspace_t *nsptr;
     pmix_cb_t cb;
     int errhandler_ref;
+    bool use_symlink;

     if (NULL == proc) {
         return PMIX_ERR_BAD_PARAM;
@@ -337,7 +338,13 @@ PMIX_EXPORT pmix_status_t PMIx_Init(pmix_proc_t *proc)
     /* setup the path to the daemon rendezvous point */
     memset(&address, 0, sizeof(struct sockaddr_un));
     address.sun_family = AF_UNIX;
-    snprintf(address.sun_path, sizeof(address.sun_path)-1, "%s", uri[2]);
+    use_symlink = (strlen(uri[2]) > sizeof(address.sun_path)-1);
+    if (use_symlink) {
+        snprintf(address.sun_path, sizeof(address.sun_path), "/tmp/pmixlink-%d", getpid());
+        symlink(uri[2], address.sun_path);
+    } else {
+        snprintf(address.sun_path, sizeof(address.sun_path)-1, "%s", uri[2]);
+    }
     /* if the rendezvous file doesn't exist, that's an error */
     if (0 != access(uri[2], R_OK)) {
         pmix_argv_free(uri);
@@ -397,6 +404,9 @@ PMIX_EXPORT pmix_status_t PMIx_Init(pmix_proc_t *proc)
     PMIX_WAIT_FOR_COMPLETION(cb.active);
     rc = cb.status;
     PMIX_DESTRUCT(&cb);
+    if (use_symlink) {
+        unlink(address.sun_path);
+    }

     if (PMIX_SUCCESS == rc) {
         pmix_globals.init_cntr++;
diff --git a/opal/mca/pmix/pmix112/pmix/src/server/pmix_server.c b/opal/mca/pmix/pmix112/pmix/src/server/pmix_server.c
index a81df69..7b7b90d 100644
--- a/opal/mca/pmix/pmix112/pmix/src/server/pmix_server.c
+++ b/opal/mca/pmix/pmix112/pmix/src/server/pmix_server.c
@@ -63,7 +63,7 @@ pmix_server_globals_t pmix_server_globals = {{{0}}};

 // local variables
 static char *myuri = NULL;
-static struct sockaddr_un myaddress;
+static char * myaddress;
 static char *security_mode = NULL;
 static char *mytmpdir = NULL;

@@ -218,22 +218,11 @@ static pmix_status_t initialize_server_base(pmix_server_module_t *module)
     }

     /* now set the address - we use the pid here to reduce collisions */
-    memset(&myaddress, 0, sizeof(struct sockaddr_un));
-    myaddress.sun_family = AF_UNIX;
-    if (0 > asprintf(&pmix_pid, "pmix-%d", pid)) {
+    if (0 > asprintf(&myaddress, "%s/pmix-%d", tdir, pid)) {
         return PMIX_ERR_NOMEM;
     }
-    // If the above set temporary directory name plus the pmix-PID string
-    // plus the '/' separator are too long, just fail, so the caller
-    // may provide the user with a proper help... *Cough*, *Cough* OSX...
-    if ((strlen(tdir) + strlen(pmix_pid) + 1) > sizeof(myaddress.sun_path)-1) {
-        free(pmix_pid);
-        return PMIX_ERR_INVALID_LENGTH;
-    }
-    snprintf(myaddress.sun_path, sizeof(myaddress.sun_path)-1, "%s/%s", tdir, pmix_pid);
-    free(pmix_pid);
     if (0 > asprintf(&myuri, "%s:%lu:%s", pmix_globals.myid.nspace,
-                     (unsigned long)pmix_globals.myid.rank, myaddress.sun_path)) {
+                     (unsigned long)pmix_globals.myid.rank, myaddress)) {
         return PMIX_ERR_NOMEM;
     }

@@ -301,7 +290,7 @@ PMIX_EXPORT pmix_status_t PMIx_server_init(pmix_server_module_t *module,
     pmix_list_append(&pmix_usock_globals.posted_recvs, &req->super);

     /* start listening */
-    if (PMIX_SUCCESS != pmix_start_listening(&myaddress, sockmode, sockuid, sockgid)) {
+    if (PMIX_SUCCESS != pmix_start_listening(myaddress, sockmode, sockuid, sockgid)) {
         PMIx_server_finalize();
         return PMIX_ERR_INIT;
     }
@@ -401,7 +390,7 @@ PMIX_EXPORT pmix_status_t PMIx_server_finalize(void)
     pmix_usock_finalize();

     /* cleanup the rendezvous file */
-    unlink(myaddress.sun_path);
+    unlink(myaddress);

     cleanup_server_state();
     pmix_output_verbose(2, pmix_globals.debug_output,
diff --git a/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_listener.c b/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_listener.c
index c5ba87a..9ecf007 100644
--- a/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_listener.c
+++ b/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_listener.c
@@ -72,13 +72,15 @@ static pthread_t engine;
 /*
  * start listening on our rendezvous file
  */
-pmix_status_t pmix_start_listening(struct sockaddr_un *address,
+pmix_status_t pmix_start_listening(const char *path,
                                    mode_t mode, uid_t sockuid, gid_t sockgid)
 {
     int flags;
     pmix_status_t rc;
     socklen_t addrlen;
     char *ptr;
+    struct sockaddr_un address;
+    bool use_symlink;

     /* create a listen socket for incoming connection attempts */
     pmix_server_globals.listen_socket = socket(PF_UNIX, SOCK_STREAM, 0);
@@ -87,21 +89,32 @@ pmix_status_t pmix_start_listening(struct sockaddr_un *address,
         return PMIX_ERROR;
     }

+    address.sun_family = PF_UNIX;
+    use_symlink =  (strlen(path) > sizeof(address.sun_path)-1);
+    if (use_symlink) {
+        snprintf(address.sun_path, sizeof(address.sun_path), "/tmp/pmixlink-%d", getpid());
+        symlink(path, address.sun_path);
+    } else {
+        snprintf(address.sun_path, sizeof(address.sun_path), path);
+    }
     addrlen = sizeof(struct sockaddr_un);
-    if (bind(pmix_server_globals.listen_socket, (struct sockaddr*)address, addrlen) < 0) {
+    if (bind(pmix_server_globals.listen_socket, (struct sockaddr*)&address, addrlen) < 0) {
         printf("%s:%d bind() failed\n", __FILE__, __LINE__);
         return PMIX_ERROR;
     }
     /* chown as required */
-    if (0 != chown(address->sun_path, sockuid, sockgid)) {
-        pmix_output(0, "CANNOT CHOWN socket %s: %s", address->sun_path, strerror (errno));
+    if (0 != chown(path, sockuid, sockgid)) {
+        pmix_output(0, "CANNOT CHOWN socket %s: %s", path, strerror (errno));
         goto sockerror;
     }
     /* set the mode as required */
-    if (0 != chmod(address->sun_path, mode)) {
-        pmix_output(0, "CANNOT CHMOD socket %s: %s", address->sun_path, strerror (errno));
+    if (0 != chmod(path, mode)) {
+        pmix_output(0, "CANNOT CHMOD socket %s: %s", path, strerror (errno));
         goto sockerror;
     }
+    if (use_symlink) {
+        unlink(address.sun_path);
+    }

     /* setup listen backlog to maximum allowed by kernel */
     if (listen(pmix_server_globals.listen_socket, SOMAXCONN) < 0) {
diff --git a/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_ops.h b/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_ops.h
index 1b94644..3c47929 100644
--- a/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_ops.h
+++ b/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_ops.h
@@ -147,7 +147,7 @@ typedef struct {
     } while (0)


-pmix_status_t pmix_start_listening(struct sockaddr_un *address,
+pmix_status_t pmix_start_listening(const char *address,
                                   mode_t mode, uid_t sockuid, gid_t sockgid);
 void pmix_stop_listening(void);

ggouaillardet on 11 Feb 2017

Wow - that's a pretty major change. For the record:

this problem was identified before we even released v2.0.0.
it has been discussed on the user and devel mailing lists since that time
the problem is due to something Apple introduced back in Yosemite that "bit" quite a few software packagers, not just OMPI
the simplest fix (setting TMPDIR in your environment to something reasonable) was provided a long time ago and comes straight from the Apple user forum
we introduced a more complete "out-of-the-box" fix in PMIx last fall that has simply not made its way into the OMPI v2.1 series due to RM decisions

Introducing yet another session directory that users have to cleanup whenever something fails seems like an annoying solution fixing an equally annoying problem. I'd suggest we:

update the FAQ to tell OSX users how to work around the problem; and
update OMPI v2.1 to a more recent version of PMIx that removes the need for the workaround

rhc54 on 12 Feb 2017

@rhc54 what this POC does when the rendezvous filename is too long is basically

ln -s <rendezvous filename> /tmp/pmixlink-<serverpid>
bind(/tmp/pmixlink-<server-pid>)
unlink(/tmp/pmixlink-<server-pid>)

so unless the program gets killed between the symlink creation and the unlink, no extra cleanup is required

ggouaillardet on 13 Feb 2017

I don't understand how that can work when the client (which can be multiple clients, invoked separately due to comm_spawn) have to find the rendezvous file.

Let's not create more problems - we have a workaround that follows the recommendation of the Apple forum, and a permanent solution. If we don't like the workaround, then let's use the permanent solution.

rhc54 on 13 Feb 2017

fwiw, the rendezvous file is always the same. the trick is we use a symlink but only for the bind syscall, so it fits the sun_path field of the sockaddr_un struct

ggouaillardet on 13 Feb 2017

It doesn't work that way - the client has to fill the sun_path field too, and the addr looks for that path to make the connection. So the symlink would have to be setup by the server, and must remain so long as any client is going to connect.

rhc54 on 13 Feb 2017

i got that too, and that is why anyone (server or client) is using (when needed) its own (getpid() based) symlink to the one and only (and sometimes very long) rendezvous filename.

bottom line, we only need the symlink to make bind() happy with the sun_path length restriction, and then the symlink is removed (and we do __not__ remove the rendezvous filename)

note the ``uri``` that is passed to the client(s) is still the rendezvous filename, and __not__ the symlink.

ggouaillardet on 13 Feb 2017

@ggouaillardet I honestly think you are missing the client side of the picture. The client _must_ put the rendezvous filename into a sockaddr_un path in order to connect to the server. Thus, the length of the filename given to the client is also restricted - and it must be only the symlink. Otherwise, the filename is too long for the sockaddr_un, it gets truncated, and therefore the connection cannot be completed.

Honestly, we really _did_ go thru this at length. 😄

rhc54 on 13 Feb 2017

@rhc54 well, we both missed something

i use the same trick (use a temporary and short symlink pointing to the one and only rendezvous file) in __both__ client and server
note the symlink is created __before__ the socket in bound (e.g. it points to nothing at that time).

OS X Yosemite is happy to bind() the socket to the symlink, and the created unix socket file is the rendezvous file.
On the other hand, Linux refuses to `bind() the socket and fails with errno 98 (Address already in use)

here is the previous patch plus some extra debug output that illustrates this

diff --git a/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c b/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
index 226b84c..0a0a12a 100644
--- a/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
+++ b/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
@@ -245,6 +245,7 @@ PMIX_EXPORT pmix_status_t PMIx_Init(pmix_proc_t *proc)
     pmix_nspace_t *nsptr;
     pmix_cb_t cb;
     int errhandler_ref;
+    bool use_symlink;

     if (NULL == proc) {
         return PMIX_ERR_BAD_PARAM;
@@ -337,7 +338,14 @@ PMIX_EXPORT pmix_status_t PMIx_Init(pmix_proc_t *proc)
     /* setup the path to the daemon rendezvous point */
     memset(&address, 0, sizeof(struct sockaddr_un));
     address.sun_family = AF_UNIX;
-    snprintf(address.sun_path, sizeof(address.sun_path)-1, "%s", uri[2]);
+    use_symlink = (strlen(uri[2]) > sizeof(address.sun_path)-1);
+    if (use_symlink) {
+        snprintf(address.sun_path, sizeof(address.sun_path), "/tmp/pmixlink-%d", getpid());
+        symlink(uri[2], address.sun_path);
+        printf("client: using symlink %s -> %s\n", address.sun_path, uri[2]);
+    } else {
+        snprintf(address.sun_path, sizeof(address.sun_path)-1, "%s", uri[2]);
+    }
     /* if the rendezvous file doesn't exist, that's an error */
     if (0 != access(uri[2], R_OK)) {
         pmix_argv_free(uri);
@@ -397,6 +405,9 @@ PMIX_EXPORT pmix_status_t PMIx_Init(pmix_proc_t *proc)
     PMIX_WAIT_FOR_COMPLETION(cb.active);
     rc = cb.status;
     PMIX_DESTRUCT(&cb);
+    if (use_symlink) {
+        unlink(address.sun_path);
+    }

     if (PMIX_SUCCESS == rc) {
         pmix_globals.init_cntr++;
diff --git a/opal/mca/pmix/pmix112/pmix/src/server/pmix_server.c b/opal/mca/pmix/pmix112/pmix/src/server/pmix_server.c
index a81df69..7b7b90d 100644
--- a/opal/mca/pmix/pmix112/pmix/src/server/pmix_server.c
+++ b/opal/mca/pmix/pmix112/pmix/src/server/pmix_server.c
@@ -63,7 +63,7 @@ pmix_server_globals_t pmix_server_globals = {{{0}}};

 // local variables
 static char *myuri = NULL;
-static struct sockaddr_un myaddress;
+static char * myaddress;
 static char *security_mode = NULL;
 static char *mytmpdir = NULL;

@@ -218,22 +218,11 @@ static pmix_status_t initialize_server_base(pmix_server_module_t *module)
     }

     /* now set the address - we use the pid here to reduce collisions */
-    memset(&myaddress, 0, sizeof(struct sockaddr_un));
-    myaddress.sun_family = AF_UNIX;
-    if (0 > asprintf(&pmix_pid, "pmix-%d", pid)) {
+    if (0 > asprintf(&myaddress, "%s/pmix-%d", tdir, pid)) {
         return PMIX_ERR_NOMEM;
     }
-    // If the above set temporary directory name plus the pmix-PID string
-    // plus the '/' separator are too long, just fail, so the caller
-    // may provide the user with a proper help... *Cough*, *Cough* OSX...
-    if ((strlen(tdir) + strlen(pmix_pid) + 1) > sizeof(myaddress.sun_path)-1) {
-        free(pmix_pid);
-        return PMIX_ERR_INVALID_LENGTH;
-    }
-    snprintf(myaddress.sun_path, sizeof(myaddress.sun_path)-1, "%s/%s", tdir, pmix_pid);
-    free(pmix_pid);
     if (0 > asprintf(&myuri, "%s:%lu:%s", pmix_globals.myid.nspace,
-                     (unsigned long)pmix_globals.myid.rank, myaddress.sun_path)) {
+                     (unsigned long)pmix_globals.myid.rank, myaddress)) {
         return PMIX_ERR_NOMEM;
     }

@@ -301,7 +290,7 @@ PMIX_EXPORT pmix_status_t PMIx_server_init(pmix_server_module_t *module,
     pmix_list_append(&pmix_usock_globals.posted_recvs, &req->super);

     /* start listening */
-    if (PMIX_SUCCESS != pmix_start_listening(&myaddress, sockmode, sockuid, sockgid)) {
+    if (PMIX_SUCCESS != pmix_start_listening(myaddress, sockmode, sockuid, sockgid)) {
         PMIx_server_finalize();
         return PMIX_ERR_INIT;
     }
@@ -401,7 +390,7 @@ PMIX_EXPORT pmix_status_t PMIx_server_finalize(void)
     pmix_usock_finalize();

     /* cleanup the rendezvous file */
-    unlink(myaddress.sun_path);
+    unlink(myaddress);

     cleanup_server_state();
     pmix_output_verbose(2, pmix_globals.debug_output,
diff --git a/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_listener.c b/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_listener.c
index c5ba87a..f1ec2b2 100644
--- a/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_listener.c
+++ b/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_listener.c
@@ -72,13 +72,15 @@ static pthread_t engine;
 /*
  * start listening on our rendezvous file
  */
-pmix_status_t pmix_start_listening(struct sockaddr_un *address,
+pmix_status_t pmix_start_listening(const char *path,
                                    mode_t mode, uid_t sockuid, gid_t sockgid)
 {
     int flags;
     pmix_status_t rc;
     socklen_t addrlen;
     char *ptr;
+    struct sockaddr_un address;
+    bool use_symlink;

     /* create a listen socket for incoming connection attempts */
     pmix_server_globals.listen_socket = socket(PF_UNIX, SOCK_STREAM, 0);
@@ -87,21 +89,45 @@ pmix_status_t pmix_start_listening(struct sockaddr_un *address,
         return PMIX_ERROR;
     }

+    address.sun_family = PF_UNIX;
+    use_symlink =  (strlen(path) > sizeof(address.sun_path)-1);
+    if (use_symlink) {
+        char * cmd;
+        snprintf(address.sun_path, sizeof(address.sun_path), "/tmp/pmixlink-%d", getpid());
+        symlink(path, address.sun_path);
+        printf("server: using symlink %s -> %s\n", address.sun_path, path);
+        asprintf(&cmd, "ls -l %s %s", address.sun_path, path);
+        system(cmd);
+        free(cmd);
+    } else {
+        snprintf(address.sun_path, sizeof(address.sun_path), path);
+    }
     addrlen = sizeof(struct sockaddr_un);
-    if (bind(pmix_server_globals.listen_socket, (struct sockaddr*)address, addrlen) < 0) {
-        printf("%s:%d bind() failed\n", __FILE__, __LINE__);
+    if (bind(pmix_server_globals.listen_socket, (struct sockaddr*)&address, addrlen) < 0) {
+        char * cmd;
+        printf("%s:%d bind() failed with error %s (%d)\n", __FILE__, __LINE__, strerror(errno), errno);
+        asprintf(&cmd, "ls -l %s %s", address.sun_path, path);
+        system(cmd);
+        free(cmd);
         return PMIX_ERROR;
     }
     /* chown as required */
-    if (0 != chown(address->sun_path, sockuid, sockgid)) {
-        pmix_output(0, "CANNOT CHOWN socket %s: %s", address->sun_path, strerror (errno));
+    if (0 != chown(path, sockuid, sockgid)) {
+        pmix_output(0, "CANNOT CHOWN socket %s: %s", path, strerror (errno));
         goto sockerror;
     }
     /* set the mode as required */
-    if (0 != chmod(address->sun_path, mode)) {
-        pmix_output(0, "CANNOT CHMOD socket %s: %s", address->sun_path, strerror (errno));
+    if (0 != chmod(path, mode)) {
+        pmix_output(0, "CANNOT CHMOD socket %s: %s", path, strerror (errno));
         goto sockerror;
     }
+    if (use_symlink) {
+        char * cmd;
+        asprintf(&cmd, "ls -l %s %s", address.sun_path, path);
+        system(cmd);
+        free(cmd);
+        unlink(address.sun_path);
+    }

     /* setup listen backlog to maximum allowed by kernel */
     if (listen(pmix_server_globals.listen_socket, SOMAXCONN) < 0) {
diff --git a/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_ops.h b/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_ops.h
index 1b94644..3c47929 100644
--- a/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_ops.h
+++ b/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_ops.h
@@ -147,7 +147,7 @@ typedef struct {
     } while (0)


-pmix_status_t pmix_start_listening(struct sockaddr_un *address,
+pmix_status_t pmix_start_listening(const char *address,
                           mode_t mode, uid_t sockuid, gid_t sockgid);
 void pmix_stop_listening(void);

i also tested loop_spawn from the ibm test suite, and it works fine as well on OS X Yosemite

for example

TMPDIR=$TMPDIR/aaaaaaaaaaaaaa/bbbbbbbbbbbbbbbb/ccccccccccccccc/ddddddddddddd/eeeeeeeeeee mpirun -np 2 --tag-output ./hello_c
server: using symlink /tmp/pmixlink-95777 -> /var/folders/my/nrglry8n6ddbz2vdtxzxz3rw0000gn/T//aaaaaaaaaaaaaa/bbbbbbbbbbbbbbbb/ccccccccccccccc/ddddddddddddd/eeeeeeeeeee/openmpi-sessions-501@dhcp-40-112_0/50766/pmix-95777
ls: /var/folders/my/nrglry8n6ddbz2vdtxzxz3rw0000gn/T//aaaaaaaaaaaaaa/bbbbbbbbbbbbbbbb/ccccccccccccccc/ddddddddddddd/eeeeeeeeeee/openmpi-sessions-501@dhcp-40-112_0/50766/pmix-95777: No such file or directory
lrwxr-xr-x  1 gilles  wheel  175 Feb 13 13:49 /tmp/pmixlink-95777 -> /var/folders/my/nrglry8n6ddbz2vdtxzxz3rw0000gn/T//aaaaaaaaaaaaaa/bbbbbbbbbbbbbbbb/ccccccccccccccc/ddddddddddddd/eeeeeeeeeee/openmpi-sessions-501@dhcp-40-112_0/50766/pmix-95777
lrwxr-xr-x  1 gilles  wheel  175 Feb 13 13:49 /tmp/pmixlink-95777 -> /var/folders/my/nrglry8n6ddbz2vdtxzxz3rw0000gn/T//aaaaaaaaaaaaaa/bbbbbbbbbbbbbbbb/ccccccccccccccc/ddddddddddddd/eeeeeeeeeee/openmpi-sessions-501@dhcp-40-112_0/50766/pmix-95777
srw-rw-r--  1 gilles  staff    0 Feb 13 13:49 /var/folders/my/nrglry8n6ddbz2vdtxzxz3rw0000gn/T//aaaaaaaaaaaaaa/bbbbbbbbbbbbbbbb/ccccccccccccccc/ddddddddddddd/eeeeeeeeeee/openmpi-sessions-501@dhcp-40-112_0/50766/pmix-95777
[1,0]<stdout>:client: using symlink /tmp/pmixlink-95780 -> /var/folders/my/nrglry8n6ddbz2vdtxzxz3rw0000gn/T//aaaaaaaaaaaaaa/bbbbbbbbbbbbbbbb/ccccccccccccccc/ddddddddddddd/eeeeeeeeeee/openmpi-sessions-501@dhcp-40-112_0/50766/pmix-95777
[1,1]<stdout>:client: using symlink /tmp/pmixlink-95781 -> /var/folders/my/nrglry8n6ddbz2vdtxzxz3rw0000gn/T//aaaaaaaaaaaaaa/bbbbbbbbbbbbbbbb/ccccccccccccccc/ddddddddddddd/eeeeeeeeeee/openmpi-sessions-501@dhcp-40-112_0/50766/pmix-95777
[1,1]<stdout>:Hello, world, I am 1 of 2, (Open MPI v2.0.3a1, package: Open MPI gilles@KA15-002 Distribution, ident: 2.0.3a1, repo rev: v2.0.2-38-g646486b, Unreleased developer copy, 139)
[1,0]<stdout>:Hello, world, I am 0 of 2, (Open MPI v2.0.3a1, package: Open MPI gilles@KA15-002 Distribution, ident: 2.0.3a1, repo rev: v2.0.2-38-g646486b, Unreleased developer copy, 139)

on the other hand, the server fails to bind() on Linux

ls: cannot access /tmp/aaaaaaaaaaaaaaaa/bbbbbbbbbbbbbbbbbbb/ccccccccccccccccccccccc/ddddddddddddddddddddddddd/eeeeeeeeeeeeeeeeeeeeeeee/ffffffffffffffffffffff/gggggggggggggggggggggggg/hhhhhhhhhhhhhh/openmpi-sessions-1000@c7_0/27677/pmix-8341: No such file or directory
lrwxrwxrwx 1 gilles gilles 222 Feb 13 13:50 /tmp/pmixlink-8341 -> /tmp/aaaaaaaaaaaaaaaa/bbbbbbbbbbbbbbbbbbb/ccccccccccccccccccccccc/ddddddddddddddddddddddddd/eeeeeeeeeeeeeeeeeeeeeeee/ffffffffffffffffffffff/gggggggggggggggggggggggg/hhhhhhhhhhhhhh/openmpi-sessions-1000@c7_0/27677/pmix-8341
ls: cannot access /tmp/aaaaaaaaaaaaaaaa/bbbbbbbbbbbbbbbbbbb/ccccccccccccccccccccccc/ddddddddddddddddddddddddd/eeeeeeeeeeeeeeeeeeeeeeee/ffffffffffffffffffffff/gggggggggggggggggggggggg/hhhhhhhhhhhhhh/openmpi-sessions-1000@c7_0/27677/pmix-8341: No such file or directory
lrwxrwxrwx 1 gilles gilles 222 Feb 13 13:50 /tmp/pmixlink-8341 -> /tmp/aaaaaaaaaaaaaaaa/bbbbbbbbbbbbbbbbbbb/ccccccccccccccccccccccc/ddddddddddddddddddddddddd/eeeeeeeeeeeeeeeeeeeeeeee/ffffffffffffffffffffff/gggggggggggggggggggggggg/hhhhhhhhhhhhhh/openmpi-sessions-1000@c7_0/27677/pmix-8341
[c7.kmc.kobe.rist.or.jp:08341] [[27677,0],0] ORTE_ERROR_LOG: Error in file ../../../src/ompi-v2.0.x/orte/orted/pmix/pmix_server.c at line 262
[c7.kmc.kobe.rist.or.jp:08341] [[27677,0],0] ORTE_ERROR_LOG: Error in file ../../../../../../src/ompi-v2.0.x/orte/mca/ess/hnp/ess_hnp_module.c at line 666
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  pmix server init failed
  --> Returned value Error (-1) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
server: using symlink /tmp/pmixlink-8341 -> /tmp/aaaaaaaaaaaaaaaa/bbbbbbbbbbbbbbbbbbb/ccccccccccccccccccccccc/ddddddddddddddddddddddddd/eeeeeeeeeeeeeeeeeeeeeeee/ffffffffffffffffffffff/gggggggggggggggggggggggg/hhhhhhhhhhhhhh/openmpi-sessions-1000@c7_0/27677/pmix-8341
../../../../../../../src/ompi-v2.0.x/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_listener.c:108 bind() failed with error Address already in use (98)

bottom line, this works only on OS X (i tested only Yosemite so far)
the initial request is we hope something that works out of the box everywhere, and so far, it does work out of the box everywhere __but__ OS X, so i believe this approach (protected with #ifdef __APPLE__) addresses the initial request.
i also understand you might not want to do that, so i will not push for this further.

feel free to discuss this with @hppritcha and @jsquyres and all during the weekly call.

__[EDIT]__
i made an other POC that also works on Linux.
on the server, bind() to /tmp, make a (hard) link() to $TMPDIR and then unlink() the socket in /tmp
client can keep using symlinks or could also use hard links
note this works only if /tmp and $TMPDIR are on the __same__ filesystem

ggouaillardet on 13 Feb 2017

Yeah, I just don't think this is a good solution. For one thing, you are assuming that we can just use /tmp for the symlink. We know from experience that this isn't a valid assumption - we have plenty of places where that directory isn't available to people. So the first thing we'd have to change is to provide a mechanism by which the user can specify the shortened tmpdir for the symlink.

And if we do that - then what's the point of all the extra logic? We'd already have a shortened directory we can use, so we can simply use it.

Personally, I think the correct solution is to simply upgrade to the permanent solution and avoid the whole mess. I believe @dsolt is working towards that anyway for other reasons, so this may become a moot point.

rhc54 on 13 Feb 2017

@rhc54 and I talked about this on the phone.

It feels like putting a workaround for /tmp is just that -- it's a band-aid for the real problem.
The real solution is to not use this problematic mechanism (i.e., unix-domain sockets) when we can avoid it -- use loopback TCP sockets, instead. master already has this fix (which I think is down in PMIx).

So let's do the Real fix rather than a workaround, if possible:

For 3.x (i.e., master), we're already set.
For 2.1.x, it seems like we can roll this into the PMIx that goes into that branch.
For 2.0.x, @rhc54 is going to take a whack at porting the loopback TCP sockets solution to that branch and see what it looks like. Hopefully, it's easy/not risky, and that can be the final solution.

jsquyres on 13 Feb 2017

@jsquyres note the default is to use usock instead of tcp.
and unlike what i wrote earlier, PMIx does fall back to tcp if the rendezvous file name is too long.

ggouaillardet on 14 Feb 2017

The default should/will be to use tcp - it was only temporarily set back to usock until we fixed the performance issue (which you did awhile ago).

rhc54 on 14 Feb 2017

I updated the OS-x part of the FAQ to describe workaround for 2.0.x series.

hppritcha on 14 Feb 2017

Good morning all,

This is my first post on github.com, please excuse me if this is not at the right place!

I'm trying to install OpenMPI on a Mac laptop under Sierra to realise some parallel computations, but am experiencing some problems when trying the command mpirun.
I've followed the same installation process as on my Ubuntu desktop (which is working just fine) and specified the different paths in my .bashrc file. The error (on the Mac) is the following:

ORTE_ERROR_LOG: Bad parameter in file orted/pmix/pmix_server.c at line 262
ORTE_ERROR_LOG: Bad parameter in file ess_hnp_module.c at line 666

I ended up on this discussion while looking for a solution to this issue and am hoping to find an answer here :) Is there yet a procedure to follow to get round this error for Mac users? Also, where is this issue coming from?

Thank you very much for your support if you can help me!
Regards,
Willow94

Willow94 on 9 Mar 2017

@Willow94 Please see https://www.open-mpi.org/faq/?category=osx#startup-errors-with-open-mpi-2.0.x.

jsquyres on 9 Mar 2017

@jsquyres Thank you for your answer, I hadn't seen this comment on openmpi.org before.

I'm trying to set the TMPDIR environment variable to /tmp, as adviced, by entering the following in my .bashrc file: export TMPDIR="$TMPDIR:/tmp", without any success. Am I doing anything wrong?

PS: sorry if that's basics, I'm completely new to this kind of things and trying to learn more about it so I can do it by myself next time

Thanks again

Willow94 on 9 Mar 2017

@Willow94 all you need is

export TMPDIR=/tmp

then you can

echo $TMPDIR

to confirm the correct value.
(if you only set your .bashrc, you will likely have to close your terminal and open a new one)

ggouaillardet on 9 Mar 2017

👍3

@ggouaillardet Thanks a lot, works perfectly now!

Willow94 on 9 Mar 2017

I'm going to close this as we've done all we intend to do. The switch to TCP solved the problem long-term, and the workarounds seem adequate for OMPI releases prior to v3.0

rhc54 on 30 May 2017

Ompi: OS-X: need a workaround for long session dir name

Most helpful comment

All 29 comments

Related issues