Ompi: There are not enough slots available in the system to satisfy the 1 slots

[cshanbo]

Hi all,
I tested a multi-node task with openmpi in this project.

As In the title, when I comment --oversubscribe in this line, there will be an exception, There are not enough slots available ... even when I only launch one task on each node.

I know it may raise this error when multiple tasks are launched, but is this normal of raising the exception when there's only one task?

Thank you

[rhc54]

It always helps us to properly respond if you tell us which version of OMPI you are using, whether this is a managed environment or not, and if not, then what was in your hostfile?

[cshanbo]

Thank you for your reply.

I'm using the latest version of openmpi on this page.
I launch the processes in a python script here. Basically, it's a bash command like
mpirun -np 2 -H host1,host2 ...

[ggouaillardet]

@cshanbo is mpirun invoked on one of the nodes listed in the -H option ?
what if you manually ?
mpirun -np 2 -H host1,host2 hostname
if it fails, what if you ?
mpirun -np 2 -H host1:1,host2:1 hostname
it do not think that should be necessary, but since there were quite some changes in this area, i might be wrong

[rhc54]

Adding the colon qualifiers isn't necessary. We automatically assign 1 slot for every time you list the host. My best guess is that there is something in your Python wrapper that is messing up the cmd line. Try printing out the actual cmd line you are executing, and then try invoking it manually to see what happens.

[cshanbo]

I tried to invoke the process from command line without python wrapper, still got the same error.
The command is simply like:

mpirun -mca mpi_warn_on_fork 0 -mca orte_base_help_aggregate 0 -np 2 -H nmyjs_104_22,nmyjs_104_37 -x PATH -x LD_LIBRARY_PATH -x PLATOON_TEST_WORKERS_NUM -x THEANO_FLAGS=floatX=float32,device=cpu /search/odin/chengshanbo/anaconda2/bin/python -u -m platoon.channel.controller test_multi_node PLATOON_LOGS/test_multi_node/2017-03-14_16-43-35 --multi

The THEANO_FLAGS, etc, parameters are needed by my codes.

Error

There are not enough slots available in the system to satisfy the 1 slots
that were requested by the application:
  /search/odin/chengshanbo/anaconda2/bin/python

Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 1 slots
that were requested by the application:
  /search/odin/chengshanbo/anaconda2/bin/python

Either request fewer slots for your application, or make more slots available
for use.

[rhc54]

Let's break the problem down into some smaller steps. First, let's get mpirun working correctly. Try running mpirun -np 2 -H <host1>,<host2> hostname

[zhangciba]

Hi every body.Is this problem solved?I meet the same problem.

[cshanbo]

Hi @zhangciba I instead using the --oversubscribe flag to "solve" the problem

[zhangciba]

I just submit the open-mpi jobs through Torque PBS system and I set the '#PBS -l nodes=32:ppn=16' parameter in the job script. And I also set the -np and -hostfile at the mpirun command. So I meet the problem. I remove the -np and -hostfile parameter in the mpirun command ,then the problem is solved.

[ggouaillardet]

You need to make sure Open MPI was built with tm support.
If you configure --with-tm, it will abort if there is no such support.

What if you simply

mpirun hostname

In your script ?
Do you see 512 tasks on 16 nodes ?
If you only see 16 tasks on the (first) node, that would strongly hint tm support was not built.

[zhangciba]

@ggouaillardet Thank you for your detailed answer.

I have checked my Open MPI and find it was configured with the 'tm' support.

Below is my job script named hostname.pbs

#!/bin/bash
#PBS -q medium
#PBS -l nodes=2:ppn=16
#PBS -N hostname

mpirun -np 32 -hostfile /home/zhangchi/study/jobs/hostfile hostname

the content of /home/zhangchi/study/jobs/hostfile

node234

why I put node233 and node234 in hostfile ?
At that time the queue was not busy. And the free node list is like this

node234
node235
...
...

So I think my job will be scheduled into these two nodes.

and the property of the queue medium was like this

#
# Create queues and set their attributes.
#
#
# Create and define queue medium
#
create queue medium
set queue medium queue_type = Execution
set queue medium max_queuable = 20
set queue medium max_user_queuable = 3
set queue medium max_running = 20
set queue medium acl_user_enable = True
set queue medium acl_users = ionization
set queue medium acl_users += zhangchi
set queue medium resources_max.ncpus = 16
set queue medium resources_max.nodes = 4
set queue medium resources_default.nodes = 1
set queue medium resources_default.walltime = 240:00:00
set queue medium max_user_run = 2
set queue medium enabled = True
set queue medium started = True

I submited this job using hostname.pbs and get the job number 5402, then the job completed soon.
And the job was actually running on node233 and node234
I checked the output file named hostname.o5402. It was like this.

[zhangchi@grid jobs]$ cat hostname.o5402
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 32 slots
that were requested by the application:
  hostname

Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------

Then I submit the same script the second time, I got different messages.
The job was running on node181 and node234.

[zhangchi@grid jobs]$ cat hostname.o5409
--------------------------------------------------------------------------
A hostfile was provided that contains at least one node not
present in the allocation:

  hostfile:  /home/zhangchi/study/jobs/hostfile
  node:      node233

If you are operating in a resource-managed environment, then only
nodes that are in the allocation can be used in the hostfile. You
may find relative node syntax to be a useful alternative to
specifying absolute node names see the orte_hosts man page for
further information.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
ORTE was unable to reliably start one or more daemons.
This usually is caused by:

* not finding the required libraries and/or binaries on
  one or more nodes. Please check your PATH and LD_LIBRARY_PATH
  settings, or configure OMPI with --enable-orterun-prefix-by-default

* lack of authority to execute on one or more specified nodes.
  Please verify your allocation and authorities.

* the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base).
  Please check with your sys admin to determine the correct location to use.

*  compilation of the orted with dynamic libraries when static are required
  (e.g., on Cray). Please check your configure cmd line and consider using
  one of the contrib/platform definitions for your system type.

* an inability to create a connection back to mpirun due to a
  lack of common network interfaces and/or no route found between
  them. Please check network connectivity (including firewalls
  and network routing requirements).
--------------------------------------------------------------------------

Because that was my first time using Torque PBS to submit jobs.So I did not know how to write the job script. After search the same question on the web. I corrected the script like this(just remove the -np and -hostfile option).

[zhangchi@grid jobs]$ cat hostname.pbs
#!/bin/bash
#PBS -q medium
#PBS -l nodes=2:ppn=16
#PBS -N hostname

mpirun hostname

[zhangchi@grid jobs]$ qsub hostname.pbs
5405.grid
[zhangchi@grid jobs]$ cat hostname.o5405
node236
node236
node236
node236
node236
node236
node236
node236
node236
node236
node236
node236
node236
node236
node236
node236
node234
node234
node234
node234
node234
node234
node234
node234
node234
node234
node234
node234
node234
node234
node234
node234

The job could work normally and I think that not only the real resource lack but also the wrong way of using job script will lead to the problem 'not enough slots available'.

[ggouaillardet]

Looks like you are all set !

Open MPI was correctly built with tm support.
A consequence is you can only run on nodes that were allocated to your job.
Generally speaking, you should not need a hostile, but if you need one, it should be built inside your submission script by using the content of the $PBS_NODEFILE file.

[mmokrejs]

I think openmpi should include in the error message path to the $PBS_NODEFILE and its contents too. My file contains just one hostname and openmpi is assumed to figure out the machine has 24 cpus. But, somehow it does not.

Forcing mpirun -H r21u24n600.foo.bar:24 or mpirun --oversubscribe works. I use openmpi-3.0.0 with torque enabled.

$ qstat -f 7637057.isrv5
Job Id: 7637057.isrv5
    Job_Name = STDIN
    Job_Owner = [email protected]
    resources_used.cpupercent = 3
    resources_used.cput = 00:00:06
    resources_used.mem = 36520kb
    resources_used.ncpus = 24
    resources_used.vmem = 558448kb
    resources_used.walltime = 00:24:28
    job_state = R
    queue = qprod
    server = isrv5
    Account_Name = xx
    Checkpoint = u
    ctime = Tue Mar 20 09:46:08 2018
    Error_Path = /dev/pts/0
    exec_host = r21u24n600/0*24
    exec_vnode = (r21u24n600:ncpus=24)
    Hold_Types = n
    interactive = True
    Join_Path = n
    Keep_Files = n
    Mail_Points = a
    mtime = Tue Mar 20 09:49:06 2018
    Output_Path = /dev/pts/0
    Priority = 0
    qtime = Tue Mar 20 09:46:08 2018
    Rerunable = False
    Resource_List.fairshare = 997450
    Resource_List.ncpus = 24
    Resource_List.nice = 15
    Resource_List.nodect = 1
    Resource_List.place = free
    Resource_List.select = 1:ncpus=24
    Resource_List.walltime = 48:00:00
...
    Submit_arguments = -I -q qprod -A xx -l select=1:ncpus=24,
    walltime=48:00:00

Maybe the lshw support is broken?

[ggouaillardet]

@mmokrejs can you please post your PBS script ?

[mmokrejs]

There was none, it was interactive session, hence `qsub -I -q qprod -A xx -l select=1:ncpus=24,walltime=48:00:00

But OK, initially I hit this through PBS script jobs, an example is:

#PBS -S /bin/bash
#PBS -q qprod
#PBS -N test
#PBS -l select=1:ncpus=24,walltime=48:00:00
#PBS -A xx
#PBS -m ea

cd $PBS_O_WORKDIR || exit 255

export NCPUS=24
export OMP_NUM_THREADS=24

export threads=24
export mpithreads=24

I also unset the mpithreads and other variables too with no luck. And it used to work fine with openmpi-2.1.1.

The rror mesage is:

There are not enough slots available in the system to satisfy the 24 slots
that were requested by the application:
  ABYSS-P

Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------

But I get same for mpirun -np 2 ls inside the interactive session on a machine with 24 physical cores (amd64).

[ggouaillardet]

What is your mpirun command line ?

This behavior typically occurs when tm support was not built.

Can you

mpirun --mca plm_base_verbose 10 --mca ess_base_verbose 10 hostname

And post the output ?

Do you observe the same behavior both in interactive and batch mode ?

[ggouaillardet]

BTW, are you sure ncpus is appropriate here and ppn should not be preferred ?

[mmokrejs]

We have PBSpro 13.1.1.162303 here. Yes I observe this behavior with both approaches. Other users are still using older openmpi versions. I got new interactive job session for the below task (on another host):

$ mpirun --version
mpirun (Open MPI) 3.0.0

Report bugs to http://www.open-mpi.org/community/help/

$ mpirun --mca plm_base_verbose 10 --mca ess_base_verbose 10 hostname
[r4i2n10:22979] mca: base: components_register: registering framework ess components
[r4i2n10:22979] mca: base: components_register: found loaded component env
[r4i2n10:22979] mca: base: components_register: component env has no register or open function
[r4i2n10:22979] mca: base: components_register: found loaded component hnp
[r4i2n10:22979] mca: base: components_register: component hnp has no register or open function
[r4i2n10:22979] mca: base: components_register: found loaded component pmi
[r4i2n10:22979] mca: base: components_register: component pmi has no register or open function
[r4i2n10:22979] mca: base: components_register: found loaded component singleton
[r4i2n10:22979] mca: base: components_register: component singleton register function successful
[r4i2n10:22979] mca: base: components_register: found loaded component tool
[r4i2n10:22979] mca: base: components_register: component tool register function successful
[r4i2n10:22979] mca: base: components_register: found loaded component slurm
[r4i2n10:22979] mca: base: components_register: component slurm has no register or open function
[r4i2n10:22979] mca: base: components_register: found loaded component tm
[r4i2n10:22979] mca: base: components_register: component tm has no register or open function
[r4i2n10:22979] mca: base: components_open: opening ess components
[r4i2n10:22979] mca: base: components_open: found loaded component env
[r4i2n10:22979] mca: base: components_open: component env open function successful
[r4i2n10:22979] mca: base: components_open: found loaded component hnp
[r4i2n10:22979] mca: base: components_open: component hnp open function successful
[r4i2n10:22979] mca: base: components_open: found loaded component pmi
[r4i2n10:22979] mca: base: components_open: component pmi open function successful
[r4i2n10:22979] mca: base: components_open: found loaded component singleton
[r4i2n10:22979] mca: base: components_open: component singleton open function successful
[r4i2n10:22979] mca: base: components_open: found loaded component tool
[r4i2n10:22979] mca: base: components_open: component tool open function successful
[r4i2n10:22979] mca: base: components_open: found loaded component slurm
[r4i2n10:22979] mca: base: components_open: component slurm open function successful
[r4i2n10:22979] mca: base: components_open: found loaded component tm
[r4i2n10:22979] mca: base: components_open: component tm open function successful
[r4i2n10:22979] mca:base:select: Auto-selecting ess components
[r4i2n10:22979] mca:base:select:(  ess) Querying component [env]
[r4i2n10:22979] mca:base:select:(  ess) Querying component [hnp]
[r4i2n10:22979] mca:base:select:(  ess) Query of component [hnp] set priority to 100
[r4i2n10:22979] mca:base:select:(  ess) Querying component [pmi]
[r4i2n10:22979] mca:base:select:(  ess) Querying component [singleton]
[r4i2n10:22979] mca:base:select:(  ess) Querying component [tool]
[r4i2n10:22979] mca:base:select:(  ess) Querying component [slurm]
[r4i2n10:22979] mca:base:select:(  ess) Querying component [tm]
[r4i2n10:22979] mca:base:select:(  ess) Selected component [hnp]
[r4i2n10:22979] mca: base: close: component env closed
[r4i2n10:22979] mca: base: close: unloading component env
[r4i2n10:22979] mca: base: close: component pmi closed
[r4i2n10:22979] mca: base: close: unloading component pmi
[r4i2n10:22979] mca: base: close: component singleton closed
[r4i2n10:22979] mca: base: close: unloading component singleton
[r4i2n10:22979] mca: base: close: component tool closed
[r4i2n10:22979] mca: base: close: unloading component tool
[r4i2n10:22979] mca: base: close: component slurm closed
[r4i2n10:22979] mca: base: close: unloading component slurm
[r4i2n10:22979] mca: base: close: component tm closed
[r4i2n10:22979] mca: base: close: unloading component tm
[r4i2n10:22979] mca: base: components_register: registering framework plm components
[r4i2n10:22979] mca: base: components_register: found loaded component isolated
[r4i2n10:22979] mca: base: components_register: component isolated has no register or open function
[r4i2n10:22979] mca: base: components_register: found loaded component rsh
[r4i2n10:22979] mca: base: components_register: component rsh register function successful
[r4i2n10:22979] mca: base: components_register: found loaded component slurm
[r4i2n10:22979] mca: base: components_register: component slurm register function successful
[r4i2n10:22979] mca: base: components_register: found loaded component tm
[r4i2n10:22979] mca: base: components_register: component tm register function successful
[r4i2n10:22979] mca: base: components_open: opening plm components
[r4i2n10:22979] mca: base: components_open: found loaded component isolated
[r4i2n10:22979] mca: base: components_open: component isolated open function successful
[r4i2n10:22979] mca: base: components_open: found loaded component rsh
[r4i2n10:22979] mca: base: components_open: component rsh open function successful
[r4i2n10:22979] mca: base: components_open: found loaded component slurm
[r4i2n10:22979] mca: base: components_open: component slurm open function successful
[r4i2n10:22979] mca: base: components_open: found loaded component tm
[r4i2n10:22979] mca: base: components_open: component tm open function successful
[r4i2n10:22979] mca:base:select: Auto-selecting plm components
[r4i2n10:22979] mca:base:select:(  plm) Querying component [isolated]
[r4i2n10:22979] mca:base:select:(  plm) Query of component [isolated] set priority to 0
[r4i2n10:22979] mca:base:select:(  plm) Querying component [rsh]
[r4i2n10:22979] mca:base:select:(  plm) Query of component [rsh] set priority to 10
[r4i2n10:22979] mca:base:select:(  plm) Querying component [slurm]
[r4i2n10:22979] mca:base:select:(  plm) Querying component [tm]
[r4i2n10:22979] mca:base:select:(  plm) Query of component [tm] set priority to 75
[r4i2n10:22979] mca:base:select:(  plm) Selected component [tm]
[r4i2n10:22979] mca: base: close: component isolated closed
[r4i2n10:22979] mca: base: close: unloading component isolated
[r4i2n10:22979] mca: base: close: component rsh closed
[r4i2n10:22979] mca: base: close: unloading component rsh
[r4i2n10:22979] mca: base: close: component slurm closed
[r4i2n10:22979] mca: base: close: unloading component slurm
[r4i2n10:22979] [[46329,0],0] complete_setup on job [46329,1]
r4i2n10
[r4i2n10:22979] mca: base: close: component tm closed
[r4i2n10:22979] mca: base: close: unloading component tm
[r4i2n10:22979] mca: base: close: component hnp closed
[r4i2n10:22979] mca: base: close: unloading component hnp
$

$ ml

Currently Loaded Modules:
  1) GCCcore/6.3.0                       5) hwloc/1.11.7-GCC-6.3.0-2.27      9) gompi/2018a                                  13) bzip2/1.0.6     (H)  17) Tk/8.6.6                           21) sparsehash/2.0.3-foss-2018a
  2) binutils/2.27-GCCcore-6.3.0         6) zlib/1.2.11                     10) FFTW/3.3.7-gompi-2018a                       14) ncurses/6.0     (H)  18) Python/2.7.14-GCC-6.3.0-2.27-base  22) SQLite/3.21.0
  3) GCC/6.3.0-2.27                      7) OpenMPI/3.0.0-GCC-6.3.0-2.27    11) ScaLAPACK/2.0.2-gompi-2018a-OpenBLAS-0.2.20  15) libreadline/7.0      19) Py/2.7                             23) ABySS/2.0.2-foss-2018a
  4) numactl/2.0.11-GCC-6.3.0-2.27 (H)   8) OpenBLAS/0.2.20-GCC-6.3.0-2.27  12) foss/2018a                                   16) Tcl/8.6.6            20) Boost/1.66.0-foss-2018a-Py-2.7

  Where:
   H:  Hidden Module

$ ml OpenMPI/2.1.1-GCC-6.3.0-2.27
$ ompi_info | grep ras
                 MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component v2.1.1)
                 MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v2.1.1)
                 MCA ras: loadleveler (MCA v2.1.0, API v2.0.0, Component v2.1.1)
                 MCA ras: tm (MCA v2.1.0, API v2.0.0, Component v2.1.1)

$ ml OpenMPI/3.0.0-GCC-6.3.0-2.27
$ ompi_info | grep ras
                 MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component v3.0.0)
                 MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v3.0.0)
                 MCA ras: tm (MCA v2.1.0, API v2.0.0, Component v3.0.0)
$

[jmiezitis]

Hi mmokrejs,

Did you resolve your issue. I have been battling a similar problem and found that using:
qsub -I -l select=2:ncpus=28:mpiprocs=28
Would avoid the problem. The key change for me was to specify mpiprocs

[zrss]

i encountered an unexpected case as hostfile slots mismatch -np, for example

# hostfile
worker-0 slots=1

# but mpirun
mpirun -np 2

and it turns out that error there are not enough slots ...

[rhc54]

i encountered an unexpected case as hostfile slots mismatch -np, for example

# hostfile
worker-0 slots=1

# but mpirun
mpirun -np 2

and it turns out that error there are not enough slots ...

Yes, that is correct. Your hostfile explicitly states that there is only one slot on this node - i.e., you are saying that only one process is allowed to be executed on that node. Then you turn around and ask that we launch two processes...and get an error. 😄

[rafaeltiveron]

Hi,

I'm new in MPI (mpiexec) usage, so I try to run a process that can run in more than one thread. So, I used hwthreads. But, the problem is that hwthread is just limited to one node, to slots available just in this node. So, I would like to use more slots in other nodes to speed the process.

My example:

mpiexec -np 1 --cpus-per-proc 24 --map-by slot --bind-to hwthread:overload-allowed -use-hwthread-cpus --hostfile /cluster/hostfile john -test

Using "John" just for example, now a process has until 24 threads to deal with, because my hostfile specified just 24 slots for the first node to use. But I would like to use 40 threads (sum of 2 nodes with 24 slots each other) and I've got the same message. Is that possible to run in this condition with another configuration?

Thanks

[jsquyres]

@rafaeltiveron Not to be pedantic, but this issue was closed about a year ago -- I only happened to notice your new comment on it by chance. Can I ask that you re-ask your question either in a new github issue, or on the Users mailing list? https://www.open-mpi.org/community/lists/ompi.php

Thanks!