Distributed: Feature request: Worker port ranges and/with nprocs support

[stuartarchibald]

I think it would be quite useful for dask-worker to support --nprocs with --foo-port=[range] so that systems with specific port ranges opened can have a number of processes opened from one call applied to a specified range opposed to having to write a shell loop to do the same.

Further, general support for port ranges and failing over to the next available port in a range if the current is taken would probably be quite useful in a similar situation as that described above.

Thanks.

[mrocklin]

Does anyone with some networking background know of a good way to ask the OS to connect to a port within a certain range?

cc @pitrou

[pitrou]

@stuartarchibald, I assume you mean --nprocs=N --start-port=X will bind to ports X, X+1, X+2, etc? Are you expecting a specific pattern (e.g. first all worker ports, then all Bokeh ports), or is it indifferent?

[mrocklin]

I suspect he's looking for something like --port=5000:6000. Often people are told that they need to stay within a particular range of ports.

[stuartarchibald]

I think being able to express a port range for each component would be ideal thanks. Something like:

dask-worker --nprocs=16 --worker-port=5000:5100 --http-port=6000:6100

which would start 16 worker processes bound to free ports between 5000 and 5100 and the same with http but s/5/6/g. This is with view of use cases where open ports are restricted to some particular range but some external reason requires computation to be undertaken in separate processes.

[mrocklin]

I see two options:

1. Find a way to ask the operating system for an unused port within a range
2. Assume that the system giving us the port range has already made them free for us, and just choose an increasing sequence (erring loudly if this fails)

[stuartarchibald]

I think either of these would be fine, the latter, with consistent and accessible complaining (for easy parsing), is probably easiest place to start?

[pitrou]

If we change the syntax only slightly, we can fit this feature in the URL syntax, for example tcp://0.0.0.0:6000-6099 would listen somewhere in the [6000, 6099] port range. What do you think?

[pelson]

The addition of a port range would be awesome. I'm in the situation where I have a range of ports opened on my firewall, and I just want to use any available port in the given range.

@pitrou - looks like a good suggestion. Looks fairly similar to the iptables approach. The --http-port / --bokeh-port interface wouldn't be able to use the tcp:// form though, so would need to do both (not that I need the bokeh interface in this instance).

[dhirschfeld]

This would be a very useful feature! I've just hit this limitation when trying to deploy on Azure where I'm currently using a powershell script like below:

$ports = @(50800, 50801, 50802, 50803)
foreach ($port in $ports) {
    Start-Process -NoNewWindow dask-worker.exe -ArgumentList '--nthreads=1','--nprocs=1',"--worker-port=$port",'tcp://10.1.0.4:8786';   
}

...after having opened an appropriate port range.

It would be nice to simply be able to specify it at the command line like:

dask-worker --nthreads=1 --nprocs=4 --worker-port=50800:50899 tcp://10.1.0.4:8786

and have dask simply assign consecutive addresses in the range (potentially automatically skipping any already in use)

[stuartarchibald]

@pitrou The suggested URL syntax form seems good to me, neat idea, also saves having custom parsers etc.

[julioasotodv]

So, I believe that this has not been implemented yet, right?

The only current way to specify ports for more than one process is to actually create multiple dask-workers in one machine...

[mrocklin]

Correct

On Thu, Feb 8, 2018 at 10:58 AM, Julio Antonio Soto <
[email protected]> wrote:

So, I believe that this has not been implemented yet, right?

The only current way to specify ports for more than one process is to
actually create multiple dask-workers in one machine...

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/distributed/issues/899#issuecomment-364156933,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AASszHT_cudJyMvSRGIVtJCmblKU8TVgks5tSxmZgaJpZM4MKnBs
.

[julioasotodv]

Ok thanks

[gdurif]

Is there any news regarding that feature ?
Thanks

[stuartarchibald]

Thanks for implementing this @jrbourbeau!