Cmdstanr: Not all parallel jobs launching

[bbbales2]

When I go to run parallel inferences not all the chains start and then the call just hangs without ever returning.

This could just be my computer. I'll try to reproduce it at home, but I can get failures with something as simple as num_chains = 2, num_cores = 1, but it's easier to reproduce (the failures are random) with something like:

model$sample(num_chains = 10,
             num_cores = 10,
             data = data,
             save_warmup = 1,
             num_warmup = 100,
             num_sample = 0,
             metric = "dense_e",
             save_diagnostics = TRUE,
             term_buffer = 0,
             init_buffer = 100,
             window = 0)

As an example, output looks like:

...
Running ./accel_splines 'id=10' random 'seed=721749581' data 'file=/tmp/Rtmp2WZqkV/standata-17c1295a0363.json' output \
  'file=/tmp/Rtmp2WZqkV/accel_splines-stan-sample-10-17c12e1147ab.csv' \
  'diagnostic_file=/tmp/Rtmp2WZqkV/accel_splines-stan-sample-diagnostic-10-17c1457833ad.csv' 'method=sample' 'num_samples=0' \
  'num_warmup=100' 'save_warmup=1' 'algorithm=hmc' 'metric=dense_e' 'engine=nuts' adapt 'engaged=1' 'init_buffer=100' \
  'term_buffer=0' 'window=0'
Chain 4 Iteration:  1 / 100 [  1%]  (Warmup) 
Chain 5 Iteration:  1 / 100 [  1%]  (Warmup) 
Chain 8 Iteration:  1 / 100 [  1%]  (Warmup) 
Chain 9 Iteration:  1 / 100 [  1%]  (Warmup) 
Chain 10 Iteration:  1 / 100 [  1%]  (Warmup) 
Chain 10 Iteration: 100 / 100 [100%]  (Warmup) 
Chain 9 Iteration: 100 / 100 [100%]  (Warmup) 
Chain 10 finished in 3.1 seconds.
Chain 9 finished in 3.2 seconds.
Chain 4 Iteration: 100 / 100 [100%]  (Warmup) 
Chain 4 finished in 3.7 seconds.
Chain 5 Iteration: 100 / 100 [100%]  (Warmup) 
Chain 5 finished in 3.8 seconds.
Chain 8 Iteration: 100 / 100 [100%]  (Warmup) 
Chain 8 finished in 4.2 seconds.

Github wouldn't let me upload R scripts so I'm just putting the files on discourse: https://discourse.mc-stan.org/t/problems-running-multiple-chains-in-cmdstanr/11997

[jgabry]

When I go to run parallel inferences not all the chains start and then the call just hangs without ever returning.

Always when you run in parallel or just for particular examples? I assume the latter?

This could just be my computer

Yeah curious what happens if you try on your other computer. I’ll test on my Mac too.

[bbbales2]

Confirmed this as a bug on my laptop (Edit: which is different than the computer I reported this on, so confirming this is a bug, not an issue with my computer I think)

[bbbales2]

Yo @rok-cesnovar, I see your thumbs up. Are you working on this now? If not I'ma dig in (need this for an experimental warmup package I'm working on)

[bbbales2]

I guess that's kinda rude cause it's like I'm asking you to make my priorities your priorities -- don't do that! Just lemme know.

[rok-cesnovar]

I am on it. I think I have a fix just need to confirm.

[rok-cesnovar]

It seems that it wasn't the issue I thought it was :/.

If one uses "too many cores" (starts too many processes in parallel), some processes go sleeping and never return. I can see them in the process list and they don't take up any resources. Will investigate further.

For now, the solution that worked for me was is to limit the num_cores to the number of physical cores. But that is obviously not the way to go long term.

[rok-cesnovar]

Also, I am unable to reproduce this on Windows, even with 20 chains with num_cores set to 20. On Ubuntu I can reproduce this every time. So it might be a Unix specific thing.

[bbbales2]

I can get it to trigger with num_cores < number of physical cores in my computer.

The sleep state thing sounds interesting. I Googled what that meant and found this: https://unix.stackexchange.com/a/47259

I did this on one of my running programs and got:

write(2, "\n", 1

So somehow the stdout buffer is filling up?

[bbbales2]

I found this in the processx manual:

"Also, because $read_output_lines() could still block if no complete line is available, this function always fails if the process has a pty. Use $read_output() to read from ptys."

Could we be printing lines so long that they individually fill up the output buffer?

[rok-cesnovar]

Nice finds.

I can confirm its the stdout getting blocked somehow on unix. If I comment out the calls to read_output_lines and process_sample_output and start the process specifying to not process the stdout all chains finish.

[rok-cesnovar]

@bbbales2 can you try one simple thing? It seemed to help me, not completely sure yet.

Remove the stderr argument here https://github.com/stan-dev/cmdstanr/blob/master/R/run.R#L322

[bbbales2]

Oh man lol, stderr, good point. Just looked it up, the '2' in write(2, "\n", 1 means stderr. This fixed it!

[bbbales2]

Just used the internet to check your time zone. Don't feel the need to fix this tonight. We can get this tomorrow or whatever.

[rok-cesnovar]

Haha :) No worries, I am used to debugging at these hours :)

So yeah, I can see why we got those. Replace that with
stdout = "|",
stderr = "2>&1"

and you will see why. We are not processing stderr right.

[bbbales2]

Hoooray! Make a pull and I will happily check and review it!