Rnaseq: Add config file for bianca cluster

[orzechoj]

The uppmax config file doesn't work for the bianca cluster. Since we can only have 7Gb RAM per core on bianca, using the uppmax config file gives the error error: CPU count per node can not be satisfied.

Would it hurt to add another config file?

[ewels]

Aha! No not at all, we can do that. Note that the institutional config files are about to be removed though, in preference of the centralised https://github.com/nf-core/configs/ - but we can add a bianca config there if necessary.

I'm a little lost though, bianca has 16 cpus and 128GB memory per node right? Are there jobs being submitted that are asking for more than that?

[orzechoj]

Here (http://www.uppmax.uu.se/support/user-guides/bianca-user-guide/) it says:

Bianca has two node types, thin being the typical cluster node with 112 GB memory and fat nodes having 256 GB or 512 GB of memory.

I agree this is a bit odd..

[ewels]

hah, and here (https://uppmax.uu.se/resources/systems/the-bianca-cluster/) it says:

Bianca provides 3200 cores in the form of 200 dual CPU (Intel Xeon E5-2630 v3) Huawei XH620 V3 nodes with 128GB memory. Eight "fat" nodes have 256 GB or 512 GB of memory.

I guess the VM system needs some memory allocation or something..? Doesn't really matter anyway, we just set --max_mem to 128GB (can do this on the command line too in the short term).

[orzechoj]

My problem was when running e.g. STAR, which requests 80Gb and 10 CPUs. This configuration was not allowed, since we can at most use 7Gb per CPU. So I had to change this to 80Gb and 12 CPUs.

[ewels]

Ack, I hate this memory-per-cpu thing, it makes life very tricky. Why can't the scheduler automatically assign additional CPUs to get to the requested memory amount??

[apeltzer]

Yeah I don't get it - should we (@orzechoj maybe as you know most about that system?) add a config for bianca then to nf-core/configs ?

Could also do it, but I don't know much about that specific system and can't test it there unfortunately...

[ewels]

The problem is that this requires per-process configuration, so can't be in nf-core/configs as it's pipeline specific. So needs a more general fix if possible.

[orzechoj]

I can make a config file for bianca for the RNA-seq pipeline. But I'm not sure how to make a more general fix for all pipelines. Let me know what you think is best.

[ewels]

A really _really_ crude way of handling this is to add the slurm failure error number to this line in the base config: https://github.com/nf-core/rnaseq/blob/37f260d360e59df7166cfd60e2b3c9a3999adf75/conf/base.config#L18

Then when the job fails, it will be resubmitted with double the resources. 2x80GB is over the max_memory so it will be throttled to the maximum available for Bianca, and I _guess_ that it should work.

This is of course horribly inefficient though - failing and resubmitting jobs by design and also over-requesting resources beyond what is required. I've asked for help on the SciLifeLab #uppmax channel to see if there's something that can be done with the way that the slurm scheduler works instead.

[drpatelh]

Is it ok to close this @orzechoj ? We now have pipeline specific configs on nf-core/configs! I do agree though this memory per CPU definition is annoying and we should probably have a parameter to deal with this in the pipeline template.

[drpatelh]

Ok. Sorry, I asked and closed it anyway but feel free to reopen if you so wish :)