Msprime: Bus error when running WG sim with recombination map

[ShyamieG]

Hello all,

I've been trying to get some msprime simulations running on our HPC at UC Davis but have been running in to a persistent error. The job will run for between 40-60 hours and then fail, with nothing more than a "bus error". I've been going back and forth with the sys admins about this issue, but unfortunately have not been able to resolve it. I am wondering if any of you might be able to help me figure out what the problem might be.

Here are my python script, the bash script I use to launch it, and the recombination map I'm using. I sort of hacked the latter based on the guidance I found here.

I have tried several things to see if I can get the error to go away. Excluding the WF portion of the simulation still results in the error, as does triting out tree files rather than the full .vcf. However, the same script runs to completion and writes out vcfs within a few hours if I split the map into parts (ex. chr1-4, chr5-9, chr10-14, and chr15-22).

I have been able to confirm that memory usage is not an issue - at peak, the job uses less than 3GB and I usually request much more than that.

Any hints would be greatly appreciated! Let me know if I can provide any other information that would be of help.

[molpopgen]

Would you be able to tell us what version you are using. You can get this by:

import msprime
print(msprime.__version__)

[benjeffery]

Hi @ShyamieG, thanks for the report.
Just to check simple things first, is it possible the device you are writing to becomes full, or the file over some size limit? That can sometimes result in a SIGBUS.

[ShyamieG]

Thank you both for your prompt responses!

@molpopgen I'm running msprime 0.7.4

@benjeffery I can look into this - I think there should be plenty of space on the device. But are you saying that if msprime is trying to write out a file that is larger than some limit it can throw this error?

[molpopgen]

Thanks. I'll see if I can replicate on my local machine.

[benjeffery]

Bus error is a system error. Msprime is triggering it somehow, but it is not an error that is specific to msprime. This could def be a bug in msprime, don't get me wrong!

Apparently SIGBUS can be raised when reading past the end of an mmap'd file or unaligned memory access.

[molpopgen]

Yeah, that's why if figured trying it on different hardware may help.

[ShyamieG]

Bus error is a system error. Msprime is triggering it somehow, but it is not an error that is specific to msprime. This could def be a bug in msprime, don't get me wrong!

Apparently SIGBUS can be raised when reading past the end of an mmap'd file or unaligned memory access.

Got it. Okay, I can look into this.

[molpopgen]

It ran to completion on my local machine:

/usr/bin/time -f "%e %M" bash run.sh 
done reading the rec map...
105428.37 5422176

The resulting VCF file is 1.2G

it would seem that this is something specific to your HPC nodes, then.

[benjeffery]

I've just kicked off my own recreation. @ShyamieG I assume you're on python 3.6?

[ShyamieG]

@molpopgen Very interesting. Maybe I should just try on my own machine, too. Thanks for looking in to this!

[ShyamieG]

@benjeffery I'm on python 3.7.4.

[molpopgen]

It took 30 hours on my machine.

[benjeffery]

My recreation has also completed (I did python 3.7.7 in the end so not exactly the same version). It took 26 hours to make a 1.2GB VCF. No bus error.

What's the quickest you can get it to error? If it isn't too bad we could run a debug build and get a stacktrace.

[ShyamieG]

My recreation has also completed (I did python 3.7.7 in the end so not exactly the same version). It took 26 hours to make a 1.2GB VCF. No bus error.

What's the quickest you can get it to error? If it isn't too bad we could run a debug build and get a stacktrace.

That's good to know! I'm giving it a whirl too on my own computer now and it's been running for about a day. If that finishes, I have other simulations coded up to look at increasing Ne, decreasing Ne, single pulse of admixture, etc. that I am hoping to run once I have figured how to get around this bug. For the steady Ne sim, the quickest it has given the error is around 40 hours, unfortunately.

[jeromekelleher]

Any luck reproducing this @ShyamieG? I think if we can't reproduce this outside your HPC environment it's most likely to be a problem there rather than an issue with msprime.

[ShyamieG]

@jeromekelleher No, I haven't been able to reproduce this error - it successfully ran on my personal machine (although it did take about 70 hours). It definitely seems to be an issue specific to the HPC I'd been using. This has been really helpful in confirming that. Thanks again to you, @benjeffery and @molpopgen. I'll close this thread now.

[molpopgen]

A few final thoughts:

When you cannot reproduce a bizarre HPC issue on a local machine, it helps to consider the typical differences between the two. The two most important usually are:

1. The HPC uses some fancy networked and distributed file system, and the local machine typically used some local (directly attached) storage.

2. The HPC compiler tool chain may be badly managed. For example, dependencies could have been compiled by different versions of a compiler, which can occasionally lead to funny business.

The way to rule out 2 is to build using conda, for example, where the tool chain is part of the environment.

A way to test 1 is to ask if your HPC nodes have the option of using local storage during a run. Typically, you are asked to write to your own, or your lab group's storage, which is often networked. However, many HPC support a more complex work flow of writing locally first (to a hard drive on the compute node itself) and them move the file to your own space at the end.

[ShyamieG]

Interesting, thanks for that information. I installed msprime via conda, so does that rule out #2?

I will ask about #1. The code does seem to work on my machine, but it takes a lot longer to get results, so I'd obviously still prefer to run on a HPC.

[grahamgower]

It's also possible the HPC uses a hardware architecture with strict alignment requirements. Is this a simple x86_64 system? (what does uname -a output?)

[molpopgen]

Interesting, thanks for that information. I installed msprime via conda, so does that rule out #2?

Probably, yes. Assuming you properly loaded your environment, etc..

[jeromekelleher]

It's also possible the HPC uses a hardware architecture with strict alignment requirements. Is this a simple x86_64 system? (what does uname -a output?)

Interesting theory @grahamgower!

[ShyamieG]

It's also possible the HPC uses a hardware architecture with strict alignment requirements. Is this a simple x86_64 system? (what does uname -a output?)

Hi @grahamgower. This is the output of that command:

Linux augrabies 4.15.0-91-generic #92~16.04.1-Ubuntu SMP Fri Feb 28 14:57:22 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[grahamgower]

Thanks @ShyamieG. Looks like my suggestion was a red herring. I guess itanium, sparc, and other non-x86 HPC systems are now properly extinct.

[ShyamieG]

Interesting, thanks for that information. I installed msprime via conda, so does that rule out #2?

Probably, yes. Assuming you properly loaded your environment, etc..

I think I've figured this out! I had installed msprime via conda, but not in its own environment. When I did that, and loaded the environment before running the msprime python code, my simulations successfully completed on the HPC with no error.

Thanks, everyone, for your ideas to help me solve this issue! I had been stuck on it for months.

[jeromekelleher]

Weird! Thanks for letting us know @ShyamieG, that's really helpful.