Msprime: Core dumped error

[TPPSellinger]

I got the following error :
~~
python3: lib/object_heap.c:136: object_heap_free_object: Assertion `self->top < self->size' failed.
Aborted (core dumped)
~~
After running the following code :
~~~~
import matplotlib
import numpy as np
import pandas as pd
import scipy.special
import scipy.stats
from matplotlib import lines as mlines
from matplotlib import pyplot
import msprime
import msprime.cli as cli

sample_size=20
Ne=10*4
r=11e-8
m=1.251e-8
L=10 * 8
strengh=10

population_configurations=[msprime.PopulationConfiguration(initial_size=Ne,sample_size=sample_size)]
demographic_events = [
msprime.PopulationParametersChange(
time=1000, growth_rate=-0.00025584278811045
),
msprime.PopulationParametersChange(
time=10000, growth_rate=0
),
]
for x in range(1,11):
ts=msprime.simulate(population_configurations=population_configurations,recombination_rate=r,mutation_rate=m,length=L,demographic_events=demographic_events)
name="Figure_1_Decrease_strength_"+str(strengh)+"_"+str(x)+".vcf"
with open(name, "w") as vcf_file:
ts.write_vcf(vcf_file, 2)
vcf_file.close()
~~~~
The error does not always occur, it might run couple of iterations before crashing.

[jeromekelleher]

Thanks for the bug report @TPPSellinger, this is very helpful. Can you tell us what version of msprime you're using please? (If you put print(msprime.__version__) into your script above somewhere it'll write it out).

[TPPSellinger]

My version was : 0.7.5.dev202+ng73eccfe

[benjeffery]

I've done 8 iterations on 0.7.5.dev237+gbcef1bd9 without a crash, once the script completes I will checkout 0.7.5.dev202+ng73eccfe to try to recreate.

[TPPSellinger]

I have also just did 10 iterations (version 0.7.5.dev202+ng73eccfe again). Also without a crash.

[TPPSellinger]

I am now running with a sharper demographic scenario. It might help unveil something.

[benjeffery]

Running with a fixed random seed would be good - as crash may be dependant on it (I really hope not though!)

[TPPSellinger]

True, I didn't think of that. I will run simulations with fixed seed and let you know if anything comes out of it.

[jeromekelleher]

When debugging this stuff it's a good idea to specify the random seed so we can reproduce. Something like

for seed in range(1, 1000):
    print("seed =", seed)
    ts = msprime.simulate(..., random_seed=seed)

Once we have a seed that reproduces the problem, fixing the issue will be straightforward.

[jeromekelleher]

Any luck on reproducing this for an exact seed folks?

[benjeffery]

I did ~30 iterations without fault, can restart it if we think it's worth it.

[jeromekelleher]

Let's see if Thibaud has any updates.

[TPPSellinger]

I also did ~ 30 iterations without any issue.

[TPPSellinger]

Here are the seed I work with so we don't compute the same simulations :
~
for seed in range(800,1000):
~
Should I let the code run or stop ?

[jeromekelleher]

Let it running if you don't mind @TPPSellinger, might take a while to show up.

[benjeffery]

Cool, to put this to bed I'll run 8 processes in parallel starting at seed 100,000.

[TPPSellinger]

I got one, It's a different error but it could still help :

seed = 841
python3: lib/msprime.c:743: msp_insert_individual: Assertion `node != NULL' failed.
Aborted (core dumped)

[jeromekelleher]

Thanks @TPPSellinger, I've reproduced this. It takes a while to hit it - I'm a bit puzzled on what's happening, but it's definitely a bug. Having an exact seed to check is a big help, thanks!

[jeromekelleher]

It would be nice if we could reproduce the other assertion trip as well though. @benjeffery , are you still running the your sims?

[benjeffery]

Still haven't caught anything, but then I keep hitting the OOM killer when I try to do anything else on the machine.

[jeromekelleher]

OK, don't run them on your desktop, we have some servers that we can dedicate a few threads to this and not miss them.

[benjeffery]

I have a few crashing seeds - but all of the type lib/msprime.c:743: msp_insert_individual: Assertion 'node != NULL' failed.'
I'm finding the quickest one to fail and will look into it.

[benjeffery]

Here are the seeds with the time they fail:
107005 41m
106022 68m
104001 41m
102024 62m
101001 63m

[jeromekelleher]

Just to update that I'm working on this, and it's a doozy. Some very subtle and tricky numerical issues with recombination going on here. I'm mainly just very happy this surfaced now rather than after release though, so thanks @TPPSellinger!

[TPPSellinger]

I hope it's solvable. I will keep you inform if I find any other issues.

[TPPSellinger]

Hi, I was wondering if this issue had been solved since I got another error but the name is slightly different (c:743 becomes c:751) :
~~
python3: lib/msprime.c:751: msp_insert_individual: Assertion `node != NULL' failed.
Aborted (core dumped)
~~
My version (which might not be the most recent one) was 0.7.5.dev308+ngcfc263e9.

[jeromekelleher]

I'm still working on this @TPPSellinger - see #1079 for progress. Hopefully I'll get some time to finish it out this week :crossed_fingers:

[jeromekelleher]

This should be fixed now @TPPSellinger - would you mind trying out your big simulations please?

[TPPSellinger]

Nice ! I will launch some analysis and let you know if anything happens (might take a while).

[TPPSellinger]

Quick questions, do you think the correction affects in anyway the simulation results or it just fixed the crashing (to know if I have to relaunch previous simulations) ?

I didn't put them on the github but I also had the same error (before the fix) while simulating with multiple merger coalescent models (and high number of recombination). Would this issue also be fixed ?

[jeromekelleher]

There must have been some subtle effects on the results of the simulations. I doubt it would be significant, but if I were you I'd rerun them. If you can also try out the multiple merger models with lots of recombination too, that would be great (that is almost certainly the same issue - I hope!).

[TPPSellinger]

Okay I will run everything, might take a week or two to have enough repetitions.

[TPPSellinger]

Most of them finished. No crash observed. I did something like ~60 iterations. Should I push it to 100 for safety ?

[jeromekelleher]

That's great,thanks @TPPSellinger. I've done about 1600 reps of your initial example now, so I think we're good.

[TPPSellinger]

I also checked for multiple merger. No issues found.