Minimap2: Memory: segmentation fault

[teshomem]

Hi,

What is the expected memory usage of minimap2? I am using cluster and my jobs keep crashing because i was not able to specify the memory accurately.

Is it also multithreaded application?

Best,
Teshome

[lh3]

How much memory have you specified? The peak memory varies greatly with data. Mapping pacbio reads to human genome requires ~7GB. Mapping nanopore reads takes some more. Long RNA-seq mapping and pacbio assembly alignment may use >20GB at the peak.

Is it also multithreaded application?

See option -t:

  Input/Output:
    -a           output in the SAM format (PAF by default)
    -Q           don't output base quality in SAM
    -L           write CIGAR with >65535 ops at the CG tag
    -R STR       SAM read group line in a format like '@RG\tID:foo\tSM:bar' []
    -c           output CIGAR in PAF
    --cs[=STR]   output the cs tag; STR is 'short' (if absent) or 'long' [none]
    -Y           use soft clipping for supplementary alignments
    -t INT       number of threads [3]
    -K NUM       minibatch size for mapping [500M]
    --version    show version number

[teshomem]

?Great, i overlooked the -t option. We specified 5GB per CPU which is 15GB for three threads. So it seems 3 threads is the default.

Best,
Teshome

---

From: Heng Li [e-mail]
Sent: Wednesday, December 6, 2017 3:01 PM
To: lh3/minimap2
Cc: Teshome Dagne Mulugeta; Author
Subject: Re: [lh3/minimap2] Memory: segmentation fault (#75)

How much memory have you specified? The peak memory varies greatly with data. Mapping pacbio reads to human genome requires ~7GB. Mapping nanopore reads takes some more. Long RNA-seq mapping and pacbio assembly alignment may use >20GB at the peak.

Is it also multithreaded application?

See option -t:

Input/Output:
-a output in the SAM format (PAF by default)
-Q don't output base quality in SAM
-L write CIGAR with >65535 ops at the CG tag
-R STR SAM read group line in a format like '@RG\tID:foo\tSM:bar' []
-c output CIGAR in PAF
--cs[=STR] output the cs tag; STR is 'short' (if absent) or 'long' [none]
-Y use soft clipping for supplementary alignments
-t INT number of threads [3]
-K NUM minibatch size for mapping [500M]
--version show version number

-
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/lh3/minimap2/issues/75#issuecomment-349648032, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIZyP5JH_9xqhdhI8XM2mNrWXrwHRSQ4ks5s9p4lgaJpZM4Q36Y9.

[lh3]

You need to specify the number of threads explicitly when submitting jobs. No scheduler adjusts the peak memory based on the actual CPU uses. Ask your sysadmins.

[prishly]

Hi,
I thought my problem might be of a similar kind. I'm using guest Ubuntu OS with 16 GB RAM, 77GB free disk space and 3 CPUs (1 thread each) allocated to it. When I try to align my raw ONP genomic reads (around 2.5 Gbp in total, in a 5.2 Gb fastq file) against hg38 reference and threads set to 1 or 2, I keep getting Killed or Segfault error. Running with valgrind throws a lot of messages like "Warning: set address range perms: large range [0x1ed50028, 0x2ed50058) (noaccess)" and later "Address 0x2006f7c0 is 0 bytes after a block of size 524,288 alloc'd".
I wonder what could be the bottle neck (disk space or memory) and is it possible at all to run jobs like mine with the resources that I mentioned? Simple tasks like fastq files with only several reads less than 100 bp against a short reference run OK.

[lh3]

Depending on your data, 16GB might not be enough, especially considering that OS and other process on the machine take part of memory as well. Could you send me the full valgrind log file? Thanks!

[prishly]

I just copied valgrind output in a txt file attached
valgrinf_minimap.txt

[lh3]

Thanks for the log file. That segfault was caused by insufficient memory during indexing. Indexing takes more memory than mapping. One possible solution is you index the genome on a machine having more RAM with:

minimap2 -x map-ont -d hs38-ont.mmi /mnt/hs38.fa

and then on your 16GB machine, call minimap2 this way:

minimap2 -ax map-ont hs38-ont.mmi /mnt/mcf_albac106.fastq

I will also try to add more meaningful error messages when the machine doesn't have enough memory.

[atongsa]

map a 3G genome assembly to ref genome

• max memory 16.711 GB
• log below
  Real time: 422.835 sec; CPU: 1189.459 sec; Peak RSS: 16.711 GB