Dada2: Segmentation fault when multithread using in assignTaxonomy
May be related to #684 and #726.
Currently on dada2 version 1.14. First observed when trying to use the UNITE database, but can also replicate using the standard RDP trainset (rdp_train_set_16.fa.gz).
I tried to update RcppParallel using BiocManager (on version 4.4.4), which appears to be downstream of the fix described in the previous issues.
Output from sessionInfo():
> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] dada2_1.14.0 Rcpp_1.0.3
loaded via a namespace (and not attached):
[1] plyr_1.8.5 RColorBrewer_1.1-2
[3] pillar_1.4.3 compiler_3.6.1
[5] GenomeInfoDb_1.22.0 XVector_0.26.0
[7] bitops_1.0-6 tools_3.6.1
[9] zlibbioc_1.32.0 lattice_0.20-38
[11] lifecycle_0.1.0 tibble_2.1.3
[13] gtable_0.3.0 png_0.1-7
[15] pkgconfig_2.0.3 rlang_0.4.2
[17] Matrix_1.2-17 DelayedArray_0.12.1
[19] parallel_3.6.1 GenomeInfoDbData_1.2.2
[21] stringr_1.4.0 hwriter_1.3.2
[23] dplyr_0.8.3 Biostrings_2.54.0
[25] S4Vectors_0.24.1 IRanges_2.20.1
[27] stats4_3.6.1 grid_3.6.1
[29] tidyselect_0.2.5 glue_1.3.1
[31] Biobase_2.46.0 R6_2.4.1
[33] jpeg_0.1-8.1 BiocParallel_1.20.1
[35] latticeExtra_0.6-29 reshape2_1.4.3
[37] ggplot2_3.2.1 purrr_0.3.3
[39] magrittr_1.5 Rsamtools_2.2.1
[41] matrixStats_0.55.0 scales_1.1.0
[43] BiocGenerics_0.32.0 GenomicRanges_1.38.0
[45] GenomicAlignments_1.22.1 ShortRead_1.44.1
[47] assertthat_0.2.1 SummarizedExperiment_1.16.1
[49] colorspace_1.4-1 stringi_1.4.3
[51] RCurl_1.95-4.12 lazyeval_0.2.2
[53] RcppParallel_4.4.4 munsell_0.5.0
[55] crayon_1.3.4
fanli-gcb
All 29 comments
Was dada2 (or any other part of your R installation) installed via bioconda?
benjjneb
on 6 Jan 2020
No, I don't believe so although I'm not much of a conda user:
fanli@kickseed:~$ conda list | grep "dada2"
fanli@kickseed:~$
Similarly for r but grep is fairly useless there.
fanli-gcb
on 6 Jan 2020
Can you try running through the basic dada2 worfklow with multithreading, including filterAndTrim(..., multithread=TRUE) and dada(..., multithread=TRUE)? Using just a few samples.
Do those previous multithreaded functions work normally? Is it only assignTaxonomy(..., multithread=TRUE) where the segfault happens?
benjjneb
on 6 Jan 2020
filterAndTrim, learnErrors, and dada were all successfully run with multithreading enabled
I will try some more variations of dataset and parameters to see if I can narrow down any more.
fanli-gcb
on 6 Jan 2020
I will try some more variations of dataset and parameters to see if I can narrow down any more.
Thanks. Ultimately some reproducible code running on the smallest input data possible would be ideal, but any additional information is welcome.
benjjneb
on 6 Jan 2020
Did a bit more digging, and I believe the issue is related to the UNITE taxonomy files. Here is a small (21x100) seqtab object that can be used to reproduce the error.
minst.zip
minst <- readRDS("minst.rds")
# this segfaults
taxa <- assignTaxonomy(minst, "/share/DADA2/sh_general_release_dynamic_02.02.2019.fasta", multithread=16)
# this does not segfault
taxa <- assignTaxonomy(minst, "/share/DADA2/rdp_train_set_16.fa.gz", multithread=16)
I tried the following:
library(dada2)
minst <- readRDS("~/Desktop/minst.rds")
taxa <- assignTaxonomy(minst, "~/tax/sh_general_release_dynamic_02.02.2019.fasta.gz", multithread=16) # No segfault
and got no segfault. sessionInfo:
> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.6
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] dada2_1.14.0 Rcpp_1.0.3
loaded via a namespace (and not attached):
[1] plyr_1.8.5 RColorBrewer_1.1-2 pillar_1.4.3 compiler_3.6.1
[5] GenomeInfoDb_1.20.0 XVector_0.24.0 bitops_1.0-6 tools_3.6.1
[9] zlibbioc_1.30.0 lattice_0.20-38 lifecycle_0.1.0 tibble_2.1.3
[13] gtable_0.3.0 png_0.1-7 pkgconfig_2.0.3 rlang_0.4.2
[17] Matrix_1.2-17 DelayedArray_0.10.0 rstudioapi_0.10 parallel_3.6.1
[21] GenomeInfoDbData_1.2.1 stringr_1.4.0 hwriter_1.3.2 dplyr_0.8.3
[25] Biostrings_2.52.0 S4Vectors_0.22.1 IRanges_2.18.3 stats4_3.6.1
[29] grid_3.6.1 tidyselect_0.2.5 Biobase_2.44.0 glue_1.3.1
[33] R6_2.4.1 jpeg_0.1-8.1 BiocParallel_1.18.1 latticeExtra_0.6-29
[37] reshape2_1.4.3 ggplot2_3.2.1 purrr_0.3.2 magrittr_1.5
[41] Rsamtools_2.0.3 matrixStats_0.55.0 scales_1.1.0 BiocGenerics_0.30.0
[45] GenomicRanges_1.36.1 GenomicAlignments_1.20.1 ShortRead_1.42.0 assertthat_0.2.1
[49] SummarizedExperiment_1.14.1 colorspace_1.4-1 stringi_1.4.4 RCurl_1.95-4.12
[53] lazyeval_0.2.2 RcppParallel_4.4.4 munsell_0.5.0 crayon_1.3.4
Any chance your UNITE file might be malformed in some way? What's the checksum?
> md5 sh_general_release_dynamic_02.02.2019.fasta
MD5 (sh_general_release_dynamic_02.02.2019.fasta) = 50094b8d644ddedf7a08e7cb53e5e69c
Doesn't appear to be the case. That's really odd. I'll dig around some more...
fanli@kickseed:/share/DADA2$ md5sum sh_general_release_dynamic_02.02.2019.fasta 50094b8d644ddedf7a08e7cb53e5e69c sh_general_release_dynamic_02.02.2019.fasta
fanli-gcb
on 10 Jan 2020
I was able to run successfully on a different system with an older version of dada2_1.10.1 and Rcpp_1.0.3:
> sessionInfo()
R version 3.5.3 (2019-03-11)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] dada2_1.10.1 Rcpp_1.0.3
I am going to try some different versions on the original machine to see if I can track down what's going on.
fanli-gcb
on 28 Jan 2020
Downgrading to BioC 3.9 versions seems to work on the original machine.
Strangely enough, both BioC 3.9 and 3.10 versions work on a different machine.
I may surrender on this one and just use a BioC 3.9 environment for now..
fanli-gcb
on 29 Jan 2020
Let's leave this issue open, but I'll wait on another report like this to take a harder look.
benjjneb
on 29 Jan 2020
Hi,
I also have an issue where I get a segmentation fault while running assignTaxonomy(multithread=12) on my own sequence table. However, it ran just fine when using the files provided by @fanli-gcb.
To address above comments, none of my packages come from Bioconda. The system I am using has 16 cores and 128GB of RAM, so resources should not be an issue. This segfault issue is reproducible whether I run it in command-line R or in Rstudio, occurs regardless of which database I use (I have tried UNITE as above and also PR2), and goes away when I do not include the multithreading flag. I have not yet tried downgrading any packages. All of my packages on my system are up to date. I tried the following:
> library(dada2)
Loading required package: Rcpp
> sessionInfo()
R version 3.6.2 (2019-12-12)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] dada2_1.14.0 Rcpp_1.0.3
loaded via a namespace (and not attached):
[1] plyr_1.8.5 RColorBrewer_1.1-2
[3] pillar_1.4.3 compiler_3.6.2
[5] GenomeInfoDb_1.22.0 XVector_0.26.0
[7] bitops_1.0-6 tools_3.6.2
[9] zlibbioc_1.32.0 lattice_0.20-38
[11] lifecycle_0.1.0 tibble_2.1.3
[13] gtable_0.3.0 png_0.1-7
[15] pkgconfig_2.0.3 rlang_0.4.4
[17] Matrix_1.2-18 DelayedArray_0.12.2
[19] parallel_3.6.2 GenomeInfoDbData_1.2.2
[21] stringr_1.4.0 hwriter_1.3.2
[23] dplyr_0.8.3 Biostrings_2.54.0
[25] S4Vectors_0.24.3 IRanges_2.20.2
[27] stats4_3.6.2 grid_3.6.2
[29] tidyselect_1.0.0 glue_1.3.1
[31] Biobase_2.46.0 R6_2.4.1
[33] jpeg_0.1-8.1 BiocParallel_1.20.1
[35] latticeExtra_0.6-29 reshape2_1.4.3
[37] ggplot2_3.2.1 purrr_0.3.3
[39] magrittr_1.5 Rsamtools_2.2.1
[41] matrixStats_0.55.0 scales_1.1.0
[43] BiocGenerics_0.32.0 GenomicRanges_1.38.0
[45] GenomicAlignments_1.22.1 ShortRead_1.44.1
[47] assertthat_0.2.1 SummarizedExperiment_1.16.1
[49] colorspace_1.4-1 stringi_1.4.5
[51] RCurl_1.98-1.1 lazyeval_0.2.2
[53] RcppParallel_4.4.4 munsell_0.5.0
[55] crayon_1.3.4
> minst <- readRDS("minst.rds")
> taxa <- assignTaxonomy(minst, "sh_general_release_dynamic_02.02.2019.fasta", multithread=12)
UNITE fungal taxonomic reference detected. # did not segfault for me
> taxa2 <- assignTaxonomy(my_huge_seq_table, "sh_general_release_dynamic_02.02.2019.fasta", multithread=12)
UNITE fungal taxonomic reference detected.
Segmentation fault (core dumped)
taxa2 <- assignTaxonomy(my_huge_seq_table, "sh_general_release_dynamic_02.02.2019.fasta")
UNITE fungal taxonomic reference detected. # did not segfault for me
brymerr921
on 30 Jan 2020
OK, there seems to be something going on here. @brymerr921 Can you try to share a minimal script/datafile that can reproduce this segfault? (don't need to include the training fasta)
benjjneb
on 31 Jan 2020
Great(?) news! I am now able to reproduce the segfault using @brymerr921 datafile.
Don't know the cause yet, but this is step 1 to figuring it out.
benjjneb
on 1 Feb 2020
I am also experiencing this issue and reverted to v1.12 for now. I noticed that I could often (but not always) run assignTaxonomy successfully once, but it would crash my R session on the second or third time, even if the inputs were exactly the same.
dleopold
on 7 Feb 2020
I think 4 independent reports make it pretty clear something is going on. There was only one commit that touched the assignTaxonomy code between 1.12 and 1.14, so that will need some scrutiny: 2317b9bce4bbf51984a7399fdc66f983f45df646
Also... is it true that these segfaults are only being seen when using the UNITE database?
benjjneb
on 7 Feb 2020
I was using the UNITE database when I noticed the problem.
dleopold
on 7 Feb 2020
Same, only noticed it when using the UNITE database. Also, using the same seqtab that segfaults against UNITE does NOT segfault against RDP on my system.
fanli-gcb
on 8 Feb 2020
Mine segfaulted also when I used the PR2 database.
On Fri, Feb 7, 2020, 4:07 PM Fan Li notifications@github.com wrote:
Same, only noticed it when using the UNITE database. Also, using the same
seqtab that segfaults against UNITE does NOT segfault against RDP on my
system.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/benjjneb/dada2/issues/916?email_source=notifications&email_token=ABZE36XVYOHF2TDNN2GLTALRBXZTVA5CNFSM4KDATNI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELFBEBY#issuecomment-583668231,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABZE36TGCHRT6ODUX5W4TS3RBXZTVANCNFSM4KDATNIQ
.
brymerr921
on 8 Feb 2020
I got segfaults with dada2 1.14 while using GTDB r89 from this Zenodo repository. I've tested with only one sequencing dataset, but this same dataset ran fine with dada2 1.12.1 & Rcpp 1.0.3 from BioConductor 3.9.
hmontenegro
on 9 Feb 2020
I am getting this same error, but I am using the silva taxonomy database.
Naomi1030
on 17 Feb 2020
I've tracked down the cause, the random number generation in this line of code is not thread safe https://github.com/benjjneb/dada2/commit/2317b9bce4bbf51984a7399fdc66f983f45df646#diff-92a077efe384576f8ce7c39641f996ecR100
When that line is replaced with something deterministic, the segfault goes away.
benjjneb
on 18 Feb 2020
Great to hear that you found the issue! Have been running into this with several Tax DBs over the last couple of weeks (custom, 18S eukaryote silva) on a new machine and was convinced it was a formatting issue. It drove me crazy because I could not identify the problem since RStudio just hard crashes ('fatal error') and I assumed it was an issue there. Running in the terminal finally gave me a useful error message and the right google results.
Going back to 1.12 seems to have fixed the crash.
simeross
on 18 Feb 2020
Going back to 1.12 seems to have fixed the crash.
Yes, reverting to 1.12 or running with multithread=FALSE are both effective workaround right now. I'll try to get the fix out for 1.14 with multithread=TRUE soon.
benjjneb
on 18 Feb 2020
I believe this is now fixed on the taxonomy-segfault-fix branch: 997389b78f7f55116b4ba43ed0bfd3b569149aa0
Basically I replaced the R-based random number a C++11 random number generator that is reinitialized everytime the function is called, which should then be threadsafe.
For those who were having segfaults, I would much appreciate if you had the time to test if this update fixes your problem on the same dataset!
To install the version on the fixed branch:
library(devtools)
devtools::install_github("benjjneb/dada2", ref="taxonomy-segfault-fix")
packageVersion("dada2") # Should be 1.15.1
Thanks! This branch fixed the issue for me.
Tested with:
- mulithread = 18
- taxdb: 18S eukaote silva 132 from https://zenodo.org/record/1447330
- approx. 10.000 ASVs
- Ubuntu 18.04
- R v3.6.2
- RStudio v1.2.5033
simeross
on 20 Feb 2020
The 1.15.1 taxonomy-segfault-fix branch seems to have fixed the issue for me as well.
With dada 1.14.0:
> library( dada2, lib="/home/hmon/R/dadabug" )
Loading required package: Rcpp
> packageVersion("dada2")
[1] ‘1.14.0’
> load( "dada2.RData" )
> gtdb16Sr89 <- "~/db/dada2/GTDB_bac120_arc122_ssu_r89.fa.gz"
> tax <- assignTaxonomy( seqtabnochim, gtdb16Sr89,
+ tryRC = F, minBoot = 95, multithread = 10 )
Segmentation fault
Now with dada2 1.15.1:
> library( dada2, lib="/home/hmon/R/dadafix" )
Loading required package: Rcpp
> packageVersion("dada2")
[1] ‘1.15.1’
> load( "dada2.RData" )
> gtdb16Sr89 <- "~/db/dada2/GTDB_bac120_arc122_ssu_r89.fa.gz"
> tax <- assignTaxonomy( seqtabnochim, gtdb16Sr89,
+ tryRC = F, minBoot = 95, multithread = 10 )
>
Agree, fixed for me as well.
> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] dada2_1.15.1 Rcpp_1.0.3
loaded via a namespace (and not attached):
[1] plyr_1.8.5 RColorBrewer_1.1-2
[3] pillar_1.4.3 compiler_3.6.1
[5] GenomeInfoDb_1.22.0 XVector_0.26.0
[7] bitops_1.0-6 tools_3.6.1
[9] zlibbioc_1.32.0 lattice_0.20-38
[11] lifecycle_0.1.0 tibble_2.1.3
[13] gtable_0.3.0 png_0.1-7
[15] pkgconfig_2.0.3 rlang_0.4.3
[17] Matrix_1.2-17 DelayedArray_0.12.2
[19] parallel_3.6.1 GenomeInfoDbData_1.2.2
[21] stringr_1.4.0 hwriter_1.3.2
[23] dplyr_0.8.3 Biostrings_2.54.0
[25] S4Vectors_0.24.3 IRanges_2.20.2
[27] stats4_3.6.1 grid_3.6.1
[29] tidyselect_0.2.5 glue_1.3.1
[31] Biobase_2.46.0 R6_2.4.1
[33] jpeg_0.1-8.1 BiocParallel_1.20.1
[35] latticeExtra_0.6-29 reshape2_1.4.3
[37] ggplot2_3.2.1 purrr_0.3.3
[39] magrittr_1.5 Rsamtools_2.2.1
[41] matrixStats_0.55.0 scales_1.1.0
[43] BiocGenerics_0.32.0 GenomicRanges_1.38.0
[45] GenomicAlignments_1.22.1 ShortRead_1.44.1
[47] assertthat_0.2.1 SummarizedExperiment_1.16.1
[49] colorspace_1.4-1 stringi_1.4.5
[51] RCurl_1.98-1.1 lazyeval_0.2.2
[53] RcppParallel_4.4.4 munsell_0.5.0
[55] crayon_1.3.4
Thanks for the help in diagnosing this issue.
The fix has now been pushed to both the release (1.14.1) and devel (1.15.1) branches, so please upgrade to those versions. I've closed the issue, but if anyone sees a re-occurence on the updated release/devel branches, please re-open!
benjjneb
on 24 Feb 2020
Related issues
FelixErnst
·
10Comments
HarryBatsang
·
11Comments
moleconet
·
3Comments
rebbec
·
9Comments
valdeanda
·
4Comments