Dada2: Integer overflow

[FelixErnst]

Hi,

Similiar to #432 I get a warning on an integer overflow when running dada

dadaForward <- dada(filesForwardFiltered, err = errForward,
                                      multithread = TRUE)
...
## Warning in rval[, 1:ncol(tt)] + tt: NAs produced by integer overflow

The datasets constists of ~2300 samples and it occurs 4 times for the forward reads and three times for the reverse reads.
The location is quite easy to pin point: accumulateTrans

https://github.com/benjjneb/dada2/blob/82f4660b450038413e169a5a8e6a8b103fa0e067/R/errorModels.R#L379

I guess a switch to a double matrix might solve the issue here as well.

Please let me know, what you think and wether this a problem for the analysis. Thanks!

Felix

[benjjneb]

Thanks for the report. Datasets are only getting bigger, so we should probably try to fix this.

Good news though, this should have no impact on your output. This only effects the re-estimation of the error rates that is performed by default, which you aren't using (I assume you are using the error rates from learnErrors).

[FelixErnst]

Thats it correct. Would you time to explain the difference?

I am looking at the dada function, in which accumulateTrans is used, but I cannot follow, what is happing in there and what is not.

For me it looks like this:

Since the default error function is used, cur gets evaluated.
https://github.com/benjjneb/dada2/blob/f1e53cd1f0660d632a37df387a8475f16a99fcbb/R/dada.R#L374
Why is err not used then? err gets transfered to rval2, which is the return value.

So NA get generated and end up in the result making it potentially inaccurate or invalid

[benjjneb]

err is in the dada-class return object as $err_out. But, that post hoc re-estimation of the error-rates isn't used by any downstream functions. Only the inference of the sample composition is used, and that depends on the input error rates, not the post-hoc re-estimation.

This would be a problem if learnErrors used all the samples in your dataset rather than a subset of samples with sufficient (nbases) information, but that's not how learnErrors operates by default, and its almost never useful to push such large amounts of data through learnErrors to reach this behavior, which is why it hasn't been a priority to fix.

[FelixErnst]

If you want me to test a possible fix, let me know.

[benjjneb]

Sure. Would need confirmation it gives the exact same output (outside of the overflow cases) to incorporate into the package though, as the error-rate estimation is central to the method (learnErrors).

[FelixErnst]

Can do. Just let me know, where to pull from.

[benjjneb]

We devel on master https://github.com/benjjneb/dada2/tree/master here at dada2, so target pull requests there.

Releases are branched off of master (rather than the other way around as is more commonly done).

[FelixErnst]

Ok. let me know, if there is a fix and I will test it.

fyi: https://github.com/joey711/phyloseq/issues/1361

[FelixErnst]

@benjjneb just to clarify: Would you have time to fix this?

[benjjneb]

It's an open bug so it will get fixed eventually. I don't see it as a high priority though, since it doesn't have any affect in any practical situation yet described, so not sure it will happen soon.