dada2 aws ami?

[johnchase]

Would there be any interest in an AMI for dada2? This could also include phyloseq, or other packages that are commonly used with the microbial ecology R pipeline.

I have found it difficult to get dada2 running on AWS, mostly compiling the c++ packages. While I have been able to get dada2 running on AWS I often still encounter errors.

Having an AMI would be great to remove some issues getting it installed and running on AWS. I don't know if this is of interest to others in the community but figured I would mention it.

[benjjneb]

Yes, there is interest. I haven't moved into the cloud yet, but I'd be happy to help if there are some simple fixes that could make this easier (eg. providing a multithread argument to the fastqFilter functions).

@joey711 probably has some wisdom to share as well, as I think his cloud dada2 pipeline was running on AWS.

[joey711]

I have had a lot of experience using/building on AWS, and dada2 did not give me any unusual problems. The OpenMP error on the linked post is something I have never encountered. I don't remember the exact Linux distro I was using, but it was Ubuntu and vanilla Bioconductor installed. The standard BioC installation worked fine for dada2, as did a devtools::install_local() command before the dada2 release on BioC.

It sounds like the error to which you're referring is not at installation, but upon calling a particular multithreaded IO function from ShortRead package. I didn't see a fix on that thread that would solve this problem from within an AMI even if we made one, because users would need to know to use that modified command to address that particular error (again, which I haven't yet encountered on AWS-builds).

Rather than creating and maintaining one or more public AMIs that have RStudio Server, BioC, dada2, phyloseq installed, it might make more sense to simply post a MRE of the install code for building one. There are already publicly available RStudio Server AMI, and BioC supported AMI.

Unless there's some extra/unusual/tedious step to building dada2 from one of the above AMI releases, it seems like we'd be better off just pointing users to these docs and post the extra few lines of installation code in the same place for convenience.

@johnchase Is there a C++ compile error you can point to? It probably warrants a separate Issue post if-so.

[johnchase]

@joey711 I haven't posted issues with installation here because they are typically related to dependencies, rather than dada2. As I said I have been able to get it running, but if I happen across something again I will post it here.

In terms of the error I linked to I would think that an AMI would solve this problem as it would keep environments consistent, but I may be mistaken.

The AMIs that you mentioned are a good idea, perhaps a short note on the installation page pointing to one of the existing AMIs would suffice. I wanted to put this issue out there basically to see if others were running into problems using AWS. If I am the only one of many it sounds like it would not be necessary. On the other hand if many new users (who are unfamiliar with R like myself) run into issues it could be a good way to get them started.

[benjjneb]

Given the pre-existing Bioconductor AMI, which has most of the dada2 dependencies, I do see the value in the @joey711 suggestion of building a MRE for installing dada2 (+phyloseq?) on AWS. Avoids some maintenance that the Bioconductor AMI should take care of, although not as simple to use as an all-in-one AMI.

Either way, I see real value in this. I could see myself, as an AWS novice, using it.

[joey711]

@benjjneb makes sense to me.

@johnchase I don't think you're the only one. And it is a great idea to have _something_ to support and document dada2/phyloseq on AWS. The only thing I'd push back on is hosting/maintaining a separate suite of AMIs (one for each region) ourselves when BioC is already doing most of the work, and will be better documented for exceptions like the one you mentioned. For instance, I'd be surprised if there wasn't something on the BioC forums addressing your ShortRead OpenMP error already. Anyway, I think a small addendum to the BioC/AWS instructions should suffice, and be region-agnostic.

I'm going to the BioC devel day tomorrow, so this is well-timed. Maybe I can post a prototype example here tomorrow, and you guys can tell me if you have any problems with it.

[audy]

I wrote a Dockerfile for dada2 so that I could run on AWS. Installation was pretty straightforward: (1) install R. (2) install bioclite + dada2.

[joey711]

Nice! You beat me to it. I ended up being too engaged at BioC-devel to do something else O:-)

[joey711]

I just looked at the file (I'm a docker novice):

FROM ubuntu

RUN apt-get update && apt-get install -y \
  r-base

RUN echo 'source("https://bioconductor.org/biocLite.R"); biocLite("dada2")' | R --no-save --vanilla

That's awesomely concise!

Thanks again, @audy

[audy]

Actually, I had to add a few things. I'll come back with an updated and tested version.

[audy]

it ended up being a little more complicated to get everything: https://gist.github.com/audy/ac42c28d8f547bc752d79e721ba2f50f

@johnchase running the two apt-get commands followed by ./install-dada.R on a fresh install of Ubuntu should work.

[zachcp]

On a related note, I've packaged DADA2 for use with Bioconda, a repository of precompiled binary packages used with the conda package manager that aims for cross-platform support and non-root installation. The buildbot uses CENTOS5 and the binaries are intended to be widely (universally?) compatible. There are also builds for Win/OSX but the mileage there varies.

I haven't checked this on AWS but any it should be possible to get DADA2 + Phyloseq installed from scratch on any 64bit Linux image with the following:

``` [bash]

download and install Miniconda

See: http://conda.pydata.org/miniconda.html

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

add some repos and install dada2/phyloseq

conda config --add channels r
conda config --add channels bioconda
conda install bioconductor-dada2
conda install bioconductor-phyloseq
```

I think one nice advantage to using conda is that you can specify your entire environment in a single config file. This can include things like vsearch, seqtk, etc.

[johnchase]

👍 on the conda channel, that is awesome, no issues on the install. I wish I knew this was out there earlier!

[benjjneb]

I know more people are interested in a straightforward description of getting a dada2 amazon instance up and running. My understanding is that the stating point would be the Bioconductor AWS AMI, but would any of the AWS users be willing to write up a simple how-to on getting from that to running dada2 for the not-overly-technically-inclined?

Would be great to add to the website documentation.

[crfitzpat]

First off - DADA2 is brilliant and I’m really blown away by its robustness, thanks to all who made it work! I’m not very savvy when it comes to high performance computing but I’ll share my journey of running DADA2 on an EC2 instance. It might be useful for trad ecologists out there like me.

I had de-multiplexed fast files from 4 MiSeq runs. I multiplexed 192 high diversity samples per run and on average had 20 million reads per run.

Here is what I did:

1. Set-up an Amazon Web Service account
2. Make a key pair (go to your EC2 dashboard) and save it somewhere on your computer
3. Go to Louis Aslett’s website and click on the RStudio AMI link for your region: http://www.louisaslett.com/RStudio_AMI/
4. Launch an instance of you choice. A few notes on this: make sure you use a security group with incoming SSH and HTTP protocols enabled. Make sure you add additional EBS storage space to your root volume, 200 GiB worked for me (it will be one of the options as you are guided through instance configuration). Also I used m4.2XL instances and I went through the entire pipeline from filtering-taxonomy assignment in about 10 hours = <$5 for one of my MiSeq runs. Maybe there is a better option but this instance seemed like a good balance between memory and processor number.
5. Open a terminal window and run ssh protocol to log into your new virtual machine and change your RStudio password. This post explains how: http://strimas.com/r/rstudio-cloud-1/
6. Before entering the RStudio server in your browser run a screen command in terminal so that your analyses will keep going even if your local machine is disconnected:http://stackoverflow.com/questions/35945506/rstudio-server-on-ec2-not-persistent-when-closing-browser-tab
7. Follow steps 1 and 2 from the DADA2 installation guide (this takes a couple minutes). I had no problems with getting the latest version of DADA2 to work (decline updates). The AMIs are updated regularly though.
8. Aslett’s AMIs come with a built in package that sync’s your virtual machine with dropbox (this can take a few minutes). I dumped all my fastq files into separate db folders for each run. Using dropbox was really easy to move objects in and out of R.
9. Start your pipeline.

What was impossible on my local machine I finished using 4 parallel instances in 10 hours for about $20.