Nextflow: Support Azure Batch as additonal executor
New feature - Azure Batch Support
Microsoft Azure is still a growing compute environment. Azure Batch itself does support singularity and docker images and containers. Especially singularity is a unique feature.
Usage scenario
Execute container workload within Azure.
Suggest implementation
add additional executor by using the azure-java-sdk.
tbugfinder
All 15 comments
Related https://github.com/Azure/azure-sdk-for-java/issues/7158#issuecomment-590496687
pditommaso
on 25 Feb 2020
@tbugfinder this would be a useful feature. Many institutes rely on Azure (as an extension to all others office 365 services).
On the other way, Azure also offers a free Kubernete service (you pay for the nodes). Did you already evaluate what would be the advantages of using Azure batch over a more standard Azure Kubernetes service?
arnaudceol
on 27 Feb 2020
Any thoughts on this? from Nextflow team? I agree with the utility of this, my institute requires Azure for any cloud resources!
GHAStVHenry
on 24 Apr 2020
Not in the very short term but there are some chances to have in the second half of the year.
pditommaso
on 26 Apr 2020
@pditommaso just wanted to check in to see if there is any progress here. Thank you!
andersgs
on 19 Sep 2020
Still pending :/
pditommaso
on 23 Sep 2020
@pditommaso to get this to work we would have to use Azure BATCH API
and mimic AWS Batch
vsmalladi
on 16 Oct 2020
I think we are aligned on this 馃槃
pditommaso
on 16 Oct 2020
Great will learn more about Azure batch api
vsmalladi
on 16 Oct 2020
@pditommaso you have documentation on how you worked out gcp or aws implementation?
vsmalladi
on 16 Oct 2020
Hi @vsmalladi
are you trying to develop it? I could support with prereqs but not sure about the nf pieces.
IMHO, sdk docu and https://batch-shipyard.readthedocs.io/en/latest/00-introduction/ should help.
tbugfinder
on 16 Oct 2020
Hi @tbugfinder ,
Yes trying to develop and build it out
vsmalladi
on 16 Oct 2020
@pditommaso you have documentation on how you worked out gcp or aws implementation?
Unfortunately nothing more than the cloud provider docs
pditommaso
on 17 Oct 2020
I'm working with a group that is interested in this feature, just for maximum portability of our Nextflow pipeline, so I've subscribed!
lvclark
on 19 Oct 2020
We are starting to work soon on this. Stay tuned.
pditommaso
on 19 Nov 2020
Related issues
ewels
路
4Comments
apeltzer
路
7Comments
Crabime
路
6Comments
bede
路
3Comments
lindenb
路
3Comments
Most helpful comment
We are starting to work soon on this. Stay tuned.