Moto: Support for AWS Batch API

[rofinn]

I don't currently have time to work on this, but it'd be nice if moto could support the AWS Batch API.

[gkiar]

Recently, @JackDanger added support for all AWS managed policies (addressing https://github.com/spulec/moto/issues/1118), including BatchServiceRoles which I have been using in my tests.

What would be the engineering-cost to support the Batch API, at least basic functionality therein (such as creation of env/queue/jobdef/jobsubmission)?

An example of my current (at time of posting) test setup and failure can be found here, and in the issue linked above are instructions for how to install the code and run this (though, the commit id has changed to 4afd492fc849b6a0cb18fa4a16e04a5742726c50)

[JackDanger]

@gkiar Because all of us contributing to Moto have limited time the way it's worked best is if somebody implements just the slimmest, most barely-working version of some API endpoints that they need and then leaves TODOs for other folks to pick it up from there. It's seemed like everybody has time to push things just one step forward but none of us quite have the energy to spend a day (or more) tackling a full integration of many API endpoints.

So if you are interested in creating just the barest first pass at implementing AWS Batch (maybe using #1172?) then the other folks on this issue can step in and push it along when you get tired. Teamwork for the win :)

[gkiar]

Sounds great - I'll chug along on this when I find the bandwidth and open a PR :)

[terrycain]

Am working on this i #1197, if you have any code feel free to jump in on the PR. @rofinn Would be helpful if you had a couple of examples / test cases, mainly as I dont use batch at all.

[rofinn]

So the primary workflow I'm using is:

1) describe jobs
2) submit jobs
3) fetch logs from cloud watch logs

I'm fine assuming that the resources have already been setup by a cloudformation template.

[terrycain]

Ahh so your expecting the cloudwatch to create the batch environment, you mind giving me a cf template that would get you to stage 1 as i'll have to add cf support too

[rofinn]

No, cloudwatch wouldn't create the batch environment, but when you submit a batch job the STDOUT and STDERR from the ECS task will be sent to cloud watch logs. The "logGroupName" is "/aws/batch/job" and the "logStreamName" can be extracted from the describe jobs response (of running jobs) at resp["jobs"][0]["container"]["logStreamName"].

It's a bit verbose, but this is more or less the kind of template we use for deploying AWS Batch clusters. https://s3-us-west-2.amazonaws.com/cloudformation-templates-us-west-2/Managed_EC2_Batch_Environment.template

[terrycain]

Sorry I meant cloudformation would create the batch environment... which is whats in the template :smile:

[terrycain]

As for when the cloudformation would work, it'll probably have to wait till ive implemented the majority of batch, then the cf integrations, but its slowly getting there.

[terrycain]

1197 - linking issue to PR for my reference

[rofinn]

FWIW, thank you for working on this. I look forward to having better tested code as a result :)

[gkiar]

@terrycain as another data point, the workflow I'm using boto3 for is the following:

1. ensure roles exist with correct privileges (IAM covers this now thanks to @JackDanger's PR)
2. create compute environment if no exact match exists already else get ID of the match
3. repeat 2 for queue
4. repeat 2 for job definition(s)
5. submit job(s)
6. fetch log(s) using job ID(s)

[terrycain]

When im finished the batch bit, 1-5 should work. I need to look into how our ecs implementation works to see what it will do with 6

[terrycain]

Also what sort of jobs are you submitting? What do the Ids actually look like, and how are they represented in cloudwatch

[gkiar]

@terrycain I'm not sure I understand your question - sorry for my ignorance. What do you mean by "sort of jobs"?

In Cloudwatch, my logs are accessible under /aws/batch/jobs at: jobdefname/default/4ac45446-a511-4b6c-a9fb-02df93a4de83 (obviously, the id is an example id).

[terrycain]

Thanks, sorry was being an idiot, the jobs are just tasks to run containers so technically what they do is irrelevant as long as their output goes to c/w

[terrycain]

@JackDanger @thehesiod Thoughts, am just about to implement the SubmitJob part for AWS batch.

Was thinking of essentially running docker like the lambda does as batch jobs need to specify a docker image anyway. The only issue I see is, if people run their normal images they might try and contact aws directly... I could just pass in bogus access key and secret key to the container which would nerf that but then we have other issues to deal with.

[thehesiod]

ya I think you want to do it that way since it seems like it's all docker based anyways.

ya it's a rather daunting task because the image can be running an AWS SDK in any language. I think if they're using moto the client using moto would have to figure out how they want to cause the AWS calls to go to moto, in my blog I talk about how I achieved that...It will be easier once proxy support is added to moto...or perhaps adding my patch utilities to moto.

speaking of which, I think moto should be switched to be based on asyncio (aiohttp + aiodocker) so that multiple requests can occur in parallel. Right now once an lambda is triggered, the moto server is blocked until the lambda finishes.

[terrycain]

That explains why the async invoke of a lambda isnt async, that or its doing some wizardry that I'm missing. Whilst we're in the sync land, I suppose we could just make use of threading as long as its done well enough it should be alright.

I'm up for asyncifying moto @JackDanger you alright with this?

[thehesiod]

ya threading is possible as well, but more dangerous :)

[terrycain]

Yeah agree with that. Though I reckon it should be alright if we use daemon threads. Will test to see what docker does if thread is killed whilst a container is running.

[terrycain]

@gkiar Mind giving me a simplified version of what your doing with boto3?

[gkiar]

Hey @terrycain - I'd written a Jupyter Notebook a while back that does essentially what I need. It requires an admin-level credentials.csv, and shouldn't assume that configuration has or hasn't happened, but will check and either do pieces that were missing or return ARNs for things that aren't. Please let me know if I should simplify this more for you.

[terrycain]

Thats great thanks ;-)

[terrycain]

Progress :smile: I have a proper simple implementation... which works... but only runs an alpine container spamming echo messages :smiley:

[terrycain]

Should have basic support in #1197, it has a hardcoded containert for now which writes hello world to cloudwatch.

Cloudformation should also work

[terrycain]

If you get any issues then report them here

[gkiar]

Amazing, @terrycain - thanks! I'll try to start playing with this soon and will report back. Really appreciate your work!

[spulec]

Closing this with the merging of https://github.com/spulec/moto/pull/1197