Aws-parallelcluster: Not able to schedule GPU jobs in slurm using parallel cluster

[Palak-15]

Environment:

• AWS ParallelCluster : aws-parallelcluster-2.4
• OS: centos 7
• Scheduler: slurm
• Master instance type: t2.micro
• Compute instance type: p2.xlarge

Bug description and how to reproduce:
AWS parallel cluster does not create/contain gres.conf file , due to which slurm is unable to schedule any gpu jobs.

Additional context:

• gres.conf file is missing

[sean-smith]

Not only gres.conf, but also accounting is required for GPU scheduling. I'm going to mark this as a feature request.

[Palak-15]

Hello @sean-smith ,

Can you tell me that when will this gonna update?

[demartinofra]

We don't have an exact timeline for that but most likely this feature is going to be addressed in one of the upcoming releases of ParallelCluster.

In order to better design support for this feature could you please share how you would like to submit gpu related jobs (e.g. sbatch -G 2 -N 2)

[Palak-15]

Hello @demartinofra ,

You can run gpu related jobs by running this command -
sbatch -N 2 -n 4 --gres=gpu

[barrettabelmode3]

Hi

This is a month or so since the last reply. @sean-smith - Is this work underway or if it was on the backburner ? I was looking to do myself, but I didnt want to create duplicate work. my usecase is exactly the same. Commercial molecular dynamics packages rely on a gres gpu resource and handle the multi gpu scheduling ( https://www.schrodinger.com/kb/1844 )

( I am willing to expedite this with beers to the team as well lol )
Thanks for all your work.

[sean-smith]

@barrettabelmode3 We've done the work and it'll be released shorty (I swear :-) in the next release: See https://github.com/aws/aws-parallelcluster-cookbook/pull/409

You can buy @rexcsn that beer next time you're in Seattle!

[barrettabelmode3]

Thanks If your at reinvent that would work too... I need accounting so this will solve alot for me. Is there a RTFM / cheatsheet for trying cookbook409. Have users excited for the next release.

[rexcsn]

@sean-smith @barrettabelmode3 @Palak-15 Just some clarification, the next release will support submitting jobs with GPU options. Contrary to to our early understandings, accounting is NOT required to setup GPU functionalities in slurm.
As a result, the next release will not include accounting :(

We are working on prototypes, but there is no concrete release timeline for slurm accounting as of now.

In the meantime, if you want to experiment with setting up accounting with an existing cluster, here are some tips we have found useful:

• Install mysql packages
• Slurm scans for installed packages when building, and skips plugins that do not have the required packages installed. So after installing mysql we need to rebuild slurm from tarball to get the accounting storage plugins
• After rebuilding slurm we should see accounting_storage_mysql.* in /opt/slurm/lib/slurm
• Follow the official slurm tutorial: https://slurm.schedmd.com/accounting.html

[barrettabelmode3]

@rexcsn - Thanks for this. I yanked out a previously written ansible playbook as a post run to do this but it is exceedingly high on the hack-o-meter.

@Palak-15 happy to provide it if you really need i.e. GPU and accounting

Thanks either way gents @rexcsn @sean-smith

[demartinofra]

A quick update: GPU scheduling is now supported in ParallelCluster 2.5.0

I'm going to resolve this issue that was related to GPU scheduling and open a new one to track accounting support: https://github.com/aws/aws-parallelcluster/issues/1494