Aws-parallelcluster: Accounting Support in Slurm

[demartinofra]

Opening this in order to track requests related to Accounting support in Slurm.

Quoting a reply from https://github.com/aws/aws-parallelcluster/issues/1318:
We are working on prototypes, but there is no concrete release timeline for slurm accounting as of now.
In the meantime, if you want to experiment with setting up accounting with an existing cluster, here are some tips we have found useful:

• Install mysql packages
• Slurm scans for installed packages when building, and skips plugins that do not have the required packages installed. So after installing mysql we need to rebuild slurm from tarball to get the accounting storage plugins
• After rebuilding slurm we should see accounting_storage_mysql.* in /opt/slurm/lib/slurm
• Follow the official slurm tutorial: https://slurm.schedmd.com/accounting.html

[dapritchard]

In the meantime, is it possible to enable accounting without reinstalling Slurm by using the following approach as described in https://slurm.schedmd.com/accounting.html?

Storing the accounting information into text files is very simple. Just configure the appropriate plugin (e.g. AccountingStorageType=accounting_storage/filetxt and/or JobCompType=jobcomp/filetxt) and then specify the pathname of the file (e.g. AccountingStorageLoc=/var/log/slurm/accounting and/or JobCompLoc=/var/log/slurm/job_completions). Use the logrotate or similar tool to prevent the log files from getting too large. Send a SIGUSR2 signal to the slurmctld daemon after moving the files, but before compressing them so that new log files will be created.

When I follow these directions (and place the logs in a shared volume) then I see output written to the completions file, but not the accounting file.

[rexcsn]

Some update on this issue.

Slurm scans for installed packages when building, and skips plugins that do not have the required packages installed. So after installing mysql we need to rebuild slurm from tarball to get the accounting storage plugins

This step is only required on Ubuntu OSs, other OSs should already come with the accounting plugin and should not require rebuilding slurm from source.

@dapritchard Thank you for pointing out how to setup accounting with text file. I am able to enable simple accounting with text files by changing AccountingStorageType=accounting_storage/filetxt and AccountingStorageLoc=/var/log/slurm/accounting in slurm.conf.
I am also able to record job completion data by changing JobCompType=jobcomp/filetxt and JobCompLoc=/var/log/slurm/job_completions.

Job data is recorded in both accounting and job completion files, please double check to make sure you have the above parameters and right permissions for the directories used for accounting and job completion.

Note that in the past we have hit issue when trying to setup GPU scheduling with accounting_storage/filetxt, so if you are using GPU scheduling, simple accounting with text files might not be an option, and you should try to setup accounting with slurmdbd.

[rexcsn]

Some updates:

We recently released a blog post detailing how to setup Slurm accounting using Amazon Relational Database Service(RDS) with ParallelCluster, link is here: https://aws.amazon.com/blogs/compute/enabling-job-accounting-for-hpc-with-aws-parallelcluster-and-amazon-rds/

In general, you can follow the same procedure to setup Slurm accounting using a tradition MySQL database with ParallelCluster; using Amazon RDS just simplifies the process of setting up the MySQL database.

There is currently no plan to enable Slurm accounting natively with ParallelCluster, so please refer to the blog post as a guide to setup accounting.

We will close this issue now, but feel free to drop a comment if you would like to see Slurm accounting enabled natively with ParallelCluster so we can continue gauge interest for this feature. Thank you.

[dapritchard]

@rexcsn although these instructions are incredibly clear and detailed, I've still never been able to get accounting to work over multiple tries with different AMIs. Because of these difficulties, I would find it very helpful if accounting could be enabled natively with ParallelCluster.