E3sm: Discussion: what project usage information would be useful?

[worleyph]

I am writing scripts to pull information out of the performance_archive to summarize, among other things, project utilization. There is a lot of information there, and I would appreciate some suggestions on what would be most useful.

I started on NERSC, but that is most complicated environment (multiple machines, multiple allocations), so I moved this over to Anvil (the simplest environment).

I "archive" the performance_archive every two weeks, and have been looking at one of these snapshots for development purposes. The summary I have currently is:

 (2017-12-2 20:08:32) - (2017-12-17 15:16:02)
 ---------------------------------------------
 Jobs completed: 51
   Total days simulated: 48945
   Total node hours: 60262.4247222222 (70.7056490933031 % of maximum)
 Jobs failed: 35
   Total days simulated: >= 2373 (19 jobs)
   Total node hours: >= 3451.56111111111 (4.04970211323608 % of maximum)

So, question for POCs and project managers - what other information would be useful here? Note that this script can be applied to the current "active" performance archive as well, though differentiating between failed and currently running jobs has not been implemented in the current version of the Anvil script.

I also have detailed per job summaries, for example,

 Job 1800299 (171215-091933) submitted at 2017-12-14 11:49:03:
 User: maltrud; Account: ACME; Queue: acme; Case:  20171206.newCongel.A_WCYCL1850S.ne30_oECv3.anvil
 Nodes: 50; PPN: 36; Cores: 1800; Time Limit: 129600
 Compset: A_WCYCL1850S; Resolution: ne30np4_oEC60to30v3_ICG; Source: v1.0.0-beta.2-1435-g4f3a0e5
 Queue Wait: 77419 seconds (0 days 21 hours 30 minutes 19 seconds)
 Simulation: 1825 days
 Job  Total  : 92616 seconds
  - Job Init   : 13 seconds
  - Case Ovhd  : 62.6330000000016 seconds
  - Run Init   : 110.692 seconds
  - Run Loop   : 92429.520 seconds (4.67 SYPD)
  - Run Final  : 0.155 seconds

I plan on summarizing the per job information for each case (instead of each job), to capture performance and performance variability, but am still working on this.

[philipwjones]

@worleyph This looks great. The piece I'd be most interested in is captured in your last sentence: ...summarizing the per job information for each case (instead of each job), to capture performance and performance variability...

[worleyph]

Latest experiment ....

 ...
 USER: maltrud
   CASE: 20171122.runoffNN.A_WCYCL1850S.ne30_oECv3.anvil

     COMPSET: A_WCYCL1850S; RESOLUTION: ne30np4_oEC60to30v3_ICG; NUMNODES: 51
       Jobs completed: 1
       Days Simulated: 1825
       SYPD for TOTAL: 4.65 (average) 4.65 (max) 4.65 (min)
     COMPSET: A_WCYCL1850S; RESOLUTION: ne30np4_oEC60to30v3_ICG; NUMNODES: 51
       Jobs failed with no simulation days: 1
       Jobs failed with simulation days: 1
       Days Simulated: 1274

   CASE: 20171204.ksno+emis.A_WCYCL1850S.ne30_oECv3.anvil

      COMPSET: A_WCYCL1850S; RESOLUTION: ne30np4_oEC60to30v3_ICG; NUMNODES: 50
       Jobs completed: 10
       Days Simulated: 18250
       SYPD for TOTAL: 4.688 (average) 4.73 (max) 4.67 (min)
     COMPSET: A_WCYCL1850S; RESOLUTION: ne30np4_oEC60to30v3_ICG; NUMNODES: 51
       Jobs completed: 1
       Days Simulated: 1825
       SYPD for TOTAL: 4.7 (average) 4.70 (max) 4.70 (min)
     COMPSET: A_WCYCL1850S; RESOLUTION: ne30np4_oEC60to30v3_ICG; NUMNODES: 51
       Jobs failed with no simulation days: 1
       Jobs failed with simulation days: 1
       Days Simulated: 1099

   CASE: 20171206.newCongel.A_WCYCL1850S.ne30_oECv3.anvil

     COMPSET: A_WCYCL1850S; RESOLUTION: ne30np4_oEC60to30v3_ICG; NUMNODES: 50
       Jobs completed: 10
       Days Simulated: 18250
       SYPD for TOTAL: 4.659 (average) 4.69 (max) 4.64 (min)

Now moving back to NERSC, and also looking at look at performance across USER and CASE for same compset, resolution, and numnodes (though this may miss out on other differences).

[sarats]

@worleyph When you are ready to share, could you please point to your Perl scripts and instructions on running the script to generate summary?

[worleyph]

I have something working on both NERSC and Anvil now. If you want to try them out:

NERSC:

 % cd /project/projectdirs/acme/OLD_PERF/performance_archive_acme_2018_01_14
 % perl nersc_summary_report.perl --help

Anvil:

 % cd /lcrc/group/acme/OLD_PERF/performance_archive_anvil_acme_2018_01_14
 % perl anvil_summary_report.perl --help

You can also run these in the "working" performance_archive subdirectories (or any of the older snapshots in OLD_PERF).

Once I have working scripts on Theta and Titan (and Mira?) I'll probably have a required --center parameter and only have one script. The center-specific information has to do with different otuput from qstat on the different system and sqs at NERSC.

These scripts provide some "functionality" but I have made little attempt to clean them up or to optimize them yet, and I apologize to anyone trying to read them. I am also not finished adding options, and would appreciate any suggestions.

Things I am considering include:

a) adding a filter to summarize only simulations that requested more than some indicated number of simulation days (to filter out short test jobs).

b) documenting location of inputdata (for detailed job information) or selecting based on this? Cori-KNL seems to care about this?

c) doing something with the per day timing information in cpl.log? Not sure what to do - max, min, average, median, ??? Since periodic output skews this, not sure what statistic it makes sense to extract.

[worleyph]

Now have a single script with support for multiple systems:

 $ perl e3sm_summary_report.perl --help
 Usage:

 perl e3sm_summary_report.perl

 within the top level of a performance_archive at NERSC, 
 at ALCF, at OLCF, or on Anvil.

 Purpose:
 Script to summarize throughput for E3SM jobs at NERSC 
 run Cori-Haswell, Cori-KNL, or Edison, jobs at ALCF run 
 on Theta, jobs at OLCF run on Titan, or jobs run on Anvil.

 Optional arguments:
   -h, --help      show this help message and exit 
   -u, --user [user name]
                   limit summary to jobs submitted by indicated user
   -a, --account [account name]
                   limit summary to jobs charged against indicated account
   -s, --system [{anvil, cori-haswell, cori-knl, edison, theta, titan}]
                   limit summary to jobs run on indicated system
   -r, --runlength [number of simulation days]
                   limit summary to jobs requesting to run more than
                   indicated number of simulation days
   -d, --detail [{0,1,2,3}]
                    0: basic summary (default)
                    1: also summarize by (compset,resolution,number of nodes)
                    2: also summarize by compset for each user
                   >2: also summarize per job

I still need to look at how to summarize cpl.log information of per day timings, and will see if I can differentiate between jobs that fail because they ran out of time and those that failed for other reasons.

I haven't yet looked at summarizing performance data from Mira. There won't be any generated going forward, but it might be useful to be able to look at the performance of old runs?

[worleyph]

I added one more capability ... the script now looks through the per simulated day cost (from cpl.log) and determines median/minimum, average/median, and maximum/minimum, as one way to look for performance perturbations. (Other suggestions would be appreciated.)

I also added two new optional paramters to the script:

   -m, --medminratio [ratio of dt median to dt min]
                   output job information when median time
                   to calculate a simulation day exceeds the
                   minimum time by more than the indicated ratio
   -p, --avgmedratio [ratio of dt average to dt median]
                   output job information when average time
                   to calculate a simulation day exceeds the
                   median time by more than the indicated ratio

If a job exceeds one of these thresholds, it outputs an "Exceeded Threshold" comment and also writes out the full array of simulation day cost times. We can use this to identify jobs that may to worthwhile looking at more carefully. (If you can think of better names for these options, please add a comment.)

Note that I did not add a threshold on the maxmimum/minimum. If this is big enough, it will affect average/median, but I can add this if people think that it would be useful. In the meantime, you can just

 % perl e3sm_summary_report.perl -d 3 | grep Variability

to get a feeling for what performance variability is typical and what is not.

[worleyph]

Decided that medminratio and avgmedratio aren't sufficient ... looking at defining high cost simulation days as some ratio greater than the minimum and counting how many of these occur in each month. First glance, this provides more information, but still working on this.

[worleyph]

Added ..

   -c, --highcostdef [dt cost relative to minimum]
                   increase in simulation day cost as compared
                   to the minimum that defines a high cost 
                   simulation day (default is 2.0)
   -n, --highcostnum [{0,1,2,...,31}]
                   number of high cost days in 31 consecutive days
                   that will trigger output of job information
                   (default is 32)

so can get output like ...

     DT Performance Variability:
       Median/Min = 1.06195393169182, Average/Median = 1.05310396409873, Max/Min = 4.90945194598888
       (35 out of 5840) >= 2*Min; Largest number of high cost days in 31: 2

Output could be worded better, but this run had one occurrence whether a simulated day cost more than twice the minimum and was not an end-of-month day. End-of-month was around 4.9X more expensive than the minimum.

The E3SM project runs too many different compsets, node counts, etc. to make it simple to identify interesting cases to look at (and "natural" variability is sensitive to compset and resolution as well as to node count and system), but the script helps some. I'll try to start reporting some performance-like data on the allocation usage confluence page, to see if anyone is interested.

[ndkeen]

I'm still trying to parse this in general, but could you _please_ always add the jobid as this simple integer can be useful, always tied to the job timestamp, and not always easy to find.

[worleyph]

Sorry - not sure what you are asking for. Add it to what? Job id is part of the detailed output. With your recent change to add job id to lid, it shows up multiple times now.

[ndkeen]

OK, I just didn't see jobid in the output you pasted above with the comment starting: "Latest experiment..."

[worleyph]

Please try this yourself. Recent comments have only been snippets, to highlight the new stuff.

[worleyph]

Also, the "latest experiment" output was a summary of multiple jobs, so there is no single job id. This reflects the heirarchy of the performance_archive: user/case/lid .

[worleyph]

I've stopped working on scripts at the moment. They do what I want (for the moment). Seems like I am the only user at the moment. If anyone wants to look at these, just send me a note. They are also in the performance_archive subdirectories for each system/allocation that we are currently tracking.