E3sm: current E3SM master is broken on Summit

[worleyph]

A fresh checkout of E3SM is failing on Summit. There are two issues.

1) A system change (?) now requires that we load netcdf and netcdf-fortran after we load spectrum-mpi.

2) Even with this change, jobs fail at runtime, almost immediately. For some reason, .case.run is not visible when the jobs is launched. E.g.

 $ ./case.submit 
 ...
 Submitting job script bsub -q batch -W 02:00 -P CSC190ACME .case.run --resubmit
 Submitted job id is 96451
 Submitted job case.run with id 96451

$ ls -ltr
 ...
 Not_Specified.96451

$ cat Not_Specified.96451
 /ccs/home/worley/.lsbatch/1528413328.96451: line 8: .case.run: command not found

 ------------------------------------------------------------
 Sender: LSF System <lsfadmin@batch1>
 Subject: Job 96451: <Not_Specified> in cluster <summit> Exited
 ...

This works fine with acme-split-2018-05-18-74-g53a912d3c .

[worleyph]

Note that for the working version of master, the output from ./case.submit is ...

 ...
 Submitting job script bsub   -q batch -W 00:30 -P CSC190ACME .case.run 
 Submitted job id is 96446
 Submitted job case.run with id 96446

Note the absence of the --resubmit flag.

[sarats]

Just noting that the resubmit option is added in the recent CIME merge:
Relevant changed files:
https://github.com/E3SM-Project/E3SM/commit/2873c2005452dcf97abcd7d43f1ae4ab5bb91485#diff-675d8b79b55d21adf06d4aef6c64bc5e
https://github.com/E3SM-Project/E3SM/commit/2873c2005452dcf97abcd7d43f1ae4ab5bb91485#diff-34cfd5af7fac190878cbfc6590ad6002

I was stuck at an all-day workshop today, will try to dig further tomorrow.

[sarats]

Status update:

I can reproduce this problem with today's master too. It still can't find .case.run and wrote output to Not_specified.98232.

On a hunch, I submitted .case.run.sh directly using
bsub -q batch -W 0:15 -P CSC190ACME .case.run.sh

At least I got output and error files like e3sm.stderr.98386. It didn't load modules. So I added

source .env_mach_specific.sh

to .case.run.sh. Trying to see if I can run this manually.

[worleyph]

@jgfouca and @sarats , I modified CIME (lib/CIME/XML/env_batch.py), as follows

         ...
         if job in ["case.st_archive", "case.run"]:
             if job == "case.st_archive" and no_batch:
                 supported["resubmit"] = "--resubmit"
             else:
 #pw               supported["submit_resubmits"] = "--resubmit"                                                                
                 supported["submit_resubmits"] = ""
         return supported
         ...

to remove --resubmit from the batch submission command. After doing this, my Summit test job worked fine. So ...

a) bsub is not successfully parsing

 bsub -q batch -W 02:00 -P CSC190ACME .case.run --resubmit

but does work with

 bsub -q batch -W 02:00 -P CSC190ACME .case.run

Sounds like something different is needed to pass --resubmit to .case.run as part of the bsub invocation.

b) What does --resubmit do? And why is it now the default?

Thanks.

Pat

[sarats]

@worleyph Thanks for figuring this out Pat!

I tried to see if quotes help but that didn't work either.
bsub -q batch -W 00:10 -P CSC190ACME .case.run "--resubmit"

Anyway, just got the email that they are ending 'Summit Early Access', so no more jobs from today and we need to copy out any files we care about from AlpineTDS (project dir/scratch) by Friday 5 pm.

[worleyph]

@sarats , I have not received any notification that I need to copy things out of the file system. Just checking, does this include /gpfs/alpinetds/proj-shared/csc190/e3sm ?

[worleyph]

@sarats, will we have a mechanism for further E3SM porting to Summit and for testing?

[sarats]

@worleyph I have forwarded you the email I got.

Unfortunately, it includes /gpfs/alpinetds/proj-shared/csc190/e3sm and anything else under /gpfs/alpinetds.

Right now, access is limited to OLCF personnel working on acceptance. If/Once our early science proposal is accepted, we will potentially get access once the machine is accepted (timeline is unclear).

[jgfouca]

@worleyph , --resubmit tells case.run to do the resubmission if RESUBMIT is set to >0.

[jgfouca]

@sarats , didn't I have to give you the --allow-pnl option because summit cannot pass any arguments to submitted scripts?

[sarats]

@jgfouca That's for create_test if I recall correctly.

[jgfouca]

@sarats , yes, I'm just remember that this is the machine where you can't pass flags to submissions.

[jgfouca]

@worleyph , this PR made the flag necessary:

https://github.com/ESMCI/cime/pull/2511

[sarats]

@jgfouca Just wondering, can this be done another way: option in env_run.xml etc. instead of a command-line argument?

[mfdeakin-sandia]

I originally had implemented the PR that way, but it was decided adding xml variables wasn't the right way to pass arguments to the script. CIME has another mechanism for passing arguments to scripts using an environment variable; I can look into getting it to use that instead on summit?

[jgfouca]

@mfdeakin-sandia , that would be great if you have some time.

[worleyph]

@sarats, adding the additional information we discussed by e-mail. Access to Summit will continue for CAAR projects (which we are one), so removing files and data from /gpfs/alpinetds is not required by this Friday (though this is essentially scratch space, so backups are important in any case). When acceptance is over, the CAAR projects will again be able to submit jobs, but this may not be until after June 30.

[worleyph]

@jgfouca , @mfdeakin-sandia , I'm probably not understanding what is going on ... but if --resubmit is the default for all E3SM jobs, perhaps it should have been made it the default within CIME, and then instead have a flag to disable resubmission? (--no_resubmit, --ignore_resubmit, ... ???) Seems like the most common action should be the default. When would we run without --resubmit?

[jgfouca]

@worleyph , I was thinking the same thing. @mfdeakin-sandia , can you comment to this?

[mfdeakin-sandia]

That's a fair point, and would make it so you can at least run a case, but doesn't solve the underlying issue, which is relatively straightforward to address (in a slightly hacky LSF specific manner).
I'll add that change in the PR I make for this.