E3sm: case.submit --skip-preview-namelist fails the run

[tangq]

I am not sure how this option works. Perhaps, I didn't use it correctly.

On theta, I used ./case.submit --skip-preview-namelist to submit the run and got an error and failed the run.

.case.run: error: unrecognized arguments: ARGS_FOR_SCRIPT=--skip-preview-namelist

However, ./.case.run -h does show:
--skip-preview-namelist
Skip calling preview-namelist during case.run
(default: False)

[jgfouca]

That should work. Are you using current master?

[tangq]

@jgfouca , I used yesterday's master (v1.0.0-beta.2-3534-g3918a448f).

[jgfouca]

@tangq , almost forgot the most important question: what machine are you on?

[rljacob]

Theta

[rljacob]

Your mira credentials should work there.

[amametjanov]

@tangq can you try removing this line from case-dir's env_batch.xml:

<batch_env>-v</batch_env>

This is expanding job arguments incorrectly making it look like this

Jobid: 181734
qsub -A ClimateEnergy_2 -t 04:00:00 -n 825 -q default --mode script .case.run -v ARGS_FOR_SCRIPT=--skip-preview-namelist

with unnecessary -v ARGS_FOR_SCRIPT= (Note: case.test is not affected).
@jgfouca do you recall why <batch_env>-v</batch_env> was added to Cobalt systems? Env-flags are different than with PBS schedulers.

Update: it looks like batch_env for Cobalt should be <batch_env>--env</batch_env>.

[tangq]

@amametjanov , I think you found the cause of this issue. For the runs I am doing now, I will just use case.submit. Thanks.

[jgfouca]

@amametjanov , I need to clean up our batch config. I'll fix this in the process.

[amametjanov]

It also looks like by default all tests are run with --skip-preview-namelist, but that doesn't do anything. Below is what needs to be done for Theta and to enable this option both for case.run and case.test:

azamatm@thetalogin5:~/repos/ACME-theta> git diff
diff --git a/cime/config/acme/machines/config_batch.xml b/cime/config/acme/machines/config_batch.xml
index b68261069..d8197a11f 100644
--- a/cime/config/acme/machines/config_batch.xml
+++ b/cime/config/acme/machines/config_batch.xml
@@ -39,7 +39,7 @@
     <batch_query>qstat</batch_query>   
     <batch_submit>qsub</batch_submit>  
     <batch_cancel>qdel</batch_cancel>  
-    <batch_env>-v</batch_env>
+    <batch_env>--env</batch_env>
     <batch_directive></batch_directive>
     <jobid_pattern>(\d+)</jobid_pattern>
     <depend_string> --dependencies</depend_string>
@@ -61,7 +61,7 @@
     <batch_query>qstat</batch_query>   
     <batch_submit>/projects/ccsm/acme/tools/cobalt/dsub</batch_submit>
     <batch_cancel>qdel</batch_cancel>  
-    <batch_env>-v</batch_env>
+    <batch_env>--env</batch_env>
     <batch_directive>#COBALT</batch_directive>
     <jobid_pattern>(\d+)</jobid_pattern>
     <depend_string>--dependencies jobid</depend_string>
diff --git a/cime/scripts/lib/CIME/XML/env_batch.py b/cime/scripts/lib/CIME/XML/env_batch.py
index bc268f021..8da7da2b0 100644
--- a/cime/scripts/lib/CIME/XML/env_batch.py
+++ b/cime/scripts/lib/CIME/XML/env_batch.py
@@ -464,16 +464,20 @@ class EnvBatch(EnvBase):
                "Unable to determine the correct command for batch submission.")
         batchredirect = self.get_value("batch_redirect", subgroup=None)
         submitcmd = ''
-        for string in (batchsubmit, submitargs, batchredirect, get_batch_script_for_job(job)):
+        for string in (batchsubmit, submitargs):
             if  string is not None:
                 submitcmd += string + " "

-        if job == 'case.run' and skip_pnl:
+        if (job == 'case.run' or job == 'case.test') and skip_pnl:
             batch_env_flag = self.get_value("batch_env", subgroup=None)
             if not batch_env_flag:
-                submitcmd += " --skip-preview-namelist"
+                submitcmd += " --skip-preview-namelist "
             else:
-                submitcmd += " {} ARGS_FOR_SCRIPT='--skip-preview-namelist'".format(batch_env_flag)
+                submitcmd += " {} ARGS_FOR_SCRIPT='--skip-preview-namelist' ".format(batch_env_flag)
+
+        for string in (batchredirect, get_batch_script_for_job(job)):
+            if  string is not None:
+                submitcmd += string + " "

         if dry_run:
             return submitcmd

Note however that there are also updates to LID with job id's as noted here and that fails the post_run_check. @jgfouca please take a look at that.

[jgfouca]

@amametjanov , thanks for discovering this.

[jgfouca]

This will be fixed in ESMCI with this PR:
https://github.com/ESMCI/cime/pull/2373