E3sm: Can't continue job from someone else's directory

[PeterCaldwell]

For big production runs which could take months to complete, it would be nice if one person could start the run, then someone else could continue it (e.g. if the first person went on vacation or simply needed a break). This capability currently doesn't work.

To test this functionality, I had Az give me group write permissions for /projects/ClimateEnergy_2/azamatm/ACME_simulations/20180404.A_WCYCL1950_HR.ne120_oRRS18v3_ICG.theta , then issued "./case.submit --no-batch" from case_scripts/ (by the way - ./case.submit --help has no mention of the --no-batch option). I got the following error:

    Calling /gpfs/mira-home/azamatm/repos/ACME-theta/cime/src/drivers/mct/cime_config/buildnml
Traceback (most recent call last):
  File "./case.submit", line 76, in <module>
    _main_func(__doc__)
  File "./case.submit", line 73, in _main_func
    batch_args=batch_args)
  File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 127, in submit
    custom_success_msg_functor=verbatim_success_msg)
  File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 1494, in run_and_log_case_status
    rv = func()
  File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 125, in <lambda>
    batch_args=batch_args)
  File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 78, in _submit
    check_case(case)
  File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 139, in check_case
    create_namelists(case) # Must be called before check_all_input_data
  File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/preview_namelists.py", line 88, in create_namelists
    run_sub_or_cmd(cmd, (caseroot), "buildnml", (case, caseroot, compname), case=case)
  File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 285, in run_sub_or_cmd
    getattr(mod, subname)(*subargs)
  File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/src/drivers/mct/cime_config/buildnml", line 389, in buildnml
    shutil.copy(os.path.join(confdir,"drv_in"), rundir)
  File "/usr/lib64/python2.7/shutil.py", line 120, in copy
    copymode(src, dst)
  File "/usr/lib64/python2.7/shutil.py", line 91, in copymode
    os.chmod(dst, mode)
OSError: [Errno 1] Operation not permitted: '/projects/ClimateEnergy_2/azamatm/ACME_simulations/20180404.A_WCYCL1950_HR.ne120_oRRS18v3_ICG.theta/run/drv_in'

[PeterCaldwell]

By the way - I don't expect this to get fixed before the release. Just making an issue so I don't forget the problem. Also, @maltrud and @jonbob apparently experienced similar problems a few months ago.

[jgfouca]

@PeterCaldwell that looks like a simple permissions problem. If @amametjanov gives you full access to his run directory, it should work.

[amametjanov]

@jgfouca i did chmod -R 775 /projects/ClimateEnergy_2/azamatm/ACME_simulations/20180404.A_WCYCL1950_HR.ne120_oRRS18v3_ICG.theta and that dir contains case-, bld- and run-directories. Is there something written to the $CIMEROOT dir and that dir also has to have g+rwx permissions?

[PeterCaldwell]

In support of Az' comment, when I ls -l the run/directory, I see that all files have group ClimateEnergy_2 (which I'm a member of) and group rwx permission.

-rwxrwxr-x1 azamatm ClimateEnergy_2 5913 Apr 5 22:31 drv_in

Permissions for the run/ directory are similar.

[jgfouca]

It looks like the chmod component of shutil.copy is failing, which can happen if the current user doesn't own the file.

[jgfouca]

Out of curiosity, change the shutil.copy calls in the mct buildnml to shutil.copyfile and see if that works.

[amametjanov]

OK, can you try again?

azamatm@thetalogin1:~/repos/ACME-theta> git diff cime/src/drivers/mct/cime_config/buildnml
diff --git a/cime/src/drivers/mct/cime_config/buildnml b/cime/src/drivers/mct/cime_config/buildnml
index 04c15a224..88203ae74 100755
--- a/cime/src/drivers/mct/cime_config/buildnml
+++ b/cime/src/drivers/mct/cime_config/buildnml
@@ -386,15 +386,15 @@ def buildnml(case, caseroot, component):
     # copy drv_in, drv_flds_in, seq_maps.rc and all *modio* fiels to rundir
     rundir = case.get_value("RUNDIR")

-    shutil.copy(os.path.join(confdir,"drv_in"), rundir)
+    shutil.copyfile(os.path.join(confdir,"drv_in"), rundir)
     drv_flds_in = os.path.join(caseroot, "CaseDocs", "drv_flds_in")
     if os.path.isfile(drv_flds_in):
-        shutil.copy(drv_flds_in, rundir)
+        shutil.copyfile(drv_flds_in, rundir)

-    shutil.copy(os.path.join(confdir,"seq_maps.rc"), rundir)
+    shutil.copyfile(os.path.join(confdir,"seq_maps.rc"), rundir)

     for filename in glob.glob(os.path.join(confdir, "*modelio*")):
-        shutil.copy(filename, rundir)
+        shutil.copyfile(filename, rundir)

 ###############################################################################
 def _main_func():

[PeterCaldwell]

Failed in a different way (can't copyfile a directory?):

Calling /gpfs/mira-home/azamatm/repos/ACME-theta/cime/src/drivers/mct/cime_config/buildnml
Traceback (most recent call last):
File "./case.submit", line 76, in
_main_func(__doc__)
File "./case.submit", line 73, in _main_func
batch_args=batch_args)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 127, in submit
custom_success_msg_functor=verbatim_success_msg)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 1494, in run_and_log_case_status
rv = func()
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 125, in
batch_args=batch_args)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 78, in _submit
check_case(case)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 139, in check_case
create_namelists(case) # Must be called before check_all_input_data
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/preview_namelists.py", line 88, in create_namelists
run_sub_or_cmd(cmd, (caseroot), "buildnml", (case, caseroot, compname), case=case)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 285, in run_sub_or_cmd
getattr(mod, subname)(*subargs)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/src/drivers/mct/cime_config/buildnml", line 389, in buildnml
shutil.copyfile(os.path.join(confdir,"drv_in"), rundir)
File "/usr/lib64/python2.7/shutil.py", line 83, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 21] Is a directory: '/projects/ClimateEnergy_2/azamatm/ACME_simulations/20180404.A_WCYCL1950_HR.ne120_oRRS18v3_ICG.theta/run'

[amametjanov]

OK, modified to have file-to-file copy:

azamatm@thetalogin1:~/repos/ACME-theta> git diff cime/src/drivers/mct/cime_config/buildnml
diff --git a/cime/src/drivers/mct/cime_config/buildnml b/cime/src/drivers/mct/cime_config/buildnml
index 04c15a224..f2430c107 100755
--- a/cime/src/drivers/mct/cime_config/buildnml
+++ b/cime/src/drivers/mct/cime_config/buildnml
@@ -386,15 +386,15 @@ def buildnml(case, caseroot, component):
     # copy drv_in, drv_flds_in, seq_maps.rc and all *modio* fiels to rundir
     rundir = case.get_value("RUNDIR")

-    shutil.copy(os.path.join(confdir,"drv_in"), rundir)
+    shutil.copyfile(os.path.join(confdir,"drv_in"), os.path.join(rundir,"drv_in"))
     drv_flds_in = os.path.join(caseroot, "CaseDocs", "drv_flds_in")
     if os.path.isfile(drv_flds_in):
-        shutil.copy(drv_flds_in, rundir)
+        shutil.copyfile(drv_flds_in, os.path.join(rundir,"drv_flds_in"))

-    shutil.copy(os.path.join(confdir,"seq_maps.rc"), rundir)
+    shutil.copyfile(os.path.join(confdir,"seq_maps.rc"), os.path.join(rundir,"seq_maps.rc"))

     for filename in glob.glob(os.path.join(confdir, "*modelio*")):
-        shutil.copy(filename, rundir)
+        shutil.copyfile(filename, os.path.join(rundir,os.path.basename(filename)))

 ###############################################################################
 def _main_func():

[PeterCaldwell]

Still failed. I'm confused why the code needs to copy this file at all...

Calling /gpfs/mira-home/azamatm/repos/ACME-theta/cime/src/drivers/mct/cime_config/buildnml
Traceback (most recent call last):
File "./case.submit", line 76, in
_main_func(__doc__)
File "./case.submit", line 73, in _main_func
batch_args=batch_args)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 127, in submit
custom_success_msg_functor=verbatim_success_msg)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 1494, in run_and_log_case_status
rv = func()
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 125, in
batch_args=batch_args)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 78, in _submit
check_case(case)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 139, in check_case
create_namelists(case) # Must be called before check_all_input_data
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/preview_namelists.py", line 88, in create_namelists
run_sub_or_cmd(cmd, (caseroot), "buildnml", (case, caseroot, compname), case=case)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 285, in run_sub_or_cmd
getattr(mod, subname)(*subargs)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/src/drivers/mct/cime_config/buildnml", line 397, in buildnml
shutil.copyfile(filename, os.path.join(rundir,filename))
File "/usr/lib64/python2.7/shutil.py", line 69, in copyfile
raise Error("%s and %s are the same file" % (src, dst))
shutil.Error: /projects/ClimateEnergy_2/azamatm/ACME_simulations/20180404.A_WCYCL1950_HR.ne120_oRRS18v3_ICG.theta/case_scripts/Buildconf/cplconf/esp_modelio.nml and /projects/ClimateEnergy_2/azamatm/ACME_simulations/20180404.A_WCYCL1950_HR.ne120_oRRS18v3_ICG.theta/case_scripts/Buildconf/cplconf/esp_modelio.nml are the same file

[amametjanov]

I had an error in that *modelio* file copy, can you re-try?

[amametjanov]

@jgfouca is there a machine where you have both acmetest and jgfouca users?

[PeterCaldwell]

Still failed. Looks like the error we got with os.copy, suggesting that os.copyfile doesn't help?

Calling /gpfs/mira-home/azamatm/repos/ACME-theta/cime/src/drivers/mct/cime_config/buildnml
Finished creating component namelists
Traceback (most recent call last):
File "./case.submit", line 76, in
_main_func(__doc__)
File "./case.submit", line 73, in _main_func
batch_args=batch_args)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 127, in submit
custom_success_msg_functor=verbatim_success_msg)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 1494, in run_and_log_case_status
rv = func()
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 125, in
batch_args=batch_args)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 78, in _submit
check_case(case)
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/case_submit.py", line 139, in check_case
create_namelists(case) # Must be called before check_all_input_data
File "/gpfs/mira-home/azamatm/repos/ACME-theta/cime/scripts/Tools/../../scripts/lib/CIME/preview_namelists.py", line 105, in create_namelists
shutil.copy2(file_to_copy, docdir)
File "/usr/lib64/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
File "/usr/lib64/python2.7/shutil.py", line 98, in copystat
os.utime(dst, (st.st_atime, st.st_mtime))
OSError: [Errno 1] Operation not permitted: '/projects/ClimateEnergy_2/azamatm/ACME_simulations/20180404.A_WCYCL1950_HR.ne120_oRRS18v3_ICG.theta/case_scripts/CaseDocs/esp_modelio.nml'

[amametjanov]

OK, some progress (got past the previous error), seem to be Python-specific errors

shutil.copy2(src, dst)
Similar to shutil.copy(), but metadata is copied as well – in fact, this is just shutil.copy() followed by copystat(). This is similar to the Unix command cp -p.

azamatm@thetalogin1:~/repos/ACME-theta> git diff cime/scripts/lib/CIME/preview_namelists.py
diff --git a/cime/scripts/lib/CIME/preview_namelists.py b/cime/scripts/lib/CIME/preview_namelists.py
index 116fe408c..37b01d6a0 100644
--- a/cime/scripts/lib/CIME/preview_namelists.py
+++ b/cime/scripts/lib/CIME/preview_namelists.py
@@ -102,9 +102,9 @@ def create_namelists(case, component=None):
                    "*streams*txt*", "*stxt", "*maps.rc", "*cism.config*"]:
         for file_to_copy in glob.glob(os.path.join(rundir, cpglob)):
             logger.debug("Copy file from '{}' to '{}'".format(file_to_copy, docdir))
-            shutil.copy2(file_to_copy, docdir)
+            shutil.copyfile(file_to_copy, os.path.join(docdir,os.path.basename(file_to_copy)))

     # Copy over chemistry mechanism docs if they exist
     if (os.path.isdir(os.path.join(casebuild, "camconf"))):
         for file_to_copy in glob.glob(os.path.join(casebuild, "camconf", "*chem_mech*")):
-            shutil.copy2(file_to_copy, docdir)
+            shutil.copyfile(file_to_copy, os.path.join(docdir,os.path.basename(file_to_copy)))

[PeterCaldwell]

@amametjanov - it sounds like you've figured out how to test things all by yourself. Is this true? If so, I'll continue monitoring this thread but won't try doing any more runs for a while.

[amametjanov]

No, this kind of a problem needs to be fixed by the same person with tho different usernames on a machine, which i don't have, or by two people at the same desk (hackathon).

[rljacob]

If its a permission problem, could it be that you need to use setfacl?

It used to be the case that many paths in CIME (or the previous unnamed system) had $USER or $HOME in them and those would not resolve until you accessed them. So if a different user tried to run things, the script system would get really confused looking for things that didn't exist in the different user's paths.

I believe "create_clone" is what should be done here but not sure.

[rljacob]

I'll add that "create_clone" was created because its not possible to just copy (using cp) someone's $CASEDIR and start running their case.

[jgfouca]

@amametjanov , no, there's no machine where I use jgfouca and acmetest. I could try to work with @wadeburgess to get this working.

[PeterCaldwell]

I suspect this problem could be solved by a single person who has write access to both the case/run directories of someone else and also to the code that case directory is trying to access. That person could then both tinker with the code and try runs on their own.

[PeterCaldwell]

@rljacob - create_clone would mean that some pieces of our production would be in one place and other pieces would be somewhere else. This would be a disaster for provenance and for our ability to perform diagnostics etc.

I can see how $USER references in case.submit and its derivatives would cause problems. Those problems are what I expected to see when I tried continuing Az's case... maybe we are just crashing too early to see $USER problems this time. I also think these kind of references are bad code that could/should be fixed. The point of this issue was to say that ability to run from each other's directories would be a nice feature we should work towards. Certainly not something we should focus on now instead of getting ready for the release, but something to keep in the back of our minds...

[wlin7]

If using create_clone, with the same casename, a quick non-conventional approach can be replacing the rundir with one that is a symbolic link to the original one. From there, all output should still be on the same place.

[rljacob]

Similarly, if everyone points to the same short-term archive dir then the output will also land in one place.

[PeterCaldwell]

Thanks @wlin7 and @rljacob - these are good ideas. I think running from different directories but sharing short-term archiving space would still be a headache in terms of restart files and case directories not being in a single searchable space. Wuyin's idea is slightly better in that respect (and maybe good enough?) but would still result in case directories being spread in multiple places. There's a reason Chris G is archiving the entire parent directory for a run to hpss rather than just the stuff put in the short term archive... Having the executable and the case_scripts directories can be very useful.

[jgfouca]

Fixed by: https://github.com/ESMCI/cime/pull/2475