E3sm: maint-1.0 cannot restart from DECKv1 restart files

[golaz]

This problem was first discovered by @zshaheen and I have independently confirmed it.

While the maint-1.0 branch does BFB reproduce piControl when starting from year 0001, it cannot restart from restart files archived from the original simulation. I was trying to rerun the model starting at year 0210 from restart files that were retrieved from HPSS. The model fails during initialization of the land component with the following error:

 Reading restart file
 20180129.DECKv1b_piControl.ne30_oEC.edison.clm2.r.0210-01-01-00000.nc
 Reading restart dataset
 Opened existing file
 20180129.DECKv1b_piControl.ne30_oEC.edison.clm2.r.0210-01-01-00000.nc
           4
 ncd_inqvid: variable eflx_dynbal_amt_to_dribble is not on dataset
 ERROR: Unknown error submitted to shr_abort_abort.

The corresponding traceback is:

645: Image              PC                Routine            Line        Source             
645: e3sm.exe           0000000004C24038  Unknown               Unknown  Unknown
645: e3sm.exe           0000000002F29662  shr_abort_mod_mp_         114  shr_abort_mod.F90
645: e3sm.exe           0000000001EC7B73  restutilmod_mp_re         272  restUtilMod.F90.in
645: e3sm.exe           00000000026B7395  annualfluxdribble         463  AnnualFluxDribbler.F90
645: e3sm.exe           00000000022EECC7  energyfluxtype_mp         720  EnergyFluxType.F90
645: e3sm.exe           0000000001EBB3F8  restfilemod_mp_re         497  restFileMod.F90
645: e3sm.exe           0000000001D681B5  clm_initializemod         702  clm_initializeMod.F90
645: e3sm.exe           0000000001D54DFB  lnd_comp_mct_mp_l         236  lnd_comp_mct.F90
645: e3sm.exe           0000000000429D98  component_mod_mp_         267  component_mod.F90
645: e3sm.exe           00000000004194D4  cime_comp_mod_mp_        1180  cime_comp_mod.F90
645: e3sm.exe           0000000000426FD2  MAIN__                     92  cime_driver.F90

Comparing code from the original simulation (2e145acfa7eaf89d2480c3967a150d4e15cc86e1) and maint-1.0 (f618db17a05c828d9c0f3015c3f124a7f201f371), line 720 in EnergyFluxType.F90 is new:

    call this%eflx_dynbal_dribbler%Restart(bounds, ncid, flag)

and so is the file AnnualFluxDribbler.F90.

[thorntonpe]

In master as of mid-November, this variable was being written/read on restarts, but was not being used elsewhere in the code. There are at least two other examples of variables like this on master:
qflx_liq_dynbal_dribbler
qflx_ice_dynbal_dribbler

I would expect these to give you the same problems. A simple fix would be to add a "move on" check if readvar comes back .false. for any of these variables.

[bishtgautam]

@thorntonpe, Since all three dribblers (eflx_dynbal_dribbler, qflx_liq_dynbal_dribbler, qflx_ice_dynbal_dribbler) are not used in the code, I propose we simply comment out their restart calls in a branch starting from maint-1.0. What do you say?

(Though, I wonder how the dribbler code got added to maint-1.0 in the first place).

[golaz]

@thorntonpe and @bishtgautam : thanks for the quick feedback. I'll try the code modifications suggested by @bishtgautam and report on them later.

[thorntonpe]

@bishtgautam , I agree removing the restart calls is better.

[rljacob]

@thorntonpe what PR added that? It was Nov, 2018?

[thorntonpe]

@rljacob I don't know how/when that came on, or who added it. It is on master as of Nov 14, 2018. I can track it down later, if you'd like.

[bishtgautam]

• The DECK experiments were run using master at 2e145ac from Jan 29, 2018.
• PR-#2024 added 165a6aa (code related to three dribblers) to master on Feb 26, 2018.
• v1.0.0 was made on Apr 20, 2018.
  This explains why the dribbler code is on maint-1.0.

[rljacob]

Our production test on the maint-1.0 branch uses this restart set:
https://web.lcrc.anl.gov/public/e3sm/inputdata/e3sm_init/20171228.beta3rc13_1850.ne30_oECv3_ICG.edison/0331-01-01/

Why does it work but other restarts from the DECK run do not?

[golaz]

I tried @bishtgautam's branch. The land model now initializes but then fails during the first time step with a segmentation fault:

645: forrtl: severe (174): SIGSEGV, segmentation fault occurred
645: Image              PC                Routine            Line        Source
645: e3sm.exe           0000000004CFCB91  Unknown               Unknown  Unknown
645: e3sm.exe           0000000004CFACCB  Unknown               Unknown  Unknown
645: e3sm.exe           0000000004CA4374  Unknown               Unknown  Unknown
645: e3sm.exe           0000000004CA4186  Unknown               Unknown  Unknown
645: e3sm.exe           0000000004C226E6  Unknown               Unknown  Unknown
645: e3sm.exe           0000000004C2E0F6  Unknown               Unknown  Unknown
645: e3sm.exe           00000000044BA0A0  Unknown               Unknown  Unknown
645: e3sm.exe           0000000001ED8EAE  subgridavemod_mp_         984  subgridAveMod.F90
645: e3sm.exe           0000000001E25E4D  histfilemod_mp_hi        1062  histFileMod.F90
645: e3sm.exe           0000000001E259E4  histfilemod_mp_hi         978  histFileMod.F90
645: e3sm.exe           0000000001D60447  clm_driver_mp_clm        1236  clm_driver.F90
645: e3sm.exe           0000000001D4F41D  lnd_comp_mct_mp_l         464  lnd_comp_mct.F90
645: e3sm.exe           00000000004272F4  component_mod_mp_         728  component_mod.F90
645: e3sm.exe           000000000040CEDD  cime_comp_mod_mp_        2655  cime_comp_mod.F90
645: e3sm.exe           0000000000427022  MAIN__                    103  cime_driver.F90

[bishtgautam]

@golaz, Can you provide steps on how to reproduce your workflow?

[bishtgautam]

Why does it work but other restarts from the DECK run do not?

@rljacob, Maybe because @golaz is using a different restart file (20180129.DECKv1b_piControl.ne30_oEC.edison.clm2.r.0210-01-01-00000.nc) as compared to the test (20171228.beta3rc13_1850.ne30_oECv3_ICG.edison.clm2.r.0331-01-01-00000.nc)

[rljacob]

Yes but my understanding is both restart files come from runs with code bases prior to PR #2024 . Could it be because the production test is a hybrid run? It also sets check_finidat_year_consistency = .false.

[golaz]

@bishtgautam, here is the path to the run e3sm script on Edison:

/global/homes/g/golaz/E3SM/scripts/run_e3sm.DECKv1b_piControl_0210.ne30_oEC.edison.csh

Output is under

/global/cscratch1/sd/golaz/E3SM/simulations/piControl/0210/20180129.DECKv1b_piControl.ne30_oEC.edison

[bishtgautam]

@golaz, Can you open permissions for the /global/cscratch1/sd/golaz/E3SM/simulations/piControl/0210/20180129.DECKv1b_piControl.ne30_oEC.edison?

[bishtgautam]

@golaz, Using run_e3sm.DECKv1b_piControl_0210.ne30_oEC.edison.csh, I got a different error than the one you mentioned. My error was:

(seq_infodata_Init)  restart file from rpointer= 20171228.beta3rc13_1850.ne30_oECv3_ICG.edison.cpl.r.0331-01-01-00000.nc
(seq_infodata_Check)  case_name =20180129.DECKv1b_piControl.ne30_oEC.edison: rest_case_name =20171228.beta3rc13_1850.ne30_oECv3_ICG.edison:
 ERROR:
 (seq_infodata_Check) : invalid continue restart case name = 20171228.beta3rc13_

I suspect the error is because the case that I created is not named the same as the restart case and CONTINUE_RUN=.true.. Should I simply change the case_name in the run script to be 20171228.beta3rc13_1850.ne30_oECv3_ICG.edison?

[golaz]

@bishtgautam : I believe I fixed the permission problems. I think your other problem was related to not having the proper restart file in you run directory. In the run script, I create a link to them. They are under

/global/cscratch1/sd/golaz/E3SM/simulations/piControl/0210/20180129.DECKv1b_piControl.ne30_oEC.edison/restart

[golaz]

@rljacob : as you pointed out, we use a hybrid restart when starting from beta3rc13 files, and a continuation run when restarting from actual model files. I don't know how that would relate to the dribbler code, but that seems to be the most likely hypothesis.

[golaz]

@rljacob and @bishtgautam : do you have suggestions on how to move forward if this is not easily fixed?

We need to rerun a portion of the Water Cycle DECKv1 simulations before Edison is decommissioned and the end of March, so we really don't have much time.

@rljacob : is undoing the bad PR an option?

[bishtgautam]

@golaz, Since using run_e3sm.DECKv1b_piControl_0210.ne30_oEC.edison.csh, I'm was not able to reproduce your error, I tried an I-compset. The I-compset ran successfully using bishtgautam/lnd/dribbler-restart-fix branch and had following settings in user_nl_clm

finidat = '/project/projectdirs/acme/inputdata/e3sm_init/20171228.beta3rc13_1850.ne30_oECv3_ICG.edison/0331-01-01/20171228.beta3rc13_1850.ne30_oECv3_ICG.edison.clm2.r.0331-01-01-00000.nc'
check_finidat_pct_consistency = .false.

Can you make necessary modifications to run_e3sm.DECKv1b_piControl_0210.ne30_oEC.edison.csh or provide another run script file that I can use to reproduce your error?

[golaz]

What modifications do you need me to make in the run script?

[bishtgautam]

I had to set set old_executable = false. But, this was not enough to reproduce your error.

Now, I have additionally modified set case_name = 20171228.beta3rc13_1850.ne30_oECv3_ICG.edison and compiling a new case.

[rljacob]

@bishtgautam would doing a restart vs. hybrid run affect if the dribbler code is invoked?

I think its fine to undo the bad PR on the maint-1.0 branch so you can get running but still don't think we understand what is happening here.

[bishtgautam]

@golaz, Here is an update. The model successfully ran a 5-day simulation using bishtgautam/lnd/dribbler-restart-fix and with following modifitions to run_e3sm.DECKv1b_piControl_0210.ne30_oEC.edison.csh:

• set old_executable = false
• set case_name = 20171228.beta3rc13_1850.ne30_oECv3_ICG.edison

[golaz]

@bishtgautam : but did it restart from DECKv1 restart files at year 210 or the older beta3rc13 files?

[bishtgautam]

The simulation used 20171228.beta3rc13_1850.ne30_oECv3_ICG.edison.clm2.r.0331-01-01-00000.nc

[zshaheen]

@bishtgautam, as @golaz, we need the simulation to use the DECKv1 restart files starting from year 210. Even if these files were in the run directory, the rpointer* files are changed to use the ones you've mentioned above. The fix for this was to manually recopy the rpointer* files that use the 210 years right before the job to run the model is submitted to the run queue.

[bishtgautam]

@zshaheen, I manually copied the /global/cscratch1/sd/golaz/E3SM/simulations/piControl/0210/20180129.DECKv1b_piControl.ne30_oEC.edison/restart/rpointer.* into the /global/cscratch1/sd/gbisht/E3SM/simulations/piControl/0210/20171228.beta3rc13_1850.ne30_oECv3_ICG.edison/run. But the rpointer files that I copied get overwritten during the run. See the run log

>more 20171228.beta3rc13_1850.ne30_oECv3_ICG.edison.run.12277656
Generating namelists for /global/cscratch1/sd/gbisht/E3SM/simulations/piControl/0210/20171228.beta3rc13_1850.ne30_oECv3_ICG.edison/case_scripts
 - Prestaging REFCASE (/project/projectdirs/acme/inputdata/e3sm_init/20171228.beta3rc13_1850.ne30_oECv3_ICG.edison/0331-01-01) to /global/cscratch1/sd/gbisht/E3SM/simulations/piControl/0210/20171228.beta3
rc13_1850.ne30_oECv3_ICG.edison/run
Copy rpointer /project/projectdirs/acme/inputdata/e3sm_init/20171228.beta3rc13_1850.ne30_oECv3_ICG.edison/0331-01-01/rpointer.ice
Copy rpointer /project/projectdirs/acme/inputdata/e3sm_init/20171228.beta3rc13_1850.ne30_oECv3_ICG.edison/0331-01-01/rpointer.lnd
Copy rpointer /project/projectdirs/acme/inputdata/e3sm_init/20171228.beta3rc13_1850.ne30_oECv3_ICG.edison/0331-01-01/rpointer.rof
Copy rpointer /project/projectdirs/acme/inputdata/e3sm_init/20171228.beta3rc13_1850.ne30_oECv3_ICG.edison/0331-01-01/rpointer.atm
Copy rpointer /project/projectdirs/acme/inputdata/e3sm_init/20171228.beta3rc13_1850.ne30_oECv3_ICG.edison/0331-01-01/rpointer.ocn
Copy rpointer /project/projectdirs/acme/inputdata/e3sm_init/20171228.beta3rc13_1850.ne30_oECv3_ICG.edison/0331-01-01/rpointer.drv
Creating component namelists
   Running /global/u2/g/gbisht/E3SM/code/maint-1.0_dribbler-restart-fix/components/cam/cime_config/buildnml

Any suggestions how to avoid overwritting of rpointer files?

[zshaheen]

@bishtgautam, in the run script we used, under the section:
else if ( $model_start_type == 'continue' ) then
and after the calls to $xmlchange_exe, we added the code to cp the restart files we needed.

You can try that.

[golaz]

@bishtgautam : this is the result of CIME trying to do too much magic here. CIME will always copy the restartart files and corresponding rpointer files to start from year 1. The trick is to copy the rpointer file after CIME. I added code for that near line 1184 in my run e3sm script (/global/homes/g/golaz/E3SM/scripts/run_e3sm.DECKv1b_piControl_0210.ne30_oEC.edison.csh)

  # Chris Golaz: addition for restarting from pre-existing restart files
  $xmlchange_exe --id RUN_TYPE --val "startup"
  pushd ${case_run_dir}
  cp ../restart/rpointer.* .
  cp -s ../restart/*.nc .
  popd

[golaz]

@rljacob: if it's easy for you to create a branch of maint-1.0 than undoes the PR, I'm happy to test it out.

[bishtgautam]

@golaz, You are correct that CIME is trying to do too much. But, the CIME is overwritting the rpointer files when the job begins after I do ./case.submit. What should the be the value of RUN_TYPE?

[golaz]

That's frustrating. Try RUN_TYPE set to "startup" and CONTINUE_RUN set to "true".

[klindsay28]

Does setting GET_REFCASE=FALSE stop CIME from overwriting the rpointer files?

[bishtgautam]

RUN_TYPE=startup and CONTINUE_RUN=false failed because the case_name and rpointer filenames don't match. I have submitted a new run to test out @klindsay28's suggestion.

[golaz]

Could be another example of CIME trying to prevent users from doing stupid things!?! You may have to use the same case_name as the original run... at least that's what I had to do.

[bishtgautam]

@klindsay28's suggestion definitely stops CIME from overwriting the rpointer files.

@golaz, Yes I do need to create a case with the same name as that used in the rpointer files. Thus, I'm creating a new case named 20180129.DECKv1b_piControl.ne30_oEC.edison.

[rljacob]

I made this point somewhere but now can't find it (email? github?).

On the maint-1.0 branch, the block of code starting here: https://github.com/E3SM-Project/E3SM/blob/maint-1.0/cime/config/e3sm/allactive/config_compsets.xml#L289

and through line 311 is the CIME "magic" that forces any case using the low res water cycle config to be a hybrid restart with a start from 0331-01-01 of beta3rc13. Its set when you run "create_newcase" and the only way to stop it doing that is to delete those lines. Or after create_newcase is run, change the rpointer files and set GET_REFCASE to false as @klindsay28 said.

[rljacob]

@golaz the branch off of maint-1.0 that undoes the dribbling is the one @bishtgautam already has a PR for: bishtgautam/lnd/dribbler-restart-fix

[golaz]

@rljacob : I tried bishtgautam/lnd/dribbler-restart-fix and it didn't work for me (see above https://github.com/E3SM-Project/E3SM/issues/2727#issuecomment-461154447). I'm happy to try again if there is a reason to believe I did something wrong.

Note that bishtgautam/lnd/dribbler-restart-fix only comments out reading new variables from restart files. It doesn't actually revert the PR that brought all the changes in, so there might be something else going on.

[bishtgautam]

@golaz, Finally, I'm able to reproduce the same error as you reported. Now, I'm debugging what is causing the error.

[bishtgautam]

Update: The error is in the land code is where values are accumulated in the history buffers. If the call to update history buffers is commented out, the code runs fine, but all land output would be garbage. The debugging is slow on Edison because jobs at times sit in the queue with the following Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions

@golaz Btw, why is the new simulation not being run with the hash as the original simulation?

[golaz]

@bishtgautam : my understanding is that the original hash will no longer compile on Edison because of environment changes that have happened since.

[golaz]

@bishtgautam and @rljacob : I was talking to @tangq yesterday and he made a few suggestions to get around the problem of the broken restarts in maint-1.0. Since we did archive the original build and case_scripts sub-directories to HPSS, Qi suggested that it might be possible to simply reuse them instead of trying to build a new executable based on maint-1.0.

I worked on this yesterday and this morning, and I think I finally got something that works. I wanted to place the rerun simulations in a new directory structure so that I could run multiple segments at once. It took a lot of contorsion to get CIME to do this because full paths are hard-wired in so many places, including locked and hidden files.

But I did eventually get a 16 day test run this morning that BFB reproduced the original simulation. I'm planning to go ahead and prepare the production reruns using this methodology.

While I may no longer need it, it still would be good to fix the maint-1.0 branch. I would suggest to also add a new test so it doesn't break in the same way again.

[bishtgautam]

@golaz, Great that you found a workaround for now. I too have an alternate workaround that I'm testing at the moment. My workaround starts with the original hash (2e145ac) and incorporates machine updates for Edison (bishtgautam/scripts/edison-updates-for-DECKv1-sims branch). I'm testing out the new branch now and will post on how things go.

Regarding debugging the error, I first tried write-statements to isolate the issure and then used the DDT debugger. The DDT debugger is instead reporting an error in the atm code. I might have the DDT settings incorrectly and I'm working on it.

[rljacob]

If Chris has a solution then we need to focus on getting maint-1.0 to work. That is the released code and anyone should be able to use it with any v1.0 restart set.

[bishtgautam]

My job is successfully running on Edison now. So, bishtgautam/scripts/edison-updates-for-DECKv1-sims in another alternate. I will now focus on maint-1.0.

[rljacob]

Turns out a hybrid initial run does not read the land or atmosphere restarts. That's why the old restart set seemed to work with maint-1.0 and our maint-1.0 test, which just does the hybrid initial run, never triggered anything. Hybrid reads the ocean restarts and does plain initial starts with atm and lnd.

If you do a hybrid initial run and then keep running with the restarts produced, there's no problem since the restarts are written with the maint-1.0 code. So anyone redoing CMIP6 pi runs from the beginning will have no problems. But, as Chris found, trying to do a branch run from a random v1.0 complete restart set won't work.

[golaz]

Hmm... That would bad news if our initial hybrid restart did not read the land initial conditions. The motivation of reusing initial conditions from previous simulations was to have better spun-up land and ocean models.

Was something not configured right, or is this a general feature of hybrid simulations?

[rljacob]

Its a feature of hybrid runs. I was wrong though. While the "run type" is "initial", it does read the provided restart file
initial data =
20171228.beta3rc13_1850.ne30_oECv3_ICG.edison.clm2.r.0331-01-01-00000.nc

And there are even some non-fatal messages about dribble data not being found.

 Reading restart dataset
 Opened existing file 
 20171228.beta3rc13_1850.ne30_oECv3_ICG.edison.clm2.r.0331-01-01-00000.nc
      327680
 ncd_inqvid: variable eflx_dynbal_amt_to_dribble is not on dataset
 can't find rootfr in restart (or initial) file...
 Initialize rootfr to default
 ncd_inqvid: variable qflx_liq_dynbal_amt_to_dribble is not on dataset
 ncd_inqvid: variable qflx_ice_dynbal_amt_to_dribble is not on daaset

[golaz]

I checked the lnd log file of the original simulation, and there is clear evidence that the land model did read restart files from the previous simulations:

    initial data   = 
 20171228.beta3rc13_1850.ne30_oECv3_ICG.edison.clm2.r.0331-01-01-00000.nc

Reading initial conditions from 20171228.beta3rc13_1850.ne30_oECv3_ICG.edison.clm2.r.0331-01-01-00000.nc (GETFIL): attempting to find local file 20171228.beta3rc13_1850.ne30_oECv3_ICG.edison.clm2.r.0331-01-01-00000.nc (GETFIL): using 20171228.beta3rc13_1850.ne30_oECv3_ICG.edison.clm2.r.0331-01-01-00000.nc in current working directory

I don't know how to reconcile this with your statement that atm and lnd initial conditions are not read in hybrid restarts.

[rljacob]

See comment right before yours. I was wrong but the lnd.log is misleading.

[golaz]

I did see your comment before mine but I don't understand it.

Looking at the lnd.log file of the original DECKv1 simulation, I'm virtually certain that the that the land restart files from the preceding spin-up simulation were read and used. So I don't know how to reconcile that with your general statement that land restarts are not used for hybrid restarts.

[rljacob]

I retracted that statement. The land model does read the restart in a hybrid run.

[chengdang]

I'm running maint-1.0 and had similar errors, while I am not sure if this is related.
I'm able to run a fully coupled E3SM for 2 months from initial files:

/project/projectdirs/acme/inputdata/e3sm_init/20171228.beta3rc13_1850.ne30_oECv3_ICG.edison/0331-01-01

Then I got errors for the continue run:

 7109    0:  (t_initf) Read in prof_inparm namelist from: drv_in
  7110    0:  (t_initf) Using profile_disable=          F
  7111    0:  (t_initf)       profile_timer=                      4
  7112    0:  (t_initf)       profile_depth_limit=               20
  7113    0:  (t_initf)       profile_detail_limit=              12
  7114    0:  (t_initf)       profile_barrier=          F
  7115    0:  (t_initf)       profile_outpe_num=                  1
  7116    0:  (t_initf)       profile_outpe_stride=               0
  7117    0:  (t_initf)       profile_single_file=      F
  7118    0:  (t_initf)       profile_global_stats=     T
  7119    0:  (t_initf)       profile_ovhd_measurement= F
  7120    0:  (t_initf)       profile_add_detail=       F
  7121    0:  (t_initf)       profile_papi_enable=      F
  7122    0:  ERROR: Unknown error submitted to shr_abort_abort.
  7123    1:  ERROR: Unknown error submitted to shr_abort_abort.
  7124    3:  ERROR: Unknown error submitted to shr_abort_abort.
  7125    4:  ERROR: Unknown error submitted to shr_abort_abort.
  7126    5:  ERROR: Unknown error submitted to shr_abort_abort.
  7127    6:  ERROR: Unknown error submitted to shr_abort_abort.
  7128    8:  ERROR: Unknown error submitted to shr_abort_abort.
  7129    9:  ERROR: Unknown error submitted to shr_abort_abort.
  7130   10:  ERROR: Unknown error submitted to shr_abort_abort.
  7131   11:  ERROR: Unknown error submitted to shr_abort_abort.
  7132   12:  ERROR: Unknown error submitted to shr_abort_abort.
  7133   13:  ERROR: Unknown error submitted to shr_abort_abort.
  ...

I modified the $case//Buildconf/camconf/atm_in to produce monthly cam.h0 files instead of the default daily output. Could this be the reason? If so, may I know why and how to properly output monthly files? Thanks.

[bishtgautam]

@chengdang, I believe you cannot change the output frequency of any component in for a CONTINUE_RUN=TRUE.

[bishtgautam]

FYI, I have been able to reproduce the failure in an I-compset run. Thus, I'm debugging this issue in a land-only run and will provide an update as I go along.

[bishtgautam]

@golaz, @zshaheen, I have updated PR #2728, tested it on Edison, and it ready to be merged into maint-1.0. When either one of you get a chance, could you test it out?

cc: @rljacob

[golaz]

Thanks, @bishtgautam. @zshaheen is trying your new branch now. I will try it myself as well once I can run on NERSC again (quota issue).

[zshaheen]

@bishtgautam I tested the branch and it all seems good. I'm going to be doing more runs with this and will get more data points. If @golaz can test this as well, we'll have a more definite answer. Thanks yaar 👍

[zshaheen]

@bishtgautam, Chris and I have one request. Can you keep your branch until after Edison goes down? The reason being is that this seems to work, and maybe merging it into the maint-1.0 branch might cause issues down the line. We have about 1 month to finish this.

We still need to rerun some of the corrupted data here.

[bishtgautam]

@zshaheen, Sure, I won't delete the my branch.

[rljacob]

You're going to use this branch and not use maint-1.0?

[zshaheen]

You're going to use this branch and not use maint-1.0?

This seems to work and we'll start making a bunch of batch jobs. It would suck if some other changes were made to maint-1.0 in the middle of all of this. Our one month time constants means that we'd like to minimize external issues, for now.

[rljacob]

After merging, the only difference between maint1.0 and Gautum's branch is the addition of PR #2691 which concerns theta. So you should have no problem.