E3sm: MPAS-O problem causing SMS_P12x2.ne4_oQU240.A_WCYCL1850.sandiatoss3_intel.allactive-mach_mods diffs

[jonbob]

PR #3057 introduced code that has caused threading tests, especially SMS_P12x2.ne4_oQU240.A_WCYCL1850.sandiatoss3_intel.allactive-mach_mods, to continuously diff with baselines.

[jonbob]

Testing a fix, which intel inspector says fixed this specific race condition. It passed a comparison of results from SMS_P12x2.T62_oQU240.CMPASO-NYF.compy_intel ten times, but I'll test next with SMS_P12x2.ne4_oQU240.A_WCYCL1850.compy_intel.allactive-mach_mods

[jonbob]

Also got the same result ten times for SMS_P12x2.ne4_oQU240.A_WCYCL1850.compy_intel.allactive-mach_mods

[rljacob]

Add more info for the record:
SMS_P12x2.ne4_oQU240.A_WCYCL1850.sandiatoss3_intel.allactive-mach_mods would have a baseline diff on both next and master. If we blessed the diff, it would keep diffing because of race conditions producing different answers from run to run.

SMS_Ld2.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6.cori-knl_intel.allactive-v1cmip6 would also diff with baselines (on next) even after blessing and no further non-BFB changes.

PET_Ln9_PS.ne30_oECv3_ICG.A_WCYCL1850S.sandiatoss3_intel.allactive-mach-pet would occasionally fail on next and master. (A fail, not a diff) The two runs would produce different answers sometimes.

What we need to add to the test suite is a test that would consistently fail (not just diff with baselines) with the broken version of the code.