E3sm: land albedos significantly greater than 1

[maltrud]

@jonbob ran into trouble with the ocean model (POP in this case) exiting due to negative net shortwave flux. while investigating this problem, we have seen widespread negative shortwave fluxes over land. this is presumably due to land albedos greater than 1 in many areas. is this expected behaviour?

i have attached an image from the coupler history to help illustrate this. on the left is the net shortwave (a2x_Faxa_swnet) and on the right is the land direct visible albedo (all of the albedos appear to be the same). it appears to us that in some cases (note it took 42 years of simulation before this happened) that if a large albedo can create a large (in magnitude) anomaly in shortwave, it can drag the net flux for the ocean negative as well.

[bishtgautam]

I would say negative net shortwave radiation is an unexpected behavior. Out of curiosity, is the land model receiving positive downwelling shortwave radiation (i.e. Faxa_swndr, Faxa_swvdr, Faxa_swndf, Faxa_swvdf)? Are the restart files saved to reproduce this error?

[maltrud]

@bishtgautam , i've never seen the downwelling shortwave negative while investigating this, only the net. shouldn't albedo > 1 also cause you some concern?

these images are from Chris G's latest run on edison:
https://acme-climate.atlassian.net/wiki/display/SIM/20170628.POP.A_WCYCL1850.ne30_g16.edison

[bishtgautam]

I'm trying to understand if the land model is the source of error or the forcing from the atmosphere to the land is incorrect.

Btw, do the images of a2x_Faxa_swnet correspond to 20170628.POP.A_WCYCL1850.ne30_g16.edison.cpl.hi.0042-01-01-00000.nc?

[bishtgautam]

One more thing. @golaz's note on 20170628.POP.A_WCYCL1850.ne30_g16.edison page state that the model ran for 51 years. Which log file in /global/cscratch1/sd/golaz/ACME_simulations/20170628.POP.A_WCYCL1850.ne30_g16.edison/run corresponds to the model exiting due to negative shortwave radiation?

[maltrud]

sorry, i have been a bit unclear. it was Jon's run on anvil (20170605.POP.A_WCYCL1850.ne30_g16.anvil) that died in year 42. Chris' run didn't die (likely due to a different coupling frequency and using CESM1_MOD instead of RASM_OPTION1) since the ocean didn't see negative shortwave in this run. but the fact remains that there are lots of negative shortwave fluxes and land albedos > 1 in Chris' run, too.

the images are from 20170628.POP.A_WCYCL1850.ne30_g16.edison.cpl.hi.0051-01-01-00000.nc

[jonbob]

@bishtgautam - I guess another question is if albedos that much greater than 1 are OK, whether or not their occurrence causes the model to fail?

[bishtgautam]

Based on a quick look through the land model code, I don't believe there is a check to ensure albedo is less than 1.0.

[jonbob]

Thanks @bishtgautam - do you think we need one? It would seem non-physical to allow it to exceed 1, but I'm not sure what impact that would have on the coupled model.

[PeterCaldwell]

Do any of you know whether negative net surface shortwave is the result of albedo > 1 or whether the albedo is diagnosed from the net SW and is as a result greater than 1? This would clarify where the problem is.

[bishtgautam]

1. Using notes from 20170628.POP.A_WCYCL1850.ne30_g16.edison page, I have been able to run a 5-days simulation.
2. Now, I have to a 5-days simulation in queue that uses restart files available in /global/cscratch1/sd/golaz/ACME_simulations/20170628.POP.A_WCYCL1850.ne30_g16.edison/run.
3. If the step-2 is successful, I will attempt to diagnose which land grid cell is returning an albedo > 1.0 and/or verify if the problematic land grid cell is receiving positive downwelling shortwave radiation.

Regarding @PeterCaldwell's comment: The computed albedo is used to estimate net shortwave radiation.

[jonbob]

I don't know -- but can try to look. From what I could tell from the cpl history files, the lnd actually is sending albedos greater than 1 to the cpl -- and doesn't get any albedos returned from the cpl. That would infer that the out-of-bounds values are due to a lnd issue. The negative SW's are coming into the cpl from the atm, but not getting sent to the lnd -- at least in the file I'm looking at. It is always possible the albedos are a chicken-and-egg thing -- due to an inconsistency of some kind in return values from the cpl -- but I don't have a clue how they are determined in ALM. @bishtgautam ?

[jonbob]

And it's not just a handful of cells -- here's ncview output from one of my cpl history files

ncview copy.pdf

[jonbob]

In that output, the negative SW's in the atm are in cells between numbers 12000 and 15000. But that doesn't correspond to the largest albedos

[PeterCaldwell]

Thanks for the explanation (from both @bishtgautam and @jonbob ) that the albedos are causing the negative surface SW. I'm curious whether this is only a problem when using POP, or whether the v1 master also has albedos > 1. Is it easy to check this?

[jonbob]

@PeterCaldwell - I'll look through some non-POP runs and check

[bishtgautam]

What I don't get is, how can the choice of ocean model result in negative albedos on land? If POP is causing negative albedos on land, I suspect the atmosphere model must be somehow involved in all of this.

[jonbob]

I checked and the lnd is sending albedos much greater than 1 regardless of ocean model

[rljacob]

Isn't the land abledo calculation entirely based on intrinsic land surface properties? It doesn't matter what the atmosphere does.

[bishtgautam]

I checked and the lnd is sending albedos much greater than 1 regardless of ocean model

Ok, there goes my theory regarding atmosphere model being somehow involved.

[PeterCaldwell]

Another sanity-check question I had is whether we were using the correct version of the land model for our simulations. Looking at cime/config/acme/allactive/config_compsets.xml shows that we are in fact using CLM4.5 for all conceivable simulations the coupled team has been looking at.

[bishtgautam]

Isn't the land abledo calculation entirely based on intrinsic land surface properties? It doesn't matter what the atmosphere does.

@rljacob: I believe you are correct.

[jonbob]

@PeterCaldwell - could this be a bug for CAM? I checked the cpl history, and it is sending the out-of-range albedos to the atm. And CAM's driver doesn't limit them during the import phase...

[rljacob]

I don't think its a "bug" for a model to be missing a check on the range of a variable it reasonably expects to never exceed that range.

[jonbob]

@rljacob - agreed. I didn't mean it that way as much as that it could create real problems for CAM

[PeterCaldwell]

I agree that land albedos > 1 could "create real problems for CAM" in the sense that they could cause big and non-credible changes in model behavior. That said, I can't think of any place in the atm model where having land albedo>1 or having land be a source of SW would cause a particular parameterization to break.

[rljacob]

We've done decades-long simulations with F and B-cases and CLM45. Either non-POP components are very tolerant of this or its a recent bug.

@maltrud in your plot above, the bad sw values are near land. Is that always the case or did you see this in open ocean?

[jonbob]

It's Mat's day off, but I looked at the output with him. POP is set up to abort if it gets a negative sw value, and it failed only at a single point -- the cell with two lnd edges. He plotted the negative sw points and otherwise they were all over lnd

[maltrud]

@rljacob they appear to be near land. to my eye, it also seems to happen right around sunrise when the downwelling shortwave is small. the ocean point which caused the abort in @jonbob's run is surrounded on 2 sides by land in the Gulf of Tonkin.

a couple of things to reiterate:
1) the dependence on ocean model arises only because POP has an explicit check for negative shortwave--if it finds even 1 cell then it aborts. MPAS-O doesn't have this explicit check (though i think this shows that maybe it should).
2) it took 42 years for POP to abort (and hasn't happened in @golaz's run) so it isn't a huge issue in that sense. but (to me) the main issue here is that when Jon and i looked into this, we found these strange land albedos, and widespread negative shortwave on land which seems to be significant.

[rljacob]

"right around sunrise" means the zenith angle might be involved. Its possible the land uses that as part of its shortwave albedo calculation.

[thorntonpe]

The incoming SW could get very close to zero on the last timestep of the day, in which case rounding errors could lead to spurious albedo calculations, especially in places/times where the albedo is already close to 1. We may need to add a check for that and limit to 1 in cases where the incoming SW is below a specified threshold.

From: Robert Jacob [mailto:[email protected]]
Sent: Friday, July 7, 2017 3:55 PM
To: ACME-Climate/ACME [e-mail]
Cc: Thornton, Peter E. [e-mail]; Assign [e-mail]
Subject: Re: [ACME-Climate/ACME] land albedos significantly greater than 1 (#1614)

"right around sunrise" means the zenith angle might be involved. Its possible the land uses that as part of its shortwave albedo calculation.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHubhttps://github.com/ACME-Climate/ACME/issues/1614#issuecomment-313777596, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AHvfVLJi3NKNoBUXRTt8g5W60hY1y4QIks5sLoz8gaJpZM4OQG9l.

[bishtgautam]

In that case, #1554 (which updated the computation of cosine of solar zenith angle) could be the culprit.

[bishtgautam]

The land model definitely uses the cosine of solar zenith angle in the computation of albedo (SurfaceAlbedoMod.F90#L349)

[bishtgautam]

@jonbob: My jobs on Edison are taking a lot of time to get picked up. How can I reproduce similar model behavior using restart files for 20170605.POP.A_WCYCL1850.ne30_g16.anvil on Anvil?

[jonbob]

@bishtgautam - I don't believe #1554 was the culprit. The POP run started from a codebase before that made it to master

[jonbob]

@bishtgautam - we have anvil pretty full right now.

[bishtgautam]

In that case, I will use Edison for the time being. But, my job on Edison cannot use restart files from Chris's run. Specifically, CAM fails to restart (/global/cscratch1/sd/gbisht/ACME_simulations/20170628.POP.A_WCYCL1850.ne30_g16.edison/run/atm.log.170707-134142)

[jonbob]

Good morning, @bishtgautam - I can help get your run going on edison today using Chris' restart files. I missed this on Friday

[bishtgautam]

Here is an update so far:

• Successfully run a 5-day simulation using restart files from Chri's run.
• The land model produces albedo > 1 within the 5-day run.
• So far it appears that land model is passing albedo > 1 when the sun is below the horizon.
  
  
  
  • When the sun is below the horizon, the land model skips the computation of surface albedo at ALM's patch-level and set albedo to 1.d0.
  
  
  • A possible source of error could be the averaging of albedo from ALM's patch to grid level.
  
  

Remaining question: Why the upscaling from ALM's patch-level to grid-level is producing albedo > 1 only when the sun is below the horizon?

[rljacob]

If the sun is below the horizon, is the shortwave calculation even done? If so then we also need to know how these albedos are getting in to daylit areas.

[thorntonpe]

Can you tell if this happens only when the sun is very close to (but below) the horizon?

[bishtgautam]

@rljacob: As I understand, when the sun is below the horizon, the land model skips albedo computation and attempts to return albedo = 1.0. A value of 1.0 for albedo would ensure that any downwelling shortwave radiation from the atmosphere is reflected back and energy conservation is maintained for the coupled system.

@thorntonpe: When albedo > 1.0 is returned by the land model, it does not matter how below the sun is below the horizon.

After more debugging, it appears that albedo > 1.0 is being returned when patch-level values for urban patches are aggregated to grid level AND the sun is below the horizon.

• garr(g,j) = garr(g,j) + parr(p,j) * scale_p2c(p) * scale_c2l(c) * scale_l2g(l) * pft%wtgcell(p) is first computed at subgridAveMod.F90#L693
• Then, garr(g,j) = garr(g,j)/sumwt(g) at subgridAveMod.F90#L704

The scale_c2l has a non-intutive values for urban columns at subgridAveMod.F90#L639

PS: I have also reported the bug to the CLM community.

[akturner]

Not particularly relevant to this discussion but I vaguely remember that some parts of the coupled system (presumably related to the sea ice if I've heard about it) have positive shortwave flux with the sun below the horizon - this is presumably the pre-dawn/post-sunset glow, which I imagine being relatively significant during polar winter.

[PeterCaldwell]

Is there a plan to get this bugfix onto master? And do you know whether it has a significant effect on model climate? @rljacob - this seems like something that should be added to the next beta tag?

[rljacob]

I think @bishtgautam is testing a fix. If the right fix can't be found, just clamp the albedos to 1 for a temporary fix.

[jonbob]

@rljacob - we could limit it in the cpl?

[bishtgautam]

Following things are still outstanding before I plan to issue the PR to fix:

• More testing is underway to make sure a5a9c95 fixes albedo for all scenarios.
• Need someone with expertise in the land urban model to review the fix. I believe ACME lacks expertise in the urban model. Thus, I have been interacting with colleagues at NCAR.

Not sure how one would determine if the bug fix is CC or just non-BFB.

[bishtgautam]

With more testing, I'm confident that a5a9c95 fixes albedo for all scenarios. Does anyone have a suggestion on how to figure out if the bugfix is a CC?

[PeterCaldwell]

@salilmahajan , @huiwanpnnl , @singhbalwinder are testing automated methods to check for climate-changing modifications as part of the CMDV-SE project. They might be interested in taking up this case.

Aside from that, I'd suggest running the model for a few years with/without the bugfix and seeing if things look different. I'm not sure whether this fix affects the land model in standalone mode. If not, I'd suggest running in F configuration and using AMWG diagnostics to evaluate. I can do that if you can't.

[bishtgautam]

@PeterCaldwell: Can you suggest the F --compset and --res values? I will submit runs with and without the bug-fix. Since I don't have any experience running and interpreting outputs of AMWG diagnostics, it would best if you run the AMWG package.

[PeterCaldwell]

compset: FC5AV1C-04P2
res: ne30_ne30
How long to run depends on how big the difference is - we may be able to see it in a year, or it may require 10 yrs. I'd suggest running for 3 yrs as a compromise between good statistics and efficiency.
Let me know when the runs are done and I'll apply AMWG diagnostics to them.
Note that I have no idea what PE layout you'll get when you try to run this - it could suck. If it crashes let me know.

[bishtgautam]

I will submit the cases on Edison today and provide updates on their progress as the jobs proceed.

[jonbob]

@bishtgautam - we have several potential climate-changing commits coming, in the atm and ocn as well. Because the albedo issue is a real bug and not a science tweak, I think you could just get it onto master as soon as possible. We discussed this on the latest Critical Path call and came to that understanding -- though it's also useful to quantify the impact.

[rljacob]

Also, you can do both: get this on master while studying the impact.

[bishtgautam]

@jonbob: Over the weekend, two simulations finished on Edison. I had plans to look at the results today and issue a PR, but Edison is down.

[PeterCaldwell]

@bishtgautam - can't you look at the results on cori? Or did you save the data to edison scratch instead of cori scratch?

[bishtgautam]

@PeterCaldwell: A good suggestion. The files are accessible from Cori. I will compare the results.

[PeterCaldwell]

Did you want me to run AMWG diagnostics on your simulation? If so, let me know the locations.

[bishtgautam]

@PeterCaldwell, Only a single year of simulation completed so far. Is it worthwhile running AMWG diag for only a single year of simulation?

[bishtgautam]

ALM relative differences with and without the bug fix are less than 1% for all variables in the first 12-months for a FC5AV1C-04P2. Thus, I don't believe the bug fix would be a CC and it is not worth running the AMWG diag.

[PeterCaldwell]

Hmm. I suspect you're right, but it's easy to run AMWG and perhaps changing urban albedos has a bigger impact on atmospheric radiation than on land? What's the path?

Also, we should probably document this bugfix on confluence somewhere. I've been dumping stuff in subpages of https://acme-climate.atlassian.net/wiki/display/ATM/Atmosphere+Bug+Report+Page for lack of a better place. Should I create a page for this bug too, or do you have somewhere in the land confluence pages?

[bishtgautam]

Here is the path to output:

• Without bugfix
  /global/cscratch1/sd/gbisht/acme_scratch/edison/FC5AV1C-04P2.ne30_g16.2da48f9/run

• With bugfix
  /global/cscratch1/sd/gbisht/acme_scratch/edison/FC5AV1C-04P2.ne30_g16.a5a9c95/run

The land group doesn't have a confluence page to track bugs.

[PeterCaldwell]

Ok. I'm working on AMWG diags. I made a confluence bug page at https://acme-climate.atlassian.net/wiki/display/ATM/Land+Albedo+%3E+1

[PeterCaldwell]

Ok, it seems clear that this bugfix doesn't have (big) climate effects (see above above confluence page for details). I hope we're already merging it ASAP, but I think we can now do so with confidence that it won't screw up other things.