Scoring Antarctic surface mass balance in climate models to refine future projections

Gorte, Tessa; Lenaerts, Jan T. M.; Medley, Brooke

doi:https://doi.org/10.5194/tc-14-4719-2020

Articles | Volume 14, issue 12

https://doi.org/10.5194/tc-14-4719-2020

© Author(s) 2020. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/tc-14-4719-2020

© Author(s) 2020. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 14, issue 12

Research article

|

23 Dec 2020

Research article |

| 23 Dec 2020

Scoring Antarctic surface mass balance in climate models to refine future projections

Tessa Gorte, Jan T. M. Lenaerts, and Brooke Medley

Download

Final revised paper (published on 23 Dec 2020)
Supplement to the final revised paper
Preprint (discussion started on 17 Dec 2019)
Supplement to the preprint

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'review', Anonymous Referee #1, 14 Jan 2020
- AC1: 'Response to Reviewer 1', Tessa Gorte, 25 Mar 2020
RC2: 'Review of "Scoring Antarctic surface mass balance in climate models to refine future projections" by Gorte et al.', Anonymous Referee #2, 16 Jan 2020
- AC2: 'Response to Reviewer 2', Tessa Gorte, 25 Mar 2020
RC3: 'Review: Scoring Antarctic surface mass balance in climate models to refine future projections', Anonymous Referee #3, 29 Jan 2020
- AC3: 'Response to Reviewer 3', Tessa Gorte, 25 Mar 2020

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

ED: Reconsider after major revisions (further review by editor and referees) (26 Mar 2020) by Michiel van den Broeke

AR by Tessa Gorte on behalf of the Authors (08 May 2020) Author's response Manuscript

ED: Referee Nomination & Report Request started (19 May 2020) by Michiel van den Broeke

RR by Anonymous Referee #3 (27 May 2020)

Suggestions for revision or reasons for rejection

Review
Scoring Antarctic surface mass balance in climate models to refine future projections
Tessa Gorte, Jan T. M. Lenaerts, and Brooke Medley

Submitted to The Cryosphere
Overall, this revised version is much improved – really nice job on revising. My primary concerns have been addressed, and I recommend that this manuscript be accepted for publication with minor technical corrections and revisions, as listed below.

Minor
Lines 25-27 repetitive, delete one sentence (or the other)
Line 47 MB = mass balance? GMSL = global mean sea level? Define
Line 99 MERRA-2 reference?
Line 245 “added retroactively figures 2-3 for comparison? Not in Figure 2…

Figure 1. These colors appear blue on my screen (not purple)?

Figure 2 and text – why are mean, trends shown from 1801-2000 when the model scoring is for 1850-2000? Or am I missing something?

Lines 264-280 Although this is better than the first submission, I still find sections in here become overwhelming and read as a list of numbers. I would prefer having a table presenting the numbers for any readers who want the actual values (this makes it also easier for future work to find values in the paper and cite or compare!) with written text more of a summary (with details in the table). Figure 4 also summarizes quite nicely. This is more of a stylistic recommendation and the authors may choose to disagree and keep as is…..

Line 299 should be “Figure 5A, B” or simply “Figure5”

Lines 302-303 states that the STD for the reanalysis is 6.6% and reconstruction 2.9%, but the figure caption states that 6.6% is the reconstruction. Which is it?

Lines 413-415
It seems like the CESM-LE performs, on the whole, better than the CESM CMIP5 contributions. Why do you think this is the case? A detailed analysis is beyond the scope of this paper – just find this to be very interesting and wonder why this might be (at most 1-2 sentences)? I think there are two possibilities here (although maybe more?). One, there are only 5 individual CMIP5 CESM runs, compared with 35 LE – and only one of the 5 is a clear outlier (FASTCHEM). Thus, the differences in scores and ranges may not be significant and only show that scoring itself is susceptible to internal variability (and what a large range!), and cautions against using single-ensembles to estimate model performance. Another possibility may lie in forcing differences. I believe the CESM-LE does not have the same ozone forcing as the CESM CMIP5 contributions, and I would expect SH to be the region this would matter most (I don’t recall exactly the difference but I think the CESM LE is much closer to obs for ozone forcing, particularly in the SH?).

Also, I am not sure if the shorter overlap of time (LE 1920-2000 and CMIP5 1850-2000) should impact the EOF analysis (the trend, yes maybe). If the modes of variability are robust under strong external forcing, then the LE – with 30 ensembles – should have plenty of samples to adequately estimate EOFS (I’m actually not as convinced of single-ensemble members). The exceptions will be if the EOFs are not robust under anthropogenic forcing, or if there are mulit-decadal modes missing in the CESM LE due to identical ocean initial conditions (although not convinced they would be resolved in single-runs of the different CESM CMIP5 simulations).

Supplementary material
Figure 6. Put “coarser” and “finer” resolution in figure caption and maybe on figure itself (otherwise easily confusing).
Table 1. I think resolution is confusin here. Suggest you make 4 columns: model, resolution (given as lat x lon in traditional manner), total resolution (for correlation calculations eg. Lat/lon), and score. Also many scores are absent?

Table 4. Perhaps elevated to a table in the main text?
What is “GMSL rise buffering”?
Gt yr-2 should be G yr-1 in column one for change in SMB in time

Figure 4. The yellow dot for CESM FASTCHEM is extremely difficult to see (at first I thought it was missing) – for some reason even more difficult in this figure than in Figure 3 in the main text (although hard there too). Suggest changing the color scale a bit or outlining this dot in another color or something to make it visible.

Hide

RR by Anonymous Referee #2 (10 Jun 2020)

Suggestions for revision or reasons for rejection

Review of revision 1 of “Scoring Antarctic surface mass balance in climate models to refine future projections” by Gorte et al.

Although the revised version goes part of the way to responding to my original comments, a number of key concerns remain. These are detailed below summarized against each of my previous major comments that still need addressing (1.1, 1.2, 1.3 and 1.4). I think the manuscript has improved, but as before, these remaining major comments need to be addressed and I have left minor points for future assessment.

Previous comment 1.1: Comparing GCMs with reconstructions.
I am satisfied that analysis of HighResMIP not actually suitable due to the uncoupled nature and shorter duration. I am also satisfied that resolution seems to explain a small amount of inter-model variance in SMB values (just 16%). I would be happy to leave it at that, but the authors seem to contradict themselves on this point.
Specifically, their analysis of the range of resolutions across the CMIP5 and CMIP6 models shows a statistically significant relationship between resolution and SMB (Fig. 5 of supplementary material). They state in their response to reviewers that ‘This result is further exemplified when looking at total scores from the same model run at different resolutions.’ This would seem to strengthen the importance of resolution as a factor that should be considered. The authors then seem to contradict themselves by stating that ‘we conclude that there is no significant correlation between model resolution and total Score’, which is based on splitting the population into high and low resolution subsets in Figure 6 of supplementary information. I don’t follow the rationale for doing this, in particular 6b seems to show a collection of models with almost identical grid spacing. My suggestion is to leave out Figure 6, and use Figure 5 to show that the resolution effect is, although statistically significant, relatively small.

Previous comment 1.2: A lack of mechanistic explanation for why each of the 5 criteria are relevant for improving reliability of projections
Not really. No clear explanation of each criteria.
Leave-one-out can help to see how much difference an evaluation makes to projections. In a response to a later point below the authors state that “In our analysis, we make the significant assumption that the past ability to capture SMB correlates to higher skill in projecting AIS SMB into the future.” If the authors follow the above suggestion this need not to be an assumption. It would not be too much work to test whether past ability to capture SMB correlates to higher skill in projecting AIS SMB by looking at projections and biases across the different models to quantify the extent to which they may be correlated.

Previous comment 1.3 The methodological framework for model weighting.
A more mathematical description of the criteria is important since I still don’t follow some of the key points. Two examples are:
i. A point that still isn’t clear is the consequences of situations with small observational uncertainty. The author’s response states that “In regards to the point wherein the reconstruction uncertainty approaches zero, if this is the case, then all the models would score highly on this criterion, that is correct.” However, my interpretation is that most models would score badly and one or two models might score very well if they happened to be within one standard deviation of the reconstruction uncertainty. Writing out an equation would really help to clarify.
ii. Related to this I’m still not clear on what is meant by “a model that fits entirely within the reconstruction uncertainty” (in 2nd para of Section 3.1). This (I think) needs to be written down mathematically to really clarify. In addition, I couldn’t see a clear explanation of the reconstruction uncertainty of AIS-integrated SMB? Is it 5th-95th percentile range for example?

Previous comment 1.4 The role of internal climate variability in trend and spatial EOF analysis.
Probably the most major concern is that the role of internal variability still isn’t comprehensively addressed. This is of fundamental importance to the success of the method and must be addressed. Two key aspects that illustrate this are:
i. The results of the CESM LENS analysis show a very large range in scoring values for different ensemble members of the same model. This indicates a very strong element of chance (associated with internal variability) in the selection of the top subset (now 8) models, which can be inferred from inspection of Figure 7 (I can see this referenced in the main text). This leads on to the second aspect -
ii. It appears to me that the role of internal climate variability is likely significantly under-estimated in the reconstruction uncertainty of the criteria involving trends and EOFs. The real world can be thought of as one ensemble member and therefore the trends and patterns of variability could include a strong element of chance. One approach that is used in detection and attribution studies is to use models to estimate internal climate variability, since there often aren’t other alternatives. Essentially, even a perfect reconstruction of what happened in the real world would not necessarily represent the forced climate response and additional uncertainty associated with internal climate variability is still a major consideration. The authors should separate these out into observational uncertainty and internal variability uncertainty.

Hide

RR by Anonymous Referee #1 (11 Jun 2020)

ED: Reconsider after major revisions (further review by editor and referees) (15 Jun 2020) by Michiel van den Broeke

AR by Tessa Gorte on behalf of the Authors (29 Jul 2020)

ED: Publish subject to revisions (further review by editor and referees) (03 Aug 2020) by Michiel van den Broeke

ED: Referee Nomination & Report Request started (10 Aug 2020) by Michiel van den Broeke

RR by Anonymous Referee #2 (24 Aug 2020)

Suggestions for revision or reasons for rejection

Review of revision 2 of “Scoring Antarctic surface mass balance in climate models to refine future projections” by Gorte et al.

The authors have done well to address most of the comments raised on the first revision. However, on the role of internal climate variability in defining/applying the scoring criteria further major revision is required. On a positive note, it was very useful to see the equations for scoring presented in the response document. In fact, this has helped in suggesting a way forward towards publication within the existing scoring framework. Below, the steps towards this are explained, incorporating the key general points that:
• internal variability is an important consideration climate model evaluation;
• in this case the scope of ‘uncertainty’ needs to be expanded to include ‘reconstruction uncertainty’ and ‘internal variability’; and
• the CESM Large Ensemble (or indeed other multi-realisation datasets) provides a possibility for estimating the contributions of internal variability to observational uncertainty.

So, on the first point, the recent Science Advances paper by Tokarska et al. (2020) provides just one prominent example of the concept that internal variability is a key aspect of observational uncertainty. To quote from the Materials and Methods section of their paper:
‘The overall observational uncertainty is calculated as the sum in quadrature of the above three effects: structural [data] uncertainty, internal variability, and uncertainty of the blending effect [comes from blending different types of data, e.g. land and SST]. Uncertainty from internal variability dominates the trend uncertainty.’
This is just one example of the very common practise in model evaluation of including internal variability as part of observational uncertainty. Their structural data and blending uncertainties are conceptual parallels to the reconstruction uncertainty used in the scoring criteria. The internal variability component deals with the fact that there is only one realisation of reality and therefore any model-observation comparison could be influenced by this. The authors are correct in their response that the reconstruction uncertainty is not affected by internal variability, but the broader observational uncertainty may be.

My suggestion is that the ‘uncertainty’ term in the scoring criteria must take into account both structural uncertainty and internal variability. I.e.:

uncertainty = reconstruction uncertainty + internal variability

In some cases internal variability may be small, but in others such as EOF patterns or regional trends it will likely be significant (as indicated by the range in overall scores across CESM-LENS members). In the absence of real-world data, model estimates of internal variability are often used (e.g. in D&A studies such as Gillett et al., 2008). Variables with larger internal variability will have less distinguishing power between models (and ensemble members).

To help explain the rationale for this, an extreme example would be the use of 5-years of observations/reconstructions to evaluate the representation of trends in a climate model. Any estimate of an underlying real-world forced climate signal from such a short period would be highly unreliable and any model-observation agreement would be highly likely to be by chance. Therefore preferentially weighting some models over others based on reproduction of short fluctuations would not be a reliable way to test the strength of their long-term forced response. This lack of distinguishing power would be taking into account if internal variability were included in the scoring equation as per above.

The implication from CESM-LENS assessment is that effects such as in the above example mean that some models are being preferentially weighted over others in an unreliable fashion (if one is interested in identifying the best models for projected long-term climate change response). If internal variability is adequately taken into account then, for example, the scores from different ensemble members of CESM-LENS would likely not differ substantially.

As mentioned above, in the absence of any better information one could use CESM LENS ensemble to estimate the internal variability in the parameters of interest. Some will be small (e.g. mean climatologies), but others may be larger. Due to the limited time periods available from CESM LENS, another option would be to use CMIP models with large numbers of ensemble members.

References:
Tokarska, K. B., Stolpe, M. B., Sippel, S., Fischer, E. M., Smith, C. J., Lehner, F., & Knutti, R. (2020). Past warming trend constrains future warming in CMIP6 models. Science Advances, 6(12), eaaz9549. https://doi.org/10.1126/sciadv.aaz9549
Gillett, N. P., Stone, D. a., Stott, P. a., Nozawa, T., Karpechko, A. Y., Hegerl, G. C., … Jones, P. D. (2008). Attribution of polar warming to human influence. Nature Geoscience, 1(11), 750–754. https://doi.org/10.1038/ngeo338

Hide

ED: Publish subject to revisions (further review by editor and referees) (26 Aug 2020) by Michiel van den Broeke

AR by Tessa Gorte on behalf of the Authors (23 Sep 2020)

ED: Referee Nomination & Report Request started (05 Oct 2020) by Michiel van den Broeke

RR by Anonymous Referee #2 (19 Oct 2020)

Suggestions for revision or reasons for rejection

Review of revision 3 of “Scoring Antarctic surface mass balance in climate models to refine future projections” by Gorte et al.

The authors have gone a long way to addressing the final concern on the role of internal variability in the observational uncertainty. There are however a few more clarifications on the edits that would be required before recommending acceptance. The review classification of 'major revision' is mainly a reflection that the below comments are important in terms of correctly communicating the rationale for the reformulated observational uncertainty calculation (and could potentially involve some revision of figures), but ultimately if they are addressed I fully expect to be able to recommend acceptance. It would be a great help if the authors could respond clearly to each one of the below comments in turn.

The comments below relate to two broad points: (i) The explanation of internal variability in the observational uncertainty needs to be developed more. (ii) I was surprised that the figures and scores (specifically Figures 1, 3, 4, 7 and supplementary Table 1) all look identical to the previous version. If they have been calculated to include ensemble (natural) variability I would have expected some very slight changes at least. It would be useful if the authors could show some examples of the each term in the scoring before and after the inclusion of internal variability.

Specific points:

1. Line 77: The statement
- “the intra-ensemble variability takes into account the internal variability of climate models”
needs to be revised to something like
- “the intra-ensemble variability uses climate models to estimate uncertainty in observations due to internal variability of the climate system”.
This is required because the point of the new term in the equation is to capture the fact the observational uncertainty is a combination of reconstruction uncertainty and internal climate variability (estimated here from the LENS ensemble).

2. The explanation relating to the above text should to reference other literature, for example Tokarska et al. (2020) that I mentioned in my previous round of revisions. The authors are welcome to use some of the text from this previous review to help expand the explanation.

3. The notation of equation 1 needs to be changed to reflect the above point. For example as I suggested in my previous round of revisions:
“uncertainty = reconstruction uncertainty + internal variability”

Maybe it would be clearer as follows:
“observational uncertainty = reconstruction uncertainty + internal variability”

4. Apologies if I’m missing something, but it’s not clear what is shown in the numbers in blue in supplementary Figure 5. The average and 1-sigma range in 5B looks like it’s about 1.95 +/- 0.35 Gt yr^-2. How does this relate to the numbers displayed (“obs. unc. = 1.33 Gt yr^-2” and “recon. unc. = 1.38 Gt yr^-2”)?

5. More generally, from Figure 5B in particular the internal variability aspect looks quite large. What is needed is each of the three terms of equation 1 shown explicitly in each panel, which would greatly help in clarifying and comparing the different terms.

6. Figure 7 and 8 of the supplementary material are missing captions.

References:
Tokarska, K. B., Stolpe, M. B., Sippel, S., Fischer, E. M., Smith, C. J., Lehner, F., & Knutti, R. (2020). Past warming trend constrains future warming in CMIP6 models. Science Advances, 6(12), eaaz9549. https://doi.org/10.1126/sciadv.aaz9549

Hide

ED: Publish subject to revisions (further review by editor and referees) (19 Oct 2020) by Michiel van den Broeke

AR by Tessa Gorte on behalf of the Authors (26 Oct 2020) Manuscript

ED: Publish subject to technical corrections (02 Nov 2020) by Michiel van den Broeke

AR by Tessa Gorte on behalf of the Authors (02 Nov 2020) Manuscript

Short summary

In this paper, we analyze several spatial and temporal criteria to assess the ability of models in the CMIP5 and CMIP6 frameworks to recreate past Antarctic surface mass balance. We then compared a subset of the top performing models to all remaining models to refine future surface mass balance predictions under different forcing scenarios. We found that the top performing models predict lower surface mass balance by 2100, indicating less buffering than otherwise expected of sea level rise.