Air temperature observations over land at 2 m were taken from the WATCH Forcing Data methodology applied to the ERA-Interim (WFDEI) data set . These were generated by applying monthly bias corrections from Climatic Research Unit (CRU; ) to the Era-Interim reanalysis data . Air temperatures are available at 0.5∘ resolution and were aggregated to monthly and annual means.

This data set consists of a Northern Hemisphere subset of the Canadian Meteorological Centre (CMC) operational global daily snow depth analysis . The analysis is performed using real-time, in situ daily snow depth observations and optimal interpolation with a first-guess field generated from a simple snow accumulation and melt model driven with temperatures and precipitation from the Canadian forecast model. The analysed snow depths are available at approximately 24 km resolution for the period between 1998 and 2016 and were converted from daily to monthly means. It should be noted that snow depths exhibit high spatial variability and are difficult to measure because of land surface heterogeneity. In addition there are few evaluation data available for the Arctic.

The International Permafrost Association (IPA) map of permafrost presence gives a historical permafrost distribution compiled for the period between 1960 and 1990. It separates continuous (90 %–100 %), discontinuous (50 %–90 %), sporadic (10 %–50 %) and isolated (< 10 %) permafrost. This distribution was generated from the original 1:10000000 paper map, and the version used here was regridded to a 0.5∘ resolution.

The ESA Climate Change Initiative permafrost (CCI-PF) reanalysis data set is a recently developed data set that supplies the mean annual ground temperature at the top of the permafrost (MAGT) and the probability of permafrost for each grid cell. These were derived from an equilibrium model of permafrost at 1 km resolution and provide a snapshot of the 2000–2016 period. The model is driven by remotely sensed land surface temperatures, downscaled ERA-Interim climate reanalysis data, tundra wetness classes and a land cover map. These data were within ∼±2 ∘C of in situ borehole measurements. This CCI-PF analysis of permafrost extent is within the range of but slightly lower than the estimate of . We use a version of the CCI-PF which has been regridded to a 0.5∘ resolution.

An alternative way of deriving an observational-based estimate of permafrost presence is to derive the probability of permafrost from the observed mean annual air temperature (MAAT). An observational-based relationship was defined by , who updated . The relationship has a 50 % chance of the presence of permafrost at -4.3 ∘C. Using the relationship, we reconstructed a permafrost probability map from the WFDEI estimates of MAAT. In addition we applied the relationship to the MAAT for each model to estimate a benchmark permafrost distribution specific to each model (PFbenchmark). This model-specific PFbenchmark can be used to evaluate the land surface module independently of any climate biases in MAAT.

Table  summarises the different observational-based permafrost distributions. The permafrost-affected area (PFaff) is defined as the area where the probability of permafrost is greater than 0.01 and is expected to be very similar to the land surface area where the MAAT is less than 0 ∘C. Table  shows that this is the case for the and data set, but the CCI-PF data set has a slightly lower PFaff area. It should be noted that the observational data set was one of the data sets used to develop the relationship but the CCI-PF reanalysis data set was developed independently. The permafrost extent (PFex) is the area of permafrost weighted by the proportion of permafrost in each grid cell, and PF50% is the area of the grid cells where the probability of finding permafrost is greater than 0.5. These two definitions produce a very similar land surface area. Overall there is some uncertainty in the proportion of the land surface with MAAT<0 ∘C that contains permafrost – the observational estimates are 0.55, 0.62 and 0.77. The data have a similar value for the PFaff area but a higher probability of finding permafrost at air temperatures less than -4.3 ∘C. Overall the CCI-PF data show consistently less permafrost (Table ).

The Circumpolar Active Layer Monitoring Network (CALM; ) is a network of over 100 sites at which ongoing measurements of the end-of-season thaw depth or the ALT are taken. Measurements are available from the early 1990s, when the network was formed, to the present.

MAAT, MAGT, snow depth and MAGST (mean annual ground surface temperature at 0.2 m) were available for a range of sites in Russia and Canada . The data at Russian meteorological stations are from the All-Russian Research Institute of Hydrometeorological Information – World Data Center (RIHMI-WDC), and data at Canadian climate stations are from Environment and Climate Change Canada. This gives data from ∼330 stations . Additional data are available from the Global Terrestrial Network for Permafrost (GTN-P; ). Ground temperatures are measured in >1000 boreholes at a wide variety of depths. Years with a complete seasonal cycle were extracted from selected boreholes where the data available include MAAT, MAGST and MAGT. A full description of these data and their post-processing is included in .

Snow has a big impact on the soil temperatures and presence or absence of permafrost in the northern high latitudes . Here we use the effective snow depth, Sdepth,eff which describes the insulation of snow over the cold period. Sdepth,eff is an integral value such that the mean snow depth in each month, m (Sm in metres) is weighted by its duration: 1Sdepth,eff=∑m=1MSm(M+1-m)∑m=1Mm. It is assumed that the snow can be present anytime from October (m=1) to March (m=6) with the maximum duration, M, equal to 6 months. This weights early snowfall more than late snowfall as it will have a greater overall insulating value. The insulation capacity of the snow changes little with snow depth when Sdepth, eff increases above ∼0.25 m , and seasons with an earlier snowfall will generally have a greater Sdepth,eff than seasons with a later snowfall.

The winter offset is defined as the difference between the mean soil temperature at 0.2 m and the mean air temperature for the period from December to February. This is expected to be positive with the soil temperature warmer than the air temperature. The summer offset is defined in a similar manner for the period between June and August. This is expected to be slightly negative with the soil temperature cooler than the air temperature. The surface offset is the sum of the summer and winter offset but is dominated by the winter offset. The thermal offset is the temperature difference between the annual mean soil temperature at 0.2 m (mean annual ground surface temperature – MAGST) and the annual mean soil temperature at the top of the permafrost (mean annual ground temperature – MAGT). This is expected to be slightly negative with MAGT slightly colder than MAGST.

In this paper the preferred method of defining permafrost is to diagnose the temperature at the depth of zero annual amplitude (Dzaa). Dzaa is defined as the minimum soil depth where the variation in monthly mean temperatures within a year is less than 0.1 ∘C. If the temperature at the Dzaa is less than 0 ∘C for a period of 2 years or more, there is assumed to be permafrost in that grid cell. However, only six of the CMIP6 models have a soil profile deep enough to identify the Dzaa (Table ). In the remainder of the models, the maximum soil depth is less than the Dzaa, so an alternative method of identifying the presence of permafrost is required. For these models permafrost is assumed to be present in grid cells where the 2-year mean soil temperature of the deepest model level is less than 0 ∘C. This definition was used by , who suggested that if the mean soil temperature of the deepest model level is below 0 ∘C and assuming constant soil heat capacity, there is likely to be permafrost deeper in the soil profile. However, this definition does not explicitly recognise permafrost in the soil profile – in order to do that, the maximum soil temperature of the deepest model level should be less than 0 ∘C. The main issue with this latter method is that, if the soil profile does not extend deep enough, the deepest model level may fall in the ALT and the permafrost extent will be underestimated.

Subgrid-scale variability is not taken into account in this assessment – the models are assumed to have either permafrost or no permafrost in each grid cell. However, the observations either are very high resolution (CCI-PF is at 1 km resolution in its original format) or supply a probability of permafrost for each grid cell . Therefore in order to compare the observed extent with those from the models, we assume that any grid cell where the observations have ≥50% permafrost should be identified by the models as having permafrost and any grid cells with <50% will be identified as not having permafrost. The observed values of this threshold (PF50%) are shown in Table  and are approximately equal to the permafrost extent (PFex, also shown in Table ), which is defined as the observed area of permafrost which takes into account the proportion of permafrost in each of the grid cells.

The thawed depth from the surface is defined for each month using the depth-resolved monthly mean soil temperatures. The soil temperatures were interpolated between the centre of each model level and the thaw depth defined at the minimum depth where it reaches 0 ∘C. Some of the models have a very poorly resolved soil temperature profile which will introduce some biases into this estimate . In addition, taliks (unfrozen patches within the frozen part of the soil) will not be identified using this method. The annual maximum active layer thickness (ALT in metres) is defined as the maximum monthly thaw depth for that year.

Under increasing temperature both the ALT and the time the soil is thawed will increase via an earlier thaw and later freeze up. Therefore, defined the annual mean thawed fraction (D̃) for permafrost soils which can be expressed in units of cubic metres per cubic metre: D̃=1zmax112∫112∫0zmaxH(T(z,t))dzdt,2H(T(z,t))=1if T(z,t)>0,0if T(z,t)≤0, where t is the time in months, z is the depth in metres, T is the soil temperature at time t and depth z, and zmax is the maximum depth of the soil under consideration. Here we assume zmax is 2 m which is relatively shallow but enables the models with shallower soil depths to be included consistently within the analysis. The annual mean frozen fraction (F̃) is the frozen component of the soil and given by 3F̃=1-D̃. One advantage of using D̃ over ALT is that it enables taliks to be identified, although this will become more relevant when considering soils deeper than 2 m. In addition, D̃ is a first-order proxy for the soil carbon exposure to decomposition in any particular grid cell.

The annual thawed volume (D̃tot in cubic metres) is the sum of the area-weighted values of zmaxD̃ for each grid cell in the present-day permafrost region. Any non-permafrost grid cells are masked. Similarly the annual frozen volume (F̃tot in cubic metres) is the sum of the area-weighted zmaxF̃ for each grid cell again defined for the present-day permafrost region. For any future projections, if there is no longer freezing in a specific grid cell (which had permafrost in the present day), D̃ is set to 1 and F̃ is set to 0.

We can derive an observational-based estimate of present-day D̃tot using the available site-specific data between D̃ and MAGT described in Sect.  and shown in Fig. . The CCI-PF data set was then used in conjunction with this relationship to estimate D̃ over the permafrost region. D̃ is related non-linearly to the MAGT – the warmer the ground temperature, the bigger the annual mean thawed fraction. Therefore a second-order polynomial was fitted to the site-specific relationship between D̃ and the MAGT – green line in Fig. . The dashed black lines show the relationship for the 95 % confidence intervals. These three curves were then used in conjunction with the CCI-PF data set to derive the mean and range of D̃ for each grid cell with permafrost present. Assuming zmax is 2 m and summing over the CCI-PF permafrost area give a present-day D̃tot of 5.6×103 km3 (range 2.1–9.5×103 km3). F̃tot is then 22.2×103 km3 (range 18.4–25.8×103 km3).

Figure  suggests that in order for a land surface module to accurately represent permafrost it needs to be able to represent the insulating ability of the lying snow. This is assessed in Fig.  which shows the insulating capacity of the snow in terms of the difference between the winter air temperature and winter 0.2 m soil temperature. The offsets are a function of both MAAT (not shown) and Sdepth, eff. A low MAAT and a high Sdepth,eff gives a bigger offset (see also ). In Fig.  only grid cells where the winter mean air temperatures are between -25 and -15∘C are shown, similarly to the observed sites. This ensures the comparisons are not biased by differences in air temperatures. The available models reflect the general increase in offset with increasing Sdepth,eff for the shallow snow and the saturation of this relationship for the deepest Sdepth,eff to varying degrees of accuracy. A few of the models (FGOALS-f3-L, TaiESM1 and UKESM1-0-LL) have relationships between offset and Sdepth, eff which are indistinguishable from the observed relationship. The rest of the models have an offset at any given Sdepth,eff which is generally too small, suggesting that these models do not have enough snow insulation. The net impact of the snow offset needs to be interpreted carefully in combination with the Sdepth,eff in order to evaluate the impact of the snow insulation on permafrost dynamics. Figure  shows that Arctic snow depths are relatively shallow. Therefore, because there is a non-linear relationship between offset and Sdepth,eff, small differences in Sdepth,eff will have a big impact on the insulating ability. The models typically tend to slightly overestimate Sdepth,eff and slightly underestimate the offset for any given value of Sdepth,eff.

Figure S2.6 shows the equivalent plots for the available CMIP5 models. A similar pattern is observed where the models are more likely to underestimate than overestimate the snow insulation. Although limited availability means it is hard to compare individual models between the CMIP5 and CMIP6 ensemble, specific models can be identified. Specifically CanESM and MIROC show improvements; MRI and GISS show little change, and CESM and NorESM show some degradation.

Figure  shows the combined impact of the three offsets on the relationship between MAGT and MAAT compared with an observational-based assessment made using the CCI-PF MAGT and the WFDEI MAAT. As expected the MAGT increases with MAAT, with the CCI-PF MAGT approximately 4.5 ∘C warmer than the WFDEI MAAT. As discussed earlier, this difference is dominated by the winter offset, but the summer and thermal offsets also contribute. Also shown are the same relationships for the models. Differences in snow insulation are reflected here. For example, despite recreating the observed relationship between winter offset and Sdepth,eff, the BCC-CSM2-MR and UKESM1-0-LL models have a much larger difference between MAAT and MAGT than the observations at the colder temperature, because there is too much snow on the ground in the high Arctic. CESM2 has a very similar relationship between MAAT and MAGT to the observations (Fig. ) despite having a smaller winter offset than is observed for any given value of Sdepth,eff (Fig. ). However, it has a larger-than-observed Sdepth,eff which increases the insulation and ensures a good relationship between MAAT and MAGT in Fig. . A comparison with the CMIP5 multi-model ensemble (Fig. S2.7) shows similar differences. It should be noted that this CCI-PF estimate of MAGT is a model-derived reanalysis and the observational uncertainties are likely underestimated in the current analysis.

An assessment of the thaw depth of the permafrost during the summer gives another indication of whether the model has the correct physics. Differences between the model and observations are apparent in Fig. . The observed relationship between the ALT and the MAAT in the CALM data set is shown in blue. Both the observed ALT and the spread of possible ALT increase with increasing MAAT. The spread of values increases because the active layers are more strongly impacted by environmental factors other than air temperature such as topography, soil type and solar radiation at the warmer temperatures. For each model the percentage of observed CALM sites where there is model-simulated permafrost is shown in the figure. The maximum observed ALT is about 4.5 m. Any models with a maximum soil depth of less than 4.5 m will be unable to represent this value (Table ). This is the case for GISS-E2-1-G, where the depth of the middle of the bottom layer is 2.7 m and the ALT is constrained to 2.7 m at the warmer temperatures. Poor vertical discretisation of the soil such as the three layers in CanESM5 can introduce large variability into the derivation of ALT. Although the average number of soil layers and the average soil depth increases between the CMIP5 and CMIP6 ensembles (CMIP6 – Table ; CMIP5 – Table S2.1), this is not universally true.

About half of the models have relationships between ALT and MAAT that are comparable with the observations. These are mainly the models with deeper soils. Other models have a very deep ALT, for example, MPI-ESM1-2-HR and IPSL-CM6A-LR. These both have sufficiently deep soil profiles but thaw too quickly in the summer, likely because these models do not represent the latent heat of the water-phase change. EC-Earth3 and UKESM1-0-LL have active layers around 2 m irrespective of MAAT. In the case of UKESM1-0-LL, this is worse than the CMIP5 version of the model where ALT was dependent on MAAT, just with a smaller uncertainty range (Fig. S2.8). This is because there is a large increase in the MAGT between the CMIP versions (MAGT is -5.8 ∘C for CMIP5 and MAGT and -0.9 ∘C for CMIP6) caused by adding a multilayered snow scheme . The inclusion of organic soils and the addition of a moss layer improve the insulating capacity and ability of the soil to hold water and will reduce this thaw depth .

MAGT and ALT and their relationships with MAAT are important diagnostics for model physics. However, permafrost extent and annual thawed and frozen volume are more relevant when exploring the impacts of permafrost dynamics under future climate change. There are observational-based estimates of permafrost extent available for evaluation, and this paper introduces an observational-based estimate of annual thawed and frozen volume by extrapolating the site-specific relationship between annual mean thawed fraction (D̃) and MAGT to the larger scale using the CCI-PF data set.

Figure  shows that the relationship between annual mean thawed fraction (D̃) and MAGT for the available CALM and GTNP data sets is relatively well constrained. As expected, D̃ increases with increasing MAGT. Figure  also shows the ability of the models to replicate this relationship. At the warmer temperatures the models tend to show much more variability than the observations. In models such as EC-Earth3 and UKESM1-0-LL this is likely reflecting the sensitivity to the duration over which the soil is thawed – because in these models the ALT is very similar at all temperatures (Fig. ). Overall the models follow a similar trend of increasing D̃ with increasing MAGT. There are notable discrepancies for IPSL-CM6A-LR and MPI-ESM1-2-HR which might be expected because these two models simulate very deep ALT. D̃ can be converted to annual thawed and frozen volumes (D̃tot and F̃tot) using the monthly profiles of modelled soil temperature and compared with the observational-based estimate of D̃tot and F̃tot discussed in Sect. . Figure  summarises PFex, D̃tot and F̃tot for each of the CMIP6 models.

The top plot of Fig.  shows a comparison of PFbenchmark with the observations (empty black bars). All the models fall close to the observed values, as is expected from Table S1.2. PFex (red bar) should be compared with PFbenchmark and not the observations, and any differences between the two quantify a bias added by the land surface model. Any differences here are again dominated by the snow insulation, which is a combination of the Sdepth,eff and the relationship between MAGST, MAAT and Sdepth,eff. When compared with the CMIP5 multi-model ensemble (Fig. S2.10), the spread of values is smaller, with the differences between PFex and PFbenchmark significantly lower in some cases. Assessing Fig.  in conjunction with Figs. S1.2 and suggests some improvement in snow insulation between the CMIP5 multi-model ensemble and the CMIP6 multi-model ensemble.

Figure  also shows D̃tot and F̃tot for each CMIP6 model. These metrics are calculated for the present-day permafrost region defined by the model-specific PFex. Any differences between models are caused by a combination of differences in the MAGT and differences in the relationships between D̃ and MAGT. These lead to the considerable spread in D̃tot and F̃tot shown in Fig. . There is a tendency for the models to overestimate D̃tot and underestimate F̃tot. This arises partly because of the higher likelihood of the larger values of D̃ at the warmer air temperatures, i.e. in the discontinuous and sporadic permafrost (Fig. ). The CMIP6 multi-model ensemble has a similar uncertainty to that of CMIP5, suggesting little improvement in the quantification of D̃tot and F̃tot between ensembles.

This section summarises some basic evaluation metrics which can be applied to quantify the ability of the land surface modules to represent permafrost dynamics. Relevant climate-related metrics are summarised in Table  and shown in Table S1.1 for individual CMIP6 models and Table S2.2 for individual CMIP5 models. Table  shows a summary of the land-related evaluation metrics for the observations and the two different multi-model ensembles along with the percentage of each multi-model ensemble falling within the range with individual models shown in Tables S1.2 and S2.3. These metrics should be relatively independent of climate and reflect the behaviour of the land surface module. Of particular note is the ALT, a key metric when simulating permafrost dynamics, which is much deeper than the observations for both CMIP5 and CMIP6.

There is better agreement with the observations for the climate-related metrics (Table ) than for the land-surface-related metrics with only a very small number of models falling within the range of the observations. In addition, there is no consistency in model performance – no model performs well for every evaluation metric (Tables S1.2 and S2.3). Overall, the percentage of the models which fall within the observed range is relatively low with the majority falling outside the range of the observations (Table ). However, there are some improvements for all the metrics in the percentage of models that fall within the observed range between the CMIP5 multi-model ensemble and the CMIP6 multi-model ensemble.