Large-scale snow data assimilation using a spatialized particle filter:

recovering the spatial structure of the particles

Jean Odry 1 , Marie-Amélie Boucher 1 , Simon Lachance-Cloutier 2 , Richard Turcotte 2 , and

Pierre-Yves St-Louis 2

Review by Bertrand Cluzet

GENERAL:

I discovered and read this paper with a great pleasure. Data assimilation of snow observation, in particular using the Particle Filter (PF) has been a growing topic in the snow/hydrology modelling community over the last five to ten years. However, this algorithm suffers from two strong limitations when it comes to large scale, data-scarce problems (which are the general case in the field). The first one is the curse of dimensionality, which can be solved by localizing the PF. But this solution generates a second strong limitation: individual members of localized PFs exhibit discontinuous spatial fields and noisy spatial correlation fields which are often detrimental for the PF performance itself.

Capitalizing on a localized PF variant using a spatial interpolation of the PF weights from Cantet et al., (2019), the authors efficiently introduce an approach coming from the hydrological modelling, the Schaake Shuffle to solve for both limitations: the localisation mitigates the curse of dimensionality, and the Schaake Shuffle is used to enforce the spatial structures of a deterministic run in the individual ensemble members in an elegant way.

Overall, I find that this paper is well within the scope of the journal, and has the potential to be a significant contribution to the snow/hydrology data assimilation community and beyond, because they address an important problem in a convincing and elegant way. I must admit, however, that I’m not satisfied with the theoretical justifications for the use of the interpolation of the weights within the PF from Cantet et al. (2019) (IDWPF), instead of the classical localised (LPF, e.g. Farchi and Bocquet, 2018). I would ask for more justifications, or a comparison between the LPF and the IDWPF.

The scientific quality of the writing is sometimes lacking rigor, especially in the Sections 1-3. I would ask for a significant effort on that. Nevertheless, I really appreciated the compactness of the paper and the efficiency of the results and discussion sections, which make a very clear and straightforward demonstration of the author’s point.

To wrap up, I see a lot of potential in this paper, and despite my concerns, I am very confident that the authors will be able to address my comments in a revised version of the manuscript. Please find below some details on my main comments. I’ m pleased to provide an annotated pdf version of the manuscript with comments and suggestions throughout.

MAJOR comments

(1)

The authors an interpolation of the particle filter weights (IDWPF) from Cantet et al., (2019) rather than a classical PF localisation (LPF) (see the review from Farchi et al., (2018)). I agree that the basic idea as formulated l 71-73 (a good-performing particle at a close location must also be good locally) resembles the theory. But in the classical LPF, based on Bayes theorem, and assuming equal prior weights, the posterior weights are computed by multiplying the likelihoods of the particles at the different (independent) observation sites (Eq. 27 from Farchi et al., (2018)) and then, normalising. Here, the IDWPF averages the normalised likelihoods of the particles at the different observation sites. There is a substantial conceptual difference here: averaging instead of multiplying. I suspect that this results in less sharp and potentially suboptimal (more conservative) PF analyses. For example, if a particle is given a zero likelihood at one location, the LPF will reject it, while there is still a chance for it to survive in the IDWPF.

Moreover, the arguments in Cantet et al., and the present manuscript used to justify the used of the IDWPF instead of the LPF failed to convince me: the LPF also proposes a ‘ tapering’ method to smoothly reduce the influence of the observations with the distance (Eqs. 28-29 from Farchi et al., (2018)).

To wrap up, I’m not saying that the IDWPF method is wrong, and should be rejected. I can actually imagine that it could be more resilient to outliers in the observations, and its conservativeness could be an advantage. There may also be references in the literature to serve as base for the IDWPF. But in the present form, the justifications provided to substantially deviate from the main theory, the LPF, are too weak for me. I would suggest to make a considerable effort on justifications, or even to compare the IDWPF with the LPF. The latter would have the benefit of significantly increasing the potential impact of this paper thanks to the use of a more ‘orthodox’ method.

To help with the discussion, I’m pleased to provide a toy example comparing the IDWPF and the LPF in the form of a jupyter code attached or publicly available at:

https://github.com/bertrandcz/da_notebooks

(2)

The scientific quality and rigor of the writing is often not satisfactory in its present form, in particular in Secs. 1-3:

→ Even though there is no doubt that the authors have a deep understanding of the PF and its terminology, there is sometimes a lack of rigor in the terminology and approximations that make several sentences turn wrong, and arguments fall short (e.g. l. 54, l.207-208, l. 218-219).

→ Even though the arguments are there, the logical formulation behind certain paragraphs is too loose to be convincing. I have no doubt that the authors can address that, but a considerable effort is required here.

→ There are some approximations in the description of the literature which may induce the reader into having misconception on the references (e.g. l. 69, l. 227-228), and change the conclusions of some paragraphs

→ the observation dataset and study area (Sec. 2) must be described with more details and rigor.

I’m pleased to provide several suggestions on these points in the attached pdf.

MINOR COMMENTS:

(1)

With the Schaake shuffle as used here, the authors enforce the spatial distribution of individual particles to match those of the model, (instead of historical observations): but by doing so, don’t we miss the opportunity to adjust the ensemble to the observed spatial structures? I’d be curious about overlaying Figs 3. and 6 with observed (in-situ) values to assess that.

(2)

The abstract is lacking of a general scientific context to start with. I think that the start is too technical for the scope of TC, the notion of ‘particles’ should be introduced, and given the level of technicity of the paper, a brief sentence describing the particle filter might be required in the abstract, in particular to make the need for a reordering possible to understand. Details on the ensemble construction might be appreciated also. Would benefit from a more rigorous description of the observations and validation data sets.

(3)

When computing global metrics (Secs. 4.2 and 4.3, Fig 7,8 and 9), it could be fair and interesting to compare the assimilation products with their ensemble counterpart without assimilation, not only the deterministic run (called ‘open loop’ in the paper). Ensembles are often favored compared to deteministic runs in terms of RMSE and SWE, and it would enable the authors to put into perspective the impact of the assimilation in terms of spread-skill and CRPS.

References:

Cantet, P., Boucher, M.-A., Lachance-Coutier, S., Turcotte, R., and Fortin, V.: Using a particle filter to estimate the spatial distribution of the snowpack water equivalent, Journal of Hydrometeorology, 20, 577–594, https://doi.org/10.1175/JHM-D-18-0140.1, 2019.

Farchi, A. and Bocquet, M.: Review article: Comparison of local particle filters and new implementations, Nonlinear Processes in Geophysics, 25, 765–807, https://doi.org/https://doi.org/10.5194/npg-25-765-2018, 2018.

https://github.com/bertrandcz/da_notebooks/

This manuscript develops and evaluates methods to maintain appropriate spatial correlation when using a particle filter to estimate SWE for a model of southern Quebec. The application is clearly defined and the problems that can arise when using a particle filter with affordable size are described. As noted in the manuscript, it is well-known that particle filters can diverge and that the number of particles needed to avoid this divergence increases at least exponentially with the number of spatial degrees of freedom in the model. While the number of degrees of freedom in the SWE model being used is not explicitly investigated, indirect evidence is provided that 500 particles is insufficient for the application at hand.

The authors describe the problems that arise from the limited number of particles in terms of an inappropriate ‘scrambling’ of the resampled particles at adjacent grid points. Small differences in the impact of observations can lead to a resampled particle with quite different values at adjacent gridpoints that are believed to be strongly correlated for example. To address this, the authors propose to ‘reorder’ the association between the values and the particle index at each gridpoint. Their control procedure simply sorts the particles at each grid point so that particle 1 is associated with the smallest value of SWE for each model point. The new method proposed, referred to here as a Schaake Shuffle, uses a reference set of ‘particles’ for the model points, in this case generated by sampling periodically from a free run of the SWE model. This reference distribution might be referred to as a ‘climatological sample’ in some other earth system assimilation applications. This reference distribution provides information about the correlation structure of the free model. As a hypothetical example, it could include information that when SWE is higher in western Quebec it is usually lower in eastern Quebec. This type of information could be reflected in the particle filter assimilation after the use of the Schaake Shuffle reordering.

A number of metrics are used to assess different aspects of a basic particle filter, the naïve sorting, the Shuffle, and the free run (open loop). No error estimates are provided for most of the results so it is difficult to assess the significance, and this is something that should at least be discussed if it cannot be formally addressed. For instance, it is difficult to assess if the 4 different curves in Figure 2 are meaningfully different. It appears that the basic particle filter is an outlier, while the other three are indistinguishable, but appearances can be deceiving. It is even more difficult to assess the significance of differences in Figure 3. In this case, maybe the open loop is an outlier, but I have no idea at all whether there are any meaningful differences in the other three plots. Any guidance the authors could provide would be helpful. At this point, I would be forced to conclude that there is no evidence that the 3 particle filter methods produce significantly different estimates of pointwise SWE.

There clearly are meaningful differences between the 3 filter methods in some of the subsequent figures. Not surprisingly, the variograms in figure 4 are very different. However, more evidence about which is better could be provided in the discussion. I suspect that a solid argument could be made that the Shuffle results are probably better, but this requires knowing something about the correct answer and the authors should try to discuss how that could be known with some additional clarity. Figure 6, perhaps the most important in the manuscript, clearly shows a difference in the correlations between the base particle filter and the two correction methods. This is important since the thesis was that the correlations were damaged by the particle filter. However, no solid evidence is provided of what the answer should be for this application. I believe that the correlation scales are probably much larger than the base case, but I am not convinced that they are long as indicated by the sort and Shuffle. Again, if the authors could provide some information about what the right answer is believed to be it could strengthen the argument for using one of the new methods.

In summary, the manuscript is very clear in its description of the application, the challenges to the particle filter, and the description of the new methods. It is less clear in providing evidence about the efficacy of the new methods. It is my somewhat uninformed opinion that the Shuffle has some nice features, but stronger evidence of this would be a nice addition.

As an end note, I would suggest that state-of-the-art ensemble filters, or localized particle filters that make use of some of the advantages of ensemble filters, could be a competitive alternative for this application. Ensemble Kalman filters derive much of their power by being able to approximate the most important covarying directions in model phase space which is what the particle filter is unable to do. Localizing the ensemble filter can result in high-quality assimilated estimates of covariance with ensembles much smaller than 500 members. For instance, work by Zhang https://doi.org/10.1002/2015JD024248 and references cited therein report on ensemble Kalman filter data assimilation using multiple types of observations in a comprehensive land surface model. Work by Poterjoy documents the power of localized particle filters using a theoretically-supported approach that could extend Zhang’s results to deal better with the bounded nature of SWE, https://doi.org/10.1175/MWR-D-15-0163.1

Work by Anderson extends ensemble filters to bounded quantities like SWE while retaining the high-quality covariance estimates from localized ensemble filters https://doi.org/10.1175/MWR-D-19-0307.1

The authors might want to evaluate the efficacy of some of these methods for the Quebec SWE problem in their future work.