Convolutional neural network and long short-term memory models for ice-jam predictions

Madaeni, Fatemehalsadat; Chokmani, Karem; Lhissou, Rachid; Homayouni​​​​​​​, Saeid; Gauthier, Yves; Tolszczuk-Leclerc, Simon

doi:https://doi.org/10.5194/tc-16-1447-2022

Articles | Volume 16, issue 4

https://doi.org/10.5194/tc-16-1447-2022

© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/tc-16-1447-2022

© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 16, issue 4

Research article

|

22 Apr 2022

Research article |

| 22 Apr 2022

Convolutional neural network and long short-term memory models for ice-jam predictions

Fatemehalsadat Madaeni, Karem Chokmani, Rachid Lhissou, Saeid Homayouni, Yves Gauthier, and Simon Tolszczuk-Leclerc

Download

Final revised paper (published on 22 Apr 2022)
Preprint (discussion started on 23 Jul 2021)

Interactive discussion

Status: closed

RC1:
'Comment on tc-2021-194', John Quilty, 14 Aug 2021

Please see attached for my review.

Best regards,

John Quilty

Citation: https://doi.org/10.5194/tc-2021-194-RC1
- AC1: 'Reply on RC1', FATEMEHALSADAT MADAENI, 22 Oct 2021
  
  The comment was uploaded in the form of a supplement: https://tc.copernicus.org/preprints/tc-2021-194/tc-2021-194-AC1-supplement.pdf
  
  Citation: https://doi.org/10.5194/tc-2021-194-AC1
CC1:
'Comment on tc-2021-194', Rahim Barzegar, 22 Aug 2021

Please find attached my comments.

Citation: https://doi.org/10.5194/tc-2021-194-CC1
- AC2: 'Reply on CC1', FATEMEHALSADAT MADAENI, 22 Oct 2021
  
  The comment was uploaded in the form of a supplement: https://tc.copernicus.org/preprints/tc-2021-194/tc-2021-194-AC2-supplement.pdf
  
  Citation: https://doi.org/10.5194/tc-2021-194-AC2
RC2:
'Comment on tc-2021-194', Rahim Barzegar, 24 Aug 2021

Review report on “Convolutional Neural Network and Long Short-Term Memory Models for Ice-Jam Prediction”

Reviewer: Rahim Barzegar

Recommendation: Major revision

General evaluation:

The paper deals with using deep earning models (e.g., LSTM, CNN, and CNN-LSTM) to predict ice jams (jam or no jam) for all the rivers in Quebec. Several hydro-meteorological variables including liquid precipitation (mm), min and max temperature (°C), AFDD (from August 1st; °C), ATDD (from January 1st; °C), snow depth (cm) and net radiation (W m-2) were used as inputs to classify jam or no jam occurrence in the rivers. The dataset was divided into training and validation sets and after constructing and testing the developed models, statistical metrics were considered to assess the performance of the models. It was shown that the hybrid CNN-LSTM model outperforms among developed models.

Despite the logical and valuable results, the paper needs to be organized better. It is somewhat advisable to move some of the results and discussion to the materials and method section. I suggest that the authors broaden their discussion and don't just report performance data on models. A significant revision needs to be made to the conclusion. Overall, in my view, the paper needs a major revision.

Comments:

1) Abstract should be most informative. The abstract discusses ice jam and prediction necessity in the first half. The reader should also be able to obtain information from the data, modeling process, and validation metrics.

2) Acronyms (e.g., CNN, LSTM, CN-LSTM, F1) should be defined in the first use in the abstract.

3) Provide a reference for empirical and statistical prediction methods (threshold methods, multi-regression models, logistic regression models, and discriminant function analysis)

4) Introduce “involved hydro-meteorological variables” in line 24.

5) Why did you choose deep learning over other machine learning models for ice jam prediction? Is that all due to an automatic feature selection? Clarification is needed.

6) There is a lot of focus on time series predictions in the literature review while you should be more specific about ice jam prediction. The literature should at least include data-driven models for predicting ice jams

7) In lines 97-98: the authors state “Deep learning methods are promising to address the requirements of ice jam predictions.” Is there any research to use deep learning for ice jam prediction? If so, what is the contribution to the current research

8) Although there are several deep learning methods, why did you select CNN, LSTM, and a combined CNN-LSTM?

9) The authors consider 0 value for “NaN” precipitation values. 0 value means there is no precipitation. However, there might be precipitation and there is no measurement. Since the modeling is based on time series, it is better to impute missing values instead of considering 0 values.

10) The text contains several typos

11) Provide a reference for the statement “The most popular deep neural networks for TSC are MLP, CNNs, and LSTM.”

12) The sub-heading “Input data and study area” is not appropriate. The reader may think you mean input variables of the model. Sub-heading “Data and study area” should be suitable.

13) Detailed information, at least as a statistical description, should be provided for the data used in the study.

14) Provide a reference “As a benchmark, a CNN model with the parameters and layers similar to previous studies is developed.”

15) Section 2.5.1. Overcome overfitting is too long. It needs to be shortened

16) It is not clear how you optimize the structure of the model. Did you use any hyperparameter tuning method e.g. GridSearch, random search, Bayesian optimization? Or only a trial and errors approach?

17) The training chapter “2.5 Training” has focused only on the general statements of the modeling. The authors should give more information about the modeling process, optimum values for the parameters, loos function, etc.

18) Provided Information in “3.1 Hyperparameters optimization” is not a result! It is a model development and should be moved to the materials and method chapter as a “model development” sub-chapter. The structure of the manuscript needs to be revised

19) I suggest developing a traditional machine learning model such as MLP, random forest, … to show the efficiency of the developed deep learning models.

20) Abbreviations should be checked throughout the text. Different abbreviations (e.g. CN-LSTM, CNN-LSTM, CNLSTM) has been used.

21) Discussion should be improved significantly. The authors have focused only on the structure of the models. The results should also be connected to the physical characteristics of the ice jam. How does the model can help manage the water resources? Compare the results with the literature, not a simple report

22) Conclusion needs to be revised significantly. You should only give the main findings of the research here. Providing reference is not recommended in the conclusion section.

23) Provide the limitations of the research along with the recommendations for the future studies

24) The mean error of training in LSTM is much higher than validation (middle plot in figure 16) which might be a sign of overfitting.

Citation: https://doi.org/10.5194/tc-2021-194-RC2
- AC3: 'Reply on RC2', FATEMEHALSADAT MADAENI, 22 Oct 2021
  
  The comment was uploaded in the form of a supplement: https://tc.copernicus.org/preprints/tc-2021-194/tc-2021-194-AC3-supplement.pdf
  
  Citation: https://doi.org/10.5194/tc-2021-194-AC3

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

ED: Reconsider after major revisions (further review by editor and referees) (26 Oct 2021) by Homa Kheyrollah Pour

AR by FATEMEHALSADAT MADAENI on behalf of the Authors (08 Dec 2021) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (18 Dec 2021) by Homa Kheyrollah Pour

RR by Anonymous Referee #2 (18 Jan 2022)

RR by John Quilty (24 Jan 2022)

Suggestions for revision or reasons for rejection

Review tc-2021-194_ATC1

TITLE
Convolutional Neural Network and Long Short-Term Memory Models for Ice-Jam Prediction

RECOMMENDATION
Minor revision

REVIEWER
John Quilty

GENERAL COMMENTS

For the most part, the authors addressed the various comments I had on the previous version.
According to my earlier comments, the authors:
• Included benchmark results for several machine learning models, demonstrating that the deep learning models had higher performance than these benchmarks.
• Re-worked the manuscript structure to better separate model development details from the experiment results.
• Provided discussion on the interpretability and spatial generality (model transferability) of the deep learning models.
Despite the manuscript being improved, there are still several issues that need to be resolved (improved literature review, clearer description of model development details, etc.) as outlined in my comments below. Finally, the paper would greatly benefit from thorough editing (a number of grammatical issues still exist in the text and others were introduced in the revised manuscript).
I think the paper needs a minor revision before it can be considered for publication.

SPECIFIC COMMENTS

Line (L) numbers mentioned in my comments refer the tracked changes version of the manuscript. Additional comments can be found in the marked PDF of the authors’ revised (tracked changes) manuscript (see attachments).

1. The authors did not provide any literature review in the Introduction on the use of machine learning models for ice jam prediction, despite being requested to do so by the second reviewer. The authors cite their review article that covers this topic (Madaeni et al., 2020) but do not provide any details on the machine learning methods that have been used for ice jam prediction, which seems essential to highlight in the present study.

2. Section 2.2: it would be good to mention the software packages used for developing the machine learning models. Without this information, for example, it is difficult to know what is being referred to as ‘default values’ for the decision tree method.

3. L344-345: I think this is backwards, you do not need a loss function to evaluate model error - you need a prediction and a target. However, in many cases you need the model error to evaluate the loss function (e.g., if the loss function is mean square error or some regularized version of it). Perhaps it was meant that the loss function is used to guide the optimization problem?

4. Author’s reply to my former comment 8: If grid-search has ‘poor coverage in dimension’, its not clear how trial and error overcomes this. How did the authors know which hyper-parameter values to try in trial-and-error (i.e., those reported in Table 3)? Were the values in Table 3 decided upon based on recommendations from the literature? If so, any sources that guided these decisions would be good to cite.

The authors mention that various combinations of the hyper-parameters (L378) were applied but do not mention what combinations were explored. The authors should provide more information here to enable their experiments to be reproduced. That is, assuming someone had access to the same dataset, sufficient information should be provided by the authors to enable someone to arrive at the same (or at least similar) results.

5. Supplemental information file:
a. It would be good for all acronyms (and abbreviations, if any) to be spelled out in full at first use.
b. It’s not clear what is meant by ‘channel’. Do the authors mean ‘input’?
c. I think the authors mean ‘estimating gradients’ rather than ‘applying gradients’?
d. It appears the word ‘term’ is missing after ‘momentum’ in the first paragraph of the last section.
e. The authors should appropriately revise ‘high momentums’. Perhaps ‘when using high values for the momentum term’ would be more appropriate.

6. The referencing format is inconsistent (see, e.g., L 593-594).

7. Authors’ reply to my former comment 10: it’s not clear what is meant by ‘model implementations’ in this context. I suggest removing these words or using terms that better describe the technical matter.

8. Authors’ reply to my former comment 16:
a. Why not combine Table 11 and 12? It will make it easier for the reader to compare the performance between the deep learning and machine learning models.
b. It would be good for the authors to mention these benchmark machine learning methods in the abstract and include a sentence stating the relative improvement in performance achieved by the deep learning models (in comparison to the benchmarks).

9. Authors’ rely to my former comment 17:
a. In the authors’ response A, perhaps ‘time consuming to train’ would be more appropriate than ‘time consuming’?
b. In the authors’ response B:
i. What characteristics or the model and/or data makes the models transferable to New Brunswick and Eastern Ontario? Did you run the model on data from these provinces to verify this assertion? If so, this should be mentioned. If not, then the authors should be careful to use appropriate language. For example, the authors may instead mention that they anticipate the deep learning models developed in this research to perform well in these geographical zones for reasons X, Y, and Z.
ii. Please remove ‘pretty’ and ‘really’.
iii. In ‘correct predictions with the wrong’, replace ‘with’ with ‘for’.

References
Madaeni, F., Lhissou, R., Chokmani, K., Raymond, S., Gauthier, Y., 2020. Ice jam formation, breakup and prediction methods based on hydroclimatic data using artificial intelligence: A review. Cold Reg. Sci. Technol. 174, 103032. https://doi.org/https://doi.org/10.1016/j.coldregions.2020.103032

Referee Report: PDF

Hide

ED: Publish subject to minor revisions (review by editor) (26 Jan 2022) by Homa Kheyrollah Pour

AR by FATEMEHALSADAT MADAENI on behalf of the Authors (15 Mar 2022) Author's response Author's tracked changes Manuscript

ED: Publish as is (15 Mar 2022) by Homa Kheyrollah Pour

AR by FATEMEHALSADAT MADAENI on behalf of the Authors (23 Mar 2022) Author's response Manuscript

Short summary

We developed three deep learning models (CNN, LSTM, and combined CN-LSTM networks) to predict breakup ice-jam events to be used as an early warning system of possible flooding in rivers. In the models, we used hydro-meteorological data associated with breakup ice jams. The models show excellent performance, and the main finding is that the CN-LSTM model is superior to the CNN-only and LSTM-only networks in both training and generalization accuracy.