Out-of-the-box calving-front detection method using deep learning

Herrmann, Oskar; Gourmelon, Nora; Seehaus, Thorsten; Maier, Andreas; Fürst, Johannes J.; Braun, Matthias H.; Christlein, Vincent

doi:https://doi.org/10.5194/tc-17-4957-2023

Articles | Volume 17, issue 11

https://doi.org/10.5194/tc-17-4957-2023

© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/tc-17-4957-2023

© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 17, issue 11

Research article

|

24 Nov 2023

Research article |

| 24 Nov 2023

Out-of-the-box calving-front detection method using deep learning

Oskar Herrmann, Nora Gourmelon, Thorsten Seehaus, Andreas Maier, Johannes J. Fürst, Matthias H. Braun, and Vincent Christlein

Download

Final revised paper (published on 24 Nov 2023)
Preprint (discussion started on 28 Feb 2023)

Interactive discussion

Status: closed

RC1:
'Comment on tc-2023-34', Anonymous Referee #1, 06 Apr 2023

Please find the review comments attached.

Citation: https://doi.org/10.5194/tc-2023-34-RC1
- AC1: 'Reply on RC1', Oskar Herrmann, 31 May 2023
  
  Thank you very much for the thorough review.
  
  We collected comments from the two referees in one PDF. Our answers to the comments of referee #1 can be found on pages 1-4.
  
  Citation: https://doi.org/10.5194/tc-2023-34-AC1
RC2:
'Comment on tc-2023-34', Anonymous Referee #2, 11 Apr 2023
General Comments:
This study applied nnU-Net by Isensee et al. (2021) to a benchmark dataset from Gourmelon et al. (2022). The author designed six experiments and showed that multi-task learning benefits each task’s performance, while there is no distinct difference between early- and late-branching, and adding the task of delineating the glacier boundary does not further improve the terminus delineation. The experimental design and results presented in this manuscript might be of more interest to the deep-learning community than to glaciologists, as the errors in glacier terminus delineation produced by this study are relatively large compared to other studies (Zhang et al., 2019; Cheng et al., 2021; Baumhoer et al., 2019). These termini may not be suitable for scientific research on ice-ocean interactions due to their relatively large error. Providing more termini that are beyond the benchmark dataset with lower error might be more beneficial to the glaciology community, but I understand it might be out of the scope of this study. Also, further details on technical aspects would be appreciated.
Major comments:
The author applies a nnU-Net and improves the error from 753 ± 76 m by Gourmelon et al. (2022) to 541 ± 84 m. Still, 541 meters is a relatively large error for terminus delineation, compared with previous studies that have errors ranging from 33 to 108 meters (Zhang et al., 2019; Cheng et al., 2021; Baumhoer et al., 2019). Given the relatively large margin of error in the terminus products produced by this method, it might be challenging to conduct scientific research that relies on them. Also, providing more termini that go beyond the benchmark dataset might be more beneficial to the glaciology community, but I understand it might be out of the scope of this study.

One key advantage of using no new U-Net is that the framework can take the fingerprint of the dataset and automatically configures the framework (adjust the hyperparameters). However, it is not clear to me which hyperparameters are being adjusted automatically, how they are adjusted, and what is the final choice of these hyperparameters. If we adopt the hyperparameters from nnU-Net and apply them to standard U-Net, would there be no difference between nnU-Net and U-Net? If that is the case, I suppose it is important to clearly demonstrate how to automatically adjust the hyperparameters and how these newly derived hyperparameters improve the results.

While the author emphasizes that the method is an out-of-the-box application, it would be beneficial for the readers to have a clearer understanding of the steps taken by the author to ensure its out-of-the-box applicability. Providing further details on this aspect could enhance the credibility of the method's out-of-the-box applicability claim and contribute to its broader adoption in the research community.

Specific Comments:
Line 16: The scripts are not publicly available. As the author emphasizes that the method is out-of-the-box, it would be more beneficial if the codes are publicly available.
Line 66: What is the visual preparation? Why is the visual preparation necessary as the dataset from Gourmelon et al. (2022) is a benchmark dataset?
Line 115: Are sentences about the labels here and the sentences on Line 50 repeated?
Figure 3: For Crane, what is the black dots in the middle of the ocean?
Line 120: Are there any procedures to deal with the imbalance? It might be helpful to increase the weight of the positive pixel loss in the loss function.
Figure 5: Please explain the abbreviations of the glacier names.
Section 4.1: Additional information about the method needs to be provided. It would be beneficial to include more details on the data fingerprint and how to use the fingerprint to determine the hyperparameters in this study. The following are some specific descriptions that is unclear to me:
What is the definition of the distribution of spacing?

How to use the distribution of spacing to determine the annotation, image resampling strategy, and target spacing?

Is the annotation equivalent to labeling the images? If so, how is that related to the distribution of the spacing?

What is the image resampling strategy that is used?

Is CT or z-score normalization used in this study? How to conduct the z-score normalization with image mean and standard deviation?

Line 175: Please be more clear about the minor changes here.
Lien 180: What is the post-processing step for the fifth experiment? How to combine the three types of output to get a single glacier terminus?
Line 188: “To generate a prediction of the zones, pixels classified as front are assigned to the ocean, and the glacier zone is dilated once with a 7x7 kernel.” Do the zones here represent the segmentation zone? If so, how does this lead to a glacier terminus? Similar to the fifth experiment, how to combine the two types of output to get a single glacier terminus?
Line 219: Why the errors of the STL approaches here are larger than the error of Gourmelon et al. (2022)?
Reference
Baumhoer, C. A., Dietz, A. J., Kneisel, C., and Kuenzer, C.: Automated Extraction of Antarctic Glacier and Ice Shelf Fronts from Sentinel-1 Imagery Using Deep Learning, Remote Sensing, 11, 2529, https://doi.org/10.3390/rs11212529, 2019.
Cheng, D., Hayes, W., Larour, E., Mohajerani, Y., Wood, M., Velicogna, I., and Rignot, E.: Calving Front Machine (CALFIN): glacial termini dataset and automated deep learning extraction method for Greenland, 1972–2019, The Cryosphere, 15, 1663–1675, https://doi.org/10.5194/tc-15-1663-2021, 2021.
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., and Maier-Hein, K. H.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation, Nature Methods, 18, 203–211, https://doi.org/10.1038/s41592-020-01008-z, 2021.

Gourmelon, N., Seehaus, T., Braun, M., Maier, A., and Christlein, V.: Calving fronts and where to find them: a benchmark dataset and methodology for automatic glacier calving front extraction from synthetic aperture radar imagery, Earth System Science Data, 14, 4287– 4313, https://doi.org/10.5194/essd-14-4287-2022, 2022.
Zhang, E., Liu, L., and Huang, L.: Automatically delineating the calving front of Jakobshavn Isbræ from multitemporal TerraSAR-X images: a deep learning approach, The Cryosphere, 13, 1729–1741, https://doi.org/10.5194/tc-13-1729-2019, 2019.
Citation: https://doi.org/10.5194/tc-2023-34-RC2
- AC2: 'Reply on RC2', Oskar Herrmann, 31 May 2023
  
  Thank you very much for the thorough review.
  
  We collected comments from the two referees in one PDF. Our answers to the comments of referee #2 can be found on pages 5-9.
  
  Citation: https://doi.org/10.5194/tc-2023-34-AC2

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

ED: Reconsider after major revisions (further review by editor and referees) (03 Jun 2023) by Stef Lhermitte

AR by Oskar Herrmann on behalf of the Authors (05 Jun 2023) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (23 Jun 2023) by Stef Lhermitte

RR by Anonymous Referee #2 (12 Jul 2023)

Suggestions for revision or reasons for rejection

General Comment

I appreciate that the author addressed most of the questions and comments raised by both reviewers. My main concern is about the impact of this study. The ultimate goal of automating terminus extraction is to provide as many as quality-controlled termini for glaciological community to conduct scientific research. I understand it might not be the scope of this study to achieve this such an ultimate goal, but I concern that a high uncertainty might undermine the impact of this paper, even though this is a methodology paper. I believe the key contribution of this study is to improve the error from 753m to 541 m, compared with Gourmelon et al. (2022). However, both errors are relatively high.

Besides, I have several comments about nn-UNet, the name “out-of-the-box”, and multi-task-learning, which are described as follows.

Specific Comments

2.2 In Line 66, the author mentioned that the nn-U-Net is a framework for training the U-Net and can adjusts the hyperparameter accordingly. Therefore, in the previous comments, I was wondering which hyperparameters the framework adjust and how the it is done based on the dataset automatically (accordingly). In the response, it seems to me that the framework did not adjust the hyperparameter according to the dataset. The patch size and the batch size are decided by author based on the GPU memory, which is commonly adopted when using any other deep learning methods. So, I suggest to remove the sentence about the nn-UNet here can adjusts the hyperparameters according to the dataset.

With that being said, I am still wondering that why the nn-UNet here does not adjust the hyperparameters, as being a self-configuring method is the most important feature of the nn-UNet (Isensee et al., 2021).

2.3 If “out-of-the-box” means “unmodified”, why it is important to highlight in the title that the author use the unmodified nn-UNet? In line 72, the second contribution is the out-of-the-box application of the nn-UNet. It seems a little odd to me to highlight “unmodified” application of a method.

I tried HuggingFace link, however it shows error after I upload one image of a glacier and click submit. I would suggest to add additional information (e.g., image format) in the webpage or the manuscript to make the link work.

2.13 If I understand correctly, in the multi-task learning experiments, the task of delineating calving front only and delineating the whole glacier boundary are abandoned in the inference, but only help to adjust the parameters in the training progress.

Another related issue is about the fused label. If the author fuse the label of calving front into the zone label with an additional class, I think it is a single classification task instead of a multi-task method. It is the same with the original zone segmentation but with one more class representing the calving front. Also, as this fused label yields the best results, it might be better to change the conclusion throughout the manuscript regarding the multi-task-learning.

Reference

Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., and Maier-Hein, K. H.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation, Nature Methods, 18, 203–211, https://doi.org/10.1038/s41592-020-01008-z, 2021.

Gourmelon, N., Seehaus, T., Braun, M., Maier, A., and Christlein, V.: Calving fronts and where to find them: a benchmark dataset and methodology for automatic glacier calving front extraction from synthetic aperture radar imagery, Earth System Science Data, 14, 4287–4313, https://doi.org/10.5194/essd-14-4287-2022, 2022.

Hide

RR by Anonymous Referee #1 (21 Jul 2023)

ED: Reconsider after major revisions (further review by editor and referees) (23 Aug 2023) by Stef Lhermitte

AR by Oskar Herrmann on behalf of the Authors (28 Sep 2023) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (01 Oct 2023) by Stef Lhermitte

RR by Anonymous Referee #2 (10 Oct 2023)

ED: Publish as is (13 Oct 2023) by Stef Lhermitte

AR by Oskar Herrmann on behalf of the Authors (16 Oct 2023) Manuscript

Download

The requested paper has a corresponding corrigendum published. Please read the corrigendum first before downloading the article.

Article (22000 KB)
Full-text XML

Short summary

Delineating calving fronts of marine-terminating glaciers in satellite images is a labour-intensive task. We propose a method based on deep learning that automates this task. We choose a deep learning framework that adapts to any given dataset without needing deep learning expertise. The method is evaluated on a benchmark dataset for calving-front detection and glacier zone segmentation. The framework can beat the benchmark baseline without major modifications.