Fatal Software Failures in Spaceflight

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Dear Authors,

Please find below my comments.

The paper provides a concise review of the fatal software failures in the Space industry starting with the early days of Space exploration towards  the most recent projects. The first part of the paper emphasis the importance of the space exploration in the society  as an innovation enabler for other industry sectors (telecommunication, digitization).  The space mission are depicted from point of view of the mission attributes, supply chain, space technology with the end goal to underline the role of the flight software for space mission and also bring to front several types of software failures.  The most part of the work is dedicates to a comprehensive list of the flight software failures of space exploration.

1) The paper could also include in the list  a recent project (2019) dedicated to moon exploration, Beresheet  lunar lander, designed by SpaceIL and Israel Aerospace Industries.

Please see the link:  https://davidson.weizmann.ac.il/en/online/sciencepanorama/what-happened-beresheet

2) In the conclusion and Prospect section, in my opinion I do not think Software and software engineering, in particular, are underestimated. They play only a minor role in the perception of aerospace engineering. (lines 901-902) and that should be changed to past tense "played".

3) Also, I think it is  is appropriate to mention there exist several tools for "autocoding" that enables the user to generate the code (source files) for the flight software from the simulation diagram,( please see Matlab/Simulink  autocode generation) as modern method minimize software failures

4) Links to figures are missing (see lines 58,74).

Thank you!

 

Author Response

1) Beresheet: Thank you very much for the informative link! Beresheet is a great addition.

2) Perception of software engineering: In our experience, it is still underestimated, particularly on the higher, more political levels. But we rephrased the sentences to reflect these potentially different perceptions better.

3) autocoding: We are now mentioning autocoding in Section 4, and joined this with the already mentioned LLM autocoding, where we think it fits nicely.

4) Links to figures: Links somehow got lost in PDF export. Now fixed.

Reviewer 2 Report

Comments and Suggestions for Authors

The authors have provided a comprehensive analysis of the pivotal role of software in space missions, complemented by an extensive inventory of software-related failures. This article promises significant value to the community, especially considering the escalating importance of software in advancing space exploration endeavors.

However, I have the following suggestions which may further improve the current manuscript:

A brief subsection describing the distinctive features of flight software compared to commercial software used in everyday engineering tasks could provide valuable context.   Briefly address the primary causes of software failures, including inadequately defined requirements, immature architectures, and challenges in integrating Commercial Off-The-Shelf components and ensuring interoperability. These factors often contribute to software unreliability and mission risks.   Most importantly consider including a seperate section dedicated to mitigation strategies for software failures in space missions.   Highlight the significance of defect or bug reports as a crucial repository of information throughout the software life-cycle phases. These reports play a crucial role in tracking defects identified at various stages, including development, testing, and mission phases, thereby facilitating thorough analysis and resolution of issues.   What about the role of system engineering or model-based system engineering in mitigating software errors? Elaborate.   Consider incorporating a section on software reliability analysis, particularly focusing on Software Reliability Growth Models (SRGM). These models offer effective techniques for assessing reliability growth by utilizing data collected during the testing phase.   Please elaborate on the importance of software quality monitoring throughout the software life-cycle to enhance the likelihood of mission success?    Discuss the adoption of a product line approach in flight software development (which has been successfully implemented in several NASA missions), highlighting its benefits and implications for space missions.     Some minor comments:   Line 57 and 74: references are missing Line 137: Give a reference to 'Perrow-class failures'

 

 

Comments on the Quality of English Language

Typo and grammar check

Author Response

• A brief subsection describing the distinctive features of flight software compared to commercial software used in everyday engineering tasks could provide valuable context.
• Added a new paragraph “Distinctive features of flight software” to Section 1. Thank you for this neat idea!
• Briefly address the primary causes of software failures, including inadequately defined requirements, immature architectures, and challenges in integrating Commercial Off-The-Shelf components and ensuring interoperability. These factors often contribute to software unreliability and mission risks.  
• Short discussion of primary causes of software failures and two references to further literature added to Section 2. Thank you for motivating further literature search which brought interesting literature to light, which surely benefits the paper. However, we decided not to discuss the issue in more detail ourselves because this is not the focus of our paper.
• Most importantly consider including a seperate section dedicated to mitigation strategies for software failures in space missions.  
• Just as with the previous point, mitigation is shortly discussed in Section 2; but we also do not go into details here, in order to not lose focus too much. We focus on what went wrong.
• Highlight the significance of defect or bug reports as a crucial repository of information throughout the software life-cycle phases. These reports play a crucial role in tracking defects identified at various stages, including development, testing, and mission phases, thereby facilitating thorough analysis and resolution of issues.  
• Bug reports are indeed an important knowledge management tool; we included a reference. However, this is a common and widely used method. Therefore, we do not discuss further because it is already established. We added a sentence to explain this. Please see Section 4.
• What about the role of system engineering or model-based system engineering in mitigating software errors? Elaborate.  
• As far as we can see, Reviewer 1 also mentioned this point (“autocoding”). We agree that MBSE is a huge topic, and thank you for pointing out! However, we added only a short mention as a new method, and do not discuss it further with the same reasoning as above.
• Consider incorporating a section on software reliability analysis, particularly focusing on Software Reliability Growth Models (SRGM). These models offer effective techniques for assessing reliability growth by utilizing data collected during the testing phase.  
• We internally discussed SRGMs controversially. This topic could probably fill an entire new paper. We have added a short mention alongside autocoding in Section 4 as one more example of a New Method that needs to be analyzed for its benefits.
• Please elaborate on the importance of software quality monitoring throughout the software life-cycle to enhance the likelihood of mission success? 
• We added an explanation (“A side note on Software Product Assurance”) to Section 1. It is probably indeed important to point out the need for quality assurance. We come back to quality also in Section 4, where we mention it again.
• Discuss the adoption of a product line approach in flight software development (which has been successfully implemented in several NASA missions), highlighting its benefits and implications for space missions.
• We see software product lines primarily as a method to increase reuse; which, of course, makes them valuable. We also added a reference. However, we do not discuss in detail with the same reasoning as above.
• Missing image references: fixed
• Give a reference to 'Perrow-class failures': Two references (now 21 & 22) were already in the paper, so we just made it clearer that they can provide more information on Perrow-class failures.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Dear Authors,

I do not have other comments.

Thank you!

Best regards,

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you authors for responding to my comments and I am happy with the updated version. And I am sure the community will appreciate this good work!