Risk Management for Defense SoS in a Complex, Dynamic Environment
Round 1
Reviewer 1 Report
Second Review of the manuscript. Revision 1.
Clearly the authors have taken reviewer comments and have done due diligence to fix the manuscript. The revisions are extensive in statistical analysis, applying a series of
The authors clearly identify the 15 risks from the interviews upfront.
Better description of the risk data collection in Methodology section, obtained from interview and survey from DoD programs that have System of System context.
Several new figures have been add to better capture the statistical analysis of these 15 risks, into clusters (with low and high impact). Analysis continues to provide a series of depictions of the risk data: Pairwise Comparison, dimensionality scree plot, common coordinate space plot and dendrogram. All contribute to the separation of risk factors and their relationships.
Adequate discussion on what these results could mean to Program Manager and on what they should focus. Much improved analysis and description of the initial interview/ survey risk data.
----
Few numbering/format issues - minor. These would probably be fixed in final formatting.
page 10 Lines 443-443 typo. adjoin two groups "Risk Factors with High Impact Intensity" and "Risk Factors with High Impact Intensity". [one of those should be changed to Low Intensity]
page 13, Lines 505 and beyond. typo. These correlations are expressed in the following relationships: [the relationships start on number 19 (not #1) and continue]. Maybe this is an artifact of getting a PDF with show comments... I can't see a clean revision with edits off.
page 15 Line 607 and beyond. typo. The main conclusions starts on #26 [not #1]
Author Response
Responses to reviewers’ comments
|
Review 1 |
Response |
|
page 10 Lines 443-443 typo. adjoin two groups "Risk Factors with High Impact Intensity" and "Risk Factors with High Impact Intensity". [one of those should be changed to Low Intensity]
|
Line 434-435 – We changed one of them to Low Intensity |
|
page 13, Lines 505 and beyond. typo. These correlations are expressed in the following relationships: [the relationships start on number 19 (not #1) and continue]. Maybe this is an artifact of getting a PDF with show comments... I can't see a clean revision with edits off.
|
|
|
page 15 Line 607 and beyond. typo. The main conclusions starts on #26 [not #1]
|
|
|
Review 2 |
|
|
the author should clearly define what the purpose of the article is. |
We clarified the purpose of the article on line 42-46 |
|
The title includes Complex and dynamic environment. The article does not present the characteristics of the analyzed environment. There is no mention of this in the research subject described or in the results themselves. |
We added an explanation on line 32-39 Unlike projects in a stable environment, defense SoS are characterized in a dynamic environment in which managers must react quickly and organizations must be flexible to respond. Technology, consumer changing requirements, laws and regulations, political leaders and international conditions are all changing rapidly and dramatically in the defense arena. Moreover, in the defense highly complex environment, there are many variables that are hard to identify and measure. In this complex environment, small changes in one factor can produce a major change in another. Conditions of instability and complexity in the defense arena intensify the impact of systematic risk management of defense SoS.
|
|
INTRODUCTION There is no clearly defined purpose and structure of the article |
The purpose of the article is presented on line 42. The structure of the paper includes the usual structure that reflects the stages of the study: Literature Review, Methodology and Research Design, Results, Discussion, Conclusions and Recommendations, Study limitations and recommendations for future research.
|
|
INTRODUCTION There is no justification why the presented research is important. The author also did not specify what is new in the presented research results |
The justification is presented on line 41: Defense SoS require systematic management of risks that limits risk disruptions and their propagation throughout the systems. Therefore, risk management is one of the most important areas that must be considered when managing defense SoS. All systems have some level of inherent risk because of the uncertainty that accompanies any new endeavor. In defense industries, the riskier the system, the higher the payoff. Thus, risk is sometimes beneficial because it has the potential to increase profits. The novelty of the research results is presented on line 568 The current study presents an innovative and implementation of Heterogeneity and Homogeneity using Cochran’s Q test and I2 statistic which is validated using Multidimensional scaling which in turn is validated by Hierarchical Cluster Analysis. The above method is a loop in which it will be possible to start from any of these statistical analyzes.
|
|
LITERATURE REVIEW The author should clearly indicate the differences between risk and uncertainty she/he should even provide definitions that differentiate these concepts. On the basis of which sources / research were distinguished two types of uncerainty in defense projects? |
We added an explanation regarding the differences between risk and uncertainty on line 88: The PMBOK guide [1] distinguishes between risk and uncertainty. A risk is an unplanned event that may affect one or some of project objectives if it occurs. Conversely, uncertainty refers to a condition where the future events are not known. The source that distinguished between the two types of uncertainty is written on line 93 [1]
|
|
DISCUSSION AND CONCLUSION Points 5 and 6 require some sorting - incomprehensible numbering order. |
Thank you for the comment. We mention this comment in the track changes in the paper. |
|
The discussion did not take sufficient account of the new results presented in section 4. |
We updated the discussion. Please see lines 477- 587. |
|
Results. At the same time, the results presented in the orange part of the risk matrix are not interpreted (very low probability). The ranges for individual fields in the risk matrix should be completed - ranges of probability values qualifying for the low, moderate, high group. |
We updated Figure 10. Please see the new version of Figure 10 |
Author Response File: 
Author Response.pdf
Reviewer 2 Report
All comments in the attachment
Comments for author File: Comments.pdf
Author Response
Responses to reviewers’ comments
|
Review 1 |
Response |
|
page 10 Lines 443-443 typo. adjoin two groups "Risk Factors with High Impact Intensity" and "Risk Factors with High Impact Intensity". [one of those should be changed to Low Intensity]
|
Line 434-435 – We changed one of them to Low Intensity |
|
page 13, Lines 505 and beyond. typo. These correlations are expressed in the following relationships: [the relationships start on number 19 (not #1) and continue]. Maybe this is an artifact of getting a PDF with show comments... I can't see a clean revision with edits off.
|
|
|
page 15 Line 607 and beyond. typo. The main conclusions starts on #26 [not #1]
|
|
|
Review 2 |
|
|
the author should clearly define what the purpose of the article is. |
We clarified the purpose of the article on line 42-46 |
|
The title includes Complex and dynamic environment. The article does not present the characteristics of the analyzed environment. There is no mention of this in the research subject described or in the results themselves. |
We added an explanation on line 32-39 Unlike projects in a stable environment, defense SoS are characterized in a dynamic environment in which managers must react quickly and organizations must be flexible to respond. Technology, consumer changing requirements, laws and regulations, political leaders and international conditions are all changing rapidly and dramatically in the defense arena. Moreover, in the defense highly complex environment, there are many variables that are hard to identify and measure. In this complex environment, small changes in one factor can produce a major change in another. Conditions of instability and complexity in the defense arena intensify the impact of systematic risk management of defense SoS.
|
|
INTRODUCTION There is no clearly defined purpose and structure of the article |
The purpose of the article is presented on line 42. The structure of the paper includes the usual structure that reflects the stages of the study: Literature Review, Methodology and Research Design, Results, Discussion, Conclusions and Recommendations, Study limitations and recommendations for future research.
|
|
INTRODUCTION There is no justification why the presented research is important. The author also did not specify what is new in the presented research results |
The justification is presented on line 41: Defense SoS require systematic management of risks that limits risk disruptions and their propagation throughout the systems. Therefore, risk management is one of the most important areas that must be considered when managing defense SoS. All systems have some level of inherent risk because of the uncertainty that accompanies any new endeavor. In defense industries, the riskier the system, the higher the payoff. Thus, risk is sometimes beneficial because it has the potential to increase profits. The novelty of the research results is presented on line 568 The current study presents an innovative and implementation of Heterogeneity and Homogeneity using Cochran’s Q test and I2 statistic which is validated using Multidimensional scaling which in turn is validated by Hierarchical Cluster Analysis. The above method is a loop in which it will be possible to start from any of these statistical analyzes.
|
|
LITERATURE REVIEW The author should clearly indicate the differences between risk and uncertainty she/he should even provide definitions that differentiate these concepts. On the basis of which sources / research were distinguished two types of uncerainty in defense projects? |
We added an explanation regarding the differences between risk and uncertainty on line 88: The PMBOK guide [1] distinguishes between risk and uncertainty. A risk is an unplanned event that may affect one or some of project objectives if it occurs. Conversely, uncertainty refers to a condition where the future events are not known. The source that distinguished between the two types of uncertainty is written on line 93 [1]
|
|
DISCUSSION AND CONCLUSION Points 5 and 6 require some sorting - incomprehensible numbering order. |
Thank you for the comment. We mention this comment in the track changes in the paper. |
|
The discussion did not take sufficient account of the new results presented in section 4. |
We updated the discussion. Please see lines 477- 587. |
|
Results. At the same time, the results presented in the orange part of the risk matrix are not interpreted (very low probability). The ranges for individual fields in the risk matrix should be completed - ranges of probability values qualifying for the low, moderate, high group. |
We updated Figure 10. Please see the new version of Figure 10 |
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
The authors indicated that they had prepared an update of the discussion in line with the comments made by the reviewer. The update should be contained on lines 477-587. Unfortunately, there is no new content in the discussion in the attached manuscript. The text ends on line 508. Perhaps the article provided is not the final version that was intended to be uploaded to the system. Points 5 and 6 are still not properly prepared - incorrect numbering, wrong formatting.
Author Response
The updated discussion goes from line 493 to line 614. The new discussion presents the risk map of 46 defense SoS and the explanation of the risk factors according to their impact and probability.
The next section of the discussion presents the common results of the exploratory study and the survey results (based on figure 5-9) and gives some explanations to these results in comparison to the relevant literature.
Author Response File: Author Response.pdf
This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.
Round 1
Reviewer 1 Report
The manuscript has many positive qualities. There was clearly significant work that was accomplished collecting and analyzing data. However, there were many lacking details to accept the manuscript for publishing in the current form.
System of Systems is a concept that is brought up early in the paper but does not carry through the method and analysis. The paper uses the term "defense projects based on system of systems (SoS) 21-22, and "SoS projects" 25 and "projects using SoS". These are confusing inconsistent uses of SoS. In my career, I have never seen such things. Projects are based on requirements, needed capabilities and threats, though based on external interconnectivity, projects may be part of a larger SoS context. The authors don't define SoS early in the paper, nor do they make the case as to what about SoS is risky. They hint at it in sentence 64-65, but it is not clear if the "whole" represents the whole system or the whole SoS. SoS is eventually defined on page 4.
Section 3 method is one page. The abstract discusses "advanced data science tools" line 11, and the method should describe them. While the figure 4 is great, this section only describes the data in lines 156-176 - the 10 interviews, the pilot questionnaire for 10 experts and and the final survey. How many survey responses were collected? How was the data validated? Probably the biggest issue was there is no discussion on data analysis in the method section. No mention of Tree based classification models (creation and use).
Section 3/4. No description of the projects used in the data... what were these defense projects? (aircraft, space, Command and Control, business systems, ground vehicles, etc). The readers should be able to decide if they agree with the author that the projects are related to some SoS, or just system. Maybe some discussion or Table identifying the SoS or the SoS attributes for each of the 46 defense projects.
Figure 6 is plotted low to high; most authors would plot highest to lowest especially if highlighting the 5 highest risk. No discussion why the author would hypothesize that all risk should be uniform.
Section 4 introduces Decision Tree procedure on page 7 line 222. No discussion how this was accomplished. There is reference to independent variables, item clusters and items. Line 225 "item clusters of 19, 20,22, 25 on item 31". There is no discussion on the full list of items, their definition, nor how these items were identified. This should all be in Section 3 Method/ Research Design. Author list the Items used in Figure 7 but not all the other items. The author provides little literature review, method description and how it was applied to the data for this paper.
Section 4 continues to introduce new methods not described in the the research design. Line 252 The third step uses RSM, on the clusters and items introduced previously. The forth step used "advanced data science tools such as Bayesian SEM Path analysis and IBM Watson Analytics analysis" Line 267-268. A statement and some figures does not help the reader understand what was accomplished and what logic leads to results.
Line 290 states that "Figure 12 and 13 present the main characteristics of defense organizations that are able to avoid risk". The #1 characteristics is a Line 292 Transparent risk management methodology. This is not mentioned in either figure. How did the analysis lead to such a finding. Did the author identify this or did the analysis show this. Very confusing list - where is the logic to lead to Lines 292 -300.
Figure 12 (also backwards low to high) and FIgure 13 are unfinished with no axis labels nor legends.
Section 6 Conclusion and Recommendations, based on lack of clear description of research method, is questionable. First the "finding of the current study show the need to focus on three main challenges/ risks..." line 369. The first is uncertainty... of what?. The third focus is managing a project under constraints. All projects have constraints - in cost, schedule, requirements.
Again, I see very few references in the results and conclusions (Sections 5 and 6) to discuss what risk items, challenges, organizational characteristics, best risk practices and data analysis finds directly relate to aspects of SoS.
Overall, there was a lot of work in this study and it shows. However, the manuscript in this current form is missing a detailed section 3 methodology/research design on the data analysis, thus any findings are in question. You mention several data analytical techniques and show some result figures, but never describe the technique nor how independent variables were identified. It appears some of the findings are not supported direclty by the data analysis.
Minor Edits
line 45 ISO standards don't use commas 31,000
Formatting on the Results - lines 183 and lines 202 have extra indents on the numbered lists.
line 254 "oder", order
line 322 first use of "TTM" spell out acronym. Strangely, this is a study on defense which does not focus on TTM. That is a commercial term. Rather defense focuses on initial operating capability (IOC), which is when unit will be fully equipped and trained to operate and maintain a new system.
line 347-349 awkward. I think the sentence simply states it will be necessary to change the initial design if you change the initial design.
Reviewer 2 Report
in Attachment
Comments for author File: Comments.pdf
