Certifiable AI

Round 1

Reviewer 1 Report

This paper is motivated by the fact that the currently most successful machine learning models are very difficult to explain or interpret by human experts. Starting from a description of a newly established field, namely "explainable AI" (XAI), the authors in this paper redefine this notion of "explanation" somewhat differently, namely in the direction of `interpretation'. Interpretations are indeed sometimes possible, but the authors also show in this paper that they provide at best a subjective understanding of how a particular model works. Based on this, the authors propose an alternative or complement to existing XAIs, namely, explicitly a certified AI (CAI). The authors describe how an AI can be specified, implemented, and tested to become certified. The resulting approach combines ontologies and formal logic with statistical learning to obtain reliable AI systems that can be safely used in engineering applications. 

The paper is readable, relevant, interesting and this reviewer is very positive and recommends this paper for acceptance but gives three important addition suggestions that should still be incorporated by the authors to make the paper even better:

1) Two things are mentioned once very briefly but are definitely extremely relevant and important and should therefore also be backed up with references:

1a) legal aspects (mentioned only once on page 2).
1b) ethical aspects (also mentioned only once on page 2).

These two aspects are important insofar as the authors explicitly address a safety-critical area in the paper that absolutely requires consideration of these two aspects: medicine, or medical AI, or machine learning for medical applications. This should be supplemented and expanded.

ad 1 a) Especially in safety-critical areas, the legal aspects are not only relevant but even mandatory and lthough the European Commission proposed a new legislation for the use of "high-risk artificial intelligence" for the future, the existing European fundamental rights framework already provides some clear guidance on the use of medical AI - from a legal perspective and there is a very recent related work by Karl Stoeger et al (2021) [1], which can be a very good pointer for the interested reader, see:
[1] K. Stoeger, et al, "Medical Artificial Intelligence: The European Legal Perspective," Communications of the ACM, vol. 64, no. 11, 2021. 10.1145/3458652

ad 1b) Legislation such as the European General Data Protection Regulation (GDPR) is likely to make the use of black-box approaches (e.g., Deep Learning) in business difficult, as they often cannot explain why a machine made the decision it did - so one of the fundamental questions - especially for certifiable AI - is who will be responsible, and can we agree on common ethical principles for AI? These broad questions are well summarized in a relevant paper by Mueller et al. (2021) [2}, and should be referred to, see:

[2] H. Mueller, et al, "The Ten Commandments of Ethical Medical AI," IEEE COMPUTER, vol. 54, no. 7, pp. 119--123, 2021. 10.1109/MC.2021.3074263

2) The authors often refer to causality, here we should definitely refer to causability:

The term causability coined by Holzinger et al. (2021) [3] is neither a typo nor a synonym for causality in the sense of Judea Pearl - which, by the way, should also be referred to in causality.
The term causability was introduced in reference to usability - which is even explicitly mentioned in this paper. While XAI is about implementing transparency and traceability, Causability is about measuring the quality of explanations, i.e. the measurable extent to which an explanation of a statement reaches a certain level of causal understanding for a user with effectiveness, efficiency and satisfaction in a certain context of use. Thus, explainabilityin terms of XAI highlights technically decision-relevant parts of machine representations and machine models, i.e., parts that contributed to model accuracy in training or to a particular prediction. It does NOT refer to a human model.  Usability, according to DIN EN ISO 9241-11, is the measurable extent to which a software can be used by specific users to achieve specific goals with effectiveness, efficiency, and satisfaction in a specific context of use, and Causabilityis the measurable extent to which an explanation achieves a certain level of causal understanding for a human, see e.g. [3].

[1] A. Holzinger, et al. "Towards Multi-Modal Causability with Graph Neural Networks enabling Information Fusion for explainable AI," Information Fusion, vol. 71, no. 7, pp. 28-37, 2021. 10.1016/j.inffus.2021.01.008

In general, a very well-written and readable paper that is sure to be of interest to a wide range of readers, and this reviewer hopes that the commenters will help improve this paper even further.

Author Response

Dear Reviewer, 

thanks for your comments. 

A new section (4.3) was introduced to deal with legal and ethical aspects in a manner that is adequate to the scope of the paper. I inserted a footnote defining what I mean with causal model', referred to your new term `causabiliity’ and included your idea (graph NN) in section 4.2.

Thanks for the valuable comments which helped to improve the paper!

Sincerely,

Jobst Landgrebe

Reviewer 2 Report

I regret to inform you that I do not feel adeguate for reviewing this work since while I understand the basic assumption that we do not need to interpret how  stochastic ai systems operate, but to explain and certify them, the reasoning is mostly based on functions and mathematical models that run out of my full comprehension. So I would suggest to revert to another reviewer.

Author Response

Dear Reviewer,

thanks for your honest comment.

Sincerely,

Jobst Landgrebe

Reviewer 3 Report

It is a very interesting article with many good points that should be of wide interest to a diverse readership (this is my opinion as a researcher on the boundary of AI and human neuroscience). I just have the following minor comments:

(1) The article covers a diverse range of quite technical topics. It would really benefit from 1 or 2 summary figures and/or tables that drive home the main points in a more visual way.

(2) For the line "We do not understand how humans classify texts or images, or conduct conversations. Neither do we have mathematical causal (and thus predictive) models of the weather or of the Earth’s climate. Such models will remain out of reach for humans [37,38]."
I totally get the meaning and don't necessarily disagree. But a strong statement like "Such models will remain out of reach of humans" that will make many researchers annoyed could merit a bit more in-text expansion and scientific argument beyond just citing those two general topic references.

(3) For the end Certifiable AI section about the need for better priors, the references and background discussion here really doesn't do justice to the many other AI researchers trying to incorporate prior knowledge in AI systems a large variety of ways. This is actually the biggest issue I have with this paper and was why I put "major revisions". The author needs to be more clear about why what they are proposing with better incorporating more useful priors into XAI is better/different/equal to what the other frontier AI researchers are already doing or proposing, with a more fair and diverse list of references in this area. 

(4) Minor typo "But, as we have seen from the reviewe 322 of important XAI literature examples,"

Author Response

Dear Reviewer, 

thanks for your comments. 

1. I added an overview table (Table 1) - good suggestion!
2. I adapted the sentence (line 324), good hint.
3. I added a section on other approaches (4.2), excellent suggestion, thank you .

Your suggestions helped me to improve the manuscript. Thank your very much.

Sincerely,

Jobst Landgrebe

Round 2

Reviewer 3 Report

Overall the author adequately responded to reviewer comments.

Just some minor points:

(1) " dNNs solve these tasks in a manner which differs completely from the way humans interpret text, language, sounds, images, or smell or somatosensory input"  -- This is not entirely true, CNN were inspired by human neuroscience experiments in the visual system actually.  (e.g. https://arxiv.org/ftp/arxiv/papers/2001/2001.07092.pdf "Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future ") . Obviously silicon-based neural nets are not exactly biological neural nets, but it is important to acknowledge what is currently similar between them at different levels of abstraction and how CNNs can continue to improve to be closer to human systems.

(2) Line 379 missing a period after "impossible".

(3) The author is being slightly ungenerous towards corporate/industrial DNNs. They do have internal standards, quality checks, and significant performance testing. For example: "Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry" https://www.atlantis-press.com/journals/jase/125905766/view I encourage the author to be somewhat more specific and balanced in critiquing overall what the current state of "certifying " AI is generally versus what needs to be improved, before publication.

Author Response

Dear Reviewer,

thanks for the second round of comments. To your points:

1.  In this point we disagree (for good reasons, I think), and I added a footnote expressing and briefly justifying this (fn 10). 
2. Thank you, period added.
3.  Thanks for this. I dealt with this aspect in ll. 491-498 of the new version

Thanks again for your help!

J. Landgrebe