Visual Analysis Scenarios for Understanding Evolutionary Computational Techniques’ Behavior

Round 1

Reviewer 1 Report

 I think the "black box" nature of most machine learning and AI approaches makes them very difficult to reason about, configure, and indeed, trust. So, I found the central concept of this paper very compelling.

To get this out of the way, the paper does have some issues with respect to readability. It includes many grammatical errors such as "... looking for InfoVis techniques easy of understanding for creating scenarios permitting data correlations...". These could largely be eliminated with an editing pass by someone with experience editing written English. While in many places, these sentences are still understandable, in others, I feel the meaning is lost or obscured. 

As for the actual content, I think the authors are on the right track, but I struggled to understand some of the visualizations and to see the implications of them. The authors made a number of interesting looking visualizations, but what I rarely saw was a a clear progression from "here is the question we are trying to answer" to "here is the visualization that can help us answer it" to "this is what this tells us so we can do something different". 

The one exception is the line chart in figure 11. There, the question is clearly "how does the fitness change over time". The visualization is a clear representation, and the implication is that training can be cut down. That said, the line chart is still problematic. It is not immediately obvious what the various lines actually represent, and thirty different colors creates too many close matches to be able to distinguish between all of the lines. There seem to be some lines that have near 100% fitness right from the beginning, so it isn't clear why we are concerned with the lines that don't reach that peak. Lastly, the line highlighted by the authors as having the last fitness increase was stable for nearly 300 generations, so I'm not sure that I give much weight to the idea that since it hasn't changed again by generation 500, it is now stable. 

With the other visualizations, I struggled to understand what their implications were. The alluvial visualization and the histograms are maybe helping the user to see which components of the clustering algorithm are more successful or useful. If this is the case, the authors could be clearer about stating not just what can be observed in the views, but what it means. Why are these views interesting, and how what action could a user be expected to take as a result. Should the user be trying to bias the weights in the DAG for another iteration? 

The first treemap was very confusing to me for several reasons. One, it took me a bit to really understand the organizing metric, using the linear sequence of steps as a hierarchy took a shift in my view of the visualization, and the size and color encoding didn't make any sense until I realized they were reversed in the description. Two, the discussion primarily involved the fitness score, which is encoded as color in the visualization. Since we seem to either have a score of 0, or somewhere in the 90's, the color is just a binary, splitting the algorithms into those two categories. This doesn't really allow much in the way of fine grained comparisons (not that a treemap would be great for that kind of comparison in the first place). 

There are similar issues with all of the visualizations. My fear is that the visualizations were chosen for variety, and not to answer specific questions. I really expected either clear goals stated for each visualization, or, if the authors were doing more of an exploratory analysis, greater explanation of what the visualizations revealed. 

Finally, there is no discussion about how to generalize these results. This significantly diminishes the relevance of this work. 

I would like to see this work completed -- there are too few researchers trying to peel back the hood on these automated processes. However, before this is ready for publication, I think the language needs to be improved, the motivation and implication for the visualizations made clear, and possibly some discussion of broader implications of the work added. 

Author Response

Dear reviewer,

We are very grateful for all the considerations made for the article. We seek to adapt the article text as best as possible to make it better. Finally, all comments are answered in the PDF file.

Sincerely,

Author Response File: Author Response.pdf

Reviewer 2 Report

In general, the discussed topic is very important and prominent now. In my personal opinion, it also directs to explainable AI. However, the authors are describing that they are using InfoVis metaphors to describe or to visualize the results of genetic algorithms in combination with the so-called AutoClustering tool. Based on this, they are visually describing the evolutionary behavior of the tool AutoClustering.

 

Generally, the structure, as well as the writing style, is very good. I personally like Section 2 presenting the theoretical foundation before going into the related work in Section 3 because this helps novices to better understand all the important background very quickly. Section 2.3 describes InfoVis and relates to the Information Visualization Seeking Mantra by Shneiderman (1996, Title: The eyes have it) à Overview first, zoom and filter, then details-on-demand. In the paper, Shneiderman (1996) also Relate, History and Extract are named in the seven tasks. These three tasks may be also important for this type of visualizations.

 

Additionally, in Section 3 (Related Work), the authors are talking about visual analytics (VA). I think a description of VA also should be included in Section 2 because between InfoVis and VA there are some important differences. Visual analytics combines automated analysis techniques with interactive visualizations for an effective understanding, reasoning, and decision making on the basis of very large and complex datasets.” (Keim et al., 2010, Title: Mastering the information age). Therefore, a Visual Analytics Seeking Mantra was generated: “Analyze First - Show the Important - Zoom, Filter, and Analyze Further - Details on Demand” (Keim et al., 2006, Title: Challenges in visual data analysis).

 

In line number 145: Figure 2 and  3are… à Figure 2 and 3 are

 

In Section 4, I am missing a general-purpose description as well as a description of the goal of the section at the beginning. Therefore, only three or four short sentences are needed to directing the readers focus to the section.

 

In line number 212 the sentence starting with: “In relation to the organization of the Treemap, …” the sentence is way too long. I would suggest to split it into two or three sentences.

 

In relation to the included Figures (6 to 12), please add a more detailed description so that the figures are more or less self-contained. Additionally, a short description of the visualizations type would be helpful for the readers.

  

From this point of view, I think that the paper provides a very good overview of the need of visualization for genetic algorithms to overcome the currently existing “black box”, which is very seldom described. Thus, I think the paper is ready for publication after a very quick minor revision, great job.

Author Response

Dear reviewer,

We are very grateful for all the considerations made for the article. We seek to adapt the article text as best as possible to make it better. Finally, all comments are answered in the PDF file.

Sincerely,

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

This paper has improved greatly since I last read it. I see that the language has been cleaned up and the visualizations are now better explained and motivated. I still find myself not following everything in the paper, but I think this comes from the authors assuming a deeper familiarity with the problem domain than I have (I am from the a visual analytics background, with some experience with crossover/mutation based evolutionary algorithms). The various building block names held no real meaning to me and that made it difficult to follow some of the discussions. 

This is not to say there is no room for improvement beyond a deeper explanation of the domain problem. The histogram in Figure 8 was a good example of this. I wasn't sure what I was looking at, and I couldn't find the named examples in the histogram chart. Overall, I was hoping for a bit more discussion of actual insights from the visualizations rather than observations. It would have been nice to see more along the lines of "seeing X means that parameter Z should be tuned this way". There were occasional instances of this (the line chart being an example -- limited change means we can run fewer generations).

While I think there is still room for improvement, I no longer hold the stance that further work is essential.

Some small edits:

- line 101-105: This could be clearer. I would probably suggest a bullet list, broken out by task. As written, it is difficult to separate task from visualization, and the repetition of some of the visualization techniques becomes very confusing.

- line 110: "... is a basic procedures..."

- line 116: "...of this two procedures..."

- line 243: there is a missing word in the sentence

- Figure 13: The line charts are better, but the red lines should be explained. I assume that they represent the final changes, but the red dot now obscures that point. (I also stand by my previous stance that R6 changing in generation 406 after being stable for 400 generations significantly weakens the thought that these have actually stabilized, though it does seem to be a small change).

Author Response

Dear reviewer,

We are grateful for all your suggestions. 

Your considerations have greatly helped us to improve our work.

Sincerely,

Author Response File: Author Response.pdf

Reviewer 2 Report

Many thanks to the authors for their good and fast improvements of the paper at hand. The Adaptions of Section 2.3, especially adding Section 2.3.1 and Section 2.3.2 is very good and helpful. However, I have one more suggestion: Please add 2 or 3 sentences introducing Section 2.3.1 and 2.3.2 in Section 2.3. Additionally, the paragraph starting at line 102 seems to be not directly a part of 2.3.2. Instead, it seems to be the concluding part of 2.3. Maybe you can add an empty line or an including the word ‘Summary’: … in bold letters.

Section 2 (line 45) is only a headline, please take care that after each headline is at least one to three sentences introducing the following parts.

Additionally, I didn’t get the idea behind the bullets like in line 230. Maybe it will also work to write these things (e.g., “Type if individuals”) in bold letters at the beginning of the following paragraph. This way you might also save some space.

Generally, I think the authors have increased the quality of the paper significantly by adding all the suggestions as well as providing further own improvements. I think the paper is ready for publication, but please take care of the comments before. However, from my personal point of view, no further minor revision is needed. 

Author Response

Dear reviewer,

We are grateful for all your suggestions. 

Your considerations have greatly helped us to improve our work.

Sincerely,

Author Response File: Author Response.pdf