Variable Chromosome Genetic Algorithm for Structure Learning in Neural Networks to Imitate Human Brain

Round 1

Reviewer 1 Report

1) The authors should provide more references to other works that deals with the problem of constructing neural networks

2) It is not clear at all how the genetic operators of crossover and mutation are applied

3) The authors should  apply the proposed method to more problems from the relevant litearature

Author Response

1.      The authors should provide more references to other works that deals with the problem of constructing neural networks. ;

 

(Answer) Based on your suggestion, References and some sentences are added in line 87. :

However, VCGA uses chromosome attachment for a system that is able to evolve the structure of an ANN not only constructively but also destructively. Destructive method like pruning is necessary. Since the size of the ANN is unknown at the beginning, design method such as pruning is required. And structure learning of big neural network spends a large amount of time[15,18].

 

2.      It is not clear at all how the genetic operators of crossover and mutation are applied.

 

(Answer) In order to make it clear, sentences are modified. First of all, an example of cross-over is shown in Figure 4.(b). And some explanations are added in line 152. :

For example, mutation modifies a selected gene of a chromosome randomly as shown in Figure 4.(a). And cross-over exchanges a gene of each parent as shown in Figure 4.(b).

 

3.      The authors should apply the proposed method to more problems from the relevant literature. ;

 

(Answer) Based on your suggestion, additional experiment results are added to figure 7 and line 196. This experiment is starting from large structure of ANN. It verifies proposed the destructive approach. ;

Figure 7 plots the results of the second simulation. In this simulation, species of first generation has been fully connected with 5 neurons. The score of the highest performing species starts from 0.97 and decreases to 0.95, however, it increases again to 0.97. This system starts from completed ANN structure. However, it decreases the number of chromosomes after starting. Neurons and connections are decreasing together until the number of chromosomes reaches 11.

 

Author Response File: Author Response.pdf

Reviewer 2 Report

comments:

line 19: explain ANN 

2.  line 40, no comma needed

3. line 42: as ANNs have been ......... have become much...

line 53: proposed

line 57: Our proposed

line 69, algorithms

line 72: GA have been ...

extensive proofreading and advanced english/academic language is required

my main concern is that the contribution is not clear at all

Author Response

1.      line 19: explain ANN. ;

 

(Answer) The explanations for all the acronyms used are explained in the revised manuscript as follows:

a.      ANN (Artificial Neural Network) : in the Abstract

b.     EA (Evolutionary Approach) : in the Introduction

 

2.      line 40, no comma needed. ;

 

(Answer) As you mentioned, comma is eliminated as follows. :

Especially Convolutional Neural Network (CNN) is widely implemented to image recognition[4,5] and Recurrent Neural Network (RNN) is used for various applications such as sentence and voice recognition[6].

 

3.      line 42: as ANNs have been ......... have become much... ;

 

(Answer) As you mentioned, it is updated in line 43. :

As ANNs have been scaled up and improved, they have become much more complex[7].

 

4.      line 53: proposed. ;

 

(Answer) As you mentioned, it is updated in line 56. :

This study proposed a new genetic operation to implement structure learning of ANN;

 

5.      line 57: Our proposed. ;

 

(Answer) As you mentioned, it is updated in line 60. :

Proposed method in this study presents a first challenge through structure learning.

 

6.      line 69, algorithms. ;

 

(Answer) That sentence is eliminated during the revision process. :

 

7.      line 72: GA have been ... ;

 

(Answer) That sentence is eliminated during the revision process:

 

8.      My main concern is that the contribution is not clear at all. ;

 

(Answer) To explain contribution of our study, some sentences are added on line 53 and line 212. :

Structure learning is a very useful instrument which allows to find an appropriate ANN architecture automatically[12]. For this reason, structure learning algorithm is used to generate the topology of ANN in this study.

 

Because of this limit, previous methodology has a difficulty on design large structure. However proposed method enables evolutionary algorithm to destructive approach by using chromosome attachment. As a result, a given initial ANN can be tuned to auto-designed structure after structure learning whether initial ANN is minimum structure or not. It is important to note that the approach can learn larger ANN.

 

Author Response File: Author Response.pdf

Reviewer 3 Report

The paper presents an interesting but not novel subject, introducing a small but helpful contribution to the state of the art for designing artificial neural networks. My recommendation is that the paper should be published but not in the way it is currently written. First of all, please improve the style of the paper and polish the written English. A few comments are:

1) Rewrite the first paragraph of the Introduction (32-34), it looks like a 'spaghetti text'.
2) Please define, explain in some detail, to what is referred by (53) 'structure learning of ANN'. Maybe, the topology of the network?
3) Write what the acronym VCGA means in line 58, since it is there (besides the Summary) where the acronym is presented for the first time to the reader.
4) Delete section 2 'related works', summarize its content in a short paragraph and include it in (section 1) Introduction.
5) The word 'linkage' is repeated throughout the text. Wouldn't it be more accurate to replace it by 'connection'? The reason is that in ANN nodes representing artificial neurons are connected (not linked) among them.
6) In line 106, the expression (1) requires at left of the equal sign to write a symbol indicating fitness, for example: F = expression.
7) Paragraph 107-113 is not well written. Write it again explaining the concepts better.
8) Throughout the text a new genetic operator is mentioned several times: chromosome attachment. What does it consist of? Could you explain it? Please explain in Figure 5 what the illustration depicts, what is happening with the chromosomes? and offspring? If this operator is new, and is one of the contributions of the work, then it should be mentioned.
9) Reduce article verbosity. For example the concept of phenotype is quite complicated in biology, whereas in ANN it would be limited to the topology of the network. Please replace the term phenotype by 'ANN topology'. It also seems inappropriate to say that (117) 'Figure 2 shows two types of genotype'. It is not right, what is shown are two chromosomes or chromosomal segments.  That is, replace the term genotype by chromosome. Consequently, delete phrases such as line 127
'Chromosomes .... individual'. The work is spoiled by these sentences.
10) On lines 130 and 131, as well as in the legend figure page 5 replace Figure 2 by Figure 4. Isn't that right?
11) Why do you set the mutation, cross over and attachment chromosome probabilities to the same value? 0.2? It's the first time I've seen something like this in a genetic algorithm.
12) In reference 3 Isn't something missing?

Author Response

1.      Rewrite the first paragraph of the Introduction (32-34), it looks like a 'spaghetti text'. ;

 

(Answer) In order to make paragraph natural, some sentences are updated in line 32. :

The purpose of Artificial Intelligence (AI) is to imitate human intelligence behavior[1]. In order to imitate human intelligence, the research of AI has been developed as an algorithm to figure out a certain issue. Artificial Neural Network (ANN) is a kind of these algorithms. In other words, the development of ANN is to imitate a human brain process [2].

 

2.      Please define, explain in some detail, to what is referred by (53) 'structure learning of ANN'. Maybe, the topology of the network? ;

 

(Answer) Structure learning is am algorithm to design an ANN structure automatically. In order to explain structure learning of ANN, some sentences are added on line 53. :

Structure learning is a very useful instrument able to find an appropriate ANN architecture automatically[12]. For this reason, structure learning algorithm is used to generate the topology of ANN in this study.

 

3.      Write what the acronym VCGA means in line 58, since it is there (besides the Summary) where the acronym is presented for the first time to the reader;

 

(Answer) As you mentioned, the explanation for VCGA is explained in the revised manuscript as follows. :

Variable chromosome genetic algorithm (VCGA) is applied to this method to implement structure learning of ANN to imitate a human brain.

 

4.      Delete section 2 'related works', summarize its content in a short paragraph and include it in (section 1) Introduction. ;

 

(Answer) As you mentioned, it is updated in the manuscript at Section 1. :

A typical constructive algorithm starting with a minimal network is Genetic Algorithm (GA). GA has been the most common method used for designing neural network architecture[10–17]. In these studies, GA was used to generate the network structures depending on their purposes. It can be applied to a system having a network structure such as a Bayesian network or a neural network.

In the most researches on the automatic generation of the neural network structure, they have been carried out to control the number of neurons through a genetic algorithm in a network structure having one hidden layer[12,15,16]. In these studies, the neurons belonging to the hidden layer are increased or decreased according to generations while searching the suitable number of neurons.

Neuro Evolution of Augmenting Topologies(NEAT) is a representative EA methodology. NEAT outperform the best fixed topology methodology on challenging benchmarking reinforcement learning[14]. NEAT is used as a typical evolutionary approach[7]. It is constructive algorithm using GA that starting with a minimal structure of artificial neural network. Initial neural network has a minimal structure, however it becomes gradually more complex to search an optimal architecture with NEAT methodology.

On the other hand, NEAT methodology only can evolve the neural network architecture to larger structure, because it has to start with minimal structure. If neural network architecture in solution area is too large to increase gradually, it is time consuming works to search a right answer. Thus, destructive operation is necessary to design a more complex or larger neural network.

 

5.      The word 'linkage' is repeated throughout the text. Wouldn't it be more accurate to replace it by 'connection'? The reason is that in ANN nodes representing artificial neurons are connected (not linked) among them. ;

 

(Answer) In order to clarify the meaning, it is updated in line 102, line 108, line 124, line 127, line 134, line 138, line 143, line 152, line 155, line 163, line164, line 173, line 200, line 204, line 205, and line 206.

 

6.      In line 106, the expression (1) requires at left of the equal sign to write a symbol indicating fitness, for example: F = expression. ;

 

(Answer) As you mentioned, the expression is updated in line 114. :

 

7.      Paragraph 107-113 is not well written. Write it again explaining the concepts better. ;

 

(Answer) In order to make it clear, some sentences are modified in line 115. :

Score means the accuracy of predicted output value of ANN structure and it has a real value between 0.0 and 1.0. D means a dependency on the number of chromosomes. The larger D is, the more important the minimum structure of the ANN is. And the smaller D is, the more important the accuracy of the predicted output value is. N means relative number of chromosomes; (number of own chromosomes) / (number of average chromosomes). In this case study, the constant D was set to 0.2.

 

8.      Throughout the text a new genetic operator is mentioned several times: chromosome attachment. What does it consist of? Could you explain it? Please explain in Figure 5 what the illustration depicts, what is happening with the chromosomes? and offspring? If this operator is new, and is one of the contributions of the work, then it should be mentioned. ;

 

(Answer) Explanation of chromosome attachment is added in line 156. And contribution is added in line 165. :

The mutation in biology has two types; gene mutation and chromosome mutation. Well known mutations (in Biology) like mutation (in GA) or cross-over can be treated as a kind of gene mutations. It changes base sequence of DNA[21]. However, chromosome mutation only changes the number of chromosomes. Chromosome attachment is a kind of chromosome mutations[22]. It is also called nondisjunction in biology[23]. It makes a change that varies the number of chromosome of offspring. This is a kind of mutations and cause chromosome aberration.

 

This operation is first applied to the GA and EA through this study.

 

9.      Reduce article verbosity. For example the concept of phenotype is quite complicated in biology, whereas in ANN it would be limited to the topology of the network. Please replace the term phenotype by 'ANN topology'. It also seems inappropriate to say that (117) 'Figure 2 shows two types of genotype'. It is not right, what is shown are two chromosomes or chromosomal segments.  That is, replace the term genotype by chromosome. Consequently, delete phrases such as line 127

'Chromosomes .... individual'. The work is spoiled by these sentences ;

 

(Answer) As you mentioned, the term phenotype is replaced by ANN topology in Figure 3, line 132, line 142, and line 179. And the term genotype is replaced by chromosome type in Figure 2, line 121, line 123, line 124, line 125, line 127, line 131, line 134, line 136, line 140, line 143 and line 145.

 

10.   On lines 130 and 131, as well as in the legend figure page 5 replace Figure 2 by Figure 4. Isn't that right? ;

 

(Answer) Lines 130 and 131 were inappropriate sentences. It is eliminated.

 

11.   Why do you set the mutation, cross over and attachment chromosome probabilities to the same value? 0.2? It's the first time I've seen something like this in a genetic algorithm. ;

 

(Answer) Probabilities of mutation, crossover, and attachment were simulated several times and it will be covered in later papers. And similar figures were used in papers about NEAT. ; 0.3. And additional sentences and reference are included in revised manuscript in line 183. :

The probabilities of mutation, cross-over, and attachment are all 0.2 and the learning rate of the neural network is 0.2. This value is approximation to NEAT[14].

 

 

12.   In reference 3 Isn't something missing? ;

 

(Answer) As you mentioned, reference 3 is updated in line 238. ;

3. Survey of neural networks in autonomous driving Available online: https://www.researchgate.net/publication/324476862_Survey_of_neural_networks_in_autonomous_driving (accessed on Jul 20, 2019).

 

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The authors have performed the required changes and hence I recommend publication