DOPSIE: Deep-Order Proximity and Structural Information Embedding

Round 1

Reviewer 1 Report

Most comments of reviewers are considered in the revised version. However, couple of things are not carried out: 

1. Traditional machine learning split for training/testing should be used (80-20, which is very common), (This is a misunderstanding of the reviewer comment about using the whole data for training and testing, which means to use the whole data for training/testing as in traditional machine learning experiments: 80-20, 75-25 or 70-30 splits).

2. Related work section should be written using authors own words; a lot of copy-paste is included as shown in the attached report.

Comments for author File: Comments.pdf

Author Response

1. Unlike the standard machine learning protocol we used 

an increasing training rate from a low value (10%) to an average value (50%). 

This approach has already been used in the work "DeepWalk" (table 2):

http://www.perozzi.net/publications/14_kdd_deepwalk.pdf

2. We rewritten the related work section with different description 

of the cited works as suggested;

Reviewer 2 Report

1.       In the abstract, “with the aim of preserve its structure” should be changed to “with the aim of preserving its structure”, there are several semantic issues in the sentence “Moreover, due to the connection with clustering coefficients, adopted to encode neighbourhood information, and others structural properties…………” and also this sentence is hard to understand.

2.       In the introduction, what does “Property” mean here? Hard to understand.

3.       In the introduction, authors mention that “among the many measures clustering coefficients are adopted due to the connection between other graph properties”.  I think this is quite important, but the authors just put this sentence down and did not explain them further.

4.       In Table 6, the most accurate one for 40% should be Node2Vec

Author Response

1. We modified the abstract as suggested.

2. We replaced the word "Property" with "Capability" in order to explain 

the feature of algorithm of capturing local and global network information;

3.  We rewritten the last part of the introduction  

in order to highlight the concept of "clustering coefficients".

4.  We changed table 6 as suggested.

Round 2

Reviewer 1 Report

Experiments should show traditional splits like 90-10, 80-20, 70-30 before accepting the paper.

Author Response

Kind reviewer, 

as suggested I added the experiments with train/test percentage 90-10, 80-20, 70-30. 

Now, the experimental phase is composed of train/test percentage 90-10, 80-20, 70-30, 60-40, 50-50, 40-60, 30-70, 20-80, 10-90 with Accuracy and F-measure.

Best Regards,

Mario Manzo

Author Response File: Author Response.pdf

Round 3

Reviewer 1 Report

The authors have carried out the reviewers' comments. The new results indicate that the algorithm behavior is unstable and the performance is not explainable. 

The authors failed to analyze the results correctly which causes conflicts; for example:

At abstract they said that: "Our experimental results show that DOPSIE outperforms state-of-the-art embedding methodologies in different classification problems." This is not true according to the results. It is only true for "Email-Eu-core" dataset.

Lines 251-252: they said: "We adopted a percentage of train in the range 10-50% to avoid the phenomenon of overfitting". This is not true in machine learning, and this will not usually happen in machine learning where 80%-20% is a very common split for traditional machine learning algorithms.  To claim that there is overfitting they need to draw learning curves.

Lines 265-266: "This results highlight a specific feature of DOPSIE which occur when small fractions of the graph are labeled." 
WHICH CONFLICTS with:
Lines 279-280: they said: "This results highlight a different feature of DOPSIE which fits when a remarkable fraction of the graph are labeled." 

in the  conclusion  section, they said: "The main weak point concerns the effort of capturing topological features when the network presents few connections, when nodes are poorly connected to each other."

In summary, I think now when using more training data, the learning curve should include low error. The authors are encouraged to understand the algorithm and problem in more depth. One possible reason for the instability of results might be sampling techniques which are not explained in details in the paper.  While more data will usually result in better performance, in this study, for the BlogCatalog dataset, this was false for the proposed algorithm, and in contrast, was superior for DeepWalk algorithm. However, for smaller datasets,  the proposed algorithm performed better than those of the-state-of-the-art. This is not reasoned nor analyzed well in the paper. Moreover, learning curves will be very useful to explain results.

Author Response

Kind reviewer,

first of all thank you for the suggestions. In order to improve the clarity of the experimental phase, I repeated the tests where necessary. I confirm that the proposed method outperforms the considered state of the art. Compared to previous results, I noticed errors during the writing of test scripts that led to inaccurate performance measurements and lack of clarity. In the new version of paper, in addition to the updated results, the changes are shown in green.

Author Response File: Author Response.pdf

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

The paper is interesting but its acceptance lacks some points (major revision):

- many local graph measures exist, see, e.g.,

  @article{key,
   author = {M. Dehmer},
   title = {Information Processing in Complex Networks:
Graph Entropy and Information Functionals},
 journal = {Appl. Math. Comput.},
   year = {2008},
     volume={201},
     issue={1-2},
     pages={82-94}

   }

@article{key,
   author        = {F. Emmert-Streib and M. Dehmer},
   title         = {Networks for Systems Biology: Conceptual Connection of Data and Function},
   journal       = {IET Systems Biology},
   year          = {2011},
   volume        = {5},
      issue        = {3},
   pages={185-207}
   
 
}

@Article{junker_2006,
AUTHOR = {B. Junker and D. Kosch\"utzki and F. Schreiber},
TITLE = {Exploration of biological network centralities with CentiBiN},
JOURNAL = {BMC Bioinformatics},
VOLUME = {7},
YEAR = {2006},
NUMBER = {1},
PAGES = {219}

}

many information functionals have been proposed capturing structural information. those could be also used in conjection with the algorithm. the authors should briefly review and cite these sources (and  others) in the introduction. as it is right now, the introduction is narrowly focused on well-known measures

- the authors should describe the training and test data more properly

- error measures to evaluate the performance of the classifier must be discussed

`´conclusions must be extented

Reviewer 2 Report

This paper aims at encoding the deep order proximity and structural information in graphs by applying the concept of clustering coefficient. This is a very novel way of embedding graphs compared with the existing methods. However, I cannot see the significance of applying clustering coefficient in graph embedding, especially from the perspective of the tasks usually supported by other graph embedding methods, including node classification, link prediction and etc.  Usually, the objective of node embedding is to make sure that if two nodes are close in the graph, then they should be close in the embedding space. But I cannot see how the method in this paper can assure this. Although the authors discuss clustering coefficient has interesting relationships with transitivity and density of the graphs, I cannot see the direct relationship between these two concepts and the objective of node embedding. 

For algorithm 1, what's your objective of Lines 19-39? is there an objective function? I think this part should be more detailed.

For section 3.2, this part should cover why integrating clustering coefficient will benefit node embedding. The denotation of |NN_k| is missed.

I think the authors should cover more tasks, not just node classification. Even for node classification, the authors should give more discussions about their results as their method performs quite different on different data sets. 

Reviewer 3 Report

The paper proposes a new graph embedding method called "DOPSIE", the authors claim that the new method outperforms state-of-the-art embedding methodologies in different classification problems.

Major issues include:

Motivation is not clear, the authors are encouraged to add one or two paragraphs in the introduction to explain this issue. Also, "topological perspectives" in the abstract should be clarified. 

Related work section has many sentences and paragraphs taken as is from stated references, please re-write in your own words.

Related work section needs to be enhanced by adding more explanation and more related work, especially ref [11,12,13,27].

Methodology section needs to have a full working example to explain the algorithm at a higher level of abstraction level.

Please justify using a portion of the datasets (10%-50%).

Would like to see results when using the whole dataset for training and testing.

Accuracy by its own can sometimes fail to measure performance, please add f-measure.

A discussion section is very much expected to compare the state-of-the-art algorithms with yours.

Minor issues include:

English language mistakes.

Equations should be all numbered.

Algorithm 1 needs reformatting as some lines are Interleaved.