Employing Source Code Quality Analytics for Enriching Code Snippets Data

Abstract

The availability of code snippets in online repositories like GitHub has led to an uptick in code reuse, this way further supporting an open-source component-based development paradigm. The likelihood of code reuse rises when the code components or snippets are of high quality, especially in terms of readability, making their integration and upkeep simpler. Toward this direction, we have developed a dataset of code snippets that takes into account both the functional and the quality characteristics of the snippets. The dataset is based on the CodeSearchNet corpus and comprises additional information, including static analysis metrics, code violations, readability assessments, and source code similarity metrics. Thus, using this dataset, both software researchers and practitioners can conveniently find and employ code snippets that satisfy diverse functional needs while also demonstrating excellent readability and maintainability.

1. Introduction

Recently, the open-source model has facilitated a substantial surge in code reuse through the availability of code snippets in online repositories, such as GitHub, GitLab, or SourceForge. This approach, sometimes referred to as “component-based reuse” or “opportunistic programming”, allows developers to easily find and reuse existing code in their own projects, considerably expediting the development process [1,2]. By leveraging this paradigm, they not only reduce development time and effort but also encourage cooperation and contribution within a vast network of projects. Additionally, the relevant metadata, such as statistics from code hosting, enables easy tracking and analysis of popular open-source projects, assisting in decisions about which projects to use or contribute to.

While code reuse presents significant advantages, it also poses challenges. When practicing reuse, developers must clearly understand the workings of the source code to ensure its proper integration and maintenance. Thus, it becomes paramount to assess the quality of these code components or snippets, especially in terms of readability. A readable code snippet is not only integrated more seamlessly but is also easier to modify, fitting the unique requirements of diverse projects. Moreover, comprehending the code’s logic and structure simplifies its long-term maintenance, aiding in the swift identification and rectification of bugs or necessary updates.

There are various systems that mine code snippets and can recommend API calls [3,4,5,6,7,8,9] or even reusable code [10,11,12,13,14]. However, these systems primarily focus on the functional aspect of the developer query, answering questions like “how to read a csv file?”, and sometimes neglect quality characteristics. More specifically, out of the characteristics defined in the ISO/IEC 25010 for software quality (including, e.g., performance, reliability, security, etc.), maintainability and readability are of paramount importance for code reuse [15]. Maintainability ensures that software can be efficiently modified to address new requirements, fix bugs, or improve performance, making it adaptable and long-lived [16]. On the other hand, readability pertains to the clarity of the code. Easily understandable code simplifies maintenance tasks, reduces the introduction of bugs, and streamlines the onboarding process for new team members [17]. Emphasizing these two characteristics is pivotal for the sustainable development and evolution of software projects [18]. To facilitate reuse and produce readable and maintainable code, it is important to consider these characteristics when recommending source code snippets.

In this context, we have developed a dataset of code snippets that comprises a comprehensive analysis of both functional and quality characteristics. The dataset is based on the CodeSearchNet [19] corpus, and includes additional information in the form of static analysis metrics and violations, as well as readability assessments of the snippets. Furthermore, considering the functional aspect, our dataset reports the similarity between the code snippets to facilitate the process of finding similar code. The dataset is provided in MongoDB dump format, which facilitates reproducibility and provides advanced querying capabilities. As a result, software researchers and practitioners are enabled to easily retrieve snippets that are relevant to different challenges (e.g., code clone detection [20,21,22,23], code snippet recommendation [10,11,12,13,14], code synthesis [24,25], code summarization [26,27]), while ensuring a standard of high quality.

Our work presents a distinct contribution in the space of open-source code repositories and their subsequent reuse. As already mentioned, several approaches mine and recommend code snippets or API calls for reuse purposes [3,4,5,6,7,8,9,10,11,12,13,14]. However, most of these systems mainly focus on the functional aspects of code reuse, and do not emphasize the imperative non-functional characteristics such as readability and maintainability. This paper’s novelty emerges from the synthesis of both functional and quality characteristics in one cohesive dataset. Unlike previous works that may focus predominantly on the functional domain [28,29], we incorporate static analysis metrics, violations, and readability assessments into our dataset. Moreover, by introducing a similarity measure between code snippets, we further facilitate the seamless identification and integration of reusable code, ensuring developers not only find functionally useful code but also high-quality pieces.

The rest of this paper is organized as follows. In Section 2, we delve into the detailed methodology adopted to create, analyze, and curate the dataset of code snippets. Following that, Section 3 provides a comprehensive view of the dataset generated, encompassing its various statistics and shedding light on how it can be used to yield meaningful information. Section 4 explores the numerous ways in which this dataset can be used in answering pertinent research questions, while, lastly, Section 5 encapsulates the primary takeaways from our study and underlines its significance in the larger context.

2. Materials and Methods

Figure 1 depicts the architecture of our platform, which comprises six main components, the Metrics and Violations Analyzer, the Readability Analyzer, the Source Code Parser, the Abstract Syntax Tree Analyzer, the Tree Distance Extractor, and the Hierarchical Clusterer. The source code of our system is available online (https://github.com/AuthEceSoftEng/code-snippets-dataset, accessed on 28 August 2023) to allow for full reproducibility.

As already mentioned, our system uses the data from CodeSearchNet [19], as it is a well-crafted dataset of source code snippets and docstrings. The code snippets are initially analyzed using three different components. The Metrics and Violations Analyzer computes the values of static analysis metrics for each snippet and further detects violations in their source code. The Readability Analyzer is used to extract metrics relevant to code readability. Finally, the Source Code Parser extracts the Abstract Syntax Trees (ASTs) of the snippets. All these outputs are stored in MongoDB, along with the source code and the metadata of the snippets. The ASTs extracted from the Source Code Parser are further forwarded to the Abstract Syntax Tree Analyzer that extracts an intermediate representation for each tree (pq-Grams profile, discussed in detail in Section 2.4), which is used by the Tree Distance Extractor to compute the distance between any pair of trees (snippets). The resulting distance matrix is processed by the Hierarchical Clusterer component, which forms clusters and stores them in the database, allowing for easy retrieval of functionally equivalent snippets.

Note that our methodology is mostly language-agnostic, especially regarding the similarity between snippets, since it only requires an appropriate AST extractor from code, while the rest of the components of our architecture remain as they are. However, in this paper we provide a proof-of-concept for components written in the Java programming language due to its suitability for managing large codebases, the availability of robust libraries, and a rich ecosystem that promotes snippet reuse, further underpinning our approach. The components outlined in Figure 1 are analyzed in detail in the following paragraphs.

2.1. Metrics and Violations Analyzer

The first part of our analysis concerns the extraction of useful metrics from the code. To do so, the Metrics and Violations Analyzer employs the SourceMeter analysis tool (https://www.sourcemeter.com/ accessed on 31 August 2023) and extracts a set of different method-level metrics belonging to different categories. These categories correspond to Complexity (with metrics such as McCabe Cyclomatic Complexity, Nesting Level, etc.), Coupling (with metrics for the Number of Incoming/Outgoing Invocations), Documentation (with metrics such as the Comment Density, etc.), and Size (with metrics such as the Lines of Code, etc.). Furthermore, we employ PMD (https://pmd.github.io/ accessed on 31 August 2023) to identify source code violations belonging to different categories (Best Practice Rules, Code Style Rules, Design Rules, Documentation Rules, Error-Prone Rules, Multithreading Rules, Performance Rules, and Security Rules). We also keep track of the number of violations per category of rules as well as the number of violations per priority level (minor, major, or critical).

2.2. Readability Analyzer

The second part of our analysis concerns extracting the readability of source code snippets. The concept of code readability has been studied by several researchers, and different methods have been proposed to assess it. In an effort to provide a holistic view of readability, we extracted different sets of metrics using the tool of Scalabrino et al. [30]. More specifically, the metrics extracted involve (a) the structural metrics identified by Buse and Weimer [31] (including, e.g., the number of identifiers, the number of loops, etc.), (b) the structural metrics defined by Posnett et al. [32] (lines of code, entropy, and Halstead’s Volume), (c) the visual metrics of Dorn [33] (e.g., indentation length) (The Dorn metrics also include certain metrics that are relevant to spatial perception or natural language; however, we may also assume that these are relevant to the visual aspect of code.), and (d) the textual metrics of Scalabrino et al. [30] (e.g., Comments and Identifiers Consistency, Textual Coherence, etc.).

2.3. Source Code Parser

To further examine code reuse, we performed source code analysis to extract functionally equivalent snippets. This type of analysis can be combined with code readability to allow developers to understand and navigate different solutions for a query and select the most readable one. Furthermore, by analyzing the structure and organization of reuse candidate snippets, developers can identify patterns and potential issues, such as code duplication or complex control flow. In this context, we parsed and transformed the source code of the snippets into ASTs, i.e., code representations in tree-like structure, with each node representing a different element of the code, such as, e.g., a function or a variable. To accomplish that, we made use of the ASTExtractor tool (https://github.com/thdiaman/ASTExtractor/ accessed on 31 August 2023). It should be noted that, in our methodology, the source code parser, which translates code snippets into their corresponding ASTs, is the sole component tailored to specific programming languages. Once the code has been transformed into an AST, the next phases of our approach are language-independent, leveraging the generated trees without requiring any modifications.

As an interesting alternative, one could even modify our system to integrate a parser generator, like ANTLR [34] or Bison [35]. This way, it would be possible to parse different languages into ASTs, by providing the generator with the corresponding grammar, and thus achieve language independence.

2.4. Abstract Syntax Tree Analyzer

Given the ASTs of the snippets, the next step is to compute their similarity, which can be easily performed using a Tree Edit Distance (TED) algorithm [36]. TED is a method for comparing two trees and measuring their similarity by counting the minimum number of operations (insertions, deletions, and label changes) required to transform one tree into another. Various methods have been developed over the years to improve the complexity of the TED algorithm, but they all have a complexity of $O\left(n^2\right)$ or greater. To avoid such computational complexity, we used the pq-Grams algorithm [37] to approximate TED. This algorithm constructs a pq-Extended tree by adding null (dummy) nodes to the tree (noted using the symbol “*”). Specifically, $p-1$ ancestors are added to the root of the tree, $q-1$ children are added before the first and after the last child of each non-leaf node and q children are inserted to each leaf of T. The pseudocode for constructing the extended tree is depicted in Algorithm 1.

Algorithm 1 pq-Grams algorithm pseudocode for building the pq-Extended tree.

procedure  pqExtendedTreeConstruction($tree,p,q$)

Add p-1 ancestors to the root of the tree

For each non-leaf node in tree:

Add q-1 children before the first child

Add q-1 children after the last child

For each leaf node in tree:

Insert q children

Return the pq-Extended tree

end procedure

The parameters p and q were set to values 2 and 3, respectively. For each pq-Extended tree, we first calculated a list of all the pq-Grams patterns it contains. A pq-Grams pattern is defined as a subtree of the extended tree T that consists of an anchor node with $p-1$ ancestors and q children. Each list of pq-Grams patterns is called a pq-Grams profile. Algorithm 2 depicts the pseudocode for calculating the pq-Grams profile given an pq-Extended tree. Therefore, practically, each AST of a code snippet is first extended using the procedure of Algorithm 1, and then the extended tree is given as input into the procedure of Algorithm 2 to build the final pq-Grams profile for the AST.

Algorithm 2 pq-Grams algorithm pseudocode for computing pq-Grams profile.

procedure computePqGramsProfile($tree,p,q$)

List pqGramsProfile = empty list

For each node in tree:

If node has p-1 ancestors and q children:

Extract pq-Gram pattern consisting of the node, p-1 ancestors, and q children

Add the extracted pq-Gram pattern to pqGramsProfile

Return pqGramsProfile

end procedure

An example pq-Extended tree construction and extraction of pq-Grams profile for p equal to 2 and q equal to 3 is shown in Figure 2 (initially presented in [38]).

2.5. Tree Distance Extractor

Upon having extracted the pq-Grams profiles, one for each AST/snippet, we are now able to apply the TED metric of the pq-Grams algorithm [37]. Thus, the pq-Grams distance between two trees $T_1$ and $T_2$ is defined as follows:

$$distance(T_1,T_2)=1-2\ast \frac{|P\left(T_1\right)\cap P\left(T_2\right)|}{|P\left(T_1\right)\cup P\left(T_2\right)|}$$

(1)

where $P\left(T_1\right)$ and $P\left(T_2\right)$ are the pq-Grams profiles for trees $T_1$ and $T_2$, respectively. As shown by this equation, the distance between two trees depends on the number of mutual pq-Grams patterns contained in both their profiles divided by the union of the two lists, which results in a value between 0 and 0.5 [37]. It is obvious that the similarity between two trees can be easily calculated using the formula $1-distance(T_1,T_2)$.

2.6. Hierarchical Clusterer

We used Agglomerative Hierarchical Clustering, a bottom-up approach, to group code snippets, based on their distances calculated in the previous step. The optimal number of clusters was determined using average silhouette score, defined for each point i as

$$s\left(i\right)=\frac{b\left(i\right)-a\left(i\right)}{max\left(a\right(i),b(i\left)\right)}$$

(2)

where the calculation of the silhouette coefficient uses the mean intracluster distance and the minimum nearest cluster distance. The mean intracluster distance for the data point i is defined as the average distance of i to all other points in the same cluster as i ($C_i$)

$$a\left(i\right)=\frac{1}{|C_i|-1}\sum _{\begin{array}{c}j\in C_i\\ j\ne i\end{array}}d(i,j)$$

(3)

The minimum nearest cluster distance is defined as the average distance from the points of the nearest cluster of data point i, except for cluster $C_i$:

$$b\left(i\right)=\underset{k\ne i}{\min }\frac{1}{|C_k|}\sum _{j\in C_k}d(i,j)$$

(4)

After performing the cluster analysis, we examined the generated clusters (maximum silhouette equal to 0.78) and maintained the clusters that contained a minimum number of data points (100 snippets) to ensure that the snippet implementations are generic enough. Listings 1 and 2 depict two snippets that have been put together in the same cluster. Both snippets check for null values and call the .get() function on an object.

Listing 1. Example snippet that is in the same cluster with the snippet of Listing 2.

public static EarthEllipsoid getType(String name){   if (name == null)       return null;   return hash.get(name); }

Listing 2. Example snippet that is in the same cluster with the snippet of Listing 1.

public Object getUserProperty(Object key){    if (userMap == null)       return null;    return userMap.get(key); }

3. Results

Figure 3 depicts the schema of our database, which comprises four different collections, the snippets, the analysisMetrics, the violations, and the readabilityMetrics. The data are available as a MongoDB database (in .bson format, which is a binary-encoded serialization of JSON documents).

Our decision to provide the dataset in MongoDB dump format was based on the inherent advantages of MongoDB for handling large-scale datasets. MongoDB, as a NoSQL database, offers flexibility in storing diverse data types and is renowned for its scalability and performance. The dump format ensures that the entire database, including its structure and contents, can be easily shared, restored, and queried, promoting reproducibility. The capabilities of MongoDB in managing and querying vast datasets are well-documented in the database and big data literature [39].

It should be noted that all the collections use “url” as a “primary key” to identify the code snippet they refer to. As already mentioned, our dataset consists of four collections:

• Snippets: This collection contains the code and the docstring of each snippet, information about its origin, the AST and the id of the cluster it belongs to, which can be used to group snippets into clusters (i.e., similar snippets).
• AnalysisMetrics: This collection includes the static analysis metrics calculated by the SourceMeter analysis tool.
• Violations: This collection contains the source code violations identified by the PMD tool.
• ReadabilityMetrics: This collection includes the extracted readability metrics, which are split into four categories, with respect to the research approach they refer to (Buse and Weimer—BW, Posnett, Dorn, and Scalabrino).

Table 1. Dataset statistics.

Metric	Value
Number of Documents/Snippets	496,685
Number of Repositories	500
Number of Clusters	893
Data Size	11.3 GB (940.3 MB compressed)

depicts certain statistics about the collected dataset. The statistics presented have been chosen to provide a comprehensive overview of the scale and organization of the dataset. They practically indicate the range of code samples in our dataset, their distribution across different projects, and how often certain patterns or functionalities are repeated.

Additionally, Listing 3 illustrates a sample query that can be used for data extraction purposes; it retrieves the static analysis metrics and the readability metrics of the cluster of Listings 1 and 2. Queries such as this can be issued from multiple environments. For example, one could use MongoDB Compass (https://www.mongodb.com/products/compass accessed on 31 August 2023) to explore the data or form and issue queries. And, of course, it is also possible to issue requests using a programming language, e.g., for Python, one can use the pymongo library (https://pymongo.readthedocs.io/en/stable/ accessed on 31 August 2023).

Listing 3. Example query that retrieves static analysis metrics and readability metrics of a cluster.

db.getCollection("snippets").aggregate([     {         "$match": {             "clusterID": 1         }     },     {         "$lookup": {             "from": "analysismetrics",             "localField": "url",             "foreignField": "url",             "as": "satmetrics"         }     },     {         "$lookup": {             "from": "readabilitymetrics",             "localField": "url",             "foreignField": "url",             "as": "readmetrics"         }     } ])

Thus, using our dataset, one can extract useful insight for the relations between metrics. For instance, given the query of Listing 3, we can plot the diagram of Figure 4, depicting the Scalabrino readability value versus Comment Density (CD). If we further highlight their relation using linear regression (dashed line in diagram), we see that the two metrics are positively correlated, which is expected if we consider that comments typically enhance the readability of the code.

4. Discussion

Concerning the structure and the design of the dataset, retrieving and reproducing it is straightforward, as it only requires a MongoDB instance. As already mentioned, MongoDB ensures scalability and offers advanced storing, retrieving, and querying capabilities; thus, it is ideal for managing the varied and dynamic nature of source code snippets and the associated metrics [39]. Our dataset can be used to confront several challenges in current research. First of all, in examining source code readability, researchers can use our dataset to empirically study the various aspects that contribute to or detract from code understandability. Consider a real-world scenario wherein a developer has to choose between multiple similar implementations of a function or algorithm. With our dataset, one could identify similar code snippets and run user experiments. For example, comparing the readability of iterative versus recursive implementations of a factorial function can help determine which version is deemed more readable by a broader audience. Moreover, the data facilitate the design and development of readability models, either by using the source code directly [40,41] or the associated static analysis metrics [42,43,44]. When doing so, comparison with the state of the art is straightforward, as our dataset includes different metrics for readability [30,31,32,33].

The similarity data in our dataset make it an invaluable resource for code clone detection [20,21,22]. For instance, researchers are able to detect whether two different open-source projects inadvertently employ the same code implementation, indicating potential for reuse. By utilizing the tokens and ASTs of the provided snippets, researchers can compare different clone detection methods. Furthermore, our dataset paves the way for studies investigating correlations between code clones and metrics like static analysis or readability. For instance, a pertinent question may be whether code clones inherently suffer from reduced readability due to repeated patterns. Finally, one could even try to determine whether it is possible to identify code clones based on these metrics [23].

Concerning code similarity challenges, consider the potential application in code search engine scenarios. For example, when a developer submits a textual query looking for a “quicksort algorithm implementation”, our dataset can not only help in retrieving relevant snippets, [10,11,12,13,14,19,45], but it can also rank them based on their readability or quality score [46,47]. Moreover, it can be used for pattern matching tasks, by detecting whether certain snippets follow nano-patterns, study how these nano-snippets are implemented, and even examine their readability [48].

Venturing into the realm of code synthesis [24,25], imagine an automated tool that can take a cluster of similar code snippets and synthesize a new version by amalgamating the best parts of each snippet. For instance, if the cluster contains various ways to implement a database connection [49], the tool could create a single, clean, and readable version that incorporates the most efficient and readable elements from the cluster. Lastly, our dataset offers a fertile ground for advancements in code summarization [26,27]. With the burgeoning complexity of today’s software projects, being able to automatically summarize a lengthy function into a few descriptive lines or even comments can significantly expedite code reviews and maintenance.

All in all, our dataset can be useful for confronting several research challenges. Considering its limitations, and thus identifying possible future research directions, we may note that the data include method snippets, while complete project information is omitted. As a result, the dataset cannot be used for challenges at the project and/or library level (e.g., extracting the semantics of software libraries and using them for code comprehension [50]). Given also that the data are only drawn from GitHub, they may not always include generic solutions for all possible developer queries. As future work, we plan to link the method snippets to their original source repositories, which would offer interesting metadata, and even integrate other sources, such as Stack Overflow, which includes snippets that are generally regarded as high-quality code [51].

5. Conclusions

In this work, we have aimed to improve the research and practice on code reuse and readability by creating a dataset of code snippets that involves both functional and quality characteristics. The dataset is based on the CodeSearchNet corpus and includes additional information in the form of static analysis metrics and violations, readability assessments, and code similarity metrics. Thus, using our dataset, one can confront different research challenges in the areas of code reuse/readability, as well as combine these areas with the aim of optimizing the process of identifying, integrating, and maintaining reusable code. Future work could include incorporating additional representations of the code, such as control flow graphs, and even augmenting the dataset, e.g., by using also data from code-hosting repositories.