TC-Verifier: Trans-Compiler-Based Code Translator Verifier with Model-Checking

Abstract

Code-to-code translation, a critical domain in software engineering, increasingly utilizes trans-compilers to translate between high-level languages. Traditionally, the fidelity of such translations has been evaluated using the BLEU score, which predominantly measures token similarity between the generated output and the ground truth. However, this metric falls short of assessing the methodologies underlying the translation processes and only evaluates the translations that are tested. To bridge this gap, this paper introduces an innovative architecture, “TC-Verifier”, to formally employ the Uppaal Model-checker to verify trans-compiler-based code translators. We applied the proposed architecture to a trans-compiler translating between Swift and Java, providing insights into the verified and unverified aspects of the translation process. Our findings illuminate the strengths and limitations of using Model-checking for formal verification in code translation. Notably, the examined trans-compiler reached a verification success rate of 50.74% for the grammar rules and productions modeled. This study underscores the gaps in trans-compiler-based translations and suggests that these gaps could potentially be addressed by integrating Large Language Models (LLMs) in future work.

1. Introduction

In modern software engineering, ensuring the reliability of software systems is essential, particularly as software becomes more complex and integrated into critical applications. Formal software verification ensures that a system satisfies predefined formal specifications. These methods are now widely used in the software industry to enhance software reliability and reduce the cost associated with fixing errors during the testing phase of the software life cycle [1]. Instead, errors are eliminated at the design stage in the case of correct-by-construction software. If the system is already implemented, it can be formally verified by theorem proving, model checking, and formal assertions.

However, despite the widespread adoption of formal verification techniques in areas like low-level code verification and compiler optimization, there is a significant gap in their application to code translation between high-level languages. Formal verification was used in the compiler verification context to translate high-level code to a lower-level representation or verify compiler optimization. Early work on compiler verification has focused specifically on verifying C/C++ code [2] and LLVM intermediate representations (LLVM IR) [3]. LLVM, in this context, serves as a collection of modular and reusable compiler and toolchain technologies.

To the best of the authors’ knowledge, almost no previous research applied formal methods for code translation between high-level languages. The Trans-compiler-based high-level code translators mentioned in previous surveys [2,4], in addition to the recent trans-compilers [5,6], are evaluated by functional testing or by bilingual evaluation understudy (BLEU) score [7]. Functional testing tests the generated code’s functionality and is usually based on benchmarks with a finite number of test cases. The BLEU score is widely used to evaluate machine-translated text versus human translation, which acts as a reference [8]. The higher the BLEU score, the more similar the generated translation is to the reference or ground truth translation. CodeBleu, an enhanced version of BLEU, is calculated similarly to the BLEU metric and involves comparing the n-gram precision of the generated code with the reference code, considering the similarity of the code structures. However, studies have demonstrated that BLEU does not reliably reflect the translation quality for code migration tasks [9]. In addition, code translation also implies the presence of benchmark datasets of parallel code snippets. The available benchmarks include certain languages like C++, C#, Python, and Java. At the same time, there is a scarcity of benchmark datasets for other programming languages [10].

The mentioned evaluation methods have to be applied to the result of each run of the translator, with no guarantee for the correctness and completeness of the code translator. Correctness serves to ensure that the translated code complies with the grammar rules of the destination language and retains the same semantics as the source code. Completeness serves to ensure that all grammar rules of the source languages are fully translated.

In this work, we propose “TC-Verifier” as a formal verification architecture based on model checking that can be applied to trans-compiler-based code translators, specifically code translators based on parser generators. The modeling can be applied to the translators, whether they are implemented or under design. Model checking serves to represent a system in the form of states and transitions, then apply the verification of certain properties and specifications on the state machine representation [11]. A model checker can exhaustively and systematically explore all the possible behaviors of the state space rather than certain behaviors [12]. We introduce a novel architecture employing the Uppaal model checker to formally verify trans-compiler-based code translators.

The main contributions of this paper are given as follows: (1) presenting the TC-Verifier as an architecture for model-checking trans-compiler-based code translators that use parser generators and translation rules. (2) applying the TC-Verifier on Swift to Java trans-compiler as a case study. (3) Pointing out the strengths and limitations of the TC-Verifier approach that are inferred from the experiment.

The rest of the paper is organized as follows: Section 2 highlights the related work in recent literature, Section 3 introduces our proposed methodology, and in Section 4, we discuss the experimental results. Finally, Section 5 presents our conclusion.

2. Related Work

Verification of code translation from one programming language to another has been tackled in very few previous works. On the contrary, the formal verification of code optimization is much more adopted according to the survey undertaken by Amira et al. [2]. Two categories of verification were identified in that survey [2]: (1) Translation verification, which can be defined as verifying equivalence between input and translated code. (2) Translator verification: verifying the correctness of the translator’s behavior; hence, all translations should be correct. The latter category is the type of verification within this paper’s scope. In the following paragraphs, we summarize the Translator verification methods used in the literature.

Compiler verification is the most dominant when investigating the previous work of translators’ verification. Early work on compiler verification depended on mathematical proofs [2]. Compiler verification was recently tackled using the Coq proof assistant [13], known as the Calculus of Inductive Constructions. This method is considered under translator verification since it represents the compiler model in Coq and formally proves its correctness. This method cannot be easily applied to production compilers because it necessitates that the compilers be written in the language of a proof assistant like Coq. Therefore, proof-based verification might not be applicable for developers not familiar with the mathematical background of Theorem proving. Other methods used symbolic execution to automatically prove the correctness of compilation rules for both the source and target languages [14]. The framework’s reliance on symbolic execution can lead to scalability issues, especially for large and complex programs [15].

Zimmermann et al. [16] used abstract state machines for the incremental verification of compilers through bisimulation of two abstract state machines (ASMs) to verify the correct mapping between source and target languages. They formalized and verified the memory mapping of the initial versions with ASMs and discussed their extensibility for the next increments. The approach is demonstrated through the agile implementation and verification of a Sather-K compiler generating MIPS assembly language.

Most of the investigated literature in the context of translators’ verification depended on formal proofs or ASMs like Zimmermann et al. [16]. The formal proof approach is very complex and needs to be revisited or reapplied after updating the translator. The ASM idea of Zimmermann et al. [16] does not apply to the kind of trans-compilers we are interested in verifying. The trans-compilers [5,6,17] depend on the translation rules inside the parser generator written by developers. These trans-compilers do not use the mapping between the abstract syntax trees of the source and target languages like the compiler used by Zimmermann et al. [16]. Therefore, inspired by the work undertaken in [16], we decided to introduce TC-Verifier to model the behavior of the parser-based trans-compilers using state machines instead of the ASMs of the languages. The modeled compiler will then be verified using model-checking. This model-checking step will verify the translator’s correctness and point out its unverified or incorrect translations.

3. Methodology

This section proposes TC-Verifier as a novel approach for verifying trans-compilers through model-checking. In the following subsections, we introduce the trans-compilers’ methodology that we are interested in verifying. Then, we present our experiments with different verification tools to choose the most suitable model-checking tool for modeling and verifying the trans-compiler-based translators. Then, we detail the components and paradigms of the UPPAAL model-checker as the selected tool for our approach. In the following subsection, we demonstrate the TC-Verifier overall architecture, and we detail employing the UPPAAL model-checker to model and verify trans-compiler-based translators. In the last subsection, we explain the experimentation with TC-Verifier on a showcase trans-compiler [17] that translates from Swift to Java.

3.1. Trans-Compiler-Based Code Translators

Several trans-compiler-based translators have been implemented in recent years [2,4,5,6,17] that translate mobile applications code from one high-level language to another. The methodology behind these translators is based on the ANTLR (Another Tool for Language Recognition) [18] parser generator that receives the grammar file of the input language and generates a parser class that has all the functions of the grammar rules of the input language. The translator developer overrides these functions by the translation rules and tailors them to generate the corresponding code in the target language. We selected the trans-compiler by Muhammad et al. [17] as the showcase translator on which we apply the presented verification methodology.

3.2. Selected Verification Tool

Before using a specific tool to verify the trans-compiler, we performed an initial pilot test to check the user-friendliness of three model-checking tools. This initial test aims to choose a tool to implement the model-checking idea and prove the concept. We evaluated the tested tools according to the following points:

• User-friendly modeling: This is essential for providing a user-friendly model-checking methodology that can be applied to various types of trans-compilers regardless of the experience of the TC-Verifier users with model-checking tools.
• Flexible user-defined specifications: The specifications and properties to be checked can be written easily and do not require deep knowledge of model checking. Given that many model checkers are primarily designed to verify parallel code and address security-sensitive aspects, the chosen model checker must be adaptable enough to effectively model sequential software with user-defined specifications.
• Providing counterexamples: The results from the verification process should not only differentiate between the formally verified rules and those that remain unverified but also provide counterexamples where verification fails. These counterexamples are vital for pinpointing specific scenarios or input cases where the trans-compiler does not meet the expected criteria.
• Quality of tool documentation and tutorial availability: This documentation is essential for software developers like trans-compiler developers who are non-experts in formal verification to model their software and formalize its specifications.
• Modeling automation: capability to automate the transformation of the trans-compiler into its model form. Such automation would significantly streamline the verification process, enhancing efficiency and reducing the potential for error.

We investigated three tools for our problem context, and we evaluated them using the defined aspects. The three experimented tools are Alloy [19], Spin [20], and Uppaal [21,22].

Table 1. Comparison and evaluation of model checking tools.

Evaluation Aspect	Alloy	Spin	Uppaal
User-friendly modeling	Poor	Excellent	Moderate
Flexible user-defined specifications	Poor	Poor	Excellent
Providing counterexamples	Moderate	Excellent	Excellent
Documentation and Tutorials	Excellent	Moderate	Excellent
Automated modeling support	No	No	No

shows an evaluation and comparison resulting from the experiment of modeling the variable declaration translation process using the three tools.

Alloy, a tool that uses relational and first-order logic, requires a deep understanding of its specification language, significantly different from conventional programming languages. This complexity is further compounded by the need for users to understand how to formulate and check assertions within the system, making it less ideal for those unfamiliar with its unique structure. The analysis performed by Alloy is limited by the user-provided bounds, which may not be sufficient for the applications requiring comprehensive verification [23]. Meanwhile, Spin, which is tailored for verifying the distributed systems using Promela [24], proved cumbersome for our needs. The translation of Java code into Promela was not only time-consuming, requiring over 100 lines of code for a simple task, but also limited by Promela’s poor support for complex data types [25] and potential for state space explosions in more complex scenarios, making it unsuitable for comprehensive modeling of our trans-compiler. Although Promela allowed us to model the sequential behavior of the translator, it suffered from the scarcity of detailed tutorials and documentation and the difficulty of specifying the specifications to be checked.

Uppaal emerged as the preferred tool for modeling the trans-compiler [17] due to several key features shown in Table 1: its user-friendly interface, which includes drag-and-drop functionalities; a programming-like syntax that resembles high-level languages such as Java or C; and its capability to model state transitions effectively. Despite its inability to natively support string variables—a significant part of our trans-compiler’s functionality—the tool allows for the use of encodings to manage string data effectively. Although UPPAAL is designed for timely based systems, we could use it to model the sequential behavior of the translator by defining separate finite state machines for each input grammar rule translation process. Uppaal’s state machine approach, combined with straightforward symbolic queries for specifying properties, offers a simpler and more direct method for modeling and verifying the trans-compiler. This combination of features, along with its extensive documentation and command line utility for integration, led us to choose Uppaal over the other options as the most practical and adaptable tool for our needs.

While other model checkers are not impossible to apply, we selected Uppaal due to its programmatic advantages, compatibility with our verification methodology, and overall feasibility for the specific requirements of our trans-compiler.

3.3. Uppaal Components and Paradigms

In this subsection, we will define the important paradigms essential for understanding the Uppaal model checker tool.

• Uppaal system editor is used to create and edit the system’s state machine to be analyzed. A system description is defined by a set of process templates, global declarations, and a system definition. In the system editor, we can draw the state machine through states and edges and select options. For each edge, we can add a transition guard or condition and an update if an action is needed to occur.
• Global variables are needed by the state machine to simulate the modeled system. In the global declarations, we can also define the functions that can be called by the state machine.
• The Query is used to specify properties to be checked during simulation.
• The Verifier, also called verifyta for Uppal tool, checks the system properties by on-the-fly exploring the state space of a system in terms of symbolic states represented by constraints. The verification tool in UPPAAL also provides a requirement specification editor for specifying and documenting the system requirements.
• Traces can be used to generate counter-example traces for unsatisfied queries. These counter traces can be loaded into the symbolic simulator to be visualized and saved.
• The Symbolic simulator is a validation tool that enables the examination of the possible dynamic executions of a system. The symbolic simulator of Uppaal can visualize random traces as well as the counter traces that the verifier generates.

3.4. TC-Verifier Architecture

This paper introduces TC-Verifier for applying the Uppaal model checker to a trans-compiler-based translator. The overall architecture of TC-Verifier is given in Figure 1.

Any programming language has specifications and Grammar rules that define the correct syntax of the language. The trans-compilers [5,6,17] use a parser generator that walks the parse tree of the input language. Then, the translator developer defines translation rules between the source and input language and writes these rules in the parser walker of the input language. Whenever a source code is given to the translator, it is parsed, and the parser walker is triggered with the translation rules written inside. Eventually, the target code is generated. This part is number one in the architecture Figure 1.

In the second part of TC-Verifier, the translator developer iterates over the source language grammar and checks the translation rules for each statement in the source language grammar. The developer manually models these translation rules as states and transitions in the model checker Uppaal. If a rule or a production in the source grammar does not have a translation rule to the output language, it is modeled as a deadlock in the state machine. If a rule is covered but produces, in some cases, an incomplete or un-compilable code statement according to the output language Grammar, it is also represented as a deadlock in the state machine.

In step three, since the Uppaal model checker does not support string variables, we use a string-to-integers encoder script to encode all string variables needed to model the translation processes. The fourth step is running the state machine in the Uppaal model checker, and then Uppaal generates the XML format of the state machine model in step five.

In step six, we use prompt engineering with few shot learning to tune GPT4 to generate the query file needed to verify the state machine produced by Uppaal. Prompt engineering is the process of crafting inputs or prompts for language models to guide them toward desired responses [26,27,28]. Few-shot learning is a type of prompt engineering that involves providing the model with a few example prompts and answers within the input. This technique helps the model understand task patterns and improve response quality with minimal data [27]. This process saves time in the manual writing of the properties and queries as they must be written in the model-checker-specific syntax.

When the query file in step seven is ready, the verifier runs all the queries using an automated script, as shown in step eight. A result verified or not verified for each query is given in step nine. A proof trace is also given for the unverified cases. The output shows the verified and unverified paths for each code statement modeled by the model checker. In step ten, we implemented the automatic script to generate the result of the verification in the form of a list containing the query, the comment explaining the query and the result verified or not verified.

3.5. Verification Success Rate

Given that the translation process is modeled as a state machine, where translation correctness is verified based on whether the state machine reaches a valid final state without deadlocks, we can define an objective metric for evaluating the verifier. The formal notation of the state machine for the translation process is as follows:

• S be the set of all possible states in the state machine.
• $s_0\in S$ be the initial state.
• $s_f\in S$ be the valid final state (i.e., successful verification).
• $\Sigma$ be the input alphabet representing syntactic structures of the source language.
• $T:S\times \Sigma \to S$ be the state transition function defined by the trans-compiler rules.
• P be the set of translation paths executed for different syntactic structures.

We define the verification process as a deterministic finite state machine (FSM), where reaching $s_f$ signifies successful translation verification. We can define a verification success rate metric ($\mathcal{V}$). The verification success rate measures the proportion of input constructs that successfully reach the final state $s_f$. A higher $\mathcal{V}$ indicates better verification performance:

$$\mathcal{V}={\displaystyle \frac{|P_{\mathrm{success}}|}{\left|P\right|}}$$

(1)

where:

• $P_{\mathrm{success}}\subseteq P$ is the subset of translation paths that lead to $s_f$ (i.e., verified translations).
• $\left|P\right|$ is the total number of translation paths attempted by the symbolic queries.

After forming the Uppaal model, we can define queries to check certain properties and paths’ validity to later compute the verification success rate ($\mathcal{V}$). There are two main query types that we will use in our queries: possibly and invariantly.

The possibly query; $E<>p$ evaluates to true for a state transition system if and only if there is a sequence of action transitions s0 → s1 → …→ sn, where s0 is the initial state and sn satisfies p. p is a property that we define. For example, p can be $Process.end$ which means that the process reaches the end state. It can also be $not$ $deadlock$, which means that the state machine will not halt in a certain state. Many other properties can be defined according to the needed check. $E<>Process.end$ means there exists a path to reach the end state.

The invariantly query; $A\left[\right]p$ evaluates to true if (and only if) every reachable state satisfies the property p. The query $A\left[\right]$$Process.end$ means that for each reachable state in the state machine, the end state is reachable from that state.

Uppaal verifier symbolically checks these properties defined by symbolic queries and gives a result satisfied or not satisfied. For invariant queries, if the query is not satisfied, it gives a counter-example with a trace that proves the violation. In the case of a possibly query, it provides a trace only for the satisfied case.

Since Uppaal does not give all the counter-examples or all the violation traces, we had to write queries for all expected symbols to have a possibly query for each different symbol. This process will consume time if written manually. However, it can be easily automated since it is very systematic. We used ChatGPT 4 by open AI in this process using one-shot learning. We give one example to ChatGPT for the encoded variables, the XML model exported from Uppaal, where the format of the query needed in the prompt with the comment explains the query. We also write in the prompt the properties that we need to check in simple English language. The comment example is essential as it will be used in the interpretation of the results. Then, ChatGPT gives us all the queries needed to be placed in a .q file with the comment of each query. Since each translation process involves only a few dozen queries, we used human validation to assess the quality of the queries generated by GPT-4.

For each code statement translation state machine model, we have two main queries in addition to the path-specific queries. The first query is an invariantly query that checks for each reachable state in the state machine, the end state is reachable from that state and it never halts in any state given by $A\left[\right](not$ $deadlock$ $or$ $Process.end)$. This query is verified only if the code statement is completely covered for any possible input. The second query is a possibly query that checks whether there exists at least one path that reaches the end state given by E$<>$$Process.end$. This query is verified when there is at least one form of the code statement that is successfully translated by the translator. This query file can be imported to the Uppaal tool. The Uppaal tool runs each query one by one and if a counter-example is found, the example is loaded to the symbolic simulator and can be visualized. However, for readability purposes and since the number of queries will increase for more complex translation processes, we export the Uppaal model XML from the UI tool and pass it to a Python script. This script runs the XML model and the queries file together to iterate through the queries. The Uppaal tool can be run through the command line. The verifier functionality is accessible through the $verifyta$ command line utility in the bin directory of the UPPAAL distribution. In our experiment, we run $verifyta$ from a Python script to automate the model-checking process and to have a single click run for all the queries to be checked. The configurations of running the command line are as follows: -t, which is the trace type option set to 0 to generate some trace. The model XML file path is passed and the query file path is passed too. The Python subprocess module is used to run the command and capture its output. The capture–output is set to true, ensuring that both the standard output and the standard error are captured. The text parameter is set to True to make the output and errors returned as strings rather than bytes.

We take the output of the run and interpret the string result into a list of queries comments and results (verified/unverified) to derive meaningful insights about the model-checking process and to compute the verification success rate. The developers can also write their queries if they need to check certain behaviors that are different from the ones that are auto-generated by GPT.

3.6. Applying the Verifier Model to a Translator Case Study

In this paper, we applied the TC-Verifier on the Swift to Java translator [17] as a proof of concept for our methodology. However, our methodology can be applied to any trans-compiler-based high-level-to-high-level code translator that follows the similar translation method, depending on parsing and defining translation rules.

The code statement translation process is the behavior of the translator for each code statement in the Swift source language. A code statement, in our context, is a concrete code structure unit that has a grammar rule and can be broken down into other rules. For example, variable declaration, class declaration, and function declaration will be considered code statements. Each code statement translation process is modeled using states and transitions in Uppaal. The real translator iterates all the input code line by line in a loop. In each loop iteration, a code statement translation process is triggered. However, for simplifying, we will model each code statement translation separately. This will also allow us to have a more specific interpretation of the output of the verifier. In the following paragraphs, we will explain the modeling process for the variable declaration statement to explain our modeling methodology. The same method is applied to the rest of the statement types that we cover in our experiment.

We modeled the transition between the translator functions by transitions between states. The code starts from the main function and once the visit-java-top-rule function is visited, it goes to the next state. We represented the function visited trigger by a Boolean variable “visit-java-top”. From the visit-java-top-rule function according to the parser that the translator is following, it should visit the next function according to the type of the input statement to be translated. Figure 2 shows a model example for the variable declaration translation process.

For the variable declaration state machine, we declared a Bool $variabledeclaration$. The guard of transition checks the value of this Bool variable: if it is true, it will continue to the next state. According to the grammar of the input language, Swift in our case, the variable declaration rule has six different productions. We modeled the translator’s behavior when each of the six possible declarations was encountered. The variable declaration rule is shown in Figure 3.

We encoded the Swift identifier as a Boolean. If the swift identifier $swift\text{-}id$ is found and read by the parser, it means that it is valid. The rule of the Java identifier is the same as the Swift identifier. Therefore, the Java identifier variable $Java\text{-}id$ is assigned the Swift identifier value which is True for a valid identifier.

The translator in reality obtains the swift variable datatype and maps it through its database to the equivalent data type in Java. However, to implement this in UPPAAL, we needed variable strings, not constant ones, to model this process. Therefore, we had to do some encoding. We mapped the main datatypes of the Swift language to an array of integers named $swift\text{-}types$. The corresponding Java datatypes are mapped to integers too. A function $map\text{-}data\text{-}type$ is implemented to map each Swift data type to its corresponding Java data type. The Swift type array has eight integers. The elements from 0 to 6 represent the main swift data types like Int, Float, Double, etc. The element Swift-type[7] represents any user-defined data type. After the map function is triggered from the node $type\text{-}annotation$ to the node $java\text{-}type\text{-}return$, a guard checks the value of the Javatype variable. The transition to the next state Java-variable-dec1 will not happen unless the Javatype is assigned a value, i.e., a value that represents one of the mapped Java types like int, float, double, etc. In addition, it also checks that the Java-id Bool is given the value true.

For the real translator to form the output Java code, it concatenates the strings Java datatype, Java identifier, and semicolon. We implemented this concatenation as an addition operation and assigned it to the integer variable $Java\text{-}code$.

3.7. Experimental Setup-Statements Covered

This paper is considered the first round of iterative rounds to verify a complete trans-compiler inspired by the work performed by [16] to verify a compiler iteratively. To reach a complete verified Trans-compiler, the modeling should include all code statement types defined in the grammar of the input language. The grammar of Swift language, the input language in our case study, has nine types of statements. Declarations is one of the nine statements and it has sixteen declarations types. In this round, out of the sixteen declarations, we focused on three declaration types. The loop statement is the type of statement following the declaration statement, and we will model one type of loop statement out of four types as an example of another type of statement. the following paragraphs include details of modeling these statements:

(1) Variable declaration statement: we modeled the six different productions of the variable declaration grammar rule. The model included the built-in datatypes and their translation rules. The variable initialization was excluded in this phase as it will need modeling many other recursive statements like expressions with all their types; postfix, prefix, primary, Boolean …, etc. Each of these expressions is a code statement on its own and has several productions. Expressions will be included in the upcoming phases of the verifier and are not included in this paper. We modeled 14 different possible symbols for the variable declaration statement. The different symbols represent the different productions of the variable declaration grammar rule. In addition, some symbols represent the main built-in data types of the Swift language and a symbol that represents any other data type. For each symbol, there is a query that checks the validity of paths given this input symbol, plus two general invariantly and possibly queries explained in the last subsection and shown in

Table 2. Queries for verifying variable declaration statement.

Query	Comment
A[] (not deadlock or Process.end)	For all reachable states, we will reach end state and never halt in any other state
E$<>$Process.end	There exists a path that goes to end state
E$<>$(var_dec_type==1 and Process.end)	There exists a path to end when variable_declaration_with_pattern
E$<>$(var_dec_type==2 and Process.end)	There exists a path to end when variable_declaration_with_codeblock
E$<>$(var_dec_type==3 and Process.end)	There exists a path to end when variable_dec_with_setter
E$<>$(var_dec_type==4 and Process.end)	There exists a path to end when variable_dec_with_setterkey
E$<>$(var_dec_type==5 and Process.end)	There exists a path to end when variable_dec_with_willset
E$<>$(var_dec_type==6 and Process.end)	There exists a path to end when var_dec_with_willsetkey
E$<>$(map_data_type(swift_types[0]) and Process.end)	There exists a path to end when Int is passed
E$<>$(map_data_type(swift_types[1]) and Process.end)	There exists a path to end when Float is passed
E$<>$(map_data_type(swift_types[2]) and Process.end)	There exists a path to end when Bool is passed
E$<>$(map_data_type(swift_types[3]) and Process.end)	There exists a path to end when Double is passed
E$<>$(map_data_type(swift_types[4]) and Process.end)	There exists a path to end when Character is passed
E$<>$(map_data_type(swift_types[5]) and Process.end)	There exists a path to end when String is passed
E$<>$(map_data_type(swift_types[6]) and Process.end)	There exists a path to end when Array is passed
E$<>$(map_data_type(swift_types[7]) and Process.end)	There exists a path to end when any other type is passed

.

(2) Function declaration: The function declaration grammar rule has one production. However, the production has three optional non-terminals, as shown in Figure 4. Each non-terminal has a deeply nested grammar and is further defined by several different productions. We modeled 31 different paths that the model function declaration with all optional terms in the production. The IBaction function, which is the button action function in Swift, in addition to the override, are possible productions of the attribute term in the grammar rule. The paths also included the generic parameter clause, the where clause, the throw, the rethrow, and all built-in data types as parameters and as return types. We considered one parameter only as the same process will happen for any length of the parameters list. The function body part is excluded since it can contain any statement that has not yet been covered by the verifier in this phase. A number of 33 queries are checked for the function declaration model. They include the two main invariantly and possibly queries mentioned before and the queries for the 31 paths. The function declaration queries are shown in

Table 3. Queries for function declaration statement.

Query	Comment
A[] not deadlock or Process.end	Checks that all reachable states will reach the end state and never halt in any other state
$E<>$ Process.end	Checks that there exists at least one path that reaches the end state
$E<>$ (Ibaction == true and Process.end)	Checks that, if IBAction is true, the end will be reached
$E<>$ (override == true and Process.end)	Checks that, if override is true, the end will be reached
$E<>$(override == false and Process.end)	Checks that, if override is false, the end will be reached
$E<>$ (java-access == map-access(1) and Process.end)	Checks for private access modifier
$E<>$ (java-access == map-access(2) and Process.end)	Checks for private(set) access modifier
$E<>$ (java-access == map-access(3) and Process.end)	Checks for fileprivate access modifier
$E<>$ (java-access == map-access(4) and Process.end)	Checks for fileprivate(set) access modifier
$E<>$ (java-access == map-access(5) and Process.end)	Checks for internal access modifier
$E<>$ (java-access == map-access(6) and Process.end)	Checks for internal(set) access modifier
$E<>$ (java-access == map-access(7) and Process.end)	Checks for public access modifier
$E<>$ (java-access == map-access(8) and Process.end)	Checks for public(set) access modifier
$E<>$ (java-access == map-access(9) and Process.end)	Checks for open access modifier
$E<>$ (java-access == map-access(10) and Process.end)	Checks for open(set) access modifier
$E<>$ (generic-parameter-clause == 1 and Process.end)	Checks for generic parameter clause presence
$E<>$ (generic-where-clause == 1 and Process.end)	Checks for generic where clause presence
$E<>$ (throw == 1 and Process.end)	Checks for usage of throw
$E<>$ (rethrow == 1 and Process.end)	Checks for usage of rethrow
$E<>$ (java-type == map-type(71) and Process.end)	Checks for type mapping of Int as a parameter
$E<>$ (java-type == map-type(72) and Process.end)	Checks for type mapping of Float as a parameter
$E<>$ (java-type == map-type(73) and Process.end)	Checks for type mapping of Bool as a parameter
$E<>$ (java-type == map-type(74) and Process.end)	Checks for type mapping of Double as a parameter
$E<>$ (java-type == map-type(75) and Process.end)	Checks for type mapping of Character as a parameter
$E<>$ (java-return-type == map-type(76) and Process.end)	Checks for type mapping of String as a return type
$E<>$ (java-return-type == map-type(77) and Process.end)	Checks for type mapping of Array as a return type
$E<>$ (java-return-type == map-type(78) and Process.end)	Checks for type mapping of any other type or user-defined type as a return type

.

(3) Class declaration: class declaration is also modeled in this phase. We are interested in the signature part of the class without including the class body in this phase as it will consist of any statement type that may not be modeled yet. The class declaration grammar rule has two productions, as shown in Figure 5. In each production, there are optional parts that produce many other different forms. The class-declaration model was checked by 17 different symbolic queries. Two are the main invariantly and possibly queries. The other 15 queries include the possibilities of the optional terms like all access modifiers, general parameter clause, where clause, and type inheritance clause.

Table 4. Queries for class declaration statement.

Query	Comment
A[] (not deadlock or Process.end)	Checks that all paths will either reach the end state or not result in a deadlock
$E<>$Process.end	There exists at least one path that reaches end state
$E<>$(java-access-level==map-modifier(access-levels[0]) and Process.end)	Checks for access when access level is set to private
$E<>$(java-access-level==map-modifier(access-levels[1]) and Process.end)	Checks for access when access level is set to private(set)
$E<>$(java-access-level==map-modifier(access-levels[2]) and Process.end)	Checks for access when access level is set to fileprivate
$E<>$(java-access-level==map-modifier(access-levels[3]) and Process.end)	Checks for access when access level is set to fileprivate(set)
$E<>$(java-access-level==map-modifier(access-levels[4]) and Process.end)	Checks for access when access level is set to internal
$E<>$(java-access-level==map-modifier(access-levels[5]) and Process.end)	Checks for access when access level is set to internal(set)
$E<>$(java-access-level==map-modifier(access-levels[6]) and Process.end)	Checks for access when access level is set to public
$E<>$(java-access-level==map-modifier(access-levels[7]) and Process.end)	Checks for access when access level is set to public(set)
$E<>$(java-access-level==map-modifier(access-levels[8]) and Process.end)	Checks for access when access level is set to open
$E<>$(java-access-level==map-modifier(access-levels[9]) and Process.end)	Checks for access when access level is set to open(set)
$E<>$(java-access-level==map-modifier(access-levels[10]) and Process.end)	Checks for scenarios with no specific access level modifier
$E<>$ (swift-inheritance==1 and Process.end)	Checks that there exists a path to end with class inheritance
$E<>$ (swift-inheritance==0 and Process.end)	Checks that there exists a path to end with no class inheritance
$E<>$ (parameter-clause==1 and Process.end)	Checks for the existence of a path to the end if a parameter clause is present
$E<>$ (where-clause==1 and Process.end)	Checks for the existence of a path to the end if a where clause is present

shows the queries for the class declaration translation model.

(4) While statement: As an example of loop statements, we have modeled the while statement translation. The while statement grammar rule has only one production. consisting of the while keyword, the condition list, and the code block. The condition list has four productions, as shown in Figure 6. Queries for While statement are shown in

Table 5. Queries for while statement.

Query	Comment
A[](not deadlock or Process.end)	Checks that all paths will either reach the end state or not result in a deadlock.
E<>Process.end	There exists at least one path that leads to end.
E<>(swift_condition==1 and Process.end)	Checks whether one condition is passed and leads to the end.
E<>(swift_condition==2 and Process.end)	Checks whether multiple conditions are passed and lead to the end.
E<>(swift_condition_type==1 and Process.end)	Checks whether an expression condition is passed and leads to the end.
E<>(swift_condition_type==2 and Process.end)	Checks whether an availability condition is passed and leads to the end.
E<>(swift_condition_type==3 and Process.end)	Checks whether a case condition is passed and leads to the end.
E<>(swift_condition_type==4 and Process.end)	Checks whether a binding condition is passed and leads to the end.
E<>(swift_block==1 and Process.end)	Checks translation of a code block leading to the end state.

.

4. Results and Discussion

After modeling the trans-compiler translation process for four code statements with 67 symbolic queries checked by UPPAAL, we can interpret the results of each statement and define the translator parts that the model checker completely verifies. The model checker results also detect and define the parts that the translator fails to translate by producing uncompilable code, running into runtime errors, or producing no equivalent translations in the output Java code. The number of total attempted paths, verified paths, and verification success rate computed by Equation (1) are shown in

Table 6. Verification success rates for modeled statements.

Statement	Total Attempted Paths ($\left|\mathit{P}\right|$)	Verified Paths ($|{\mathit{P}}_{\mathbf{success}}|$)	Success Rate ($\mathcal{V}$)
Variable declaration	14	8	57.14%
Function declaration	27	15	55.55%
Class declaration	17	7	41.17%
While statement	9	4	44.44%

.

4.1. Variable Declaration Verification

For the variable declaration statement, according to the Swift grammar rules, the variable declaration rule has six different productions. The verifier clearly showed that only one production is covered by the translator. The covered production is declaration with pattern. For the other five declarations, three of them are not explicitly used by the Swift language itself anymore, the declaration with a code block, the willsetkey, and the setterkey, and therefore they were not covered by the compiler. The other two declarations, willset and setter, can have a workaround to be translated to Java but no direct correspondence.

Inside the declaration pattern, which is the only verified declaration production, the swift type can be one of the built-in types, user-defined type, or typeless. This is due to the nature of Swift language. It can infer the type of the variable from the value assigned to it. However, the Java language is a type language. Therefore, the declaration is successfully verified for seven built-in datatypes of the Swift language. Verified means that for any declaration with one of these verified types, the translator generates a correct translation. However, it is unverified for any other type including user-defined types. For example, if the input swift code has a class student declared, a variable declaration with the type student, the output java data type will be null according to the translator behavior modeled. The typeless case is not included in this phase of the verifier since it implies initialization for the variable, and this part is not modeled yet. Out of the 16 queries checked for variable declaration, 9 were verified or satisfied, as shown in

Table 7. The verifier results for variable declaration statement.

Query Comment	Result
For all reachable states, we will reach end state and never halt in any other state	NOT satisfied
There exists a path that goes to end state	Satisfied
There exists a path to end when variable_declaration_with_pattern	Satisfied
There exists a path to end when variable_declaration_with_codeblock	NOT satisfied
There exists a path to end when variable_dec_with_setter	NOT satisfied
There exists a path to end when variable_dec_with_setterkey	NOT satisfied
There exists a path to end when variable_dec_with_willset	NOT satisfied
There exists a path to end when var_dec_with_willsetkey	NOT satisfied
There exists a path to end when Int is passed	Satisfied
There exists a path to end when Float is passed	Satisfied
There exists a path to end when Bool is passed	Satisfied
There exists a path to end when Double is passed	Satisfied
There exists a path to end when Character is passed	Satisfied
There exists a path to end when String is passed	Satisfied
There exists a path to end when Array is passed	Satisfied
There exists a path to end when any other type is passed	NOT satisfied

.

4.2. Function Declaration Verification

For the function declaration code statement, 15 of the 27 checked queries were verified successfully. Using Equation (1), the success rate for function declaration is shown in Table 6. The verified paths include the button action function, override attribute, and any parameters with covered data types like those verified in the variable declaration code statement. Similarly, the return type of the function is successfully verified for the seven built-in data types shown in Table 3. The access modifier of the function is verified for three access modifiers out of ten possible access modifiers in Swift. The unverified attempted paths include the unmapped access modifiers, the generic parameter clause, the where clause, the throw and rethrow keywords, and any user-defined data type for the parameter or the return type of the function.

Table 8. The verifier results for function declaration statement.

Input Query Comment	Result
Checks that all paths will reach the end state without halting in any other state	NOT satisfied
Checks that there exists at least one path that reaches the end state	Satisfied
Checks that if IBAction is true, the end will be reached	Satisfied
Checks that if override is true, the end will be reached	Satisfied
Checks that if override is false, the end will be reached	Satisfied
Checks for private access modifier	Satisfied
Checks for private(set) access modifier	NOT satisfied
Checks for fileprivate access modifier	NOT satisfied
Checks for fileprivate(set) access modifier	NOT satisfied
Checks for internal access modifier	Satisfied
Checks for internal(set) access modifier	NOT satisfied
Checks for public access modifier	Satisfied
Checks for public(set) access modifier	NOT satisfied
Checks for open access modifier	NOT satisfied
Checks for open(set) access modifier	NOT satisfied
Checks for generic parameter clause presence	NOT satisfied
Checks for generic where clause presence	NOT satisfied
Checks for usage of throw	NOT satisfied
Checks for usage of rethrow	NOT satisfied
Checks for type mapping of Int as a parameter	Satisfied
Checks for type mapping of Float as a parameter	Satisfied
Checks for type mapping of Bool as a parameter	Satisfied
Checks for type mapping of Double as a parameter	Satisfied
Checks for type mapping of Character as a parameter	Satisfied
Checks for type mapping of String as a return type	Satisfied
Checks for type mapping of Array as a return type	Satisfied
Checks for type mapping of any other type or user-defined type as a return type	Satisfied

shows the results of the function declaration statement verification. When the unverified paths are present, the translator does not face a runtime error, and it does not generate uncompilable Java code either. We consider these paths unverified as the output Java code is missing the semantics of the input code, i.e., the output Java code becomes functionally not equivalent to the input code.

4.3. Class Declaration Verification

For the class declaration statement, 7 out of 17 queries were verified successfully. The 7 queries include the covered access modifiers. As mentioned in the function declaration 3 out of 10 access modifiers are mapped to Java. The verified queries also include an inheritance clause that is successfully translated. The unverified paths include the parameter clause, the where clause, and any unmapped access level modifier. As mentioned in the function declaration result, the parameter clause, the where clause, and the unmapped access modifiers are not verified as the translator generates no corresponding output in the Java code equivalent to these clauses. However, it does not face runtime errors or generate compilable Java code.

Table 9. The verifier results for class declaration statement.

Input Query Comment	Result
Checks that all paths will either reach the end state or not result in a deadlock	NOT satisfied
Checks whether there exists at least one path that leads to the end state	satisfied
Checks for access when access level is set to private	satisfied
Checks for access when access level is set to private(set)	NOT satisfied
Checks for access when access level is set to fileprivate	NOT satisfied
Checks for access when access level is set to fileprivate(set)	NOT satisfied
Checks for access when access level is set to internal	Satisfied
Checks for access when access level is set to internal(set)	NOT satisfied
Checks for access when access level is set to public	Satisfied
Checks for access when access level is set to public(set)	NOT satisfied
Checks for access when access level is set to open	NOT satisfied
Checks for access when access level is set to open(set)	NOT satisfied
Checks for scenarios with no specific access level modifier	Satisfied
Checks that there exists a path to the end if class inheritance is enabled	Satisfied
Checks that there exists a path to the end if class inheritance is not enabled	Satisfied
Checks for the existence of a path to the end if a parameter clause is present	NOT satisfied
Checks for the existence of a path to the end if a where clause is present	NOT satisfied

shows the verifier results for the class declaration statement model.

4.4. While Statement Verification

For the while statement, four out of nine queries were verified successfully, as shown in

Table 10. The TC-Verifier results for while statement.

Query Comment	Result
Checks that all paths will either reach the end state or not result in a deadlock.	NOT satisfied
Checks whether one condition is passed and leads to the end.	Satisfied
Checks whether multiple conditions are passed and lead to the end.	NOT satisfied
Checks whether an expression condition, the end will be reached.	Satisfied
Checks whether an availability condition is passed, the end will be reached.	NOT satisfied
Checks whether a case condition is passed, the end will be reached.	NOT satisfied
Checks whether a binding condition is passed, the end will be reached.	NOT satisfied
Checks whether the translation of a code block leads to the end state.	Satisfied

. The translation process is successful when only one condition is passed. If several conditions are passed, condition one only is translated, therefore, this path is unverified. The expression condition only is verified while the other three condition types are not verified. This is due to the absence of equivalent conditions in Java.

4.5. Comparison with BLEU Score

To evaluate the TC-Verifier and illustrate the difference between formal verification via a model checker and translation evaluation by BLEU score, we consider an input Swift statement and we prompted GPT4 to generate multiple Java translations for this statement, as shown in

Table 11. Swift array declaration and Java translations.

Language	Code
Swift	let numbers: [Int] = [1, 2, 3, 4, 5]
Translation 1	int[] numbers = {1, 2, 3, 4, 5};
Translation 2	List<Integer> numbers = new ArrayList<> (Arrays.asList(1, 2, 3, 4, 5));
Translation 3	int[] numbers = new int[]{1, 2, 3, 4, 5};
Trans-compiler translation	final int[] numbers = {1, 2, 3, 4, 5};

. GPT4 generated five translations. We tested all of them. Four translations were compiled successfully. The translation that did not compile was removed. Another translation, although compiled successfully, was also omitted since it depended on manually adding the numbers to the array one by one, and this would require multiple code statements that would lead to a very low BLEU score and would also include other grammar rules than the array declaration. Therefore, to accurately evaluate the translation, we also omitted this candidate. Then, we added the trans-compiler translation as the last row in Table 11. The trans-compiler translation is very similar to the translation number one in Table 11, except that the “final” keyword is generated as an equivalent to the “let” keyword in Swift. Any of the GPT4 did not generate the “final” keyword suggested translations. However, the generated codes are not functionally wrong or non-equivalent to the input code. It would just be more precise to use “final” and evaluating GPT4 is not the scope of this paper.

According to the model checking methodology, whenever the translator verifier decides a code statement as verified, it means that, for any example of this code statement, the translation will be correct. However, for BLEU, each translation is evaluated independently and the BLEU score depends on comparing the evaluated translation and the reference translation. For our case study translator, the TC-Verifier tool verified the array declaration shown in the example. However, the BLEU score will differ according to the reference translation as shown in

Table 12. BLEU scores for different reference translations VS TC-Verifier result.

	BLEU Reference Translation				TC-Verifier
	Translation 1	Translation 2	Translation 3	Trans-Compiler
Array declaration	52.54	31.34	88.01	100.00	Verified translation

. Each column shows the BLEU score for the translation provided by the Swift to Java trans-compiler with a different translation from the translations shown in Table 11.

As shown in Table 12 the BLEU score evaluator will give 100% only if the trans-compiler translation was set as the reference. In Benchmark datasets for code-to-code translations, there might only exist one reference translation. A translator can generate a translation with a slightly different code structure but equivalent functionality. If we assume that each translation in Table 11 was generated by a different compiler that was modeled by TC-Verifier, the TC-Verifier result would be “verified translation”. The translation rules followed for the input code statement are modeled and verified successfully.

4.6. Discussion

The verification results of the four tested statements checked by 67 symbolic queries show the following: (1) The common unverified cases are cases where there is no direct correspondence in the target language. For example, no equivalence to the where clause used in function and class declarations in the Swift language. Similarly, access modifiers like fileprivate and open access modifiers do not have any direct equivalent in Java. (2) Other than the no direct equivalent cases, there were cases that Java had an equivalent either directly or with a workaround, but were not covered by the compiler, like the usage of the throw keyword and the usage of the generic parameter clause in the class declaration. These cases show the importance of verification with model checking as it can help compiler developers to add the unverified parts that can be implemented to improve the trans-compiler. For the parts that have no direct equivalence, integration with fine-tuned LLMs can be the next step. While this paper proposes the augmentation of trans-compilers with LLMs to fill the identified translation gaps, the formal verification of LLMs themselves presents a complex challenge. Future research could explore the feasibility of formally verifying LLMs within this context.

The main limitations of the presented approach for verifying trans-compilers can be summarized as follows: (1) The hierarchical and recursive nature of the grammar rules and productions makes it sometimes impossible to complete a state machine representation of a certain rule or statement until all other related rules are modeled. This made us ignore some parts in the modeled rules and postpone them to upcoming modeling rounds, like ignoring the variable declaration with initialization, for example. (2) The modeling process that converts the translator’s behavior to a state machine is performed manually. Therefore, it must be learned by the TC-Verifier users. The modeling process is considered easy and user-friendly for trans-compiler developers. However, it can take more time for users who are not familiar with the translation process behind the translator to be verified. Automating this process will boost the time complexity of the modeling process and eliminate the need to learn manual modeling. (3) The UPPaal model checker used in the presented architecture does not support string variables, and therefore all the string variables used in the translator had to be encoded to be modeled in the Uppaal tool. (4) Migrating the TC-Verifier to be used for different source and target languages implies the cost of modeling the state machines for the new translation processes. The symbolic queries can be reused for the same source language even if the output language is changed. If the source language is changed, new queries must be defined. However, queries can be automatically retrieved either using an LLM as we presented in the approach or by an automated script that can directly retrieve the symbolic queries from the XML of the state machine.

5. Conclusions

This study presents a novel approach named “TC-Verifier” to verify trans-compilers using the Uppaal model checker to assess grammar rule compliance for high-level code translations. The verification of variable, function, class declarations, and while statements showed promising accuracy, with specific limitations observed in unsupported grammar elements and complex function bodies.

These findings underscore the practical importance of formal verification in code translation, providing an alternative to traditional evaluation methods that rely solely on output similarity measures. By ensuring grammar rule compliance, this approach could significantly improve the reliability of trans-compilers used in the industry.

Despite these contributions, this study faced limitations, including the complexity of encoding certain data types and the partial coverage of grammar rules. Furthermore, Uppaal’s constraints on string handling presented challenges that necessitated workarounds.

Future work could address these limitations by automating grammar rule encoding and exploring model-checking tools with expanded data-handling capabilities. Additionally, leveraging machine learning models to assist with automated modeling could streamline the verification process and reduce manual effort. Fine-tuning LLMs for filling the formally identified gaps of the trans-compiler translators and integrating both solutions will result in improved code-to-code translators. However, verifying LLMs will open new research questions.

This research marks an important step towards reliable, formally verified code translation between high-level languages. As further developments in formal verification methods are implemented, trans-compilers can become even more reliable, supporting safer and more consistent code translation in a variety of software development contexts.