Applying Large Language Model Analysis and Backend Web Services in Regulatory Technologies for Continuous Compliance Checks

Abstract

Regulatory technologies (RegTechs) are a set of electronic and digital technologies applied to check compliance in industrial processes. Such applications also aim to simplify the process of data collection and exchange according to the expected format over the cloud or the internet. Industrial processes are required to meet basic regulatory requirements according to law and follow a set of industry practices. Industry practices must be compliant with the basic regulatory requirements. Such applications also need a high level of privacy to protect the individual participant’s data from competitors but are revealed to the relevant regulatory agencies. However, there cannot be a standard data procurement method, as the industrial processes are different for individual businesses and often involve various stages of data collection with different aims. Also, the regulatory requirements may be changed over time. These challenges can be addressed over an online system that uses large language models (LLM) to perform continuous compliance checks. With LLMs, RegTech can be easily scaled up to meet new requirements. It can also help with data analysis and reformatting for different stakeholders in RegTech, such as producers, supply chains, regulators, and financial institutions. It can check for acceptable values with regards to RegTech through either numeric comparisons or enumerations matching. In this paper, we propose a comprehensive RegTech framework backed by LLM and web services. We propose a method to measure the accuracy of LLM in returning appropriate responses for RegTech queries and herein analyze several LLMs to conclude that they are satisfactory for basic tasks, but a dedicated LLM is needed for RegTech. Furthermore, we test the LLM’s tool-calling capabilities to identify and use dedicated functions in the form of web services to enhance the analytical accuracy and consistency of RegTech-related prompts.

1. Introduction

Primary industries, such as agriculture, construction, fisheries, and forestry, are vital to global economies [1,2,3]. However, these industries are subject to a complex web of regulations [4,5,6,7], often involving multiple jurisdictions and agencies [8]. Compliance with these regulations is crucial for ensuring product safety, environmental sustainability, and fair-trade practices [9,10,11]. Traditional compliance methods can be time-consuming, costly, and prone to human error [8,12,13,14]. This is where regulatory technology (RegTech) can streamline compliance checks and documentation with an online process enhanced by modern computing and Internet of Things (IoT) tools [15]. RegTech uses such technology to improve regulatory efficiency, reduce compliance costs, and enhance regulatory oversight [16,17,18]. The primary objective of applying RegTech solutions in primary industries is to release regulatory burdens for all stakeholders by streamlining compliance processes, reducing administrative burdens, and improving decision making.

In this paper, we consider the agriculture sector, which is a major primary industry, as a case study. It involves the typical stakeholders of RegTech: assessors, including regulators; assessees, including individual farmers; and artificial intelligence (AI) tools like large language models (LLM) and deep learning services. The main aim of this system is compliance checks.

Despite its potential, implementing RegTech in agriculture faces significant obstacles. Technically, limited connectivity and access to necessary technologies and digital skills [19], coupled with the ever-changing landscape of agricultural regulations [15], pose substantial challenges. Integrating disparate agricultural systems and addressing cybersecurity and data privacy concerns [20,21] further complicate matters. Economically, the high cost of developing and deploying RegTech solutions [22], combined with a lack of awareness and understanding among farmers [21], creates significant barriers to adoption [23]. Socially, resistance to change from those accustomed to traditional farming methods can also hinder progress [24]. Finally, the political landscape, with its complex global supply chains, multiple jurisdictions, and diverse stakeholders, makes standardizing regulatory compliance a particularly difficult undertaking. LLMs can mitigate many of these problems by providing a continuous, easy-to-use interface to technology.

1.1. Compliance Check and Its Challenges

Compliance checks comprise the following processes:

(1)

Recording information during production in industries;

(2)

Auditing and checking the correctness of the information and its adherence to the rules;

(3)

Considering exceptional situations and making appropriate decisions;

(4)

Certification of products as suitable;

(5)

Making the outcomes available to the public in an immutable manner.

A compliance check is mainly a one-to-one process between a producer, such as a manufacturer or farmer, and the corresponding industry regulator. However, there are many such producers and many levels of compliance regulators, e.g., governments and assessment bodies, as shown in Figure 1. The critical challenges of compliance on a large industrial scale are as follows:

• It is impossible for a human to remember all the rules;
• Although the majority of the compliance regulations are fixed, some critical unexpected issues can appear in a lengthy process, such as in agriculture, leading to exceptional assessments;
• Conditions external to the industrial process may change, leading to minor temporary changes in the compliance expectations;
• Data are distributed among many individual sources, especially considering that data are generated by both humans and smart IoT devices. Currently, many data are paper-based, and there is a lack of digital synchronization;
• There are privacy concerns about the sharing of data among peer producers and the regulator.

The agricultural sector confronts a multitude of compliance challenges, primarily stemming from a complex and ever-evolving regulatory landscape [25,26,27]. Frequent regulatory updates necessitate constant adaptation and vigilance from regulated entities, especially small farms, which often struggle with limited resources to prioritize and meet compliance requirements [4,9,28,29]. While digital technologies offer potential solutions, they also introduce new challenges, such as the digital divide [30,31,32], effective data management [33], non-invasive sensors [34], and cybersecurity risks [33,35,36,37]. Regulators also face obstacles such as geographic diversity of the fragmented sectors [38] and resource constraints such as EU cross-compliance, which highly relies on extensive monitoring, ref. [39] hindering effective oversight and enforcement. Challenges also exist in the auditor’s technical competence and field knowledge, and the potential negative impact of the common interests between certifying companies and their clients on audit quality may lead to the issue of certification reliability [40].

1.2. RegTech LLM–Web Service: Contributions and Limitations

This paper explores the potential of large language models (LLMs) to address real-time issues in compliance checks for industries, specifically for agriculture. We herein investigate a scenario involving regulators and farmers who must collaborate effectively. By employing an LLM as an impartial intermediary, real-time challenges can be swiftly resolved. This paper utilizes data from a cherry orchard and honeybee apiary to analyze and illustrate the practical application of LLMs in agricultural regulatory contexts. We delve into the advantages, limitations, and capabilities of LLMs in this specific use case. Also, here, we consider a continuous compliance check process that keeps on going for a lengthy period of time for the industrial process and is repeated in cycles. The key contributions of this paper are as follows:

• Basic framework/set of functionalities of different automation needed for compliance checks in a RegTech system, including the following:

  o

  A new multi-level prompt categorization with respect to difficulty in processing time series data;

  o

  Measuring accuracy with respect to consistency as needed by RegTech.

• Analysis of LLM characteristics for RegTech and its performance with regards to these basic compliance checks automation.
• A web service (WS)-based improvement on the performance of LLM.

The remainder of the paper is structured as follows: Section 2 discusses the current state of LLM, automated compliance checks, and regulatory technologies. This provides the context and motivation for the proposed LLM–web service–RegTech system. Section 3 presents the critical process of compliance check, using agriculture as an example, and the LLM–web services-based regulatory technology architecture. Section 4 presents the LLM performances, and Section 5 presents the observations and future work.

2. Related Work

This section discusses the current status of developments in LLMs, along with the compliance check process.

2.1. Role of Large Language Models vs. Traditional AI/ML

Traditional AI systems, such as rule-based systems and expert systems, are often rigid and inflexible, and they struggle to adapt to changing environments and unforeseen circumstances [41]. These systems excel at specific tasks but cannot generalize knowledge and apply it to new situations. Traditional AI systems rely heavily on structured data and predefined rules [42], limiting their ability to learn from unstructured or noisy data. Some classic AI techniques, such as expert systems, can be computationally intensive, especially for complex problems [43]. To address these limitations, modern AI approaches like natural language processing have emerged. These techniques enable AI systems to learn from data, make intelligent decisions, and adapt to changing conditions. In addition, the explainable AI (XAI) system is emphasized by many researchers, as it can enhance trust, transparency, and accountability in the decision-making process [44,45,46].

LLMs are excellent at summarizing text [47,48,49,50]. They are good at logical reasoning in the short term and when the context is clear [51]. However, they are limited in analytical capabilities if asked to perform a calculation in real time. They can understand the requirements and develop the procedure to obtain the results, but they are poor at executing the procedure to obtain the results, particularly if the data are great in quantity. In order to solve this, tool-calling abilities are being developed for LLM [52,53]. These enable the LLM to call external functions for performing a dedicated task. The LLM must create the proper input for these functions, and the function should return a proper output for LLM consumption.

In this paper, we analyze the current general purpose of LLM’s capabilities for compliance-related functionalities. We term LLM’s tool calling as “back-end web services (WS)”. Web services are well-established technologies that execute codes with inputs via an HTTP request and return a response in the form of text or other media [54]. They are suitable in terms of security and scalability when deployed over the cloud.

2.2. Compliance Checks

Compliance checks are typically time-limited manual activities between a regulator and producers that happen occasionally. In this paper, we propose a web-based compliance check mechanism that is not time-limited but a continuous automated process between the producers and online LLM and WS as well as the regulators, when needed.

2.2.1. Source of Data

Data about production can come from various sources, including human and IIoT devices. Human data can be recorded through mobile apps and desktop applications. While there are still producers relying on manual paper-based data recording, even that data can be converted to digital information. However, the data are mainly structured, i.e., specific data are recorded for specific purposes. The following are some categories of data in the case of agriculture:

• Sensors collect real-time numeric data on soil moisture, temperature, plant health, and other factors [55,56]. This is a well-established technology;
• Pest control measures with traps are still mostly manually recorded, although IoT devices are being developed for this as well [57,58]. On-site detection of pesticide residues, such as through biosensors and colorimetric sensors, is also mostly recorded manually [59,60];
• Drones and satellites capture high-resolution images of fields, allowing farmers to assess crop health, identify pest or disease outbreaks, and monitor irrigation effectiveness [61,62,63];
• QR code, RFID, and NFC offer a secure and easy way for consumers to access product information with their smartphones [64,65,66,67];
• Several advanced technologies that can directly generate digital data in the form of images or text are being developed to uniquely identify farm products, such as DNA barcodes and unique chemical makeup [68,69,70,71];
• Compliance checks in the industry can involve a variety of methods or work, including document review, field inspections, and laboratory testing, if necessary [72]. These will ultimately create documents that are usually in a structured or semi-structured format.

2.2.2. Industrial Process and General Compliance

The industrial process is a series of activities that create a product. The product must be worthy of human consumption. The process must follow the best practices. Often, there are industrial practices that are above the bare minimal requirement for safe production. The industrial process will involve human activities as well as autonomous equipment, thus involving a degree of uncertainty. Another key feature of an industrial process is that it will be finite, i.e., as a duration called a process cycle. The process cycle would be repeated again and again to produce the same product, possibly with minor variations with time. From a regulatory point of view, the aim is to ensure that the process complies with the safety, environmental, and human resource requirements. Regulatory procedures ordinarily do not judge the quality of the product or process as long as it is above the minimum requirements.

In the case of agriculture, the industrial process is the crop cycle, which takes a few months. The end product is food, which should be safe in terms of content and pests. If exported, it needs to meet biosecurity standards. It also involves both manual labor and IIoT devices to generate localized and continuous data. There are compliance requirements throughout the whole process, from seeding to packaging. Regardless of the length of the process, it is often irreversible. For example, if the agriculture process is corrupted, it impacts food safety and quality. Also, certain events may need to be recorded at a particular time, and failing to record them leaves unfillable gaps in the data for regulatory compliance checks.

Compliance efforts are hindered by three key challenges: accurate regulatory interpretation [73], seamless data collaboration or transformation [74], and efficient compliance verification. Accurate interpretation of complex regulations is essential to establish clear compliance standards. Seamless data sharing and analysis across systems and organizations is vital for effective compliance monitoring.

Traditional methods, such as third-party audits and certifications, can be costly and time-consuming, often involving on-site visits [75]. Additionally, communication barriers between producers and regulators [76], inconsistent regulatory application [25], and high turnover of supervisors or auditors [77] can further impede and complicate compliance. These third-party entities can provide valuable expertise and assistance, but they can also increase costs and introduce additional layers of complexity to the communication process [78]. Inconsistencies in the application of regulations by different regulators, leading to varying interpretations of rules, inconsistent enforcement actions, and increased compliance costs, can create uncertainty and unfairness for regulated parties. Changes in actual human staff within supervisory units can introduce variability in regulatory oversight. When experienced auditors leave, their knowledge and expertise can be lost, making it difficult to maintain consistent standards.

Automated and digital solutions may help to address these challenges by streamlining processes, improving accuracy, and enhancing transparency. By leveraging digital technologies, auditors can reduce the frequency and duration of on-site visits, leading to significant cost savings. Measures such as remote monitoring, data analytics, automated document review, and virtual audits can all contribute to a more efficient and cost-effective compliance process.

2.2.3. Automated Compliance Check

Automating compliance checks and verification processes can significantly enhance efficiency and accuracy [79]. This empowers human experts to concentrate on higher-value endeavors, such as strategic decision making and intricate problem solving. Early approaches to automation, such as “Black Box” and “Gray Box” methods [80], relied heavily on hardcoded rules and manual interpretation [79]. These methods proved to be costly to maintain and challenging to adapt, and they lacked a generalized framework for modeling rules and regulations [79]. Semi-automated methods, exemplified by the RASE (requirement (R), applicabilities (A), selection (S), and exceptions (E)) methodology, aimed to translate regulatory texts into machine-processable formats [81]. While this approach improved efficiency, it still necessitated substantial manual intervention [79]. More advanced methods leverage natural language processing (NLP) techniques [82]. Rule-based NLP methods, although effective, demand significant manual effort in data training [79]. Domain-specific regulatory texts are generally more amenable to NLP analysis than general texts [83]. Semantic and syntactic information extraction methods, frequently combined with ontology and machine learning, offer a more automated approach [84].

In the general compliance automation field, some researchers focus on provision interpretation or requirement analysis [85,86,87,88,89,90,91], while others also cover compliance checking [92,93,94]. The technologies involved in these above works include AI, NLP, ML, etc. However, these methods encounter challenges in scalability and adaptability. NLP techniques, such as the rule-based approach, often necessitate extensive manual feature engineering [95] or manual verification for information [82], while deep learning models typically require large, annotated datasets for optimal performance [96,97,98]. Large language models (LLMs) such as GPT-4 [99], PaLM [100], Galactica [101], and Llama [102] offer promising solutions to these challenges. These models excel in few-shot learning and can understand complex language structures with minimal labeled data [103,104]. They have the potential to adapt to evolving regulations and extract structured information from regulatory texts, making them useful tools for automated compliance checking [79,105,106,107,108].

Ref. [109] delved into the potential of large language models (LLMs) to enhance the explainability of AI systems. Ref. [110] explored the potential of using large language models (LLMs) like ChatGPT in the field of diagnostic medicine, particularly in digital pathology. Despite some challenges, LLMs offer significant potential to revolutionize diagnostic medicine by improving accuracy, efficiency, and accessibility [110]. In the architecture, engineering, construction, and operations (AECO) industry, the current methods for compliance evaluation are time-consuming and prone to errors [111]. Ref. [112] developed an LLM-based automatic compliance check (ACC) artifact to evaluate its performance in classifying regulations, identifying rule dependencies, and extracting information from BIM (building information modeling) models. The results demonstrate the potential of LLMs to significantly improve efficiency, transparency, and trust in building permitting.

Ref. [113] provided a comprehensive overview of the potential applications of ChatGPT and other large language models (LLMs) in the fields of accounting and finance. The review highlighted the potential benefits of LLMs in various tasks, including financial analysis, auditing, tax compliance, financial reporting, and educational purposes. Ref. [114] also discussed LLM’s transformative potential in financial analysis, risk management, fraud detection, customer service, and regulatory compliance. Ref. [115] confirmed these opportunities but also outlined the challenges, such as data quality, model bias, and ethical considerations. Ref. [105] shared the view that large language models (LLMs) have the potential to revolutionize the financial auditing industry by automating the process of regulatory compliance verification. However, they face challenges, particularly with non-English languages. The study compared the performance of different LLMs and suggested that LLMs can be a valuable tool for financial auditors, even without relying on expensive proprietary software. However, the paper focused on a pre-defined compliance check. The data used seem pre-determined, with all the levels and parameters already established. While this approach is viable, a conventional deep-learning method could achieve higher accuracy. Thus, the potential advantages of LLMs were not fully realized in this specific application. This approach may not be applicable in a dynamic environment like agriculture. The reliance on custom datasets would not translate well to the complexities and real-time needs of agricultural compliance.

In the food industry, ref. [116] demonstrated the significant potential of LLMs to improve efficiency and accuracy in legal compliance and regulatory analysis by using BERT and GPT models to classify legal provisions and automate compliance checks. The study also highlighted challenges like handling complex legal language and ensuring the reliability of LLM-generated outputs [116]. Other researchers have focused on its potential in agriculture [117,118,119,120]. Ref. [118] conducted a comprehensive review of the potential of multi-modal large language models (MM-LLMs) to revolutionize agricultural practices, including applications in crop management, soil health, livestock management, and precision agriculture. Ref. [121] highlighted the potential of tools like ChatGPT to address various challenges faced by farmers by providing accurate and timely information to enhance farm management and simplify regulatory processes in agriculture. LLMs are considered to be valuable tools to assist in understanding, interpreting, and applying agricultural safety and health regulations [122]. They can provide summaries, explanations, and practical guidelines. However, it is crucial to remember that LLMs are not a substitute for a thorough understanding of the regulations and industry’s best practices [122]. Any information or advice provided by an LLM must be reviewed and verified, especially when it is specific and detailed [122]. ChatGPT can analyze large datasets to generate valuable insights for digital and precision agriculture as well as nanotechnology applications [121]. It can also simplify complex regulatory documents, making them more accessible to farmers and stakeholders [121]. These studies provide evidence that LLM has the potential to revolutionize agriculture.

2.2.4. Compliance Cost

Every industry, such as the agricultural sector, is subject to a wide range of regulations, including those related to the environment, water resources, biotechnology, transport, and work health and safety. The sheer volume and diversity of these regulations impose a significant time burden on producers, e.g., farmers or manufacturers, who must diligently track and comply with them. This burden is further compounded by the need to report to multiple agencies at various jurisdictional levels and the frequent changes to these regulations.

The financial impact of regulatory compliance on agribusinesses is substantial. Costs can be categorized into one-off and ongoing expenses [76]. One-off costs are incurred when new regulations are introduced, while ongoing costs vary with business operations. These costs encompass direct payments to the government, expenses associated with meeting regulatory requirements, time and resources spent on compliance activities, and losses resulting from delays in applications and approvals [123]. It was found that environmental regulatory requirements such as acquiring licenses and permits were the costliest area, requiring significant external assistance and posing a significant burden on these industries [123].

2.2.5. Law Is Code and Code Is Law

“Law is Code” is a revolutionary approach that involves translating legislation, regulations, and policies into machine-readable code [124]. By encoding rules, ambiguity is reduced, interpretation becomes more manageable, and compliance becomes more straightforward. Such transformation could facilitate the development of digital services that can automatically apply and enforce rules, streamlining processes and improving efficiency. Furthermore, it enhances transparency by making rules accessible and understandable to a broader audience. “Code is Law” [125], a related concept suggesting that rules can be directly encoded into software, making code itself a form of law. This approach can further enhance the efficiency and precision of rule enforcement. However, it also raises important ethical and legal questions regarding accountability, bias, and the potential for unintended consequences.

When combined with the power of large language models (LLMs), “Law is Code” becomes even more powerful. LLMs can effectively generate, refine, and understand rules, reducing the technical burden and making the process accessible to a wider range of users. This synergy between “Law is Code” and LLMs empowers organizations to create and manage complex rule sets with greater ease and accuracy. However, the “Code is Law” principle is not easily applicable with generative AI systems due to their inherent opacity. The black-box nature of current LLM systems requires new regulatory approaches, as traditional methods based on explicit code control are no longer sufficient [126].

2.2.6. Key Issues with Compliance Checks

Expert knowledge is crucial in several key areas of traditional compliance procedures. First, understanding and interpreting complex regulations, standards, and guidelines is essential. Experts must identify critical compliance points and define specific entities (objects) and their relevant characteristics (attributes) to be checked. Second, compliance checking involves validating data against regulatory requirements, detecting anomalies, and interpreting data within the broader industry and regulatory context. Finally, handling arbitrary events requires expert decision making, clear explanations, and adaptability to changing circumstances.

However, manual compliance checks that rely on experts in primary industries can be time-consuming, costly, and prone to human error. As regulatory landscapes become increasingly complex, the need for efficient and accurate compliance solutions becomes even more critical. Additionally, manual checks can be inconsistent, as different inspectors may interpret regulations differently. Scaling manual checks to accommodate growth and increased regulatory requirements can be difficult. Furthermore, periodic inspections may not be sufficient to identify and address real-time issues.

To overcome these limitations, digital solutions offer a promising avenue for improving compliance efficiency and accuracy. For instance, IoT sensors and data analytics enable real-time monitoring of environmental conditions, livestock health, and crop growth [62,63,127,128,129]. Blockchain technology ensures product traceability and verifies ethical and sustainable practices [130,131,132]. AI and machine learning algorithms can identify patterns, predict risks, and automate routine tasks [133,134,135]. Remote sensing and satellite imagery facilitate land use monitoring and environmental compliance [136,137,138]. Drone technology offers efficient inspection, monitoring, and mapping capabilities [139,140].

These digital tools can automate routine tasks, analyze vast amounts of data, and identify potential compliance risks. Businesses and regulators can streamline compliance processes, reduce costs, and improve accuracy. For example, a cherry farm can use IoT sensors and blockchain technology to automate the tracking of pesticide use and environmental conditions, ensuring compliance with minimal manual intervention. The data collection at the producers’ end about the products and their production process is streamlined and very well developed. This can be used for continuous monitoring by AI tools in real time. However, human users are unable to ask analytical or pre-emptive questions about these processes unless they are experts in interpreting and analyzing the collected data. Often, this is due to the complex nature of processes and a single human’s inability to understand such complexities.

The current research in automated legal text processing faces significant limitations. Firstly, it heavily relies on sentence-level analysis, which can be insufficient for capturing complex legal concepts [107]. Secondly, automation strategies often lack clear justifications for their decisions and either adopt a coarse-grained approach or require substantial manual effort to build [107]. These works also focus on a one-time compliance check. However, for continuous compliance checks, a lack of clear justification in compliance automation strategies within primary industries may lead to several significant problems. It can undermine trust in the automation system, especially when decisions have far-reaching consequences for producers and consumers. It can potentially hinder maintenance efforts. In contrast to one-time compliance checks [112], we consider a continuous compliance check in this paper. It goes on for a time period during which the compliance rules may be altered due to external factors, and the events are expected in a specific order. In the single compliance check, all data products are gathered before being checked against a fixed set of rules. In the continuous process, the data may sometimes be old or experienced, and there are more chances of data corruption. This is the reason we designed our LLM-based RegTech compliance check around a time series dataset, which reflects the changing data and the relevant conditions over time.

A comprehensive and accurate compliance report generated by LLM can enhance transparency and accountability. RegTech LLM can develop mechanisms to explain the system’s outputs to its users, particularly in cases of unexpected or controversial decisions. RegTech LLM can enable human users to intervene immediately when necessary, especially for complex or high-stakes decisions, and provide a safety net.

3. Digitalization of Compliance Checks with RegTech LLM

In this section, we present the regulatory system requirements, the proposed regulatory systems supported by LLM—RegTech LLM—their key features, and expected functionalities.

3.1. Design Requirements of the Digital Regulatory Systems

The design of a regulatory system should consider the expected inputs from various sources, capabilities, and industry expectations.

3.1.1. Changes in Regulations

Most industries, like the agri-food industry, are subject to frequent regulatory changes, such as new food safety standards, labeling requirements, or traceability protocols. These changes can be complex and time-consuming to implement, especially for large-scale operations. To address this challenge, AI agents can be employed to monitor regulatory changes, interpret their impact, and automatically update systems and processes, such as in the medical device area [141]. As illustrated in Figure 2, AI agents continuously monitor regulatory databases and news feeds, utilizing NLP to extract key information from regulatory documents. They analyze these changes, assess their impact on existing traceability procedures, and update systems accordingly. Additionally, AI agents can monitor product movements in real time, flagging deviations from regulatory requirements and generating alerts for potential issues.

3.1.2. Key Regular Capabilities of RegTech LLM

With regard to the regular use of RegTech LLM, there are four key capabilities that an LLM should possess to assist in compliance checks effectively:

• Flagging: This is a process for setting alerts on variables to check if they cross a threshold. Flagging can be set with conventional software, but with LLM, it can be customized with multiple conditions based on human input;
• Analysis: The LLM must possess numerical analysis capabilities to answer user questions regarding regular and unexpected situations. This is further outlined in the next section.
• More advanced features include the following:
• Reasoning: This is an advanced feature from the farmer’s perspective, allowing the creation of conclusions based on the analysis. Reasoning requires the LLM to be trained in past decisions made by human users and the rules interpreted from the guidelines;
• Suggestions: Once the reasoning is completed, LLM can potentially provide suggestions on the next steps to follow.

In this paper, we do not delve into all the reasoning and suggestions and limit ourselves to flagging and analysis only.

3.1.3. Industry Expectations

In many compliance scenarios, businesses need to make quick decisions. In today’s fast-paced business environment, the ability to quickly adapt to changing regulations and respond to compliance challenges can be a significant competitive advantage. Delays in compliance can lead to costly fines, reputational damage, and operational disruptions. Responsive RegTech solutions can help businesses minimize these risks.

In terms of human factors, LLM can resolve the two biggest problems with respect to compliance. A key issue with compliance is having the knowledge of the requirements at the right time. While it is possible for expert workers to remember the most common rules and record the data accordingly in a timely manner, it is still difficult to remember everything. This problem becomes more complex if the compliance is checked by a non-expert or if the rules are relaxed/changed for a particular process cycle. The other issue is on-demand analysis. Often, the producers may want to try new things by analyzing their own past performance and potentially stimulating new deals to see if they would still meet the competition. However, if they do not have an expert on-site, such analysis would require additional costs. LLM can help with this by providing a comprehensive guide to all the rules, both base and temporary. The two main features are as follows:

• Responsiveness: RegTech LLM should be ready to answer in a real-time chat, such as in ChatGPT. It can respond in seconds with a human-friendly output, thus allowing users to receive fast advice. The producers do not need to wait for expensive expert advice;
• On-demand analysis: Existing compliance software can provide fixed compliance check features. LLM can give flexibility in the questions users can ask. This means an analysis can be fine-tuned to specific user situations and requirements. LLM has to be capable of the following:

  Step 1.

  Understanding the question.

  Step 2.

  Determining the procedure to obtain the results.

  Step 3.

  Performing the analysis according to step 2.

  Step 4.

  Producing a human-friendly output.

In this context, the evaluation of the RegTech LLM is different from that of conventional LLM applications.

(a)

The inputs, i.e., the prompts, need to be succinct and short, as human users are not expected to explain too much. This can be addressed by considering only 0-shot prompts;

(b)

The outputs must satisfy the question’s requirements. A majority of RegTech questions and their corresponding data will be discrete. This means the output will contain a countable number of distinct elements ($\epsilon$), and thus, if any of these elements (m) are missed by LLM or if new extra elements are added to the output (e), it will result in an incorrect response from the LLM. As such, we define the output accuracy ($\lambda$) of a RegTech LLM response as follows:

$$\lambda ={\displaystyle \frac{\epsilon -m}{\epsilon }}\times {\displaystyle \frac{\epsilon }{\epsilon +e}}={\displaystyle \frac{\epsilon -m}{\epsilon +e}}\dots$$

(1)

where $\epsilon$ can be lists of strings, sets of rows from a time series dataset, sets of rules, sets of web service functions for selection, or sets of format specifications. Variations in the outputs do not matter for RegTech LLM. If the same information is represented in multiple formats, it does not create any problems for the target users.

Ultimately, with respect to RegTech, from an industry perspective, RegTech LLM is expected to achieve the following targets:

• Product safety: This is related to the safety of using or consuming the industrial process’s product. For the agriculture sector, this means adhering to stringent food safety regulations, including Hazard Analysis Critical Control Point (HACCP) principles, to prevent foodborne illnesses and protect public health;
• Environmental protection: This is about quantifying and reducing the environmental impact of the industrial process. In agriculture, this is complying with environmental regulations to minimize the industry’s impact on natural resources, e.g., responsible pesticide use, water conservation, and soil conservation practices;
• Human resource management: This concerns the human resources in the industrial process and how their time and efforts are managed. It is important to relate human performance to the product, e.g., timeliness issues; if someone has to check and enter data manually, that should be recorded;
• Traceability: An industrial product is expected to change ownership multiple times. Thus, it is important to implement a robust traceability system to identify the source of any potential issues, such as product recalls or quality concerns.

3.2. Key Components of LLM Compliance Checks—Automation Functionalities Needed by LLM

For streamlining the compliance check process, several types of functionalities must be provided by AI and/or LLM, with or without the use of external dedicated services:

• Numeric checks: These checks are arithmetic and logical checks to determine if the recorded numeric values are within acceptable thresholds. In the context of agriculture, environmental and production data are often recorded as time series. Typically, if IIoT is implemented, the time series is generated by devices and thus records time. If IIoT is not implemented, and the data are recorded manually, the time series may be more incomplete or less orderly. However, the aim would be to continuously monitor the recorded numeric value to check if it is within the acceptable range;
• Milestone checks: This is a special case of numeric checks that check against a specific date. These checks are meant to confirm that the expert event has happened within the documented timeline. The events are not expected to be continuous but to be within time limits. Milestone checks are when we expect the event to happen within time, and compliance fails if the event does not occur by that time;
• List checks: A key aspect of compliance checks is labeling. This is obviously in the form of strings, and list checks would involve logically comparing the data entered with the expected values from the list. It is essential for users to record their data using pre-approved terms to ensure verifiability;
• Flag/binary/fault checks: Sometimes, unexpected events occur and must be flagged immediately. Unforeseen events do not inherently mean a compliance failure, as they can be remedied within a time limit. These checks are typically binary, i.e., whether there is an error or not, for example, whether there was a pest on the farm.

Apart from the above four, there is a more complex form of compliance checks when the compliance data involve more than numeric lists and date/time data.

5.

Document check: This is checking the format, relevance, and correctness of data entered in each section and pocket of data. It is invariably composed of list checks and could also have flag checks;

6.

Process checks: These higher-level checks ensure the correctness of the whole industrial process by analyzing smaller chunks of data. This would involve checking for events that should occur in an expected sequence or for missing data;

7.

Image and video checks: LLMs have multimodal capabilities to interpret and describe visual data by analyzing a given image (and possibly videos). These models, when integrated with vision modules, can process and analyze images to identify objects, recognize patterns, detect anomalies, and provide descriptive captions or insights. For RegTech LLM, computer vision algorithms are already used for visual inspection of several production-related activities, e.g., seed quality checks.

Figure 3 shows a hierarchy of different checks. A process check would comprise document and numeric checks, which have to be in the correct order with respect to time. Document checks can be compared to list checks, milestone checks, and binary checks based on writing documents (digital or scanned from paper) as well as document completeness. Numeric checks are composed of milestone checks and binary checks as well as basic arithmetic and logical checks.

In agriculture, effective compliance is crucial to ensure adherence to regulations, standards, and best practices. A robust compliance framework should incorporate a multi-faceted approach, encompassing process checks, document and numeric checks, and milestone and list checks. Process checks verify the sequence and timing of agricultural processes, such as planting, harvesting, and pesticide application, and they identify any gaps or missing data. Document and numeric checks assess the accuracy, completeness, and format of documents, such as farm records, certification documents, and regulatory reports. They also monitor continuous data, such as temperature, humidity, and soil moisture, to identify anomalies or trends that may signal non-compliance. Milestone and list checks track key events in the agricultural process and verify the accuracy and consistency of lists, such as approved pesticides, fertilizers, or seed varieties.

3.3. Architecture of the LLM–Web Services for RegTech

Figure 4 shows the RegTech LLM–web service architecture. There are multiple sources of data from producers such as farmers and shipping companies. Data can be generated by sensors as well. The source of these data is the assessee, who regularly checks for compliance. Human users would ask LLM questions under the following conditions:

(a)

Regular checks where the user asks the same question but expects updated results;

(b)

Some adverse situations have happened, and they require advice. In this situation, the question would be very specific and more difficult for the LLM to answer. As such, the prompt questions need to be detailed to provide accurate answers and specify the exact situation in which the user is asking the question;

(c)

Ask questions with relatively relaxed rules. Human users may ask about a hypothetical situation to see if making any changes in the industrial process could still satisfy compliance, even if less comfortably;

(d)

Comparative questions where human users would want to compare their own performance with historical data or, in the case of the regulator (or assessor), compare between the producers concurrently.

The assessor provides the following guidelines:

(i)

Base guidelines, which currently are in the form of text and diagram documents. These are created by governments or regulatory bodies;

(ii)

Additional temporary guidelines are in response to changes in the situation during a process cycle. These guidelines would not be part of the base guideline and are temporarily implemented in the specific cycle;

(iii)

There may be additional operational guidelines from the manufacturers of components used in subsequent productions by other manufacturers. In these cases, compliance also includes adhering to the manufacturer’s guidelines along with the regulatory guidelines.

In this paper, we only consider (i), as agriculture typically does not involve (ii) or (iii).

The LLM acts as an interface to human users, acting upon all the data sources, providing compliance features, and performing any formatting if needed. We define this as a virtual AI agent. It takes all the raw data from the assessee or data generators; i.e., the producer takes all the real-time guidelines/rules from assessors to perform the compliance checks. The output of the compliance checks can be put into any format corresponding to the assessor’s requirements from different jurisdictions. Ultimately, the output of the compliance is made available to the end consumers who only view the compliance checks.

We tested LLM’s functionalities and, unfortunately, found them inadequate for RegTech on its own. It is either too slow for human interaction or cannot handle large quantities of data. So, it needs to be backed by web services that perform specific functionalities. Such web services can be called by the LLM as part of the tool-calling features of LLM and the output used to create the final output. The web services would have limited flexibility in operation and cannot answer any specific question. The LLM, on the other hand, can determine precisely what the user needs and use the web services to calculate the output.

3.4. Key Characteristics of a Virtual AI Regulatory Agent

The virtual AI agent uses LLM to act as an assistant to both parties, facilitating communication and reducing uncertainties. Unlike traditional compliance processes, the LLM promotes a more collaborative approach. Queries in the form of prompts are submitted to the LLM, which analyzes the information and forwards it to the relevant farmer. The farmer provides additional context, and the LLM can iteratively analyze the data to reach a mutual understanding. Such queries will almost always be 0-shot; i.e., the user does not have the ability to give examples for more complex queries.

A virtual agent can monitor data continuously, albeit at intervals, for each user during each iteration of an industrial process. This continuous monitoring does not inherently need LLM. However, the reports or results generated by constant monitoring with deep learning or machines become the source of information for the LLM later when the users actually ask a query. Thus, it is important to design and record the outcomes correctly in a timely manner.

Virtual AI regulatory agents are capable of continuous operation, accumulating knowledge and experience over time. They can gain experience indefinitely. Additionally, they can share their knowledge with new regulatory human officers, ensuring knowledge preservation and training. However, it is essential to recognize that AI agents are not infallible. Human oversight will remain indispensable to ensure accuracy, fairness, and ethical decision making. Virtual AI regulatory agents with LLM can enable large-scale summarization of reported information. They can work 24/7 and consume data continuously from multiple sources, widen monitoring, and present summarized information. These conjectures can then be verified by humans quickly.

The virtual agent’s LLM interpretations must be automated and upgradable. The LLM needs to extract rules from given textual regulatory guideline sources, such as manufacturing or production guidelines, standard operating procedures, and laws. It then preserves these rules to check the data against. The models can also be retrained to remove or modify a rule. Rule identification and upgrade are developer- or regulator-centric activities. This feature of LLM is effectively a form of “Law is Code”. However, as the codes are not explicitly released for regulations, it is up to LLM to create the rules. The key challenges in this are the following:

(a)

The number of variables being tracked (α)

(b)

The number of logical conditions being tracked (ε).

3.5. Traditional Software vs. LLM for Compliance Check

LLM-based compliance checks offer significant advantages over traditional software in various aspects. Traditional software totally relies on rigid, predefined rules and static datasets, struggling to adapt to dynamic regulatory landscapes. In contrast, LLM-based software can learn from new data and evolve to meet changing compliance requirements.

Traditional software requires significant manual effort and code modifications for updates, whereas LLM-based software can be easily expanded through training and fine-tuning processes such as retrieval-augmented generation (RAG). While traditional software may not provide explanations for unexpected outcomes, LLM-based software can offer more nuanced and customized explanations derived from raw data. Traditional software is often limited by its reliance on structured data sources, while LLM-based software can process a broader range of data, including unstructured text and real-time streams. Additionally, LLM-based software leverages advanced AI techniques like NLP and machine learning to perform complex tasks and provide human-friendly interactions. LLMs can identify complex compliance issues and continuously adapt to new regulations. This allows for more accurate, efficient, and adaptable compliance checks, ultimately leading to better decision making and risk mitigation. LLM, deep learning, and traditional software are compared in Figure 5, and the differences are presented in

Table 1. Traditional Software vs. LLM for Compliance Check.

	Traditional Software	LLM
Checking Criteria	Fixed criteria for checking	Variable criteria for checking
Expandability	Limited expansion is possible; only dedicated developers can add features	Dedicated developers are needed to update the LLM model only
Explainability	It is not possible to explain any unexpected situation	Possible to re-interpret customized explanations from raw data
Data Source	Limited sources of data can be consumed in a given period of time	Unlimited data can be processed, although checking types is limited
AI	Deep learning, but no conversational paradigms	Naturally conversational and human-friendly
Upgrades	Dedicated developers are needed; it is more time-consuming	No dedicated developers are required, and it is less time-consuming

.

LLMs mainly provide a user-friendly interface. This removes the requirement for users to be experts, integrate knowledge of operational parameters, and have a deeper understanding of what could go wrong. It also eliminates the need for humans to remember all rules in real-time assessments. Current LLM-based software may require more frequent updates and maintenance and have limitations in processing large amounts of real-time data. However, future LLMs may be explicitly created for regulatory systems.

4. Results

This section presents the result of using LLM for RegTech for each role of the virtual agent mentioned above. In most cases, there is a predefined value for ε, and measurement of accuracy was based on the fractions of this, i.e., data or rules that are extracted or checked. It is to be noted that the aim of this work was to determine the suitability of LLMs in general for RegTech, and as such, we did not directly compare and contrast individual LLMs with each other. Instead, in each test, we focused on data about the best-performing LLMs for RegTech, showing that at least one LLM can perform reasonably.

4.1. Experimental Setup

The experiments conducted were comprised of raw data used as data sources and LLMs used for both extracting information/rules and acting as agents for users.

4.1.1. Data Sources

For regulations, we used the Australian Cherry Guidelines guidebook (2011) [142], which is used by farmers to grow cherries. This lays out the industry standards’ expectations as well as the minimum regulatory requirements. It contains 209 pages of text in English, tables, and some figures.

For time series data, we used a dataset of 297 rows, each containing timestamped numeric values collected at an on-field weather station, which recorded basic information such as time, temperature, and absence/presence. The system was designed to record a time series value every 5 min. However, if pests were detected, they were recorded earlier as well. Hence, a linear but variable time gap time series was used. The data were not preprocessed and were fed as raw values as saved in CSV format by the IoT system.

For format checking in Section 4.5, we used the official compliance recording forms used by Honeybee Industries and government regulators [143]. Each form contains a table with columns recording the information. There is also a two-page guideline on how to fill in the forms, i.e., expected values.

4.1.2. LLM Platforms and Models

We used a variety of platforms and models for this work. Ollama [102] is an AI tool that allows users to run and interact with LLMs locally on their own devices, ensuring privacy and security. It offers a simple command-line interface to download, customize, and use models efficiently. With a focus on accessibility and performance, Ollama was designed for developers, researchers, and AI enthusiasts seeking local AI solutions. Ollama allows one to use one of the many available LLM models. It was also used for the tool-calling feature. For hardware, we used a 2.9 GHz CPU with an Intel i5-9400 processor, 32 GB RAM, and a 16 GB GPU card.

We also used a few commercial-grade, publicly available cloud-based LLMs. Copilot is an AI coding assistant that helps developers write and debug code by providing real-time suggestions within their IDEs [144]. ChatGPT (GPT-4o), by OpenAI, is a conversational AI tool that engages users in natural dialogue, assisting with questions, problem-solving, and content creation [145]. We also used Gemini 2.0 and Claude3.

4.2. LLM Agent Monitoring Tests

The LLM agent’s monitoring comprises continuous checks of data inputs in various forms. Continuous checks are meant to provide timely feedback to users, either autonomously or when users ask questions. Such checks are like spot checks, which use limited data that have usually been available since the last check.

4.2.1. List Extraction and Checking

We start with the most straightforward analysis for the compliance check, which is the list extraction and list checking. For list extraction, we used the cherry guidebook and asked the LLMs to extract a variety of certain items like cherries from the guidebook. This is an important enumeration in the compliance check where the item names entered in any form must be from a list of pre-approved items, i.e., the cherry names. For this experiment, $\epsilon =29$ when the full document was used, and $\epsilon =23$ when only five pages were used; the difference is due to most names/strings appearing as a list on five pages, but some other names are embedded randomly in the document. To test the speed and accuracy, we changed the number of words we submitted to the LLM.

The list extraction only concerns picking the entities from the texts. For RegTech, the order of this list does not matter much, but it indicates a level of expected similarity of the extraction for every user prompt. Another indication of consistency is the item-wise lighting. In the original text, the list is embedded in the text in a specific order (not necessarily in alphabetical order), and the LLM application should ideally extract them in the same order. Assuming p(item, list) is the index of an item in a list, the actual item order in the document (D), and the list obtained after LLM (L), we used two metrics:

• List order variation rate: This is the likelihood of the list not being in the expected order. It is given by the number of times an LLM can return the perfect order divided by the number of times we ask the prompt;
• Average item displacement: This is the likelihood of an item from the expected list being out of place, given by the following:

$${\sum }_{k=1}^{\epsilon }\left(\left({\sum }_{j=1}^n\left\{\begin{array}{c}1p(k,D)\ne p(k,L_j)\\ 0p\left(k,D\right)=p(k,L_j)\end{array}\right.\right)/n\right)/\epsilon$$

where n = 10 is the number of times we ran the LLM chats independently. Even without a perfect extraction, the LLM performance in this situation was shown to be good enough for RegTech. Also, this order checking is only relevant to list extraction. Other extraction, such as rules, is not expected in any particular order. A lower value for both of these indicates that the LLM is reading the document in a consistent manner. A higher value does not necessarily mean incorrect outputs, as the output list can still contain the desired items in a different order but could be challenging to use in subsequent applications. The following are the results of the list extraction (

Table 2. List Extraction and Accuracy (with Prompt 1 from Appendix A).

LLM	Input Size	Extraction Accuracy			Extraction Time	List Order Variation	Avg. List Item Displacements
		m	e	λ
Co-pilot	Full	0.0	0.0	1.00	10.07 s	0.00%	0.00%
ChatGPT	Full	3.2	0.1	0.86	10.17 s	90.00%	72.17%
Claude	Full	1.1	0.0	0.95	10.48 s	90.00%	43.47%
Co-pilot	5 pages	0.1	0.0	0.99	2.7 s	100.00%	34.78%
ChatGPT	5 pages	1.0	0.1	0.95	5.3 s	100.00%	82.17%
Claude	5 pages	0.2	0.0	0.99	9.3 s	50.00%	29.56%

):

The results indicate that the LLMs are reasonably good for extraction with a 0-shot prompt from a general body of text. Also, all the results were obtained within a reasonable 11 s. The list variations and displacements are pretty high regardless of whether a full document was given or only the five pages. For further LLM applications, once we can correctly extract the list of strings, e.g., cherry names, the LLM can quickly check if any given cherry is on the list.

4.2.2. Numeric/Milestone/Binary Analysis for RegTech

In order to test the numeric, milestone, and binary (NMB) check, we used the time series. We created 10 prompts, as shown in Appendix A. The prompts were designed with varying levels of difficulty in obtaining the target answers:

• Level 1 prompts asked for an analysis whose answers were in the dataset itself with a 0-shot prompt, e.g., finding maximum and minimum;
• Level 2 and successive prompt levels asked for analysis that required performing mathematical operations on the raw data or data from the previous level to generate another set of derived data. These data can be used for the next level.

We prompted LLMs with a 0-shot prompt from Appendix A. Each time, the LLM started creating a Python code to generate the answer. Once the Python code was created, LLM used it to check the IoT data, which were in the form of a time series in CSV format, as provided in the prompt (297 rows of data). For each prompt for each LLM model, we ran the test in a new chat every time to isolate the conversations. We repeated this five times, each time with a new chat for each prompt for each model. Each time, the CSV data were appended to the prompts 2–11, itemized in Appendix A, when given to the LLM. Every time, the LLMs generated the code again and executed it or tried to execute it with the given data. The prompts were from three different levels:

• Level 1 (L1): Finding minimum, maximum, or row(s) greater or lower than a value, e.g., the time when the height of a tree goes above 50. The answers to these questions are not calculated but determined. The answer is already in the data given to the LLM;
• Level 2 (L2): Finding the average time gap or standard deviation and checking for secondary conditions. Answering these questions requires the LLM to perform one set of operations on the raw data provided, often estimating a new set of values, e.g., a new column based on the original values;
• Level 3 (L3): Counting the anomalies based on average time and standard deviation. This requires a level 2 operation to be performed before another round of analysis is performed.

Table 3. The accuracy (λ) of LLMs in answering NMB compliance checks on time series.

Prompt →	2	3	4	5	6	7	8	9	10	11
Level →	1	1	1	2	2	2	2	2	3	3
Copilot	1.0	0.99	1.00	0.82	0.00	0.00	0.64	0.69	0.56	0.0
ChatGPT	0.8	0.18	0.67	0.24	0.00	0.00	0.22	0.12	0.00	0.0
Gemini	0.25	0.36	0.83	0.0	0.00	0.00	0.27	0.48	0.30	0.40

shows the accuracy results using Equation (1), and

Table 4. The execution time (s) of LLMs in answering NMB compliance checks on time series.

Prompt →	2	3	4	5	6	7	8	9	10	11
Level →	1	1	1	2	2	2	2	2	3	3
Copilot	2.8	43.5	3.9	50.2	6.4	40.6	33.2	22.7	7.7	38.2
ChatGPT	25.0	23.0	9.1	36.3	55.9	53.7	25.2	12.0	23.7	45.1
Gemini	12.4	16.4	2.9	62.3	101.9	71.0	29.4	4.3	10.9	6.6

shows the average execution time. The value of ε is fixed and known for each prompt based on the fixed time series dataset providing the ground truth.

Note that for prompts 2–5, the LLM was able to produce a perfect or high-quality result at least once. However, as the responses must be consistent for RegTech, we focused only on the average accuracy. If an LLM can produce a perfect result, it is possible to update it in the future to reduce randomness and be consistent with a focus on RegTech prompts.

This means that if we ask the LLM about RegTech compliance without any RAG, it knows what to do but is limited in operational accuracy in generating a correct output. Tool calling can improve this, as shown later. The response time is also suitable for RegTech applications. It is not instantaneous and could be further impacted when there are multiple users of the same resources. It is not large enough for users to discontinue the conversation with LLM at the level 1 prompts, but it is quite large for complex prompts. This is because a code was generated most of the time successfully to answer the prompts, but the data were not fed correctly to the code, leading to poor efficiency for L2 and L3.

The LLM can handle time series where the time gap recorded is not consistent. LLMs can understand and interpret the rules but have difficulty checking. The LLM functions as an additional layer, augmenting the capabilities of existing deep learning models. It acts as an assistant, providing support and enhancing the overall performance.

4.3. LLM Agent Rule Identification and Upgradeability

From a RegTech developer’s perspective, if the LLM can convert the written text to code and apply it to a dataset, it can apply a set of rules and conclude if the regulations were met. Also, the LLM agent must be able to alter the execution of the rules according to any expected situation, i.e., understand anomalies and apply the rules accordingly. This requires the rules to be updated for a particular compliance check for a user for a specific process.

We prompted LLMs to create a set of rules for each of these tables with a 0-shot prompt. Once the rules were created, we asked via another prompt to create a Python code to check the rules, assuming the IoT data were in the form of a time series in CSV format.

Automatic rule identification is another prospective application of LLM in regulatory technology. Rule identification and extraction are complex processes depending on the source of the rules—such as whether it is from a prepared table or a text paragraph. Here, we tested the LLM’s capacity to interpret rules from a given table. In order to do this, we considered tables that contained the desired rules in the form of ranges. LLM was asked to provide a pseudo code for the rule and then convert it to a Python code. From the given guideline document, we identified four tables and asked to convert the tables to code. The tables defined thresholds for environmental parameters and/or their conversion rules for regulatory recordkeeping and further analysis. Tables had either a single variable or two variables (α = 1, 2). There was also a fixed set of rules, i.e., a set of if/else conditional statements (ε = 7, 23, 18, 16). For example, in Table 1, in [142], there is 1 variable—“temperature”, which can be between six different numeric ranges, resulting in a corresponding “chilling unit” value.

We provided Ollama models llama3.2 and llama3.1 and Copilot with the relevant pages from the guidebook, which is in PDF format. We recorded the time required and accuracy for converting to a code with varying levels of complexity 10 times for each table in isolated chats. We only considered a single 0-shot prompt for this rather than a set of prompts, as the users may not have the expertise to use multiple prompts to attain a good code. Also, LLMs are known to forget or hallucinate; hence, if the LLM is not able to obtain the perfect output in a single prompt, it may not be helpful for RegTech.

Table 5. The accuracy of rule identification by LLM with a single 0-shot prompt (with prompts 12–15 in Appendix A).

Ollama Model	α	ε	Type	Rule Identification (λ)	Rule-Code Conversion (λ)	Time (s)
llama3.2:3b	1	7	Numeric	1.00	1.00	2.1
llama3.1:8b	1	7	Numeric	1.00	1.00	3.1
Copilot	1	7	Numeric	1.00	1.00	2.5
llama3.2:3b	1	23	String	0.99	0.99	2.3
llama3.1:8b	1	23	String	0.98	0.98	2.1
Copilot	1	23	String	1.00	1.00	2.3
llama3.2:3b	2	18	Numeric	0.15	0.15	5.1
llama3.1:8b	2	18	Numeric	0.14	0.14	5.4
Copilot	2	18	Numeric	0.9	0.9	5.2
llama3.2:3b	2	16	Numeric	0.95	0.95	3.2
llama3.1:8b	2	16	Numeric	0.95	0.95	3.1
Copilot	2	16	Numeric	0.97	0.97	5.1

shows the results of the rule extraction and rule-to-code conversions. There is a difference in the accuracy of identifying the total number of rules ε correctly between 1- and 2-variable rule sets. For single variables, the numeric rule checks were very accurate; for string checks, where it had to map a set of strings with another set of strings, it was slightly inaccurate in maintaining consistency with strings that had spaces. With the 2-variable numeric check, it was pretty inaccurate with Ollama but good with CoPilot, with the second table returning a high accuracy due to one of the variables having only two possible values, which were put as an if/else conditional check every time.

In all cases, the time consumed was less than 10 s, which is suitable for RegTech applications. If the tables are fed to the LLM in stages, a perfect set of rules and codes may be possible with α > 1. However, this will increase the time consumed and also require considerable software development expertise.

A critical aspect of rule identification and execution is the ability to (a) upgrade the rules or (b) ignore some rules on demand. Eventually, after additional prompts, it will be possible to add more of the conditions to the existing rules set and code. This would follow typical practices of software coding with LLM [146,147]. However, this would require expertise from the users and is not suitable for all RegTech users.

4.4. Web Services Selection and Execution

As expected, the current LLM performance, as shown above, does not inherently meet the standards required for tasks above a certain complexity, i.e., L2 prompts and α > 1, in regulatory technologies. They can solve problems but are too inconsistent for complex tasks. However, automatic rule identification is partially successful in extracting executable code from a set of rules written in regulatory documents. These codes can be perfected and preserved as web services. The web services can be made available through cloud resources and used inside the LLM for data extraction. One of the critical aspects of the LLM for this purpose is to select the correct web service based on the user’s current prompts.

To test this, we used the same prompts from before but this time with tool calling enabled for LLM. We used Ollama models llama3.1 and llama3.2 for this. We defined four parameters to measure the effectiveness of web service selection:

• Function selection: This refers to accuracy in selecting the right function, method, or web service for the given prompt;
• Function calling: This is accuracy in correctly formatting the inputs to the selected function, method, or web service for the given prompt. The expected inputs are adequately defined as part of the web services;
• Function output correctness: This is the accuracy of the output of the selected function, method, or web service after correctly calling it. It is expected to be quite accurate, as the functions are properly hard-coded. The output of the function call is equivalent to the results described in Table 2 and Table 3 with direct analytical prompts. The web service function outputs are structured properly for human interpretation. However, this may not be easy enough, and we asked LLM to present a sentence as an explanation for the output of the web service functions;
• Function output presentation: The final criterion is the accuracy of properly presenting the information to human users in a readable sentence containing the exact outputs of the function. This is expected to be the weakest, as it depends heavily on LLM’s ability to understand the description of the functions/web services.

While LLM’s performance in selecting, preparing outputs, and executing the web service functions was very accurate, it was not good in explaining the results in this fourth aspect.

When we used raw prompts, the success rate was not very good, as seen in Table 2 and Table 3. The LLM could determine the procedure but made a substantial error in getting the inputs correct. Furthermore, the contained output of the LLM was extremely corrupted, giving unstable outputs. LLM has a default nature of writing code by itself to obtain the desired data, which is irrelevant to this RegTech application.

So, we modified the prompt to append the string “Use one of the given tools and …” at the beginning to ask the LLM to use the prompts explicitly. This increases the accuracy of the service/function selection, correctly deciding the inputs to it, executing the code, and correctly forming the human-friendly output. The experiment was run 50 times separately on two Ollama models.

The results are presented in Figure 6. For level 1 and 2 prompts, the function selection, calling, and output were near 100% for both Ollama models llama3.2:3b and llama3.1:8b. For the level 3 prompt, the accuracy in selecting the functions was not good every time. However, this was still a significant improvement from raw prompts (Table 3), as when the web service functions were correct, the output was perfect.

The function output representation was inadequate in all cases. For this, we only counted the function outputs that were correct in answering the prompt with a human-friendly sentence based on the function’s output, which is in JSON format. The time consumed for this was always less than 5 s, with an average of 3.2 s.

4.5. Formatting of Compliance Checks

The final component in the RegTech system is format checking. Often, the data are written in forms that are structured as tables and graphs. Such forms can be both digital and paper-based. However, in either case, the form can be available in PDF format, which could be fed directly to LLMs. The main parameters to determine the accuracy of LLM’s interpretation of data in the table are as follows:

o

Relevance of data entry for particular sections in the form. This is the LLM’s ability to correctly identify whether an entry in a particular section of a form is relevant to the section’s requirements;

o

The LLM can also estimate the correctness of the entries. Some of the entries can be written in paragraphs. As such, the LLM can identify whether the written text contains all required details as pre-determined by the form designer.

For the data entry experiment, we used the honeybee compliance forms [143], which are in PDF or docx format. We provided the LLM with a filled (or semi-filled) form in docx format and asked it to verify the entries. Each test was repeated five times in separate isolated chats. The results of the tests—average accuracy and average time consumed—are presented in

Table 6. Format Checking Accuracy with 0-shot prompts—Relevance.

LLM	Form 1		Form 2		Form 3		Form 4		Form 5
	λ	Time	λ	Time	λ	Time	λ	Time	λ	Time
Copilot	0.9	5.59	0.7	5.53	0.8	5.50	0.8	5.48	0.7	5.00
ChatGPT	0.9	9.55	0.8	8.41	0.7	6.30	0.8	8.03	0.8	15.9

. The values of ε are also known based on the forms used for this test. The accuracy of LLM format checking was determined with Equation (1). The time was recorded in seconds.

LLMs are quite successful in identifying relevant information in a given paragraph with respect to a given raw guidelines/section in the form. The are able to complete the checking mostly within 10 s. These results were obtained with 0-shot prompts only.

To experiment with the data entry, we used the GACC and honeybee forms [143], which are in PDF or docx format. These forms act as the base guidelines on what information is required from the users in a written paragraph. We first provided an empty form for the LLM. Then, we provided a paragraph containing the required information (in part) for the corresponding form and asked LLM to verify whether the paragraph contained all the relevant data with respect to that form. Each test was repeated five times in separate isolated chats, starting with feeding the empty form and then providing the paragraph. The results of the tests—average accuracy and average time consumed—are presented in

Table 7. Format Checking Accuracy with 0-shot prompts—Completeness.

LLM	Paragraph 1		Paragraph 2		Paragraph 3
	λ	Time	λ	Time	λ	Time
Copilot	0.84	7.50	0.87	7.12	0.77	5.22
ChatGPT	0.94	16.69	0.87	7.31	0.89	13.80

. The values of ε are also known based on the forms used for this test. The accuracy of LLM format checking was determined with Equation (1). The time was recorded in seconds.

LLMs are remarkably successful in identifying incomplete information in a given paragraph with respect to a given raw guideline. They are also able to do this mostly within 10 s.

5. Discussions and Future Works

5.1. RegTech LLM Performance

5.1.1. Industry Expectations

The experiment carried out in this paper shows that RegTech can be implemented and enhanced by using a combination of LLM and traditional software. LLM has the capabilities to identify rules, keep track of sensor and human data, and check for compliance with time series information. LLM can also perform quickly enough to continue a conversation. All the experiments were completed with 0-shot prompts, which is the likely nature of prompts from a novice user. The current LLM has the capability to understand the problem but lacks analytical ability in a large dataset. It was also unable to solve complex prompts with level 2 or higher analysis. But still, it performed with high accuracy for simple checking. This shows that the proposed RegTech virtual agent can successfully deploy LLM as an interface to the regulators and producers.

5.1.2. LLM–Web Services

To overcome the lack of analytical capabilities, back-end web services were used, and each time, LLM was successful in identifying the correct web service functions for a given prompt, creating the inputs, and obtaining the correct analysis. It was less successful in explaining the results. However, given that the results were obtained through a correct analysis, producing the results “as is” is sufficient for RegTech applications. Obviously, further work could improve the final statement produced by the LLM when presenting the rest of the web services.

5.2. Future Work

In this paper, we defined and demonstrated that LLM and web services are promising for RegTech applications. However, there is great potential for improving performance to realize actual RegTech applications with LLM.

5.2.1. k-Shot Prompts and Questions

Throughout this paper, we used only a 0-shot prompt. It is obvious that most RegTech users will use 0-shot prompt questions. However, it is possible to train such users to use k-shot prompting as well. Such prompts can increase the reliability and performance accuracy of LLMs. Future work can lay out the k-shot prompt design techniques for RegTech. The following are example scenarios on how and where few-shot prompts can be used in RegTech for agriculture.

i. 

Scenario 1: Streamlining Pesticide Usage Reporting with Few-Shot Learning

A farmer is required to submit monthly pesticide usage reports to the local agricultural regulatory authority. To simplify this process, a few-shot learning approach can be employed using a large language model (LLM). The farmer provides the LLM with a few examples of reports from previous months (the “few shots”). These examples demonstrate the required format, including specific data fields and their organization. The farmer then inputs the current month’s pesticide usage data, such as the types of pesticides used, quantities applied, and application dates. Leveraging the few-shot examples, the LLM automatically generates the new monthly report, populating it with the current data and ensuring adherence to the regulatory authority’s prescribed format.

ii. 

Scenario 2: Worker Safety Training Records

Documenting worker safety training is essential for compliance. The LLM can be trained with examples of compliant training records, including training topics, dates, and attendees. The farmer inputs the details of recent training sessions, and the LLM generates the necessary documentation.

5.2.2. Multi-Level Checking and Rule Relaxation

LLMs can enable multi-level compliance checking. It would involve a tiered approach for compliance verification, with non-linear dependencies of guidelines issued by different authorities. LLMs can keep track of these changes in rules in real time and alert relevant stakeholders if they are impacted.

Another feature of LLM is severity detection of non-compliance. If the producer meets the bare minimum of the regulatory requirements, they pass the compliance checks. However, when they fail to meet the compliance, it would be interesting for regulators to know how much they failed to comply. On the other hand, once they meet the minimum requirements, the regulator does not need to be concerned about the quality above the minimum standards. Severity detection is also helpful for early warning systems.

There may be opportunities for rule relaxation in certain circumstances. For example, less stringent regulations could be applied to small-scale, family-owned farms that pose minimal environmental or food safety risks. In emergencies, such as natural disasters or disease outbreaks, temporary regulation relaxation may be necessary to address urgent issues. With this, regulators can strike a balance between ensuring compliance and promoting agricultural innovation and sustainability.

5.2.3. Proof of Compliance

As the LLM virtual agent continuously chats with the human users in RegTech, it can keep a record of all the requests and responses. Traditional software can enable such record keeping, but LLM can also monitor for any malicious intent from the users. Also, as LLM can create a range of formatted outputs, it can create a digital certificate by combining all the recording information. This can be proof of compliance for regulators.

LLM can also act as an auditor for compliance checks, although significant improvement in reasoning skills is needed. However, as LLM can constantly interact with the data, it can build up reports, including any reasons and conditions over time.

5.2.4. LLM–Web Service Balancing

While LLMs have the potential to convert existing guidebooks into rule sets, their current capabilities are limited by factors like processing speed and data processing capacity. Hence, they are best suited for normal human conversations and basic analysis but are incapable of handling composite or derived computational tasks yet.

To overcome these limitations, we showed that LLMs can be integrated with backend web services developed by software developers to handle large datasets and complex computations. This integration will enhance the overall performance and scalability of LLM-powered applications, making them more practical and effective in real-world scenarios. Instead of having the LLM attempt to write its own code, we can instruct it to identify the necessary task and delegate it to a suitable backend service. To achieve this, the backend services should provide the LLM with properly formatted output, enabling the LLM to interpret and utilize the results effectively.

LLMs, while powerful, cannot solve all problems independently. By using a collaborative approach, where LLMs work in conjunction with specialized services, we can overcome these limitations and create more robust and effective AI solutions. In this work, we tested with only one level of web service functions, but future work can consider a chain of function selection or nested functions. In such cases, the LLM must also decide the sequence of the web service functions to be called and how to transfer the output of one to another as needed.

6. Conclusions

LLMs excel at tasks involving language and information retrieval, such as summarizing text, translating languages, and generating creative text formats. However, they may struggle with tasks that require complex reasoning, mathematical calculations, or the execution of code. For RegTech, it is essential to have speed, consistency, and accuracy in results. Thus, it is necessary to develop a new LLM dedicated to RegTech purposes. Such LLM has to be more accurate with a 0-shot prompt, but on the other hand, it does have to answer any random questions from the users, allowing it to be more focused in the response.

In this paper, we show that current LLMs are capable of the basic features needed for an LLM-supported RegTech application. LLMs can process basic analytical prompts. They can also understand how to solve complex analytical prompts and use external web services to solve those problems. LLMs can also automatically handle format checking, enumerations, and numerical checking for compliance. They can do this in a reasonable time for RegTech conversations. LLMs can also identify rules and convert them to code in publicly available human-generated bodies of text that are conventionally used by the relevant industries. Future work can focus on the development of LLM for RegTech, further refining these features.

RegTech is expected to be a fundamental component of Industry 5.0, and LLMs can make them user-friendly and easily deployable.