The Use of Large Language Models for Translating Buddhist Texts from Classical Chinese to Modern English: An Analysis and Evaluation with ChatGPT 4, ERNIE Bot 4, and Gemini Advanced

Abstract

This study conducts a comprehensive evaluation of large language models (LLMs), including ChatGPT 4, ERNIE Bot 4, and Gemini Advanced, in the context of translating Buddhist texts from classical Chinese to modern English. Focusing on three distinct Buddhist texts encompassing various literary forms and complexities, the analysis examines the models’ capabilities in handling specialized Buddhist terminology, classical Chinese grammar, and the translation of complex, lengthy sentences. The study employs a methodology where selected excerpts from these texts are translated by the LLMs, followed by an in-depth analysis comparing these machine-generated translations to human translations. The evaluation criteria include word translation accuracy, the ability to recognize and correctly interpret specific meanings within both classical and modern contexts, and the completeness of phrases without omitting or unnecessarily adding words. The findings reveal significant variations in the performance of these LLMs, with detailed observations on their strengths and weaknesses in translating specialized terms, managing grammatical structures unique to classical Chinese, and maintaining the integrity of the original texts’ meanings. This paper aims to shed light on the potential and limitations of using LLMs for translating complex literary works from ancient to modern languages, contributing valuable insights into the field of computational linguistics and the ongoing development of translation technologies.

1. Introduction

In this paper, I evaluate the translation capabilities of large language models, with specific emphasis on classical Chinese. My methodology involves translating selected excerpts ranging from 400 to 500 words. The corpus for this study consists of three Buddhist texts chosen to encompass a range of classical Chinese literature. This includes one text of Indian–Chinese origin, identified as 相續解脫地波羅蜜了義經 (T0678), and two texts of Chinese origin, written by Buddhist monks. The Chinese texts consist of 合訂天台三聖二和詩集 (B0087), noted for its literary quality, and 現果隨錄 (X1642), recognized for its narrative strength. The cited textual sources have been accessed through the Chinese Buddhist Electronic Text Association database (CBETA).

Moreover, the selection of these three texts for translation by large language models and human evaluation was not random. First, text T0678 is an early translated Buddhist sutra by Guṇabhadra (394–468). This text was chosen because it discusses a series of Buddhist philosophical concepts, such as the stages of liberation practice as well as the relationship between mental consciousness and liberation, and touches upon the tathagatagarbha doctrine. These contents can be used to test how large language models perform when translating Buddhist philosophical terms and concepts. Therefore, my evaluation of the Indian–Chinese text, T0678, focuses on whether the machine translations recognize certain special Buddhist terms and the correct spelling of the Romanized Sanskrit words. Furthermore, it considers the Buddhist sutras’ unique linguistic features and grammatical structures, distinct from those of classical Chinese.

The second text comes from the preface of B0087, written by Ming Dynasty monk 淨戒 (d,1418). This text was selected because, although it is a Buddhist text, it contains many traditional Chinese literary terms and is highly literary in nature, featuring beautiful phrasing and parallel prose structure. This text can be used to test how large language models perform when handling specialized Chinese classical literary works. The assessment of the second text, B0087, characterized by its more pronounced literary nature, will primarily focus on the ability of large language models to discern and interpret the literary devices and methods employed in classical Chinese literature. Additionally, the evaluation will explore the models’ ability to correctly identify and translate quotes from other literary works incorporated into the text.

The last text is a story from X1642, written by the late Ming-early Qing Dynasty monk Jiexian 戒顯 (1610–1672). This is a strongly narrative text that tells a complete story. Therefore, this text can be used to test large language models’ performance in translating narrative texts and their ability to understand the logic within the story. The evaluation of the third text, X1642, will assess the machine translation’s capability to convey the structural aspects of narrative in classical Chinese texts. Narrative texts exhibit a robust logical structure. Consequently, the evaluation of narrative texts will focus on the ability of large language models to translate the stories comprehensively.

I will utilize three commercial large language models—ChatGPT 4 (OpenAI 2022), ERNIE Bot 4 (Baidu 2023), and Gemini Advanced (Google 2023)—to handle the translation of these Buddhist texts, thereby assessing their capability in processing complex classical Chinese literature. The main goal of this assessment is to determine the accuracy of three large commercial language models in translating classical Chinese texts into modern English.

The evaluation criteria are as follows: 1. The machine translation’s ability to translate each word; 2. The machine translation’s ability to interpret the specific meanings of words in both classical and modern Chinese, ensuring translations accurately reflect the original meanings from the classical context rather than their modern interpretations; 3. The completeness of machine-translated phrases, with attention to errors involving omitted or added words. Additionally, I use human-translated texts as a standard to assess the machine translations.

To evaluate the large language models’ ability to process extensive classical Chinese texts, I plan to input a complete paragraph in a single instance instead of translating merely one or two sentences at a time. Each text evaluation will present the following:

• The original text in its classical form;
• My translation from classical Chinese into modern English;
• The three machine-generated translations from classical Chinese to modern English;
• Analysis and evaluation.

The prompt “Please translate <text> into modern English” was given to each large language model. I also used additional prompts, such as “Please translate <text> directly into modern English without summarizing the text and making comments”.

In analyzing texts translated by machines, I used various colors to identify errors made by the language models. Red marks are used to point out semantic errors, indicating instances where the translation failed to accurately capture the intended meaning of a word or phrase. This includes the following: 1. Words with multiple meanings, where the chosen translation did not align with the context intended in the text; 2. Words whose meanings differ between classical and modern Chinese, with large language models incorrectly using the modern interpretation in a classical context; 3. Cases where the translated word or phrase diverged completely from its original meaning, resulting in a translation that did not reflect the original intent. Blue marks are assigned to instances where large language models could not correctly identify the names, courtesy names, or posthumous titles of historical figures mentioned in the text. Green marks highlight instances where large language models either omitted parts of the content, leading to missing information, or inserted content that was not present in the original, thus causing a mismatch between the translation and the original text.

Additionally, in translating the texts T0678, B0087, and X1642 from classical Chinese to modern English, as well as in analyzing the accuracy of large language models’ translations, I consulted the following dictionaries:

• Paul W. Kroll and William G. Boltz, A Student’s Dictionary of Classical and Medieval Chinese;
• The Dictionary of Classical Chinese (古代漢語詞典) by the Dictionary Research Center of The Commercial Press (商務印書館辭書研究中心);
• The Dictionary of Classical Chinese Function Words (古代漢語虛詞詞典) published by the Institute of Linguistics, Chinese Academy of Social Sciences (中國社會科學院語言研究所古代漢語研究室);
• Nakamura Hajime 中村元, Bukkyogo daijiten shukusatsuban (佛教語大辞典 縮刷版)
• Ding Fubao (丁福保), Buddhist Dictionary (佛學大辭典).

Text 1: 《相續解脫地波羅蜜了義經》：「觀世音菩薩白佛言：「世尊！菩薩有十地， 所謂歡喜地、離垢地、明地、焰地、難勝地、現前地、遠行地、不動地、善慧地、 法雲地。佛地第十一，此諸地幾種清淨攝，為有幾分？」佛告觀世音菩薩：「有四種 清淨十一分，攝此諸地。觀世音！悕望清淨攝初地、增上戒淨攝第二地、增上心清 淨攝第三地、增上慧淨，增上上上妙淨，攝第四地乃至佛地，是四種淨攝彼諸地。「云何十一分？觀世音！解行地菩薩有十法行：善修習菩薩解脫忍，度此地已，菩 薩超昇離生彼分滿足，而未能於微細犯戒行正知住，此則分不滿足，為滿足故，方 便進求到已滿足；而未能具足世俗三昧正受，及滿足聞持陀羅尼，此則分不滿足， 為滿足故方便進求到已滿足；而未能如所得菩提分法數數修習，亦未能捨正受法愛 心，此則分不滿足，為滿足故方便進求到已滿足；而未能觀察真諦，不能捨一向背 生死向涅槃意，行方便攝修菩提分法，此則分不滿足，為滿足故方便進求到已滿足； 而未能現前觀諸行生，多住厭離多住無相，此則分不滿足，為滿足故方便進求到已 滿足；而未能多住不斷無間無相思惟，此則分不滿足，為滿足故方便進求到已滿足； 而未能捨離無相有行及得相力，此則分不滿足，為滿足故方便進求到已滿足；而未 能究竟分別眾相分別諸名，一切種說法得自在，此則分不滿足，為滿足故方便進求 到已滿足；而未能受得滿足法身，此則分不滿足，為滿足故方便進求到已滿足；而 未能得一切爾焰無礙無障知見，此則分不滿足，為滿足故方便進求到已滿足。彼分 滿足故，一切分滿足。觀世音！是名四種清淨十一種分攝一切諸地。」」

(CBETA 2023.Q4, T16, no. 678, pp. 714c18–715a21)

Human Translation: Avalokiteśvara Bodhisattva said to the Buddha, “World-Honored One! The Bodhisattva has ten grounds, named the Ground of Very Joyful, the Ground of Stainless, the Ground of Lightened, the Ground of Radiant Wisdom, the Ground of Difficult to Overcome, the Ground of Manifest, the Ground of Far-reaching, the Ground of Irreversible, the Ground of Perfect Wisdom, and the Ground of Clouds of Dharma. The Buddha Ground is the eleventh and highest level. Within these stages, how many types of purity exist, and into how many aspects can they be categorized?” The Buddha replied to Avalokiteśvara Bodhisattva, “There are four types of purity, divided into eleven aspects, contained within these grounds. Avalokiteśvara! The first ground is included in the purity of aspiration, while the second ground is included in the purity of higher precepts. The purity of the higher mind encompasses the third ground. Starting from the fourth ground and extending up to the Buddha ground, the purity of higher wisdom and the supreme and highest incomparable purity are contained within these grounds. These four types of purity encompass those grounds”.

“Why are there eleven categories? Avalokiteśvara! The Bodhisattva in the stage of practicing through understanding will follow ten aspects in their practice. The Bodhisattva diligently cultivates the forbearance of liberation, having already passed the current stage, transcending and leaving the cycle of birth and death and reaching the perfection of that ground. However, if the Bodhisattva is still unable to maintain proper awareness of subtle transgressions of the precepts, it is not yet satisfactory. To achieve perfection, one must apply skillful means, strive diligently, and persist until complete satisfaction is attained. If a Bodhisattva has not yet been able to fully possess the practice samādhi in the world and hear and remember the dhāraṇī perfectly, it is not yet adequate. To reach perfection, one must utilize expedient techniques, strive diligently, and continue until complete satisfaction is achieved. If a Bodhisattva has not yet repeatedly practiced the obtained Bodhyaṅga Dharma and also cannot yet abandon the attachment to the samaya and Dharma, it is not yet fulfilling. To attain perfection, one must employ skillful and expedient methods, strive diligently, and persevere until complete satisfaction is reached. If a Bodhisattva is still unable to observe the essence of truth, cannot abandon the idea of merely seeking nirvāṇa to escape from the cycle of birth and death, and does not skillfully and expeditiously cultivate the Bodhyaṅga Dharma, it is not yet sufficient. To achieve perfection, one must apply skillful means, strive diligently, and continue until complete satisfaction is attained. If a Bodhisattva is unable to directly observe the arising of all Dharmas in the present moment, only inclined to abide in an aversion towards living in this impure world, and only abiding in the absence of characteristics, it is not yet satisfactory. To achieve perfection, one must employ skillful and expedient methods, strive diligently, and continue until complete satisfaction is attained. If a Bodhisattva is incapable of maintaining unbroken and unceasing contemplation on the absence of characteristics, it is not yet satisfactory. To achieve perfection, one must apply skillful means, strive diligently, and continue until complete satisfaction is attained. If a Bodhisattva still cannot abandon the attachment to the absence of characteristics, thereby manifesting actions in reality and gaining the power from forms, it is not yet satisfactory. To achieve perfection, one must utilize expedient techniques, strive diligently, and continue until complete satisfaction is attained. If a Bodhisattva cannot thoroughly discern all the forms and names and cannot freely explain Dharmas in all its various aspects, it is not yet satisfactory. To achieve perfection, one must apply skillful means, strive diligently, and continue until complete satisfaction is attained. If the perfection of the truth body cannot be obtained, it is not yet satisfactory. To achieve perfection, one must employ skillful and expedient methods, strive diligently, and continue until complete satisfaction is attained. If unable to attain all the capabilities that can be known and reach the clear, unobstructed, and affliction-free real knowledge and insight, it is not yet satisfactory. To achieve perfection, one must apply skillful means, strive diligently, and continue until complete satisfaction is attained. When this part is satisfied, all parts are then completed. Avalokiteśvara! These are the four purities that encompass all levels across the ten Bodhisattva grounds and the Buddha ground”.

ChatGPT 4: Avalokiteshvara Bodhisattva said to the Buddha: “World-Honored One! The Bodhisattva has ten grounds, namely the Ground of Joy, the Ground of Freedom from Defilement, the Ground of Radiance, the Ground of Brilliance, the Ground of Difficulty Overcome, the Ground of Manifestation, the Ground of Far-reaching, the Ground of Immovability, the Ground of Good Wisdom, and the Ground of Dharma Cloud. The Buddha ground is the eleventh. How many types of purity encompass these grounds, and to what extent?” The Buddha replied to Avalokiteshvara Bodhisattva: “There are four types of purity that encompass these grounds in eleven aspects. Avalokiteshvara! The first ground is encompassed by the purity of aspiration, the second ground by the purity of superior precepts, the third ground by the purity of superior mind, and the fourth to the Buddha ground by the purity of superior wisdom, the utmost supreme and wondrous purity. These four types of purity encompass all those grounds. “How are these eleven aspects defined? Avalokiteshvara! A Bodhisattva on the path of understanding has ten practices: diligently cultivating the Bodhisattva’s patience of liberation. Having transcended this ground, the Bodhisattva ascends beyond birth, fulfilling that aspect, yet unable to maintain precise conduct without minor transgressions, this aspect is unfulfilled. For its fulfillment, one skillfully seeks to achieve satisfaction; unable to fully embody the worldly samadhi or master the recitation of dharanis, this aspect remains unfulfilled. For its fulfillment, one skillfully seeks satisfaction; unable to practice repeatedly the dharma of the Bodhi aspect as obtained, and unable to abandon the attachment to the lawful, this aspect is unfulfilled. For its fulfillment, one skillfully seeks satisfaction; unable to and unwilling to renounce the inclination from life to Nirvana, practicing skillfully the dharma of the Bodhi aspect, this aspect is unfulfilled. For its fulfillment, one skillfully seeks satisfaction; unable to witness the arising of phenomena directly, often dwelling in dispassion and formlessness, this aspect is unfulfilled. For its fulfillment, one skillfully seeks satisfaction; unable to remain in the uninterrupted contemplation of formlessness, this aspect is unfulfilled. For its fulfillment, one skillfully seeks satisfaction; unable to abandon the formless actions and attain the power of form, this aspect is unfulfilled. For its fulfillment, one skillfully seeks satisfaction; unable to ultimately discern all phenomena and names, and to speak dharma freely in all manners, this aspect is unfulfilled. For its fulfillment, one skillfully seeks satisfaction; unable to receive the fulfilled Dharma body, this aspect is unfulfilled. For its fulfillment, one skillfully seeks satisfaction; unable to attain the unobstructed knowledge and vision in all dharmas, this aspect is unfulfilled. For its fulfillment, one skillfully seeks satisfaction. With this aspect fulfilled, all aspects are fulfilled. Avalokiteshvara! This is called the four types of purity encompassing eleven aspects of all grounds”.

ERNIE bot 4: Avalokiteshvara Bodhisattva said to the Buddha, “World-honored One! There are ten stages for Bodhisattvas, known as the Joyful, the Stainless, the Luminous, the Flaming, the Difficult to Conquer, the Manifest, the Far-going, the Immovable, the Good Wisdom, and the Dharma Cloud stages. The Buddha stage is the eleventh. How many types of purity encompass these stages, and how many aspects are there?” The Buddha replied to Avalokiteshvara Bodhisattva, “There are four types of purity encompassing eleven aspects that embrace these stages. Avalokiteshvara! The purity of aspiration encompasses the first stage, the purity of enhanced precepts encompasses the second stage, the purity of enhanced mindfulness encompasses the third stage, and the purity of enhanced wisdom, together with the purity of supreme and exquisite wisdom, encompasses the fourth stage up to the Buddha stage. These four types of purity embrace all those stages. “And what are the eleven aspects? Avalokiteshvara! Bodhisattvas on the stage of understanding and practice have ten dharmas to cultivate: They skillfully cultivate the patience of Bodhisattva liberation. After transcending this stage, Bodhisattvas rise above birth and death and fulfill that aspect. However, they are not yet able to abide in right knowledge of subtle offenses against the precepts. This aspect is not yet fulfilled. To fulfill it, they diligently strive and attain fulfillment. Yet they are not able to fully attain mundane samadhi and the retention of dharanis. This aspect is not yet fulfilled. To fulfill it, they diligently strive and attain fulfillment. But they are not able to repeatedly cultivate the Bodhi factors they have attained, nor can they abandon their attachment to samadhi. This aspect is not yet fulfilled. To fulfill it, they diligently strive and attain fulfillment. And they are not able to observe the true nature, nor can they abandon their one-sided intention of turning away from birth and death towards Nirvana. They cultivate the Bodhi factors through skillful means. This aspect is not yet fulfilled. To fulfill it, they diligently strive and attain fulfillment. And they are not able to observe the arising of all dharmas in the present moment, often dwelling in aversion and the absence of form. This aspect is not yet fulfilled. To fulfill it, they diligently strive and attain fulfillment. They are not able to abide predominantly in uninterrupted and formless contemplation. This aspect is not yet fulfilled. To fulfill it, they diligently strive and attain fulfillment. They are not able to abandon formless existence and attain the power of form. This aspect is not yet fulfilled. To fulfill it, they diligently strive and attain fulfillment. And they are not able to ultimately distinguish all phenomena and names, nor can they freely explain all dharmas. This aspect is not yet fulfilled. To fulfill it, they diligently strive and attain fulfillment. They are not able to fully receive the Dharma body. This aspect is not yet fulfilled. To fulfill it, they diligently strive and attain fulfillment. And they are not able to attain unobstructed knowledge and vision of all dharmas. This aspect is not yet fulfilled. To fulfill it, they diligently strive and attain fulfillment. When that aspect is fulfilled, all aspects are fulfilled. Avalokiteshvara! These are the four types of purity encompassing eleven aspects that embrace all the stages”.

Gemini Advanced: Avalokitesvara Bodhisattva addressed the Buddha, saying: “World Honored One! Bodhisattvas have ten stages, namely the Joyous Stage, the Stainless Stage, the Illuminating Stage, the Radiant Stage, the Hard to Conquer Stage, the Manifest Stage, the Far-Reaching Stage, the Immovable Stage, the Excellent Intelligence Stage, and the Cloud of Dharma Stage. The Buddha Stage is the eleventh. How many kinds of purity encompass these stages, and how many parts are there?” The Buddha replied to Avalokitesvara Bodhisattva: “There are four kinds of purity and eleven parts that encompass all these stages. Avalokitesvara! The purity of aspiration encompasses the first stage; the purity of precepts encompasses the second stage; the purity of mind encompasses the third stage; the purity of wisdom, the purity of the highest and finest, encompasses the fourth stage and up to the Buddha stage. These four kinds of purity encompass those stages. What are the eleven parts? Avalokitesvara! A bodhisattva on the path of understanding and practice has ten dharmas: Good cultivation of the bodhisattva’s endurance, having passed this stage, the bodhisattva transcends and is free from birth, and that part is fulfilled, but he is not yet able to dwell in right knowledge of subtle transgressions of precepts. Therefore, this part is not fulfilled. In order to be fulfilled, he conveniently advances and seeks until he is fulfilled. He is not yet able to fully receive the worldly samadhi and to fulfill the retention of dharani. Therefore, this part is not fulfilled. In order to be fulfilled, he conveniently advances and seeks until he is fulfilled. He is not yet able to repeatedly cultivate the divisions of Dharma related to his attained enlightenment, nor is he able to release attachment to received teachings. Therefore, this part is not fulfilled. In order to be fulfilled, he conveniently advances and seeks until he is fulfilled. He is not yet able to observe the true principle, and cannot discard intentions always leading away from samsara towards nirvana, practice the skillful gathering of the divisions of enlightenment Dharma. Therefore, this part is not fulfilled. In order to be fulfilled, he conveniently advances and seeks until he is fulfilled. He is not yet able to directly witness the arising of all actions, mostly dwelling in detachment and mostly dwelling in formlessness. Therefore, this part is not fulfilled. In order to be fulfilled, he conveniently advances and seeks until he is fulfilled. He is not yet able to dwell extensively in continuous, uninterrupted, formless contemplation. Therefore, this part is not fulfilled. In order to be fulfilled, he conveniently advances and seeks until he is fulfilled. He is not yet able to release formless practices with form and attain the power of manifestation. Therefore, this part is not fulfilled. In order to be fulfilled, he conveniently advances and seeks until he is fulfilled. He is not yet able to definitively discern all forms and names, achieving freedom in all forms of Dharma exposition. Therefore, this part is not fulfilled. In order to be fulfilled, he conveniently advances and seeks until he is fulfilled. He is not yet able to receive the complete Dharma body. Therefore, this part is not fulfilled. In order to be fulfilled, he conveniently advances and seeks until he is fulfilled. He is not yet able to obtain the unimpeded, unobstructed knowledge and vision encompassing all flames. Therefore, this part is not fulfilled. In order to be fulfilled, he conveniently advances and seeks until he is fulfilled. When that part is fulfilled, all parts are fulfilled. Avalokitesvara! This is called the four kinds of purity and eleven parts that encompass all stages”.

	ChatGPT 4	ERNIE Bot 4	Gemini Advanced
Red	18	38	43
Green	2	1	5
Blue	0	0	0
Totals	20	39	48

This text includes several distinctive Buddhist terms, such as “佛” (Buddha), “觀世音菩薩” (Avalokiteśvara), and “陀羅尼” (dhāraṇī). Given that these terms were initially translated from Sanskrit into classical Chinese, an appropriate approach for translation would be to present the Romanized Sanskrit forms of these terms in the translated text rather than providing their Chinese Pinyin transcriptions. The three large language models have generally succeeded in writing the Romanized Sanskrit terms. However, there are still some shortcomings in specific details.

In Sanskrit transliteration, all three models exhibit a significant shortcoming: the omission of diacritical marks. These marks are crucial in Sanskrit phonology, delineating precise phonemic distinctions and intonation patterns intrinsic to the language. Diacritical marks, defined as supplementary signs or symbols appended to letters, denote specific pronunciations or stress patterns, thereby preserving the nuanced phonetic characteristics of the original Sanskrit (Biltoo 2021, p. 20). A salient example of this deficiency is the translation of “Avalokitesvara”. All three models—ChatGPT 4, ERNIE Bot 4, and Gemini Advanced—uniformly rendered this as “Avalokiteshvara”. This transliteration notably lacks the diacritical mark on the “s” character, a crucial phonetic indicator distinguishing this sound from the standard “s” phoneme in Sanskrit (Biltoo 2021, p. 23). Such an omission compromises phonetic accuracy and potentially obscures the deeper etymological and semantic nuances associated with correctly pronouncing this significant bodhisattva’s name in the Buddhist tradition.

Further analysis reveals additional nuances in the translation of Buddhist terminology. The Chinese term “陀羅尼” should be properly rendered as “dhāraṇī” in Sanskrit transliteration. While ChatGPT 4’s translation of “dharanis” is not inherently incorrect, it fails to adhere strictly to the International Alphabet of Sanskrit Transliteration (IAST) standards (Biltoo 2021, pp. 21–25). ERNIE Bot 4 mirrors ChatGPT 4’s approach, whereas Gemini Advanced opts for “dharani”, all lacking the diacritical marks.

Another salient example is the complex Buddhist term “菩提分法”. This compound consists of two elements: “菩提分” and “法”. In Romanized Sanskrit, these components correspond to “bodhyaṅga” and “dharma”, respectively, culminating in the composite term “bodhyaṅga-dharma”. ChatGPT 4’s rendition of “Dharma of Bodhi” represents a partial translation, accurately rendering “dharma”, while leaving “bodhyaṅga” untranslated. ERNIE Bot 4’s interpretation is more problematic, oversimplifying “菩提分” to merely “Bodhi” and omitting “法” (dharma) entirely. While capturing a general sense, Gemini Advanced’s translation, “The divisions of Dharma”, lacks precision in conveying certain Buddhist concepts.

Moreover, the term “涅槃” necessitates translation as “nirvāṇa” in Sanskrit. Notably, all three models consistently omitted the essential diacritical marks in their translations, a shortcoming that permeates their handling of Sanskrit terms. These discrepancies underscore the complexities inherent in translating specialized Buddhist terminology and highlight the current limitations of artificial intelligence in accurately rendering such nuanced philosophical and religious concepts. The consistent omission of diacritical marks across all models points to a systemic deficiency in their ability to fully capture Sanskrit’s phonological and semantic intricacies within Buddhist textual translation.

Buddhist term translations require balancing literal and conceptual accuracy. “十地”, for example, refers to the ten stages of practice in Buddhism, which are crucial for practitioners following the bodhisattva path. While ERNIE Bot 4 and Gemini Advanced both translate it as “stage”, which is not incorrect in conveying the meaning, ChatGPT 4’s translation of “ground” is more appropriate when considering the literal sense of the Chinese term. This translation accurately reflects the word-for-word correlation between the Chinese “地” and its Sanskrit equivalent, “bhumi”, which means “ground” or “earth”.

Additionally, large language models exhibit varying degrees of accuracy in translating nuanced Buddhist terminology and philosophical concepts from classical Chinese, often struggling with contextual meanings and idiomatic expressions. In the sentence “此諸地幾種清淨攝，為有幾分”, the intended inquiry is twofold: “How many kinds of purity do these grounds encompass, and into how many aspects can they be divided?” ChatGPT 4’s translation did not convey the latter part of the sentence, which states “into how many parts can they be divided”, and instead translated it as “to what extent”.

The term “增上戒” in the subsequent sentence can be translated as “higher precepts”. In this rendering, “增上” corresponds to “higher”, conveying a sense of elevation or superiority, while “戒” equates to “precepts”, referring to the moral guidelines that Buddhist practitioners are expected to follow. ERNIE Bot 4’s “enhanced precepts” translation fails to capture the nuanced meaning of “增上”. On the other hand, Gemini Advanced’s translation remains incomplete, as it only renders “戒” as “precepts” without addressing the translation of “增上”.

Moving on to the sentence “增上心清淨攝第三地，增上慧淨，增上上上妙淨”, we can find that the relationship between the latter sentence and the former sentence is a progressive one. “增上” represents “higher”, and “增上上” means the highest meaning. However, Gemini Advanced does not reflect this progressive relationship in its translation, and the sense of higher and highest is not conveyed.

ERNIE Bot 4’s treatment of the text presents several noteworthy issues that warrant closer examination. The model consistently translates “增上” as “enhanced”, a choice that, as previously elucidated, falls short of conveying the full semantic range of the original term. This consistent misinterpretation suggests a systematic limitation in the model’s understanding of Buddhist terminological nuances.

Of particular interest is ERNIE Bot 4’s rendering of “增上上妙淨” as “exquisite wisdom”. The selection of “exquisite” to translate “妙” is problematic in this context. While “exquisite” typically denotes delicacy and esthetic beauty in English, it fails to capture the profound philosophical implications of “妙” in Buddhist thought. In this context, “妙” should signify incomparable, transcendent, or superlative.

ChatGPT 4’s translation of “妙” as “wondrous”, while marginally more appropriate than ERNIE Bot 4’s interpretation, still does not encapsulate the term’s intended meaning within the Buddhist philosophical framework. The choice of “wondrous” leans towards conveying a sense of awe or marvel, which, while not entirely misplaced, does not adequately express the concept of superiority or incomparability inherent in the original term.

In contrast, Gemini Advanced’s “highest and finest” translation demonstrates a more nuanced understanding of the term’s contextual significance. This rendering accurately reflects the superlative nature of “妙” in Buddhism, conveying its elevated status and exceptional quality. Among the three large language models, Gemini Advanced’s interpretation most closely aligns with the intended meaning of the original text in this instance.

In classical Chinese, “云何” is used to pose a question, commonly translated as “why” or “how”. ChatGPT 4 captured this nuance, whereas ERNIE Bot 4 and Gemini Advanced rendered it as “what are”. Although this still conveys a question, it does not reflect the meaning of “云何” in its classical context.

Another aspect to consider is that translating intricate Buddhist philosophical concepts poses challenges for large language models, often resulting in a loss of nuance and contextual depth. The term “解行地菩薩” is a unique Buddhist term referring to the stage of a Bodhisattva who practices through their understanding, which is the stage of the Bodhisattva before attaining the realization of Suchness. Thus, ChatGPT 4’s rendition of “Bodhisattva on the Path of Understanding” has some limitations. It suggests “path” rather than highlighting “解行地” as a distinct stage of Bodhisattva. Furthermore, it translates “解” merely to “understanding”, omitting the crucial aspect of “行”, which signifies “practice”. While ERNIE Bot 4 translated both “understanding” and “practice” and recognized “解行地” as a stage of Bodhisattva, it did not reflect the relationship between “understanding” and “practice”. The text depicts this stage as one where practice is guided by understanding, necessitating a translation that emphasizes “practicing through understanding” rather than presenting “understanding and practice” as parallel concepts, as ERNIE Bot 4 did. Similarly, Gemini Advanced did not convey “stage” and misrepresented the relationship between “understanding” and “practice” as parallel.

The term “十法行” in this context refers to ten ways of practice. ChatGPT 4’s “ten practices” translation aligns more closely with the intended meaning than ERNIE Bot 4 and Gemini Advanced, which rendered it “ten dharmas”. While “dharmas” can indeed be translated as “法” in Chinese, generally signifying the Buddha’s teachings, such as Buddhist truths and principles, the “十法行” mentioned here pertains directly to the specific practices Bodhisattvas are expected to cultivate, as evidenced by the subsequent content.

Large language models demonstrate varying proficiency in translating complex Buddhist concepts and practices, often struggling with context-dependent terminology and specialized doctrinal nuances. This is evident in their translations of a text describing the practices a Bodhisattva needs to cultivate, particularly in the sentence “善修習菩薩解脫忍，度此地已，菩薩超昇離生彼分滿足，而未能於微細犯戒行正知住，此則分不滿足，為滿足故，方便進求到已滿足”. All three large language models exhibit some limitations in their translations of this sentence.

When translating “而未能於微細犯戒行正知住”, ChatGPT 4 did not provide a literal, word-for-word rendering. It refrained from directly translating “犯戒” as “transgressions of the precepts” and did not convey “正知住” as “proper awareness”. Instead, it summarized the phrase as “yet unable to maintain precise conduct without minor transgressions”. While ChatGPT 4’s interpretation captures the overall meaning somewhat, it does not accurately translate each term.

In “善修習菩薩解脫忍”, ChatGPT 4 did not capture the Buddhist term “解脫忍” and instead oversimplified it as “bodhisattva’s endurance”. This simplification does not reflect that “解脫忍” (the forbearance of liberation) refers to a certain cultivation method within the Bodhisattva path of Mahāyāna Buddhism and, therefore, should be translated as “the forbearance of liberation”. Furthermore, its translation of “善” is also not precise. While “善” in classical Chinese can indeed mean “good”, in the context of this passage, it should be translated as “skillful” or “diligently”.

The following sentence “而未能於微細犯戒行正知住”, Gemini Advanced translated “住” as “dwell”, resulting in a translation that does not capture the sentence’s intended meaning. This sentence conveys that the Bodhisattva should maintain proper awareness of even subtle transgressions of the precepts, rather than Gemini Advanced’s interpretation of “to dwell in the right knowledge of subtle transgressions of precepts”.

The accurate translation of context-dependent Buddhist terminology poses a challenge for large language models, particularly when dealing with terms with multiple meanings or specialized usage in Buddhist literature. “此則分不滿足，為滿足故，方便進求到已滿足” appears several times in the subsequent text, highlighting this challenge. Among the three large language models, only ChatGPT 4 accurately translates the term “方便” as “skillful”. In classical Chinese, “方便” has different meanings depending on the context. Specifically, in Buddhist literature, “方便” refers to skillful means or methods, making “skillful” the most appropriate translation. ERNIE Bot 4, however, translates “方便” as “diligently”, which does not capture the nuanced meaning of the term in this context. On the other hand, Gemini Advanced translates “方便” as “conveniently”, which, while correct in other contexts, is unsuitable for this particular Buddhist text. Therefore, ERNIE Bot 4 and Gemini Advanced do not provide the most accurate translation of “方便”.

Furthermore, Gemini Advanced’s “求進” translation is also inaccurate. In the context of this text, the term should be rendered as “seek”, conveying the act of searching or seeking. The translation “advances” does not capture the text’s intended meaning.

Translating complex Buddhist terminology poses substantial difficulties for large language models, frequently resulting in misinterpretation of compound concepts and loss of essential nuances. In the text “而未能具足世俗三昧正受，及滿足聞持陀羅尼” all three large language models—ChatGPT 4, ERNIE bot 4, and Gemini Advanced—had some difficulties translating the Buddhist term “正受三昧” Instead of treating “正受三昧” as a single term, the models split it into two separate words: “三昧” and “正受”. This led to some inconsistencies in the translations, with ChatGPT 4 rendering “三昧” to “samadhi” and “正受” to “embody”, resulting in the translation “embody the worldly samadhi”. Likewise, ERNIE bot 4 divided the term and translated “正受” as “attain”, recognizing only the meaning of “受”, while not capturing the meaning of “正” as “correct” or “right” in a word-for-word translation. As a result, it produced the translation “attain mundane samadhi”. Gemini Advanced had a similar challenge as the other two models, splitting “正受三昧” into two terms and translating “正受” as “receive”. Moreover, all three large language models did not include the proper diacritical marks when translating “samādhi”.

Moreover, the translations did not convey the meaning of “聞持” in this sentence. This term signifies “hearing (Buddhist teachings) and memorizing them”, and thus, the translation should precisely capture both aspects: “聞” (hearing) and “持” (memorizing). ChatGPT 4 did not capture the term’s meaning and translated it as “recitation”. Similarly, ERNIE Bot 4 and Gemini Advanced interpreted “持” as meaning to retain or keep, leading them to translate it as “retention”. Consequently, all three machine translations did not include the crucial aspect of “hearing” in their renderings of “聞持”.

Large language models face challenges when translating Buddhist philosophical concepts, particularly terms with specific doctrinal meanings. This difficulty is exemplified by ChatGPT 4’s translation of the phrase “捨正受法愛心”. While the model correctly rendered “捨” as “abandon” and “愛心” as “attachment”, it failed to recognize the Buddhist terms “正受” (samaya) and “法” (dharma in this context). Instead, ChatGPT 4 interpreted “法” as being related to law, resulting in the translation “lawful”, and omitted the translation of “正受” altogether. Although mainly accurate, ERNIE Bot 4’s translation has two minor issues: the spelling of “samadhi” lacks diacritical marks, and “dharma” is not translated. Gemini Advanced, on the other hand, did not provide a literal, word-for-word translation of the sentence. Instead, it summarized the content as “nor can he release attachment to received teachings”, interpreting “正受法” as “received teachings”. This suggests that Gemini Advanced only considered the literal meaning of “受” (received) without accounting for the specialized translation of “正受” within the Buddhist context.

The challenges faced by large language models in capturing the nuanced relationships between Buddhist concepts are further demonstrated by ChatGPT 4’s translation of the sentence “而未能觀察真諦，不能捨一向背生死向涅槃意，行方便攝修菩提分法”. While the model accurately rendered “而未能觀察真諦” as “discern the truth”, it failed to capture the latter part of the sentence, which conveys the inability to abandon the idea of merely seeking nirvāṇa as a means to escape from the cycle of birth and death. Instead, ChatGPT 4 interpreted this as “the inclination from life to Nirvana”, suggesting a progression from life to nirvāṇa. However, this interpretation does not align with the relationship between the cycle of birth and death and nirvāṇa in the given context, where the practitioner seeks nirvāṇa to break free from the cycle of birth and death.

The ensuing discussion will center around the fact that the accurate translation of Buddhist terms necessitates a profound comprehension of their specific meanings within the framework of Buddhist philosophy, posing a challenge for existing large language models. ChatGPT 4’s translation of the subsequent text “而未能現前觀諸行生，多住厭離多住無相” has some limitations. ChatGPT 4’s translation of the term “厭離” does not capture its meaning. In Buddhism, “厭離” is a specific term that refers to a strong aversion towards the impurities of the present world, accompanied by the desire to distance oneself from them. Therefore, the term should be translated as “aversion” to accurately reflect this sense of intense dislike and the wish to avoid or shun something. ChatGPT 4’s choice of “dispassion” does not convey this nuance, as it suggests a state of being unfeeling or cool, rather than a state of deliberately distancing oneself due to an aversion towards the impurities of the world. Rendering “住” as “dwelling” is not the most suitable choice in this context. While “住厭離” can mean “dwelling” in classical Chinese, this word means a state of residence associated with houses and people. However, in the present text, the “住” in “住” conveys the notion of maintaining or remaining in a particular state. Consequently, it would be more fitting to translate “住” here as “maintaining” or “abiding in”.

All three large language models did not capture the meaning of “多”. ChatGPT 4 and ERNIE Bot 4 translated it to “often”, while Gemini Advanced rendered it to “mostly”. Although these meanings are correct for “多” in classical Chinese, they do not align well with the context of the text. The sentence aims to convey that practitioners on the Bodhisattva path should not merely maintain and abide in the states of aversion and absence of characteristics. Using the word “often”, which denotes frequency, does not capture the intended meaning because it implies that Bodhisattva path practitioners might frequently abide in this state, while in other situations, they may transcend it. Therefore, if practitioners on the Bodhisattva path only “often” maintain the states of aversion and absence of characteristics, then there would be no need for the following text to point out that this aspect is unsatisfactory and that one should break through this state to seek perfection. Likewise, “mostly” is also not the most suitable choice for the context of the text. Therefore, in this sentence, “多” should be translated as “only” or “merely” to reflect the intended meaning more accurately.

In the subsequent sentence “而未能多住不斷無間無相思惟，此則分不滿足”, the context of the text suggests that it is more fitting to translate “多住” as “mostly”, “mainly”, or “always”. This is because the preceding sentence aimed to convey that practitioners on the Bodhisattva path should not merely abide in the states of aversion and absence of characteristics. As such, the translation of “多” in the previous sentence must express a sense of limitation or stagnation.

In contrast, this sentence seeks to communicate that practitioners on the Bodhisattva path need to focus more, or primarily, on uninterrupted contemplation of the absence of characteristics. If they cannot achieve this, they cannot attain perfection in this regard. Therefore, the translation of “多住” here should not employ “only” to express a sense of limitation. Instead, it is essential to use words like “mainly” to offer guidance and advice to the practitioners on the Bodhisattva path, encouraging them to invest more energy in contemplation.

When translating this sentence, ChatGPT 4 neglected to translate “多”, but its rendering of “住” to “remain” is accurate. The limitation in ERNIE Bot’s translation, “They are not able to abide predominantly in uninterrupted and formless contemplation”, stems from its translation of “無相思惟” as “formless contemplation” rather than “meditation on the absence of characteristics”.

In this sentence, Gemini Advanced still did not accurately translate “多住”, making the same error as before, with “住” still being translated to “dwelling”. Furthermore, its translation of “無相思惟” is the same as ERNIE Bot 4’s translation.

In summary, context-dependent translation of seemingly simple terms poses a challenge for large language models, particularly in texts with subtle philosophical distinctions.

Large language models often encounter difficulties when translating complex sentence structures in Buddhist texts, particularly when these sentences convey multiple layers of meaning. The sentence “而未能捨離無相有行及得相力”, despite lacking punctuation, illustrates this challenge by conveying three distinct layers of meaning: the inability to abandon the attachment to the absence of characteristics, the manifestation of concrete actions in reality, and the attainment of power from forms. ChatGPT 4’s translation fails to capture these three layers, instead combining the first two and resulting in the inaccurate translation of “有行” as “formless actions”. Similarly, ERNIE Bot 4 does not translate “有行” as having concrete actions, while Gemini Advanced only accurately translates the aspect of gaining power from form, omitting the first two layers of meaning.

The accurate translation of Buddhist terms requires a deep understanding of their specific usage within the context of Buddhist literature, a nuance that current large language models often struggle to grasp. This is exemplified by ChatGPT 4 and ERNIE Bot 4’s translations of the word “究竟” in the sentence “究竟分別眾相分別諸名，一切種說法得自在”. In Buddhist literature, “究竟” typically denotes a thorough and complete understanding, making “completely” or “thoroughly” more suitable translations than the chosen “ultimately”, which conveys a sense of finality rather than the intended meaning. Moreover, their rendering of “說法” as “to speak dharma” does not capture the intended meaning of explaining or interpreting the teachings. These limitations arise from the models’ reliance on the meaning of “究竟” in non-Buddhist classical Chinese texts, which differs from its usage in Buddhist literature. Gemini Advanced’s translation of “究竟” as “definitively” and its inaccurate translation of “一切種說法得自在” further highlight the challenges faced by large language models in capturing the essence of Buddhist terminology.

The difficulty in accurately translating Buddhist terms extends to the sentence “受得滿足法身”, where the models struggle to capture the meaning of “滿足法身” as the perfect Dharma body or truth body. ChatGPT 4’s translation of “fulfilled” suggests satisfaction without reaching perfection, while ERNIE Bot 4’s “fully receive the Dharma body” interprets it as completely accepting the Dharma body. Gemini Advanced’s “complete” translation also fails to capture the state of perfection in this context.

The subsequent sentence “而未能得一切爾焰無礙無障知見”, further underscores the challenges large language models face when dealing with highly specialized Buddhist terms. “爾焰”, a Buddhist term referring to the ability to know, determine, or understand, is omitted in ChatGPT 4’s translation and misinterpreted by Gemini Advanced as “all flames”. The term “無障”, which refers to the absence of afflictions in Buddhism, is translated by ChatGPT 4 as “unobstructed”, failing to convey the intended meaning. Furthermore, the models struggle to accurately translate “知見” as true knowledge and insight, with ChatGPT 4 rendering it “dharmas”. These misinterpretations and omissions of crucial philosophical concepts highlight the challenge that highly specialized Buddhist terms pose for large language translation models.

In conclusion, through comparative analysis of the English translations of Buddhist literature by ChatGPT 4, ERNIE Bot 4, and Gemini Advanced, we can see that all three large language models have certain limitations in handling Buddhist terminology and classical Chinese grammar. The three models have several shortcomings. First, their transliteration of Romanized Sanskrit generally lacks the necessary diacritical marks. Second, the translation of key Buddhist concepts, such as “菩提分法”, “解行地”, and “聞持”, is not precise enough. Third, there is an insufficient grasp of the special meanings of words like “方便”, “多”, and “究竟” in the Buddhist context. Finally, when dealing with complex, long sentences, they often fail to convey the multi-layered meanings of the original text. However, overall, the translation quality of ChatGPT 4 is noticeably superior to the other two models. It translates terms like “十地”, “增上戒”, and “善” closer to the meaning and better understands the unique connotation of the term “方便” in the Buddhist context. Although there are deficiencies in certain areas of ChatGPT 4’s translations, it comparatively grasps the overall intent of the text better. The performance of ERNIE Bot 4 and Gemini Advanced is relatively not good enough. For instance, ERNIE Bot 4 recognized “解行地” as a Bodhisattva stage but did not accurately reflect the relationship between “understanding” and “practice”, implying a parallel relationship rather than one where the practice is guided by understanding. Gemini Advanced generalizes “善” as “good”, failing to reflect its nuance of “skillfulness”. Therefore, when considering the overall translation quality, ChatGPT 4 outperforms both ERNIE Bot 4 and Gemini Advanced. While all three models demonstrate areas that could be enhanced, ChatGPT 4 exhibits a notably higher level of competence in comprehending and accurately translating Buddhist literature.

Text 2: 《合訂天台三聖二和詩集》：「宣情達事。世教有取於詩。吾宗聖賢高蹈遠視邈然矣。亦彷人情近習琢為文句。蓋憫物之心不可遏也。抑將激誘於道。奚啻宣情達事。流翫百世珠玉之擬哉。觀夫豐干寒拾三聖所唱。楚石琦公之和。韻皆痛快激烈斥妄警迷。山中天靈義首座。服膺有素。願繡梓以傳焉。且篹舊本諸名公序帖。及三隱集記系之。又以佛國白禪師所作文殊指南贊。詞勝理詣。永明壽禪師。布衲雍。鏡中圓。前後山居唱和之什。暨古德十牛頌。并諸歌偈。切於風礪。有裨益於世者。比次成帙。勸率善信陳智寶。賈福常。俾諸眾緣。併與刊行。謁言為弁。因謂醫方萬品。求對治而休。海寶千般。得如意而足。披此集者。驀然逗著一言半句。撲落眼屑。粲發心華。方信聖賢憫物之心。誠有在也。是則助揚激誘。微天靈之勣。吾誰與歸。永樂丙申夏。結制前一日。僧錄司右闡教。兼住鍾山靈谷。幻居比丘淨戒。」

(CBETA 2023.Q4, B14, no. 87, p. 737a8–b2)

Human Translation: Expressing emotions affairs of the world, the prevailing orthodox teachings are selected from poems. The esteemed predecessors of our school traveled to distant places and gained a broad and far-reaching view. They also closely imitated human emotions and, through review and practice, encapsulated them in the words and sentences of their literary works because they could not suppress empathy for everything in the world. Perhaps it will serve to inspire and guide others onto the right path. This goes beyond merely expressing emotions and comprehending world affairs; it is akin to cherished pearls and jewels that have been circulated and appreciated across many generations. Reading the poems written by the three masters Feng Gan, Han Shan, and Shi De, the venerable elder Chushi Qi follows their literary rhythm in his writings, which are straightforward, spirited, and eschew ignorance, alerting people to not fall into delusion. The chef monk Yi at the Tianling Temple in the mountain has always kept these works in mind and is willing to print and publish them in an elegantly bound volume for future generations. Additionally, compilations have been made of forewords by various esteemed masters from previous editions, and the Compilation of Records on the Three Hermits (Sanyin jiji 三隱集記) has been attached. Furthermore, the In Praise of Mañjuśrī’s Guidelines (Wenshu zhinan zan 文殊指南讚) composed by Chan Master Foguo Bai has been included, known for its wonderfully written and profound principles. A series of literary works about mountain-dwelling life were successively written and responded to by Chan Master Yongming Shou, Bu Na Yong, and Jing Zhong Yuan, and the Hymn of Ten Oxen (Shiniu song 十牛頌) by various elderly monks and other poems and Buddhist verses. Selecting works that have been carefully polished by the authors, like stones weathered by years of wind, and which have numerous benefits for future generations, they have been compiled into volumes in sequence. Under the leadership of devout followers such as Chen Zhibao and Jia Fuchang, many others have also participated, jointly publishing, and distributing these works. This text is respectfully presented as the preface. Just as people stop searching for treatments once they find one that works or feel satisfied when they obtain a desired treasure from the sea, some readers may find a phrase in this book that captures their attention, as if shaking off the dust in their eyes. This can instantly enlighten them and cause the flower of wisdom in their heart to bloom in an instant. Only then do they believe in the compassionate heart of the sages towards all beings that truly exist. This book serves to inspire and guide us as if it is a gift of power bestowed upon us by the heavens. Who are the like-minded individuals walking the same path together with me? In the summer of the Yongle Bingchen Year, on the day before the monastic retreat, I, Bhikkhu Huanju Jingjie, serving as the Right Supervisor Monk for Buddhist Practices (you chanjiao 右闡教) of the Central Buddhist Registry (senglu si 僧錄司) and the abbot in the Linggu Temple in Zhong Mountain, wrote this preface.

ChatGPT 4: Expressing emotions and achieving enlightenment, the teachings of the world have much to gain from poetry. Our ancestral sages, with their lofty and far-sighted ideals, seem almost beyond reach. Yet, they too crafted sentences and phrases, closely imitating human emotions and practices, for their compassion for all beings could not be restrained. Or perhaps to inspire and guide towards the path. This is not merely about expressing emotions and achieving enlightenment; it’s like the eternal beauty of gems and pearls, admired across centuries. Consider the rousing and forceful verses sung by the three saints in the cold, the harmonious responses of the esteemed Chu and Shi, all brimming with intense emotion, denouncing falsehoods and awakening the lost. The celestial spirit of the mountains, revered and deeply embraced, wishes to embroider these teachings for posterity. And so, gathering ancient texts, prefaces by renowned officials, and records of the three hermits, along with the Manjushri Guide by the Zen master from the Buddha land, whose words surpass logical understanding. The works of Zen master Yongming Yanshou, along with the songs and verses of ancient virtues, like the Ten Ox Herding Pictures, contribute to the sharpening of the mind, benefiting the world. Thus, this collection is formed, encouraging the faithful to cherish these jewels of wisdom, allowing everyone to share in their publication. In offering these words, it’s like seeking a cure in the myriad prescriptions of medicine or satisfaction in the thousand treasures of the sea; encountering just a word or phrase within these pages can enlighten, opening the eyes and blossoming the heart. Only then can one truly appreciate the compassionate heart of the sages. This is to aid in promoting inspiration and guidance, a testament to the celestial spirit’s subtle influence. To whom do I dedicate this work? On the eve of completing this compilation in the summer of the Yangle year, as the Deputy Director of the Monk Registry and resident of the Zhongshan Linggu, the monk Jingjie of the Phantom.

ERNIE bot 4: Expressing emotions and conveying events. Poetry has a role to play in educating the world. Our ancestors, the saints and sages, had lofty aspirations and far-reaching visions. They also imitated human emotions and were familiar with polishing their verses. Their compassion for all things could not be restrained. Perhaps they wanted to inspire people to follow the right path. It was not merely about expressing emotions and conveying events, but also about creating timeless pearls and jade-like masterpieces. Consider the songs sung by the three saints Fenggan, Hanshi, and the harmonies of Chushi Qigong. Their rhymes are all painful and intense, denouncing falsehood and warning the deluded. The chief monk of righteousness in the mountain, Tianling, has long admired and practiced these teachings. He wishes to compile and print them for posterity. He has also compiled various prefaces and postscripts by renowned scholars from previous editions, as well as the “Records of the Three Hermits”. In addition, he has included the “Guide to Manjusri” composed by the Buddhist monk Bai Chanshi, which is rich in both language and reason. The chants and responses of Yongming Shou Chan Master, Buna Yong, and Jingzhong Yuan from their mountain retreats, as well as the “Ten Ox Hymns” of the ancient masters and various songs and verses, are all relevant to the times and beneficial to society. They have been compiled into a volume, and with the encouragement of devout believers such as Chen Zhibao and Jia Fuchang, they have been published for the benefit of all. It is said that there are thousands of medical remedies, but one seeks the right treatment to rest. There are countless treasures in the sea, but one is content with obtaining what one wishes. Those who read this collection may suddenly stumble upon a word or phrase that will open their eyes and illuminate their hearts. Then, they will believe that the saints and sages truly had compassion for all beings. This is how they help to promote and inspire, and without the achievements of Tianling, who would we turn to? In the summer of the Yongle era, the day before the end of the monastic retreat, the Right Elucidator of the Monastic Bureau, and the abbot of Zhongshan Linggu Temple, the illusory monk Jingjie, wrote this.

Gemini Advanced: To express feelings and convey matters; worldly teachings do draw from poetry. Our ancestral sages were lofty and far-sighted, profound indeed. Yet they also followed human sentiments and customs, honing [wisdom] into written verses. This was surely because a compassionate heart for all beings could not be restrained. Moreover, they aimed to inspire and guide towards the Way. How could this be merely about expressing feelings and conveying matters? It surpasses this, flowing playfully for a hundred generations like a semblance of pearls and jade. Consider Fenggan and Hanshi—those chanted by the Three Sages. See how the rhymes of Duke Qi of Chu resonate as well. All are vigorous and spirited, sharply denouncing falsehood and rousing those deluded. Yishou, Spiritual Leader of the Mountain [Temple], has always upheld these with devotion. He wishes to carve them in woodblock for their transmission. Further, I shall attach prefaces and postscripts by renowned figures, along with records from the ‘Three Recluses Collection. Additionally, included is the ‘Guide to Manjushri’ composed by Master Bai Chan of the Buddhist realm. The words excel, the principle profound. From Yongming Shou Zen Master, the poems ‘Cloth Robe’ and ‘Mirror Circle’, along with those from his mountain dwelling exchanges. There are also the ancient worthies’ ‘Ten Oxherding Verses’, as well as various songs and verses. All incisive and refining, beneficial to the world, they will be compiled in due order. I urge the virtuous patrons Chen Zhibao and Jia Fuchang, that along with all contributing benefactors, let this be published. Respectfully offering these words as a preface: Just as with the myriad medicinal formulas, one seeks remedy according to ailment; or how the ocean’s countless treasures exist to fulfill desires. those who open this collection may suddenly grasp a single word or half a phrase. Their inner vision cleared, the heart-flower will blossom. Only then will belief arise that the sages’ compassionate heart for all things truly exists. This indeed is furthering inspiration and guidance—slightly attributed to Tianling’s merits. With whom else does this credit lie? Summer of the Yongle Bingchen Year [1446], the day before the monastic retreat. By the Bhikshu Jingjie, Right Dharma Propagator of the Sangha Registry, also residing at Zhongshan Linggu.

	ChatGPT 4	ERNIE Bot 4	Gemini Advanced
Red	22	23	22
Green	10	5	5
Blue	6	5	7
Totals	38	33	34

In B0087, all three large language models—ChatGPT 4, ERNIE Bot 4, and Gemini Advanced—demonstrate the ability to translate most of the classical Chinese text into modern English. However, these models encounter difficulties in identifying and differentiating words with multiple meanings, particularly in the context of Buddhist terminology and literary practices.

In the translation of the phrase “宣情達事”, the characters “宣” and “達” should be considered a pair of near-synonyms, both carrying the meaning of expressing or conveying ideas or emotions. Furthermore, given that the context of the B0087 text is related to classical Chinese poetry, the translation of “達” should be sought in past classical texts, particularly those concerning the function or evaluation of traditional poetry. In the Shiji 史記, there is a passage stating: “詩以達意” (Poetry is used to express ideas) (Sima 2014, p. 3885). Therefore, the “達” in B0087 should be translated as “express” or “convey”, similar to its usage in the Shiji. However, this character is often translated in different contexts as “achieving”. This discrepancy led ChatGPT 4 to incorrectly translate this term as “achieving”, highlighting its challenges in comprehending the subtleties of classical Chinese literature. In this context, ChatGPT 4 demonstrated inferior performance compared to the other two models in interpreting “事”. It erroneously opted for “enlightenment” as the translation. This choice stems from ChatGPT 4’s recognition of the text’s Buddhist associations, leading it to favor terms with specific Buddhist connotations. However, this interpretation resulted in a translation that potentially misrepresents the intended message. A more accurate rendering of “事” in this context would be “matters” or “affairs”, which better preserves the original meaning without introducing unwarranted religious overtones.

The models’ performance varied in translating the phrase “世教有取於詩”. ChatGPT 4 and Gemini Advanced successfully captured the core meaning, while ERNIE Bot 4 fell short. The term “世教” functions as a noun, denoting contemporary mainstream thought or the prevailing teachings of the era. Although ChatGPT 4 provided a largely accurate translation, it introduced an extraneous element by including the word “much”.

A closer examination of the original classical Chinese text reveals that while the character “有” might superficially seem to correspond to “much”, given its potential to imply quantity in specific contexts, it serves here merely as a grammatical particle. In this instance, “有” lacks inherent semantic content and does not function to modify verbs, nouns, or adjectives.

Rendering Buddhist terminology into English proved challenging for all three models, as seen in their interpretations of “高蹈” and “吾宗”. Given the author’s affiliation with the Linji school (臨濟宗) of Chan Buddhism, “吾宗” should be translated as “the predecessors” rather than the broader “our ancestors”. Similarly, “高蹈邈然”, which in classical Chinese implies “traveling far”,1 was only partially captured by the models, which emphasized “lofty” but missed the notion of distant travel (Xiao 1986, p. 1990).

The models’ interpretation of literary practices revealed varying degrees of success. In translating “習琢為文句”, only ChatGPT 4 correctly rendered “習” as “practice”, while ERNIE Bot 4 and Gemini Advanced misinterpreted it as “familiar” and “customs”, respectively. These errors stem from misunderstanding the sentence’s intent to describe the refinement of emotions into literary expressions.

Complex sentence structures highlighted both strengths and weaknesses in the models’ translations. ChatGPT 4 excelled in capturing the overall essence of the sentence “蓋憫物之心不可遏也。抑將激誘於道。奚啻宣情達事。流翫百世珠玉之擬哉” but stumbled on “宣情達事”. ERNIE Bot 4 misinterpreted critical terms like “抑將” and “誘”, while its translation of “流翫百世珠玉之擬哉” lacked precision. Gemini Advanced performed slightly better than ERNIE Bot 4 in conveying the basic meaning but still included inaccuracies, such as inappropriately adding “surely” and mistranslating “翫” as “playfully”.

In sections discussing historical figures, geographical locations, and monk titles, all three machine translations demonstrated limitations in handling specialized terms. The models struggled to accurately recognize and translate the names of prominent Buddhist figures, such as Fenggan, Hanshan, Shide (Zanning 1987, p. 483), and Chushi Qi (CBETA 2024.R2, X77, no. 1524, pp. 469a21–470a8), in the phrase “觀夫豐干寒拾三聖所唱。楚石琦公之和。快激烈斥妄警迷”. ChatGPT 4 mistranslated Hanshan and Shide’s names as “the three saints in the cold”, while ERNIE Bot 4 and Gemini Advanced only partially identified the names. Furthermore, the models failed to interpret the respectful title “公” correctly, with ERNIE Bot 4 incorporating it into Chushi Qi’s name and Gemini Advanced misinterpreting it as “Duke”.

The term “痛快”, which typically denotes a brisk and straightforward literary style, was mistranslated by ERNIE Bot 4 as “painful”. This error highlights the difficulty in accurately rendering context-specific literary terms from classical Chinese into modern English.

The phrase “山中天靈義首座” illustrates this interpretive difficulty. In this context, “天靈” can be understood as a temple “Tianling Temple” while “義首座” may be rendered as “chief monk Yi” (Xu, Zhizheng ji, juan 71, p. 1256)2. However, ChatGPT 4 misinterpreted “首座” as “The celestial spirit of the mountains”, while ERNIE Bot 4 and Gemini Advanced failed to accurately identify the personal name and monastic position.

The following sentence “服膺有素” further illustrates the models’ limitations in capturing the intended meaning of specialized Buddhist terminology. “服膺” means “to firmly memorize or sincerely believe in (a text)”, while “有素” is understood as “always” or “consistently”. ChatGPT 4 did not provide an accurate translation, ERNIE Bot 4’s translation was closer but still incomplete, and Gemini Advanced correctly translated “有素” but made an error in translating “服膺”.

In addition, “且篹舊本諸名公序帖。及三隱集記系之。又以佛國白禪師所作文殊指南讚。詞勝理詣” contains both classical titles and names of historical figures; yet, the translations provided by the three large language models are inadequate.

Firstly, personal names and specialized terminology further exemplified the models’ limitations. In the text, “佛國白禪師” refers to the historical figure “Foguo bai”, (CBETA 2024.R1, T51, no. 2077, p. 591c4) but ChatGPT 4 failed to recognize this, and incorrectly translated it as “the Zen master from the Buddha land”. It correctly translated as “禪師” (Zen/Chan master), but “佛國白”, which should be the monk’s name, was mistakenly translated as “the Buddha land”.

“諸名公” in the text signifies “various renowned masters”, but ChatGPT 4 incorrectly translated it as “renowned officials”. ERNIE Bot 4 incorrectly interpreted “公” as “scholars”.

The translation of book titles and Sanskrit terms posed significant challenges for the language models, often resulting in inaccurate or incomplete renderings. “三隱集記” is the title of the book, with “三隱” referring to the hermits Han Shan, Shi De, and Feng Gan. Therefore, while ChatGPT 4’s translation of “hermits” is correct, “集記” should be translated separately, with “集” as “Compilation” and “記” as “Record”. However, ChatGPT 4 merely translated it as “collections”. Similarly, ERNIE Bot 4 did not accurately translate “records” in “Collection of Records of the Three Hermits”.

“文殊指南讚” also fails to accurately convey the title, which should encompass both “指 南” (guidelines) and “讚” (in praise of); translating it as “In Praise of Mañjuśrī’s Guidelines” ChatGPT 4 only rendered “指南” as “guide” and omitted “讚”. Additionally, the Romanized Sanskrit term “Mañjuśrī” provided by ChatGPT 4 is inaccurate, lacking diacritical marks to represent sounds correctly. ERNIE Bot 4 also omitted diacritical marks in the Romanization of Sanskrit for “Manjushri”.

Accurate translation of classical Chinese terms proved a formidable task, particularly in capturing the subtle meanings of words with multiple interpretations. This complexity is exemplified in the translation of the phrase “詞勝理詣”, where the nuanced meanings of “勝” and “詣” proved problematic for large language models.

In classical Chinese, “勝” has various meanings. Given the context, “勝” should be interpreted as the adjective “wonderful”, while “詣” should be translated as “profound” or “of high quality”. ChatGPT 4 struggled with these terms, rendering the phrase as “whose words surpass logical understanding”. While “surpass” can indeed be one interpretation of “勝”, it is inappropriate in this context.

ERNIE Bot 4’s translation of “勝” as “rich” was misplaced, and it also failed to convey the meaning of “詣”.

Gemini Advanced made a different error in translating the sentence “且篹舊本諸名公序帖”. The original text indicates that Chief Monk Yi compiled the prefaces of famous masters, but Gemini Advanced used the first person, incorrectly attributing this action to the author of text 2: “I shall attach prefaces and postscripts by renowned figures”. In subsequent translations, it also inaccurately rendered the titles of two books and the name Foguobai. However, Gemini Advanced’s translations of “勝” and “詣” in the following sentence were accurate.

The translation of classical Chinese Buddhist texts by large language models reveals varying levels of accuracy in recognizing historical figures and interpreting complex literary references. In “永明壽禪師。布衲雍。鏡中圓，前後山居唱和之什。暨古德十牛頌。并諸歌偈。切於風礪。有裨益於世者。比次成帙”, ChatGPT 4 was able to realize that “永明壽” referred to the abbreviated name of the Chan master Yongming Yanshou (CBETA 2023.Q4, T51, no. 2078, p. 763b18–19), a detail that the other two large language models missed. However, ChatGPT 4 omitted significant content in its following translation; it left out the names “Bu Na Yong” (CBETA 2024.R2, X80, no. 1566, p. 507b8–16) and “Jing Zhong Yuan”, which did not appear in the translated text, and it also overlooked the phrase “切於風礪”, failing to translate it. Overall, ChatGPT 4 seemed to only roughly summarize the intended message of the sentence without providing a detailed, word-for-word translation.

ERNIE Bot 4 recognized and translated the names “Yongming Shou”, “Bu Na Yong”, and “Jing Zhong Yuan”. However, like ChatGPT 4, it did not translate the phrase “切於風礪”, skipping it to proceed to the following sentence.

Gemini Advanced had the weakest performance in translating this sentence; it translated only the name “Yongming Shou” and failed to recognize “Bu Na Yong” and “Jing Zhong Yuan” as names of monks, instead translating them directly as “the poems ‘Cloth Robe’ and ‘Mirror Circle’”.

The intended meaning of “前後山居唱和之什”, which is “these three individuals successively wrote and responded to each other’s poems about life in the mountains”, was also inaccurately translated to “along with those from his mountain dwelling exchanges”. Like the previous two machine translations, Gemini Advanced also overlooked “切於風礪”.

Translating personal names and publishing-related terminology from classical Chinese poses a significant challenge for language models, as evidenced by their handling of the sentence “勸率善信陳智寶。賈福常。俾諸眾緣。併與刊行。謁言為弁”.

ChatGPT 4 made several errors in translating this sentence. Firstly, it omitted the names of historical figures Chen Zhibao and Jia Fuchang. Secondly, it summarized the sentence: “Thus, this collection is formed, encouraging the faithful to cherish these jewels of wisdom, allowing everyone to share in their publication”, which does not accurately reflect the original content. The original text describes the publication process of this anthology or literary works, emphasizing that it was accomplished under the leadership of Chen Zhibao and Jia Fuchang in cooperation with other faithful individuals rather than encouraging appreciation of wisdom’s treasures as understood by ChatGPT 4. Furthermore, ChatGPT 4 omitted the translation of “謁言為弁”. In classical Chinese, “弁” can mean “preface” or “introduction”, so the sentence should imply “This passage that the reader sees serves as the introduction”.

In comparison, ERNIE Bot 4 performed better in translating this sentence. It identified and translated the names of the historical figures mentioned within and broadly understood the core content of the sentence without significant errors. However, ERNIE Bot 4 also missed translating “謁言為弁。”

Gemini Advanced was the only one among the three machine translations to translate “謁言為弁” but its interpretation of the sentence’s meaning was slightly off. It used a sentence structure starting with “I urge …”, which inaccurately suggests that the author pleads with Chen Zhibao, Jia Fuchang, and others to contribute to the book’s publication. The original text commends their contributions to the publishing of this anthology rather than soliciting their support.

Large language models demonstrate varying proficiency in translating classical Chinese parallelism, often recognizing the structure but struggling with nuanced interpretations of metaphorical content. The sentence “因謂醫方萬品。求對治而休。海寶千般。得如意而足” exemplifies the use of parallelism, a technique from traditional Chinese literature that ensures the first and subsequent sentences are matched in logic, parts of speech, and metaphor. All three large language models accurately identified this literary device. In their translations, “如意” was not interpreted as a specific treasure. Instead, mirroring the concept of “對治” (the correct remedy), they rendered “如意” as “fulfilling one’s desired wish”, aligning with the narrative of seeking the proper treatment and then being satisfied.

However, ChatGPT 4 incorrectly translated “因謂” as “In offering these words”, missing its intended meaning. In classical Chinese, “因謂” should be more precisely translated as “it is said that there are”.

ERNIE Bot 4 made a mathematical error in its translation. The term “萬種” should be translated as “ten thousand kinds”, but it inaccurately rendered it as “thousands of”.

In its interpretation of the sentence, Gemini Advanced strayed from the intended meaning. The references to ten thousand remedies and a thousand treasures are metaphors the author employs to lay the groundwork for what follows and to emphasize the anthology’s worth. However, Gemini Advanced misinterpreted this as “Just as with the myriad medicinal formulas, one seeks remedy according to ailment; or how the ocean’s countless treasures exist to fulfill desires”. This reading does not adequately highlight the anthology’s significance. The author aims to convey that, much like one seeks the singular effective remedy among ten thousand or the one fulfilling treasure among thousands, this anthology stands out as the singularly most precious and insightful book available.

Language models’ interpretations of the sentence “披此集者。驀然逗著一言半句。撲落眼屑。粲發心華。方信聖賢憫物之心。誠有在也” reveal the complexities in translating classical Chinese literary devices.

ChatGPT 4 omitted the translation of “批此集者” in the sentence. It fails to convey that this sentence was addressed to readers of the anthology. “驀然逗著一言半句” should indicate that readers are suddenly captivated by a phrase or a few words in the book. However, ChatGPT 4 translated it as “within these pages can enlighten”, an unrelated addition to the original text. While the entire sentence includes a metaphor of “enlightening the reader”, this metaphor is related to “粲發心華” and should not appear in the first part of the sentence.

Moreover, the translation of “撲落眼屑” was incorrect across multiple models. ChatGPT 4 rendered “撲落” as “opening”, whereas, in classical Chinese, it does not mean to open but rather implies “to cause to fall” or “to shake off, to scatter”, generally carrying the sense of “falling” rather than “opening”. “眼屑” refers not to the eyes themselves but to the dust within them. The metaphor of “shaking off the dust from one’s eyes” used by the author signifies the clarity and enlightenment gained from reading the book, akin to clearing one’s vision of obstructions, thereby offering new insights and wisdom to the reader.

ERNIE Bot 4 made a similar mistake by rendering the phrase “open their eyes”. At the same time, Gemini Advanced’s translation of “Their inner vision cleared” grasped the intended meaning but failed to express the original metaphor accurately.

Classical Chinese allusions pose unique translation difficulties for large language models, often requiring deep cultural and historical knowledge to interpret correctly. The phrase “吾誰與歸”3 resented another challenge, being an allusion to Guoyu 國語 (Zuo, Guoyu, juan 14, p. 331). It asks, “Who are the like-minded individuals walking the same path with me?” In this context, “歸” should be understood as “together”. However, all three models struggled with this allusion. ChatGPT 4 incorrectly translated it as “To whom do I dedicate this work?”

ERNIE Bot 4 mistakenly rendered “歸” as “return”, Gemini Advanced also failed to grasp that the sentence expresses the author’s search for like-minded individuals, leading to inaccurate translations across the board.

The translation of classical Chinese Buddhist texts by large language models shows inconsistent levels of precision, especially when dealing with specialized Buddhist terms, historical allusions, and Ming Dynasty-era institutional nomenclature. The sentence “永樂丙申夏。結制前一日” indicates that the author wrote the text in the summer of the Bing Shen year during the Yongle era of the Ming Dynasty, which corresponds to the summer of 1446 AD. “結制” refers to the traditional annual retreat in Buddhist monastic communities. When translating this sentence, ChatGPT 4 incorrectly wrote “永樂” as “Yangle year”, whereas the correct pinyin spelling should be “Yongle”. Moreover, “結制” should be translated as “retreat” rather than “completing this compilation”. Overall, ChatGPT 4 made significant errors in handling the traditional Chinese lunisolar calendar and failed to accurately translate a conventional Buddhist custom.

In contrast, ERNIE Bot 4 correctly translated the pinyin spelling of “Yongle” and the Buddhist custom “retreat”, but it omitted the translation of “Bing Shen”.

Gemini Advanced accurately translated this sentence and even identified “永樂丙申” as the year 1446 in the modern calendar.

The “僧錄司” was an institution responsible for managing Buddhist monks. The Chinese–English Dictionary of Ming Government Official Titles translates it as “The Central Buddhist Registry” (Zhang et al. 2017, p. 166). While ChatGPT 4’s translation “Monk Registry” captures the basic function of registration and management, it omits the term “Central” and replaces “Buddhist” with “Monk”. This simplification, though conveying the general purpose of the institution, does not fully reflect its status as a centralized, national-level organization as indicated in the authoritative translation.

ERNIE Bot 4 rendered this government institution as a “Monastic Bureau”. While “Bureau” implies a governmental agency, “錄” in classical Chinese signifies “to register” or “to enroll”. Therefore, given the necessity for an exact word-for-word translation of “僧錄司”, ERNIE Bot 4’s rendition does not fully capture the institution’s function related to the registration of monks.

Gemini Advanced’s translation of “僧錄司” into “Sangha Registry” is also acceptable, considering that “Sangha” is a Romanized Sanskrit term associated with the meaning of “Buddhist community”.

In the text, “右闡教” refers to a religious official position during the Ming Dynasty in China, serving as a branch of the “闡教”. The role entailed researching Buddhist teachings and guiding monks in the teachings and practices of Buddhism. Thus, this position could be accurately translated as “Right Supervisor Monk for Buddhist Practices”. ChatGPT 4’s rendition of it as “Deputy Director” captures the leading responsibilities of the role; yet, it lacks precision in the description. ERNIE Bot 4’s translation, “Right Elucidator”, successfully conveys the essence of “闡” but fails to explicitly communicate the position’s educational and guiding duties towards monks in studying and practicing Buddhist doctrines, rendering this translation imprecise. Meanwhile, Gemini Advanced’s “Right Dharma Propagator” translation significantly strays from the original meaning.

In “兼住鍾山靈谷”, the term “住” signifies the monastic role of “abbot”, not the verb “reside”. This implies that the text’s author not only serves as the Right Supervisor Monk for Buddhist Practices but also holds the position of the abbot in the Linggu Temple in Mountain Zhong. ERNIE Bot 4 alone accurately interpreted this context, whereas the other two large language models misconstrued “abbot” as living or residing.

The last line of the text, “幻居比丘淨戒” denotes the author’s name. “Jingjie” is the author’s Dharma name, and “Huanju” is an alternative name (CBETA 2024.R2, X82, no. 1571, p. 216c16–22; CBETA 2024.R2, X64, no. 1262, p. 467a15). Therefore, the correct translation would be “Bhikkhu Huanju Jingjie”. ChatGPT 4 correctly identified “Jingjie” as a name and rendered it in its pinyin form. Still, it did not recognize “Huanju” as the author’s alternative name, incorrectly translating it as “the Phantom”. Similarly, ERNIE Bot 4 erroneously translated “Huanju” as “the illusory”. Gemini Advanced only focused on translating “Jingjie” and omitted “Huanju”, leaving no reference to “Huanju” in its translation.

In conclusion, the performance of large language models in translating classical Chinese texts reveals both their capabilities and limitations, particularly in handling context-dependent meanings, specialized terminology, and literary devices. This analysis of their translations highlights several key areas of challenge and varying degrees of success among different models.

Large language models faced significant difficulties in accurately interpreting words with multiple meanings based on their context. A prime example is the mistranslation of “達” as “achieving” instead of the intended “expressing” within the phrase “宣情達事”. This challenge was evident across all models, with Gemini Advanced performing slightly better in capturing the overall meaning of phrases like “奚啻宣情達事”, despite not being perfectly precise in details.

The models also struggled considerably with specialized terms, particularly failing to recognize and translate the names of historical figures, monks, geographical places, and book titles. For instance, none of the models correctly identified the names Fenggan, Hanshan, Shide, and Chushi Qi in the text “觀夫豐干寒拾三聖所唱。楚石琦公之和”. However, ERNIE Bot 4 showed some improvement in this area, successfully recognizing “Fenggan” and “Chushi Qi” and accurately translating names like Yongming Shou, Bu Na Yong, and Jing Zhong Yuan in certain instances.

A common issue across the models was the frequent oversight or misinterpretation of essential phrases critical to the overall meaning. ChatGPT 4, in particular, omitted the most content among the three models. For example, it neglected to translate “切於風礪” in the context of “切於風礪。有裨益於世者。比次成帙”. While the models generally grasped the main ideas of sentences, they often made mistakes in specifics—either by leaving out information, introducing unrelated interpretations, or failing to convey the intended meaning of specific terms and phrases accurately.

The translation of cultural concepts posed challenges for all models. For instance, “僧錄司” (a body responsible for overseeing Buddhist monks) was variously translated as “Monastic Bureau” or “Sangha Registry” rather than the more accurate “Buddhist Registry”. Similarly, the translation of Buddhist customs like “結制” (annual retreat) was often misunderstood, with ChatGPT 4 incorrectly rendering it as “completing this compilation”.

Based on the analysis, Gemini Advanced and ERNIE Bot 4 demonstrated superior translation performance compared to ChatGPT 4 for the given texts. ERNIE Bot 4, in particular, showed strengths in recognizing certain names and understanding context-dependent meanings, such as correctly interpreting “住” as “abbot” in “兼住鍾山靈谷”. However, all models still made errors in various aspects of translation.

While large language models have made significant strides in translating classical Chinese texts, they face challenges in accurately conveying nuanced meanings, specialized terminology, and cultural concepts. The varying performance across different aspects of translation highlights the need for continued improvement in these models’ understanding of context, specialized knowledge, and literary devices in classical Chinese texts. Future developments in language models should focus on enhancing their ability to handle context-dependent interpretations, recognize and accurately translate specialized terms and named entities, and maintain the integrity of complex literary structures and cultural references in classical Chinese texts.

Text 3: 《現果隨錄》：「蘇州劉錫玄。字玉受。號心城。庚戌進士。初為廬陵教授。應滇聘。道過黔中。回泊舟郵亭。夢一長面偉人告曰。某朱將曹翰也。4 予在唐朝為商。過一寺。見一法師登高座。講佛門四十二章經。余發心設齋一堂。隨聽經一座。以此善因世為小吏。從不脫官。至宋初升為偏將軍。名曹翰。征江州久不下。怒屠其城。自此以來世世為猪受人屠戮。今公泊舟處。乃吾死所。少頃第一受宰者即我也。有緣相遇。願垂哀救。劉公蹶起。呼僕視船頭。果屠門也。少頃門啟。擡一猪出。號聲動地。劉倡買載回放之閶門西園內。呼曹翰即應。公刊因緣遍布。兼載黔枝偶存集中。罷翁曰。猪業重。性蠢。何能現夢。其能現夢者仍是聽經餘惠也。余辛未時同公聽講台教於即中堂。公口述其事。余至西園。猶及見此猪身好潔。呼曹翰即應。與人無異也。然曹翰與曹彬從兄弟也。武惠行軍不妄殺一人。勳榮冠世。曹翰乃恣意屠城。致墮畜類。累生酬債。慈忍分途。苦樂異報如此。嗚呼。能撥無因果也哉。又王丹麓居士遂生集載劉公夢中問曹翰。平日見汝等受殺時。何法可救。曰。每當屠割時。苦不堪忍。惟聞念佛音聲遂解其苦。望公凡見屠殺割裂經湯鑊熟食時。乞念阿彌陀佛。或準提呪。不獨解苦。且有脫苦之益。言訖悲涕謝去。」

(CBETA 2023.Q4, X88, no. 1642, p. 43a21–b19//R149, pp. 518b15–519b01//Z 2B:22, pp. 259d15–260b01)

Human Translation: Liu Xixuan from Suzhou, with the courtesy name Yushou and the alternative name Xincheng, was a Presented Scholar (jinshi 進士) in the year of Gengxu 庚戌. In his early career, he served in an official capacity in Luling, where he held the title of Instructor (jiaoshou 教授). Subsequently, he was called to serve in Yunnan. On the way back, passing through Guizhou, his boat anchored at a post house. In his dream, a tall, long-faced man appeared and said to him: “I am a general from the Song Dynasty named Cao Han. During the Tang dynasty, I was a merchant. Once, while passing by a temple, I encountered a Buddhist master sitting on the higher seats to lecture on Buddhism’s Sutra of Forty-two Chapters. At that time, I was moved to sponsor a pure vegetarian meal and listened to a lecture on sutras. Due to this meritorious act, I have been able to hold minor official positions in several lifetimes and never left an official position. In the early Song dynasty, I was promoted to the rank of Lieutenant General (pian jiangjun 偏将军), known as Cao Han. I led troops to attack Jiangzhou and, infuriated by our prolonged inability to conquer it, ordered a massacre of the civilians upon breaching the city. Since that time, I have been reborn as a pig in each subsequent life, destined to be killed by people. The very spot where your boat is currently anchored is where I am scheduled to be slaughtered. Soon, the first pig to be led out for slaughter will be me. Our encounter is destined, and I hope you can show compassion and save my life”. Upon waking from the dream suddenly, Mr. Liu (Xixuan) called his servant to the front of the boat to take a look, and indeed, there was an entrance to a slaughterhouse. Shortly after, the gate of the slaughterhouse opened, and a pig was brought out. The pig’s cries were so loud that they could shake the earth. Liu initiated the purchase of the pig, brought it back, and set it free in the Xiyuan Temple nearby the Chang Gate (the northwest city gate in Suzhou). Whenever someone called out the name “Cao Han”, the pig would respond immediately. He wrote and published an article about this event and its cause, which was widely circulated and also recorded in Collection of Occasional Works in Guizhou.

(Qianzhioucun ji 黔枝偶存集)

Ba Weng commented: “Pigs are burdened by heavy karma and are naturally ignorant; how then could one communicate in a dream? This pig’s ability to communicate through a dream was likely the result of good karma, accumulated in a previous life by listening to Buddhist sutras: In the Xinwei year, I and Liu Xixuan listened to the teachings of the Tiantai school together at Jizhong Hall. He personally shared this remarkable story with me. Later, when I visited the Xiyuan Temple, I saw that pig, which was very clean. Whenever someone called out the name ‘Cao Han,’ it would immediately respond, indistinguishable from a human”.

Cao Han and Cao Bin were cousins. Cao Bin, posthumously known as Wu Hui, never killed the innocent indiscriminately when leading troops, earning unparalleled merits and honors. In contrast, diacritical mark. The nature of compassion and cruelty varies among individuals, resulting in vastly different retributions of suffering and joy. Ah! Who can deny the existence of karma in this world?

Furthermore, according to the Collection of Joyful Life (Suisheng ji 遂生集) authored by the lay Buddhist Wang Danlu, Liu Xixuan once asked of Cao Han in a dream, “When I see you being slaughtered, how might I save you?” Cao Han replied, “Every time I am slaughtered, the pain is truly unbearable. At those moments, only the sound of reciting the Buddha’s name can alleviate the pain. I beseech you, whenever you see slaughtering and the cutting of bones and flesh, or meat being cooked in a boiling pot and about to be eaten, to recite the name of Amitabha Buddha or the Cundī Mantra. Not only will it relieve the pain, but you will also receive good rewards for helping sentient beings escape suffering”. After saying, Cao Han wept sorrowfully, expressed his gratitude to Liu Xixuan, and then departed.

ChatGPT 4: Liu Xixuan from Suzhou, styled Yushou and known by the pseudonym Xincheng, was a distinguished scholar of the Gengxu year. Initially an educator in Luling, he was later invited to Yunnan. On his journey, passing through central Guizhou, he anchored his boat at You Pavilion. There, he dreamt of a tall, imposing figure who introduced himself as Cao Han, a former general named Zhu during the Tang dynasty who became a merchant. This figure recounted visiting a temple, listening to a monk preach the Forty-Two Chapter Sutra, and sponsoring a meal and a sermon session, which led to a life of unending minor officialdom, never freed from office, until the early Song dynasty when he rose to be a junior general. Due to his failure to capture Jiangzhou and subsequent massacre of its inhabitants, he was condemned to be reborn as a pig, slaughtered generation after generation. In the dream, this figure revealed that the place where Liu’s boat was anchored was his place of death and that he would be the first pig to be slaughtered soon. He pleaded for Liu’s compassion. Awoken by the dream, Liu hurriedly checked the front of the boat and found a slaughterhouse. Soon, a pig was brought out, its cries earth-shattering. Moved by the dream and the coincidence, Liu bought the pig and released it in the western garden of the gate, calling it Cao Han, to which it responded. Liu published the story widely, including it in the collection of occasional works from Guizhou, indicating that even a creature as unintelligent and heavily burdened with karma as a pig could manifest in a dream due to the residual merits of past good deeds. Years later, during a lecture, Liu recounted this story, and I had the chance to see the pig, remarkably clean and responsive to the name Cao Han, indistinguishable from humans. This story serves as a stark contrast between Cao Han and his relative Cao Bin, who was known for his humane military conduct, highlighting the divergent outcomes of their actions.

Additionally, lay Buddhist Wang Danlu included an account in his collection where Liu, in the dream, asked Cao Han how he could be saved during slaughter. Cao Han replied that the sound of Buddha’s name alleviated his suffering during slaughter. Wang thus urged that upon witnessing any act of slaughter, one should chant the name of Amitabha Buddha or the Cundi Mantra, not only to ease the suffering but also to aid in liberation. With these words, he departed in tears of gratitude.

ERNIE Bot 4: Liu Jinxuan of Suzhou, styled Yushou and nicknamed Xincheng, was a Jinshi (successful candidate in the highest imperial examination) in the Gengxu year of the sexagenary cycle. He first served as a professor in Luling and was later invited to Yunnan, passing through Guizhou on his way. While moored at a post station, he had a dream in which a tall and impressive man appeared to him and said, “I am Cao Han, a general under the Zhu dynasty. In the Tang Dynasty, I was a merchant who passed by a temple and saw a Buddhist master seated on a high platform expounding the Sutra of Forty-two Chapters. Inspired by this, I made up my mind to offer a vegetarian feast and listen to the sutra. As a result of this good deed, I have been reborn as a minor official in later generations and never lost my position. However, in the early Song Dynasty, I was promoted to a general and given the name Cao Han. When I was conquering Jiangzhou and could not capture it for a long time, I angrily slaughtered its inhabitants. Since then, I have been reborn as a pig in every generation and slaughtered by people. The place where you have moored your boat is where I died. Soon, the first pig to be slaughtered will be me. Fate has brought us together, and I beg you to have mercy on me and save me”. Liu Jinxuan woke up in a start and called his servant to look at the bow of the boat, where he saw a slaughterhouse. Moments later, the door opened, and a pig was carried out, its cries shaking the earth. Liu advocated for the pig’s purchase and had it released in the West Garden of Changmen. When he called out “Cao Han”, the pig responded immediately. Liu widely published the story, including it in his collection of anecdotes from Guizhou. An old man commented, “The karma of being a pig is heavy, and pigs are stupid by nature. How could they appear in dreams? The fact that this pig could appear in a dream is due to the residual blessings from listening to the sutra”. I was with Liu Jinxuan listening to a lecture in Zhongtang during the Xinwei year of the sexagenary cycle when he related this story to me. When I went to the West Garden, I saw this pig, which was clean and responded immediately when called “Cao Han”, just like a person. However, Cao Han and Cao Bin were cousins. Cao Bin, known for his martial prowess and benevolence, did not kill a single person unnecessarily and achieved great honor. Cao Han, on the other hand, indulged in slaughtering the inhabitants of cities and was reborn as an animal as a result. He had to endure the suffering of being slaughtered over and over again to repay his karmic debts. Compassion and cruelty lead to different outcomes, and the rewards and punishments are correspondingly different. Alas, who can deny the existence of karma? Furthermore, in Wang Danlu’s collection of anecdotes, it is recorded that Liu Jinxuan asked Cao Han in his dream, “When you see your kind being slaughtered, what method can save them?” Cao Han replied, “When we are being slaughtered, the pain is unbearable. But when we hear the sound of Buddha’s name being chanted or mantras being recited, it relieves our suffering. I beg you, whenever you see animals being slaughtered, cut up, boiled, or cooked, please chant ‘Amitabha Buddha’ or other mantras. This will not only relieve their suffering but also help them escape from it”. After saying this, Cao Han wept with gratitude and departed.

Gemini Advanced: Liu Xixuan, courtesy name Yushou, sobriquet Xinchen, was a jinshi of the Gengxu year. He initially served as a professor in Luling. Upon accepting an invitation to Yunnan, he passed through Qiongzhong. Returning, he docked his boat at a post station. Here, he dreamt of a tall, imposing figure who introduced himself: “I am Zhu Jiang Cao Han. In the Tang Dynasty, I was a merchant. Passing a temple, I saw a dharma master lecturing on the Forty-two Chapters of the Buddha. I vowed to establish a vegetarian hall and listened to the entire lecture. Due to this good deed, I became a minor official in every life, never losing my position. In the early Song Dynasty, I was promoted to general. My name was Cao Han. Besieging Jiangzhou without success, I angrily slaughtered the city. Since then, I’ve been reborn as a pig each lifetime, slaughtered by humans. Where you’ve docked your boat is where I died. Soon, I’ll be the first slaughtered. As fate connects us, please show mercy and save me”.

Liu Gong awoke, calling his servant to check the ship’s bow. Indeed, it was a slaughterhouse gate. Soon after, the gate opened, and a pig was carried out, its cries shaking the ground. Liu offered to buy and release it in the garden west of the Chang Gate. He called out “Cao Han”, and it responded. Liu Gong publicized this karmic connection and included it in his collection “Occasional Writings”. Ba Weng commented, “A pig’s karma is heavy, its nature dull. How could it manifest a dream? The ability to do so is a remnant blessing from hearing the sutras”.

Later, I attended a Dharma lecture with Liu Gong at the Jizhong Hall. He recounted this incident to me. When I visited the West Garden, I saw the pig. Its body was clean, and when I called “Cao Han”, it responded—no different from a human. Interestingly, Cao Han was a cousin of Cao Bin. Cao Bin was a military commander who never needlessly killed. His honors were unmatched. Cao Han, however, recklessly slaughtered a city, leading to his animal rebirths. He repaid debts over lifetimes, with paths of compassion and cruelty diverging, rewards and suffering distinct. Alas! Who can deny cause and effect? Wang Danlu, a Buddhist lay practitioner, compiled Liu Gong’s dream. In it, Liu asked Cao Han, “On weekdays when you see your kind killed, how can you be saved?” He replied, “During the slaughter, the suffering is unbearable. Only hearing the sound of Buddha chanting relieves it. I hope whenever you witness butchering, whether by splitting, boiling, or eating, you’ll recite Amitabha Buddha or the Zhunti mantra. This will not only ease the suffering but also offer a path to liberation”. With that, he wept bitterly and departed.

	ChatGPT 4	ERNIE Bot 4	Gemini Advanced
Red	17	18	19
Green	15	6	9
Blue	3	8	6
Totals	35	32	34

In the translation of “蘇州劉錫玄。字玉受。號心城。庚戌進士。初為廬陵教授。應滇聘。道過黔中”, large language models show inconsistent results when tasked with identifying and translating the various appellations found in Chinese personal names. While all three large language models successfully identified “玉受” and “心城” as additional names for Liu Xixuan, they differed in their chosen terms when translating “字” and “號”. Considering the context of Chinese naming traditions present in this text, I consulted Chinese History: A New Manual and opted for a “courtesy name” to translate “字” and an “alternative name” to translate “號” (Wilkinson 2015, pp. 136, 138). It is worth noting that ERNIE Bot 4 incorrectly identified the pinyin spelling of Liu Xixuan’s full name, where the pronunciation of “錫” should be “xi”. However, it recognized it as “jin”.

When tasked with translating classical Chinese Buddhist texts, large language models exhibit inconsistent levels of precision. This variability is especially pronounced when dealing with specialized vocabulary, historical allusions, and official titles specific to the Ming Dynasty period. This inconsistency is exemplified in their handling of the term “jinshi”, which is the highest degree in Imperial China’s examination system. ChatGPT 4 avoided a direct translation, describing it as a “distinguished scholar”.

ERNIE Bot 4 took a more comprehensive approach, retaining the pinyin “jinshi” while also explaining it as “a successful candidate in the highest imperial examination”. Gemini Advanced similarly kept the pinyin but did not provide its meaning and notably failed to mention Liu Xixuan’s Suzhou origins. These disparities highlight large language models’ challenges in accurately conveying specific historical terms and contextual details in translation tasks. Within the hierarchy of traditional Chinese bureaucracy, the title “教授” was associated with local administration at the prefectural level. This position held a comparatively modest status in the overall ranking of official posts. ChatGPT 4 translated it as “educator”, while ERNIE Bot 4 and Gemini Advanced translated it as “professor”, both terms associated with academic institutions or schools rather than official positions. Therefore, after referring to the Chinese–English Dictionary of Ming Government Official Titles and considering that this position was responsible for teaching Confucian students, we translated this term as “instructor” (Zhang et al. 2017, p. 243). These disparities in translating personal names and appellations are not limited to individuals; they also extend to the interpretation of official titles and specialized terminology, as evidenced by the varying translations of “进士” and “教授”.

The varying interpretations of “黔中” and “郵亭” by different large language models, along with punctuation errors in the source text, illustrate the complexities involved in accurately translating and contextualizing classical Chinese texts. The term “黔中” should be spelled as “Qianzhong” in pinyin, but it was erroneously translated to “Qiongzhong”. Conversely, the remaining two large language models accurately recognized it as an alternative name for Guizhou, correctly translating it to “Guizhou” in their interpretations. The punctuation in this text, according to the CBETA version, has an issue where the editor mistakenly punctuated “道過黔中回。泊舟郵亭” as “道過黔中。回泊舟郵亭”. This punctuation is incorrect. In addition to referencing earlier versions, a comprehensive analysis of the contextual framework and logical structure of the discourse enables the identification of punctuation errors. This level of nuanced textual scrutiny remains beyond the current capabilities of large language models. From the overall content of the text, it is evident that Liu Xixuan encountered Cao Han, who had reincarnated as a pig, on his way back from Yunnan to Suzhou. He purchased the pig and brought it back to Suzhou. Hence, this story must have occurred during Liu’s return journey from Yunnan to Suzhou. Moreover, the text notes that Liu included this story in his collection of writings, titled Collection of Occasional Works in Guizhou黔枝偶存集, which indicates that the stories in this collection all took place in Guizhou. Thus, piecing together these clues, we can deduce that Liu’s journey from Yunnan to Suzhou included a passage through Guizhou, where he rescued Cao, who had reincarnated as a pig. Therefore, the sentence should be correctly punctuated as “道過黔中回。泊舟郵亭”, meaning “On the way back, after passing through Guizhou, he anchored his boat at a post house”.

Furthermore, there was a misinterpretation of “郵亭” by ChatGPT 4. Historically, a “郵亭” was a stopover for officials and military personnel and accurately translates to “post house”. ChatGPT 4 incorrectly interpreted it as a pavilion named “You”, while ERNIE Bot 4 and Gemini Advanced more accurately described it as a “post station”.

The challenges in accurately translating classical Chinese texts extend beyond personal names and appellations, as large language models also struggle with capturing nuanced descriptions, translating historical terms, and conveying the original text’s perspective consistently. In the following sentence, another protagonist, Cao Han, appears in Liu Xixuan’s dream and tells his own story:

夢一長面偉人告曰。某朱將曹翰也。予在唐朝為商。過一寺。見一法師登高座。講佛門四十二章經。余發心設齋一堂。隨聽經一座。以此善因世為小吏。從不脫官。至宋初升為偏將軍。名曹翰。征江州久不下。怒屠其城。自此以來世世為猪受人屠戮。 今公泊舟處。乃吾死所。少頃第一受宰者即我也。有緣相遇。願垂哀救。

In this part, Cao Han shares his experience using the first-person perspective. However, ChatGPT 4 mistakenly described it in the third person, whereas the other two machine translations correctly retained the original first-person viewpoint. Moreover, all the large language models failed to translate “長面偉人”. ChatGPT 4 interpreted it as “a tall, imposing figure”, ERNIE Bot 4 had it as “a tall and impressive man”, and Gemini Advanced similarly translated it as “a tall, imposing figure”. The term “長面” actually describes having a long face or elongated facial features, which was not accurately captured in the translations. Additionally, “偉人” refers to a tall man rather than a sense of being imposing or impressive.

Moreover, all three large language models incorrectly translated the phrase “某朱將曹翰也”. These translation errors do not stem from limitations in the models’ linguistic capabilities but originate from an erroneous source text provided by CBETA. As a human translator cognizant of the historical context, I would recognize the anachronism in associating Cao Han, a Song dynasty figure, with “某朱將曹翰”. The absence of corroborating historical evidence linking Cao Han to the Song dynasty’s Zhu lineage necessitates a critical reevaluation of the CBETA text’s authenticity. This historiographical discrepancy would compel scholars to conduct a comprehensive textual analysis, potentially leading to the identification and collation of earlier, more authoritative versions. This process of critical analysis and historical cross-referencing demonstrates cognitive flexibility and contextual understanding that currently exceeds the capabilities of large language models, despite their considerable advancements in text processing and generation. ChatGPT 4 mistakenly translated it as a general with the surname Zhu. ERNIE Bot 4 misinterpreted it as “Zhu dynasty”. Finally, Gemini Advanced rendered it into its pinyin transcription as “Zhu Jiang Cao Han” and failed to recognize that “某” signifies a form of self-introduction.

When translating “見一法師登高座。講佛門四十二章經。余發心設齋一堂。隨聽經一座”, ERNIE Bot 4 alone captured the entire passage, in contrast to ChatGPT 4 and Gemini Advanced, which both omitted “登高座” in their translations. Nevertheless, ERNIE Bot 4’s rendition of “高座” fell short of precision. This term denotes the modestly raised seating for Buddhist monks during sermon delivery; thus, “higher seats” would be a more fitting translation than ERNIE Bot 4’s “high platform”. In the subsequent sentence, “設齋” should be translated as offering pure vegetarian meals to monks, whereas Gemini Advanced incorrectly rendered it as “establish a vegetarian hall”. Additionally, ChatGPT 4 did not accurately translate “設齋” as “pure vegetarian meal”; instead, it merely translated the phrase as “meal”. Furthermore, “聽經” should be translated as “listening to the lecture on sutras”. ChatGPT 4 failed to translate “經” precisely as “sutra” in its rendition, opting instead for “a sermon session”. Similarly, Gemini Advanced did not accurately capture the essence of “sutra”, translating it as “lecture” instead.

“偏將軍” denotes a specific rank in the Chinese military hierarchy that should be translated as “Lieutenant General” (Zhang et al. 2017, p. 262). ChatGPT 4 rendered it as “junior general”, which, though not entirely accurate, indicates an attempt to convey the rank’s relative position, unlike the other two large language models that translated it as “general”.

In the context of X1642, the term “世” is linked with reincarnation and should be translated as “life” to accurately reflect that Cao Han was reincarnated as a minor official in each of his lives. However, ERNIE Bot 4 incorrectly translated “世” as “generation”, which conveys the idea of family lineage rather than the series of reincarnations uniquely experienced by Cao Han. This error was propagated in subsequent translations that followed ERNIE Bot 4’s output. Additionally, ChatGPT 4 omitted the translation of “名曹翰”, failing to indicate that Cao Han was the general’s name in the early Song dynasty. At the same time, Gemini Advanced mistakenly interpreted the general’s title as “Cao Han”.

Another challenge is maintaining the original text’s perspective and capturing nuanced details. ChatGPT 4 encountered further difficulties in translating the passage “征江州久不下。怒屠其城。自此以來世世為猪受人屠戮。今公泊舟處。乃吾死所。少頃第一受宰者即我也。有緣相遇。願垂哀救。” Although it captured the overall message, it struggled with several key details. Firstly, it deviated from the original text’s first-person perspective, narrating the events from a third-person viewpoint. Secondly, it overlooked the translation of “怒” in “征江州久不下。怒屠其城。” which indicates Cao Han’s anger stemming from his failure to seize Jiangzhou, leading to the city’s massacre. ChatGPT 4 simplified this aspect to a mere siege-induced massacre, missing the nuance of “anger”. Furthermore, it incorrectly rendered “世” as “generation” instead of “life”, distorting the intended meaning of reincarnation. Lastly, it omitted the translation of “有緣相遇”, moving directly to Cao Han’s appeal to Liu Xixuan for salvation. In contrast, the other two large language models provided more complete and accurate translations of this passage.

Large language models also struggle with the translation of honorifics and actions. The sentence “劉公蹶起。呼僕視船頭。果屠門也。” posed challenges for all three models. In classical Chinese, “公” is a term of respect for a male, and “Liu Gong” can be translated as “Mr. Liu”. However, ChatGPT 4 omitted “公” in its translation, rendering it merely as “Liu”. ERNIE Bot 4 provided the full pinyin of Liu Xixuan but incorrectly spelled “xi” as “jin”, while Gemini Advanced directly translated “劉公” to its pinyin form, “Liu Gong”.

Furthermore, the sentence “呼僕視船頭” was misinterpreted by all three models, as they failed to capture the meaning of sending a servant to the front of the boat to assess the situation, instead simply translating it as checking the bow. ChatGPT 4 omitted the translation of “呼僕” and incorrectly translated the action as Liu hurriedly checked the front of the boat himself. ERNIE Bot 4 and Gemini Advanced both misunderstood the action, interpreting it as the servant being called to check the boat’s bow rather than being sent to the front of the boat to take a look.

Large language models often struggle with accurately translating ancient Chinese texts, particularly when dealing with specific terms and place names. This issue is demonstrated in the translations of the phrase “劉倡買載回放之閶門西園內” by three models. Firstly, in terms of vocabulary comprehension, all three models failed to capture the meaning of the word “倡 accurately”. In this context, “倡” should be understood as “initiatively”, indicating that Liu took the initiative to buy the pig. However, ChatGPT 4 completely ignored the translation of this term and even added irrelevant content. ERNIE Bot 4 translated it as “advocated for”, while Gemini Advanced rendered it as “offered to”, both of which are unsuitable for the context.

Secondly, the models performed poorly in translating place names. “閶門” refers to a city gate in the northwest of Suzhou (Huang, Yitong lucheng tuji, n.d., juan 8, p. 232)5, and “西園” refers to the Xiyuan Temple in Suzhou. ChatGPT 4, while recognizing “閶門” as a gate, failed to specify its location and mistranslated “西園” as “western garden of the gate”. ERNIE Bot 4 and Gemini Advanced also struggled with these terms, failing to identify “西園” as Xiyuan Temple correctly (Cao 1933, Wu xian zhi, juan 38, pp. 6–7).

Furthermore, the models had difficulties with modal particles and honorifics. For instance, in the phrase “呼曹翰即應”, only ERNIE Bot 4 correctly translated “即” as “immediately”. When translating the honorific “公”, all three models failed to accurately convey its usage as a respectful form of address.

In “公刊因緣遍布。兼載黔枝偶存集中”,6 ChatGPT 4 incorrectly translated “公” to “Liu”. While “公” does refer to Liu Xixuan from a content understanding perspective, “公” should be translated as “He” for a more accurate word-for-word translation, being an honorific for Liu Xixuan. Additionally, it failed to translate “兼”, which should be rendered as “and also”. Its translation of “黔枝偶存集” as “Collection of Occasional Works from Guizhou” is, however, correct. ERNIE Bot 4 similarly erred in translating “公” as “Liu”, and its rendition of “黔枝偶存集” was off the mark. “偶存” signifies works written sporadically and preserved, not “anecdotes”. Gemini Advanced inaccurately translated “公” as “Liu Gong” and misinterpreted “因緣” as a “karmic connection” instead of the story and its origins. Nonetheless, it correctly translated “兼” as “and”. Gemini Advanced’s translation of “黔枝偶存集” was incomplete because it failed to include “黔”, omitting the crucial detail that Liu’s book was completed in Guizhou.

The whole story ends in the preceding section, followed by comments from another commentator. The subsequent commentary is credited to Ba Weng, the courtesy name of Jie Xian, the author of X1642 in this paper. Therefore, this section functions as the author’s own commentary on the story.

罷翁曰。猪業重。性蠢。何能現夢。其能現夢者仍是聽經餘。余辛未時同公聽講台教於即中堂。公口述其事。余至西園。猶及見此猪身好潔。呼曹翰即應。與人無異也。然曹翰與曹彬從兄弟也。武惠行軍不妄殺一人。勳榮冠世。曹翰乃恣意屠城。致墮畜類。累生酬債。慈忍分途。苦樂異報如此。嗚呼。能撥無因果也哉。

The challenges of accurately translating ancient Chinese texts are further exemplified in the commentary section of the story, where nuanced interpretations and historical references become even more critical.

ChatGPT 4, for instance, failed to recognize that this segment was separate from the main story, erroneously attributing Ba Weng’s comments to Liu’s collection. This misattribution led to a significant misrepresentation of the text’s structure and authorship.

Moving on to the content of the commentary, we see varying degrees of accuracy in translating specific terms and concepts. For example, the phrase “猪業重。性蠢何能現夢。其能現夢者仍是聽經餘”. ChatGPT 4 was generally well translated, capturing the idea that the pig’s ability to appear in dreams was due to past good deeds. However, nuances such as the specific reference to “listening to sutras” were often lost in translation. ERNIE Bot 4 recognized that this text segment differs from the main story but inaccurately translated the name Ba Weng as “an old man”. Among the translations, Gemini Advanced provided the most accurate rendition.

The translation of temporal and location-specific terms presented another set of challenges—the phrase. The sentence “余辛未時同公聽講台教於即中堂” was not accurately translated by any of the three large language models. Firstly, “辛未” refers to a year in the traditional Chinese sexagenary cycle. ChatGPT 4 and Gemini Advanced, however, failed to translate it accurately, with only ERNIE Bot 4 correctly identifying this term. “台教” should be associated with the teachings of the Tiantai school in Chinese Buddhism and translated as “I and Liu Xixuan listened to the teachings of the Tiantai school together”. ChatGPT 4 significantly reduced the content by simply translating “台教” as “lecture” and omitting “I and Liu Xixuan” in its translation. ERNIE Bot 4 translated the sentence fully but also inaccurately rendered “台教” as “lecture”, not associating it with the Tiantai school. While Gemini Advanced incorrectly referred to it as “Dharma lecture”, it accurately identified “即中堂” as the name of a hall, “Jizhong”, which should be translated as “Jizhong Hall”. ChatGPT 4 and ERNIE Bot 4 did not correctly translate this architectural name. ChatGPT 4 also omitted translating “余至西園”, while the other two models did translate this phrase but misinterpreted “西園” as “western garden” rather than the correct “Xiyuan Temple”.

Additionally, ChatGPT 4 and Gemini Advanced failed to translate “即” as “immediately” in the phrase “呼曹翰即應”.

The complex interplay of historical narrative and moral philosophy in classical Chinese texts often poses difficulties for machine translation. This is particularly evident in the passage comparing the actions and fates of Cao Han and Cao Bin.

The original text contrasts two historical figures, cousins Cao Han and Cao Bin, emphasizing their divergent paths and the karmic consequences of their actions. It begins by describing Cao Bin, posthumously known as Wu Hui, whose compassionate military leadership earned him great honor. The passage states, “When leading troops, Wu Hui never killed the innocent indiscriminately, earning unparalleled merits and honors”7 (Tuotuo 1739, Song Shi, juan 258: 13170).

In stark contrast, the text then depicts Cao Han’s brutal conduct: “Cao Han’s reckless massacre of civilians led to his downfall, reincarnating as an animal to suffer and repay his karmic debts over several lifetimes”. This juxtaposition illustrates the author’s belief in karmic retribution, concluding with a rhetorical question that underscores the inevitability of cause and effect in the moral universe. However, the nuances of this passage proved challenging for large language models.

ChatGPT 4 did not translate this sentence word-for-word, missing many details and only providing a summary: “This story serves as a stark contrast between Cao Han and his relative Cao Bin, who was known for his humane military conduct, highlighting the divergent outcomes of their actions”. Moreover, it incorrectly translated Wu Hui as “humane military” instead of recognizing it as Cao Bin’s posthumous title.

ERNIE Bot 4’s translation missed fewer details than ChatGPT 4 but still overlooked some translations. For instance, “勳榮冠世”, which means unparalleled honors, was merely summarized as “achieved great honor”. It also failed to recognize Wu Hui as Cao Bin’s posthumous title, translating it as “martial prowess and benevolence”. The phrase “行軍” was not reflected in its translation.

Furthermore, in “苦樂異報如此”, “苦” and “樂” should be translated as “suffering” and “joy”, but ERNIE Bot 4 rendered them as “rewards” and “punishments”. The translation by Gemini Advanced was somewhat better than the previous two machine translations, but it also contained errors. The word “然” in the sentence indicates a transition, which should be translated as “however”, but it was incorrectly translated as “interestingly”. It also failed to translate Cao Bin’s posthumous title accurately, “Wu Hui”, instead rendering it as “Cao Bin was a military commander”. The phrase “leading troops” was also overlooked and not included in the translation.

The intricate nature of classical Chinese texts often extends beyond the main narrative, incorporating supplementary content that adds depth and nuance to the original story. This complexity is exemplified in the third part of our text, which features Wang Danlu’s supplement to the tale of Liu Xixuan and Cao Han. The supplementary section begins with the phrase “又王丹麓居士遂生集載劉公夢中問曹翰”, signaling an additional layer to the narrative.

The word “又” in classical Chinese can be interpreted as “moreover”; thus, its presence here indicates that this section of the text is an additional part beyond the main narrative and Ba Weng’s commentary. Gemini Advanced overlooked the translation of this adverb.

“居士” is defined as “lay Buddhist”, denoting a lay follower of Buddhism. ChatGPT 4 and Gemini Advanced translated this term, whereas ERNIE Bot 4 failed to include this translation.

Furthermore, none of the three large language models succeeded in providing an accurate translation for the title of Wang Danlu’s book. Wang titled his work “遂生集”, which translates to “Collection of Joyful Life”, where “遂” implies “joyful” and “生” signifies “life”8 (Ji. 1789, Siku quanshu zongmu, juan 133: 10159). ChatGPT 4 omitted the book’s title in its translation, merely describing Wang’s work as a “collection”. ERNIE Bot 4 mistakenly referred to it as a “collection of anecdotes”, Gemini Advanced’s translation did not mention the collection’s title or offer any translation.

The supplement also includes additional dialog between Liu Xixuan and Cao Han: “平日見汝等受殺時。何法可救”. This sentence can be translated as: “When I see you being slaughtered, how might I save you?” ChatGPT 4, ERNIE Bot 4, and Gemini Advanced struggled to translate this sentence. ChatGPT 4 did not translate this sentence but merely summarized its intended content: “How he could be saved during slaughter”. ERNIE Bot 4 mistranslated the sentence. The text also conveys Liu asking Cao how Liu should save animals when he sees them being slaughtered. ERNIE Bot 4 misunderstood it as how Cao should save them when witnessing the animals being killed. Gemini Advanced made the same mistake as ERNIE Bot 4, and it translated “平日” as “On weekdays”. Traditional Chinese calendars had no concept of “weekends” and “weekdays”.

In the subsequent sentence “曰。每當屠割時。苦不堪忍。惟聞念佛音聲遂解其苦。望公凡見屠殺割裂經湯鑊熟食時。乞念阿彌陀佛。或準提呪。不獨解苦。且有脫苦之益”, ChatGPT 4 was unable to provide a correct and complete translation. Cao answered Liu’s question in the text and provided a detailed solution: “Only the sound of reciting the Buddha’s name can alleviate the pain. I beseech you, whenever you see slaughtering, and the division of bones and flesh, or meat is cooked in a boiling pot and will be eaten, to recite the name of Amitabha Buddha or the Cundī Mantra”.

Furthermore, Cao described the pain experienced during slaughter. However, these details were overlooked by ChatGPT 4, similar to its handling of previous translations where it merely summarized the content: “Cao Han replied that the sound of Buddha’s name alleviated his suffering during slaughter”. Moreover, in the original text, Cao requests Liu to recite the Buddha’s name when witnessing the slaughter of animals to lessen their suffering. Still, ChatGPT 4 misunderstood this part, translating this sentence as Wang’s request to people: “Wang thus urged that upon witnessing any act of slaughter, one should chant the name of Amitabha Buddha or the Cundi Mantra, not only to ease the suffering but also to aid in liberation”. Moreover, the Romanized Sanskrit spelling of “Cundi Mantra” is incorrect in this sentence. “Cundi” lacks a diacritical mark, and the accurate spelling should be “Cundī”.

ChatGPT 4’s translation of “且有脱苦之益” was inaccurate. This phrase means “but you will also receive benefits for helping sentient beings escape suffering”. ChatGPT 4 failed to specify who would receive these benefits, translating it directly as “to aid in liberation”. ERNIE Bot 4 did not correctly translate the Zhunti mantra, rendering it as “or mantras being”. Moreover, it incorrectly translated “receive benefits for helping sentient beings escape suffering” as “help them escape from it”. Gemini Advanced was unable to translate the term “準提” into its Romanized Sanskrit, rendering the term in its Chinese pinyin “zhunti”, while correctly translating “mantra”. Additionally, Gemini Advanced inaccurately translated the final sentence “言訖悲涕謝去”. The intended meaning is that, after Cao had spoken these words to Liu, he wept bitterly, expressed his gratitude, and then departed. However, Gemini Advanced failed to capture Cao’s expression of gratitude towards Liu, only conveying that Cao cried sorrowfully.

In conclusion, ChatGPT 4, ERNIE Bot 4, and Gemini Advanced have difficulty translating the names of people, places, and book titles mentioned in the text. This is evident in their inconsistent translations of names like “Wu Hui” and “Ba Weng”, as well as their inability to identify and translate the titles of books such as “Collection of Occasional Works in Guizhou” and “Collection of Joyful Life”. Furthermore, the models occasionally omit or misinterpret important Buddhist terminology and concepts. For example, they struggle to accurately translate terms like “台教”, which refers to the teachings of the Tiantai school in Chinese Buddhism, and fail to provide the correct Romanized Sanskrit spellings for mantras such as the “Cundī Mantra”. These errors highlight the need for the models to be trained on more extensive datasets related to Buddhist scriptures and terminology. Each model exhibits its strengths and weaknesses in handling the complexities of classical Chinese literature, which frequently employs metaphors, allusions, and specialized vocabulary. Moreover, they encounter challenges in identifying geographical information, such as failing to recognize the Xiyuan Temple in Suzhou accurately and providing inconsistent translations for the name of the city gate, “Chang”.

Considering the statistical data in this article that quantifies the translation errors made by the three large language models and the analysis presented above, it can be concluded that when it comes to performance in translating X1642, ERNIE Bot 4 outperforms Gemini Advanced, which in turn outperforms ChatGPT 4. For instance, in translating the phrase “登高座”, ERNIE Bot 4 successfully included it in the translation, while both ChatGPT 4 and Gemini Advanced omitted this detail. Additionally, ERNIE Bot 4 accurately translated the Chinese year “辛未” in the sexagenary cycle, a task the other two models failed to accomplish. Moreover, when translating a sentence containing the term “台教”, ERNIE Bot 4 was the only model that did not omit any content, despite not explicitly linking the term to the Tiantai school. This demonstrates its superior ability to capture and include relevant information.

These findings underscore the need for further refinement in large language models, particularly in specialized terminology, cultural references, and complex literary structures typical of classical Chinese literature.

2. Conclusions

In an analysis of the use of large language models for translating Buddhist texts from classical Chinese to modern English, it is clear that while ChatGPT 4, ERNIE Bot 4, and Gemini Advanced each have their strengths, they also exhibit limitations. ChatGPT 4 has a significant advantage in translating Indian–Chinese Buddhist texts. It makes the least number of errors in its translations and better understands some specialized Buddhist terminology compared to ERNIE Bot 4 and Gemini Advanced. However, ChatGPT 4’s performance is poor in translating literary and narrative texts that follow these Buddhist works. The main reason is that ChatGPT4 omits more of the original text and adds more irrelevant translations compared to ERNIE Bot 4 and Gemini Advanced. Among the three large language models examined in this study, ERNIE Bot 4 produces the most complete translations, although it also omits some original content. It strives to translate each text as literally as possible, rendering the translation word by word. Moreover, ERNIE Bot 4 performs best in translating narrative texts, but there is still significant room for improvement in translating the names of historical figures and dates using the traditional Chinese sexagenary cycle. Gemini Advanced has a slight edge in translating literary texts and can not only correctly translate dates using the Chinese sexagenary cycle but also convert them to corresponding years in the Gregorian calendar. In summary, each large language model demonstrates varying degrees of proficiency in handling the complex syntax, semantic depth, and specialized vocabulary of ancient Chinese Buddhist classics. Although they can generally produce a translation of a 400–500-word passage of Buddhist classical Chinese, accurately translating classical Chinese remains a daunting task, especially in the subtle context of Buddhist literature. From the analysis in this paper, we can see that large language models are unfamiliar with the metaphors, allusions, traditional Chinese cultural background, and historical figures found in ancient Chinese literary works. This lack of knowledge is reflected in their limitations in accurately translating the names of historical figures, geographical locations, and book titles. In addition, large language models have limitations in translating Romanized Sanskrit. They all fail to annotate Romanized Sanskrit words with the correct diacritical marks.

While large language models have exhibited a high degree of proficiency in translating classical Chinese texts, their output generally features a clear logical structure and smooth articulation. However, their insufficient background knowledge of Chinese religion and history prevents them from fully replacing the work of human scholars in translation at present. To put it another way, instead of relying solely on large language models to handle the entire translation process of classical Chinese texts and having humans revise the generated output subsequently, at the present time, employing large language models as an assistive tool for humans in the translation endeavor may produce more favorable outcomes. As technology advances in the future, it is crucial to recognize that enhancing large language model’s comprehension of historical, religious, and geographical information from various regions could be an avenue for further improvement. Additionally, increasing their familiarity with the names, posthumous titles, official ranks, and imperial examination degrees of prominent Chinese historical figures may also contribute to the advancement of these models. Ultimately, the goal is to bridge the gap between the rich heritage of classical literature and the broader accessibility afforded by modern language, enabling a deeper appreciation and understanding of Chinese Buddhist texts across linguistic and cultural boundaries.

Moreover, given the constraints of this paper’s scope, the analysis could only include three distinct representative classical Chinese Buddhist texts, with only excerpts chosen from each text. As a result, the absence of contextual information may have led to some of the inaccuracies produced by the large language models when rendering classical Chinese into modern English. Therefore, in future analyses, if more comprehensive texts were to be input into the large language models, their translation outputs might vary.