Knowledge-Driven Generative Design of Role-Playing Game Scenarios

Abstract

The paper addresses the problem of the generative creation of role-playing game scenarios based on a knowledge compendium. The purpose of this exploratory research was to determine the impact of elements of the generation process on the quality of the scenario. The research was conducted in a generative pipeline using large language models and a compendium that describes the world of the game. The scope of the study includes the creation of a modular system that enables ablation studies and the analysis of the influence of individual factors on the quality of the results. The experiments involved comparisons between language models, variants of knowledge compendia, and the count of user prompt steps. In addition, an ablation study, a self-bias study and a small-scale study with human respondents were conducted. The main purpose of these additional studies was to examine the methods used and identify potential problems regarding them. The ablation study supported the significance of creating a scenario skeleton in a non-random way. No indesputible self-bias was found. The human-based study showed that the LLM evaluators are, on average, less critical than the human ones, but share some similar scoring patterns. The study demonstrated statistically significant differences resulting from the choice of language model in Relevance, Coherence, Informativeness, Interactivity and Structure criteria, as well as the influence of the size of the compendium and the count of user prompt steps on the quality of the results. It was discovered that in the process of generating role-playing game scenarios, it might be beneficial to use short, non-randomly filled structures as the basis for the output scenario generation. It was found that large language models tend to score the generated scenarios higher than human respondents. There is, however, an overlap in preferences regarding the generation model between the human and the machine evaluators.

1. Introduction

In the latter half of the 20th century, Role-Playing Games (RPGs) emerged as a form of interactive entertainment centered on collaborative storytelling. Participants enact fictional characters within imagined or real-world settings, typically under the guidance of a Game Master (GM), who is responsible for describing the game world and preparing narrative frameworks for play. One common mode of play is the campaign format, understood as a continuous narrative unfolding over multiple sessions within a coherent fictional world. Research in clinical psychology [1,2], developmental psychology [3,4,5], sociology [6], and social psychology [7] indicates that RPG participation may have a positive impact on social skills, self-esteem, literacy, and collaborative competencies. These benefits highlight the practical relevance of tools that support the preparation and facilitation of RPG content. Despite this potential, the preparation of RPG scenarios remains time-consuming and conceptually complex for GMs, which motivates the development of tools that can support this process.

Recent advances in Generative Artificial Intelligence (GenAI) and Large Language Models (LLMs) have substantially expanded the scope of automatic text generation, enabling the production of long-form, coherent narratives that maintain semantic consistency and contextual grounding. Beyond literary applications, LLMs are increasingly used to generate functional narrative artifacts—texts intended to be interpreted, adapted, or acted upon by human users. Examples include educational scenarios, interactive fiction, simulation narratives, and RPG scenarios. In such contexts, textual quality cannot be evaluated solely in terms of fluency or stylistic creativity, but must also account for structure, relevance, and usability. This aligns with current research trends in the applications of deep learning to creative industries, where the challenge lies in balancing autonomy with user-centric control.

RPG scenarios constitute a distinct class of narrative artifacts. Rather than presenting a closed story, a scenario functions as a structured framework that supports interactive storytelling, player agency, and improvisation. A typical scenario combines multiple layers of information, including plot hooks, locations, non-player characters (NPCs), motivations, conflicts, and potential player interactions. These elements are commonly organized in a semi-structured form that balances explicit guidance with narrative openness. Consequently, the usefulness of a scenario depends not only on linguistic quality but also on its structural clarity, thematic coherence, and suitability for facilitating gameplay. From the perspective of automated text generation, RPG scenarios pose specific challenges. Generated content must remain coherent across multiple narrative components, respect established world constraints, and provide sufficient informational density without overwhelming the user. Excessive narrative complexity may hinder practical usability, while overly simplistic structures may limit engagement. Generating high-quality RPG scenarios, therefore, requires careful control over narrative structure, content selection, and the level of detail provided.

Maintaining narrative consistency and contextual relevance in automatically generated scenarios typically requires grounding in external knowledge. In practice, this grounding can be provided through a knowledge compendium that formalizes the fictional world, including its locations, characters, factions, and narrative constraints. Such a compendium constrains generation, reduces the risk of contradictions, and enables the reuse of world-specific entities across narrative components. At the same time, the size and structure of the compendium may influence the balance between creative freedom and informational overload, potentially affecting scenario quality.

Despite the growing availability of tools for narrative content creation, the automatic generation of RPG scenarios remains largely unsupported by systematic, domain-aware evaluation. Existing tools commonly focus on generating isolated elements using random tables or procedural rules [8,9], or on producing locations [10], items [11], characters [12], or entire worlds [13]. While effective for inspiration, such tools provide limited support for maintaining narrative coherence and contextual consistency across scenarios or campaigns.

Similarly, recent applications of LLMs to RPG content generation are often driven by ad hoc prompt engineering, offering limited transparency and reproducibility. To the best of the authors’ knowledge, no prior study has systematically examined how design choices—such as prompt structure, knowledge grounding, and model selection—interact to influence the quality and usability of generated scenarios within a modular pipeline. This gap limits the practical adoption of generative AI in RPG campaign design workflows, where consistency, relevance, and structural clarity are critical.

Therefore, the objective of this study is to investigate how selected design choices in LLM-based RPG scenario generation influence the quality and practical usability of the resulting narrative artifacts from a game master–oriented perspective. Rather than proposing a new generative model or a fully automated storytelling system, the paper focuses on analyzing how prompt structure, external knowledge integration, and model choice interact within a modular generative pipeline intended to support human-led scenario preparation. To this end, a dedicated scenario generation pipeline was designed to enable controlled experimentation and ablation studies, supporting the systematic evaluation of narrative quality across criteria relevant to RPG campaign design and gameplay.

The main contributions of this work are as follows:

1.

The design of a modular LLM-based pipeline for RPG scenario generation that enables systematic analysis of generation factors.

2.

An empirical evaluation of the influence of prompt step count, knowledge compendium variants, and LLM choice on multiple qualitative dimensions of RPG scenarios.

3.

Empirical evidence that compact, non-randomly structured narrative skeletons can lead to more coherent and practically usable generated scenarios.

4.

An ablation study assessing the impact of algorithmic selection of narrative elements on scenario quality.

The experiments led to the conclusion that shorter prompt structures outperformed longer ones across most quality parameters and that model choice can have a statistically significant impact on scenario quality. Furthermore, it was found that LLM-based evaluators tend to assign higher scores than human judges.

2. Related Work

2.1. From Procedural Content Generation to Generative AI in Games: Evolution of Narrative Systems

Research on automated content generation in games has transformed from rule-based procedural systems [14] to large language model-driven approaches [15]. This evolution resulted in a fundamental shift in how narrative artefacts are created and deployed in applied settings [16]. This is particularly relevant for role-playing game (RPG) scenario design, where generated content must support collaborative interpretation rather than algorithmic execution. In this section, role-playing game (RPG) denotes its original form as a social, narrative practice in which participants collaboratively interpret a fictional world through description, rules, and shared imagination. Such play is typically mediated by a game master. Computer role-playing games (cRPGs) are digital games inspired by selected elements of this practice, where narrative progression and mechanics are executed by a computational system rather than interpreted collaboratively. In cRPGs, content serves as an executable script, whereas in RPGs, it functions as an interpretive artefact. This distinction is critical for research on content generation, as methods developed for cRPGs might not fully translate to the open-ended, human-mediated nature of RPGs.

Early and influential work in this area focuses on procedural content generation (PCG) as a means of automating the creation of game worlds, quests, levels, and narrative structures. Early procedural approaches framed scenario generation as action planning, where sequences of actions with defined preconditions and effects are determined through backward reasoning from a goal state [17]. Stochastic goal and action selection introduced emergent diversity, while formal entity representations enabled natural language realization. Hendrikx et al. (2013) [14] provide a comprehensive survey of PCG techniques in games, identifying types of generated game content, including storyboards and stories, and proposing a taxonomy of techniques used for these content types, such as constraint satisfaction, planning, and artificial neural networks. Although primarily oriented toward digital games, this taxonomy formalized key trade-offs between authorial control, variability, and computational tractability. Hafis et al. [18] review PCG techniques used in video games between 2014 and 2018, observing that only a small fraction address cRPGs, and even fewer focus explicitly on story or narrative generation, which resisted clean categorization within the proposed taxonomy. This suggests that the field remained methodologically underdeveloped prior to the LLMs era.

More recent surveys document a fundamental methodological shift driven by the emergence of LLMs, particularly the Generative Pre-trained Transformers (GPT) family. Yang et al. [19] present a scoping review of 55 academic publications examining the use of GPT models in games research up to 2023. They identify five dominant application areas: procedural content generation, mixed-initiative game design, mixed-initiative gameplay, autonomous game playing, and game user research. Within PCG, the review shows emphasis on text-centric content, such as dialogue, quests, and narrative continuations, reflecting GPT’s strengths in natural language generation (NLG). However, applications beyond story generation—such as level, scene, or character generation—remain comparatively limited and technically challenging, often requiring structured representations, planning capabilities, or integration with non-textual game systems. An updated and substantially expanded review by Yang et al. [20] extends this analysis to 177 papers, including 122 published in 2024 alone. Despite rapid growth, the same five application categories remain dominant, suggesting conceptual consolidation rather than diversification. The review emphasizes persistent implementation challenges related to long-term planning, consistency, hallucination, and evaluation of player experience. Importantly, both reviews stress that much of the existing work remains exploratory, with limited integration into complete game systems and sparse empirical validation involving players or designers.

Taken together, these surveys trace a shift from early algorithmic PCG approaches toward data-driven, language-based generation methods, while underscoring persistent challenges related to long-term coherence, authorial control, and meaningful human–AI collaboration—challenges that are particularly salient in role-playing games. While the majority of reviewed studies focus on digital games, the identified methods and limitations directly inform RPG scenario generation research. The persistent issue of hallucinations and the lack of robust grounding mechanisms identified in these surveys underscore a critical need for structured approaches. In particular, the gap between exploratory demonstrations and production-ready systems, persistent issues with long-horizon narrative coherence, limited empirical analysis of design parameters, and the absence of systematic evaluation practices motivate the present study’s focus on controlled experimentation in knowledge-grounded scenario generation, enabling systematic analysis of factors affecting practical usability.

2.2. LLM-Based Narrative Systems: Capabilities and Deployment Barriers

The application of large language models to narrative generation has been explored across multiple domains, revealing both capabilities and persistent limitations relevant for RPG scenario creation. Empirical studies demonstrate that while LLMs can store relational knowledge within their parameters [21], their capacity to manipulate this information precisely in knowledge-intensive tasks remains limited [22]. Yu et al. [23] survey knowledge-enhanced text generation approaches, noting that reliance on parametric memory alone often leads to factual inconsistencies and discourse incoherence in long-form generation. Guan et al. [24] evaluate long-text generation capabilities, identifying challenges in maintaining long-range commonsense and discourse relations. These capabilities are essential for RPG scenarios, where narratives must adhere to the constraints of a specific, often fictional, world-state.

Attempts to formalize narrative generation mechanisms inspired by RPG practices further illustrate these constraints. Ono and Ogata [25] propose an integrated narrative generation system modelled on scenario preparation in RPGs, with particular emphasis on world settings as narrative constraints. Although computational in nature, the authors explicitly note the difficulty of automating scene development driven by player decisions. This limitation reflects a core property of RPGs: Narrative progression depends on human interpretation and improvisation, which cannot be fully specified in advance or reliably automated.

One approach to managing narrative complexity is story sifting [26], which identifies meaningful subsets of events within computer-simulated emergent narratives. The principal advantage of this method lies in its capacity to systematically extract narratively significant patterns from extensive data, effectively refining raw event logs into coherent story candidates. While such simulations enable the formal description of resulting event sequences [26], the generated structures often lack readability for human users. To address this limitation, hybrid architectures have been proposed that separate structural narrative generation (via story sifting) from surface realization performed by an LLM [27]. This separation of structure and realization is echoed in research on emotional arcs, which draws on narratological theory and treats them as organizing principles for narrative progression. Wen et al. [28] propose a framework in which emotional trajectories, such as rise and fall, guide the generation of branching story structures. While their implementation targets a computer game, the underlying idea of structuring content around emotional pacing resonates with long-standing practices in RPG design, where tension, escalation, and release are central to sustaining engagement across sessions. At the same time, the reliance on automated control of narrative progression and difficulty highlights a limitation from the perspective of RPGs: Emotional coherence cannot be enforced procedurally but must be negotiated and enacted by human participants during play.

Beyond academic research, several commercial and open-source applications demonstrate practical deployment of LLM-based narrative generation. AI Dungeon [29,30], developed since 2019, enables users to engage in text-based role-playing experiences in which the system acts as a game master, describing the fictional world and responding to player actions. The application relies on LLMs for response generation, using both role-playing-specific models such as Hermes-3 [31] and general-purpose models from the Mistral family [32]. Notably, AI Dungeon generates conversational play transcripts rather than structured scenarios designed for traditional tabletop RPGs, illustrating the distinction between real-time interactive generation and preparatory content creation. This distinction is critical: Whereas such systems generate live conversational output, RPG scenario preparation requires stable, inspectable artefacts that support planning, modification, and reuse by the game master across sessions. Other tools focus more narrowly on supporting scenario preparation. The AI-Powered Game Master Tools suite [33] generates individual scenario elements such as location descriptions, items, and adversaries based on user prompts. Particularly noteworthy is the adversary generator, which produces parameters grounded in specific rule systems [34], demonstrating system-aware content generation. Similarly, Sudowrite [35], while not designed specifically for RPGs, supports creative fiction writing through narrative suggestion and may be adapted for scenario authoring.

2.3. Quest and Scenario Generation: cRPG Methods and Transfer Limits to RPG Preparation

Methods developed for cRPGs—such as procedural quest generation, narrative planning, and constraint-based content synthesis—provide an important methodological foundation and point of comparison for the present study, although so far, as Lopez wrote in [36]:

The emerging narrative that appears by system in the ARPGs [RPGs] sessions becomes a unique narrative artifact, which stimulates the imagination of the players within a collaborative context and creative freedom. The DRPGs [cRPGs] have failed, despite all the technological advances, to emulate this gaming experience.

Several studies specifically address quest and scenario generation in cRPG-like systems. Works by Lima et al. [37,38] explore automatic quest generation using planning, genetic algorithms and graph-based representations, emphasizing logical coherence and goal satisfaction. Similar approaches are discussed by Prins et al. [39] and Breault et al. [40], who model quests as structured sequences of objectives represented by desires or goals, constrained by world state and player actions. These systems demonstrate the primary advantage of procedural quest generation and narrative planning: Algorithmic control over narrative structure can significantly improve logical consistency and reduce nonsensical outcomes compared to purely random generators.

Research by Balint and Bidarra [41] extends this line of work by integrating semantic constraints and spatiotemporal location descriptions into narrative generation pipelines. Buongiorno et al. [42] focuses on Non-Player Characters’ (NPCs) personality modelling and domain constraints in dialogue-based interactions between the player and NPC. Griffith [43] focuses on NPC traits representing personality and behaviour to create emotionally engaging NPCs. These approaches highlight the main advantage of constraint-based content synthesis: the ability to balance variability with directed narrative progression by relying on explicit state representations and predefined action schemas. From a methodological perspective, they illustrate how narrative coherence can be operationalized through formal constraints and controlled generation processes.

Recent advancements in Large Language Models have prompted a paradigm shift in Procedural Content Generation (PCG), bridging the gap between rigid rule-based systems and highly flexible, open-ended narrative generation. A comprehensive survey in [16] highlights that while traditional PCG methods ensure mechanical stability, integrating LLMs offers unprecedented dynamic storytelling capabilities. However, governing these capabilities remains a significant challenge. To address structural invalidity in purely generative outputs, recent studies, such as the schema-governed pipeline proposed in [44], demonstrate that constraining LLMs with explicitly structured knowledge schemas significantly improves narrative reliability and structural consistency in RPG environments. Furthermore, research specifically targeting tabletop RPG dynamics has explored how models balance authored scenarios with player autonomy. For instance, the evaluation of the ChatRPG system by [45] emphasizes the use of redirection strategies to maintain narrative adherence without diminishing player agency. Similarly, studies testing LLM agents as Dungeon Masters in Dungeons & Dragons [46] highlight the intricate relationship between prompt engineering, logical reasoning, and the model’s capacity to maintain a coherent, open-ended game state. These recent developments align with the premise that while LLMs offer vast generative potential, their successful application in RPG scenario preparation requires careful structural grounding.

Emergent narrative is widely recognized as a defining characteristic of role-playing games. Mumper [47] examines this phenomenon through the design and playtesting of an adventure for MÖRK BORG, applying concepts from narratology and ludology. The study demonstrates how procedural elements such as random encounter tables, travel mechanics, and environmental constraints can support narrative emergence without prescribing a fixed plot. A key finding of this study is the importance of deliberately leaving parts of the fictional world underdetermined to preserve interpretive flexibility and player agency. Excessive detail or rigid scripting risks constraining interpretation and reducing the role of the game master to that of a narrative executor rather than a participant in play.

From a literary and media studies perspective, Gryka-Zawadzka [48] analyzes RPG rulebooks and scenarios as texts situated between usability and literariness. The study emphasizes that such texts are not narratives in themselves, but tools designed to facilitate narrative events that occur during play. The narrative proper emerges only through the interaction between players and the game master, forming a triadic relationship between rulebook, scenario, and session. For content generation, this implies a fundamental limitation: Generated materials must prioritize clarity, flexibility, and interpretive affordances over narrative completeness or authorial control.

The distribution of narrative agency between players and the game master further conditions how generated content functions in practice. Svan and Wuolo [49] compare Dungeons & Dragons and Blades in the Dark, showing how differences in rules and procedures shape narrative dynamics. Blades in the Dark structurally positions the game master as reacting to player-declared intentions, while Dungeons & Dragons allows for both this kind of storytelling as well as players reacting primarily to situations framed by the game master, as in the case presented by Svan and Wuolo. These findings indicate a significant limitation for generic content generation approaches: Material that functions effectively in one RPG system may be ill-suited to another, as the same content will be interpreted and operationalized differently depending on how authority and agency are distributed.

Despite their technical sophistication, a fundamental disadvantage of applying procedural quest generation, narrative planning, and constraint-based content synthesis to tabletop RPGs is their underlying assumption of automated processing. cRPG-oriented generation systems are designed to enforce constraints automatically, which conflicts with the nature of RPG scenarios that function as human-mediated interpretive artifacts. Consequently, techniques such as exhaustive state tracking, fine-grained action schemas, or fully specified narrative graphs do not transfer directly to RPG practice. Their main disadvantage in this context is that overly complex or rigidly specified outputs may reduce usability, overwhelm the game master, or constrain player agency rather than support it. Nevertheless, these works remain highly relevant for RPG scenario generation for two reasons. First, they articulate formal solutions to problems of coherence, narrative structure, and constraint satisfaction that RPG designers routinely face in an implicit or ad hoc manner. Second, they expose the limitations of highly detailed, system-driven generation, thereby motivating alternative approaches that favor compact structure, selective grounding, and human interpretive flexibility. In this sense, research on cRPG content generation provides both a methodological foundation and a critical contrast that motivates approaches that favor compact structural representations, selective grounding with constraints that support emergent narrative while preserving the interpretative roles of players and the game master, rather than exhaustive state tracking or fully specified narrative graphs. The present study builds on these insights while explicitly targeting scenario generation as a preparatory, human-facing artifact rather than an executable narrative system.

2.4. Knowledge Grounding for Consistent Scenario Generation

The deployment of LLMs in RPGs faces the challenge of hallucinations and a lack of long-range semantic consistency, which is further amplified in long-text modeling [24]. These limitations motivate the use of explicit grounding mechanisms that externalize world knowledge and narrative constraints, shifting the burden of consistency from parametric memory to structured context representations. While LLMs demonstrate a remarkable ability to store relational knowledge within their parameters [21], their capacity to precisely manipulate this factual information in knowledge-intensive tasks remains limited [22].

To address these limitations, recent research has shifted towards knowledge-enhanced text generation, which incorporates external knowledge sources—such as knowledge bases or knowledge graphs—into the generative process [23]. The integration of Retrieval-Augmented Generation (RAG) frameworks has proven effective in increasing the fact-adherence of model outputs, ensuring that generated claims can be referenced against authoritative sources [22,50]. The roadmap for unifying LLMs and Knowledge Graphs suggests that such structured models can enhance LLMs by providing explicit external knowledge for inference and interpretability [51]. This is particularly critical for functional narrative artifacts, where text usability depends on its structural clarity and thematic adherence to a predefined setting.

The development of believable agents within these narratives requires a sophisticated architecture for memory and reflection. This architectural shift is exemplified by generative agents that utilize an architecture of observation, planning, and reflection to maintain long-term behavioral consistency. Park et al. [52] demonstrated that extending an LLM with a mechanism to store, retrieve, and reflect upon a complete record of an agent’s experiences enables the simulation of complex, emergent social behaviors. Such agent-based motivations are essential for RPG scenarios, where Non-Player Characters (NPCs) must act consistently within the narrative framework.

Role-playing game worlds are often highly elaborated, as evidenced by the size and complexity of many published sourcebooks, some of which exceed several hundred pages [53]. Typically, information contained in such rulebooks is written in natural language and thus primarily suited for human readers. However, to enable computational processing for automated scenario generation, this knowledge must be formalized into structured representations such as databases or graphs. This formalization requirement represents a key implementation challenge: converting narrative-rich, often ambiguous world descriptions into machine-readable formats while preserving the semantic richness necessary for meaningful scenario generation. By treating the RPG world-state as a structured framework, generative systems can condition their predictions on the details of the local environment, including objects, characters and their past actions [54], which can be thought of as a knowledge compendium. The compendium acts as a structured context that enforces thematic consistency and captures complex relational knowledge that LLMs cannot manipulate through weights alone [21,51] and allows for avoiding pitfalls related to deviations from the world-state that purely parametric memory often leads to [23].

Graph structures offer an effective representation of complex RPG worlds. Knowledge graphs have been used to define NPC desires, resources, and locations, enabling the generation of mission dialogues for cRPGs [55]. Annotated relations linking specific elements complement the ontology of the represented world, defining, among other things, the mutual positions of entities and the desires of individual NPCs. Alternatively, event-based graphs focus on temporal relations between occurrences [56]. However, for RPG scenario generation, temporal coherence of historical events is often less important than thematic and situational consistency, necessitating a compendium structure that prioritizes entity relationships and motivations. This domain-specific requirement distinguishes RPG content generation from general-purpose knowledge-grounded text generation and motivates the design choices explored in the present study.

Despite these advancements, there is a lack of systematic analysis regarding how the scale and structure of the grounding information—specifically, the size of the knowledge compendium, understood as the total number of elements it includes, and the number of elements retrieved during generation—influence the quality of the generated scenario. While it is established that calling external APIs or consulting external knowledge improves groundedness [50], the trade-off between creative freedom and informational overload remains under-explored in the domain of automated narrative artifact generation. This study addresses this gap by investigating how selected design choices in knowledge integration and prompt structuring interact to influence the practical usability of the resulting narrative artifacts.

2.5. Evaluation of Generated Narrative Artifacts

Evaluating the quality of generated narrative content remains one of the most challenging problems in NLG, particularly in creative and open-ended domains such as RPG scenarios. Unlike tasks with a well-defined objective function or a limited space of valid outputs, narrative generation admits a wide range of plausible, stylistically diverse, and semantically distinct realizations for the same prompt. This fundamental characteristic necessitates a comparative analysis of evaluation approaches, as methods developed for constrained generation tasks such as machine translation or summarization are often poorly aligned with human judgments of narrative quality, coherence, or usefulness in interactive settings. The evaluation of creative, unformalized text generation involves selecting appropriate metrics and quality criteria used to compare different generation pipelines or algorithms. The perception of creative text can vary among readers depending on individual preferences. However, it is possible to model the subjective assessment of text by larger groups of people across selected categories, such as coherence, engagement, and fluency. Using such metrics enables comparison of the quality of human-written versus machine-generated text, as well as comparison among different generated texts. Solutions requiring surveying extensive groups of readers are, however, described in the context of such studies as time-consuming and costly [57]. In light of the development of LLMs, research is being conducted on their capabilities for evaluating texts generated by other models [58,59,60,61]. This enables automation and acceleration of research, which can be significant in the case of large datasets.

Early automatic evaluation metrics, including BLEU [62], ROUGE [63], and METEOR [64], rely on surface-level overlap between generated text and one or more reference texts. While these metrics have proven useful in constrained generation tasks, their applicability to narrative generation is severely limited by the one-to-many nature of storytelling, where high-quality outputs may share little lexical or syntactic similarity with any reference. Numerous studies have demonstrated that such reference-based metrics correlate weakly with human judgments in creative generation tasks, particularly for longer texts and story-like structures [65,66].

In response to these limitations, a growing body of work has explored semantic and learned evaluation metrics that move beyond exact token matching. Metrics based on contextual embeddings, such as BERTScore [67] and BARTScore [68], measure similarity in a learned semantic space and have been shown to achieve higher correlation with human judgments across several NLG tasks. However, these approaches still fundamentally depend on reference texts and therefore inherit many of the same structural limitations when applied to open-ended narrative generation, where references are sparse, noisy, or inherently incomplete.

To address the reference-dependence problem more directly, several works propose reference-free or weakly supervised evaluation methods tailored to open-ended text. UNION [69], for example, trains a discriminator to distinguish human-written stories from systematically corrupted variants, capturing narrative errors such as repetition, logical inconsistency, and long-range incoherence. While such approaches show improved robustness and correlation with human judgments in story generation tasks, they require task-specific training data and assumptions about the types of errors likely to occur, which may not generalize across domains or generation paradigms. In the present work, this method was not directly employed, as certain evaluation categories used therein were deemed inadequate for RPG scenarios, while other essential categories—specifically Structure and Interactivity—were absent. Nonetheless, given the promising results described in [69], the evaluation presented in this article was based on the approach proposed therein.

More recently, LLMs have been explored not only as generators but also as evaluators of generated content. Several studies demonstrate that LLMs, when prompted appropriately, can produce quality judgments that correlate well with expert human evaluation in tasks including open-ended story generation [59]. Fu et al. propose a solution that leverages the emergent properties of GPT for automatic text evaluation [58]. The authors posited that, given the complexity of GPT models, they would be capable of evaluating texts across a broad spectrum of categories without requiring prior adaptation for such evaluations. Evaluation aspects included coherence, comprehensibility, capacity to engage the reader, depth, and textual interest. Studies showed that, based on Kendall’s absolute correlation between scores generated by different automatic methods and those assigned by humans, LLM responses are on average weakly correlated with human judgments in the context of evaluating individual stories. However, for evaluating entire systems, the average result indicates a strong correlation. Moreover, the impact of prompt structure on the degree of evaluation correlation was also tested. It was observed that simple prompts—without requests for elaboration, examples, or detailed evaluation criteria—provide the most stable results.

Frameworks such as G-EVAL [70] and GPTScore [58] leverage instruction following, chain-of-thought reasoning, and form-based scoring to perform multi-dimensional, customizable evaluations without task-specific training. These methods offer substantial practical advantages, including flexibility, scalability, and reduced cost compared to human evaluation, making them particularly attractive for iterative development and large-scale experimentation. Chiang and Lee [59] further demonstrate that LLM-based evaluation can serve as a viable alternative to human evaluation in certain contexts, achieving reasonable agreement with expert judgments while offering improved reproducibility and cost efficiency. At the same time, the use of LLMs as evaluators introduces new challenges. Several works caution that model-based evaluators may exhibit biases toward texts generated by similar models, may be sensitive to prompt formulation, and may fail to reliably assess aspects such as factual correctness, emotional impact, or long-term narrative consistency [59,66]. These concerns are especially salient in creative domains, where evaluation criteria are subjective, culturally situated, and difficult to formalize.

Beyond pointwise quality assessment, recent research emphasizes the importance of evaluating diversity alongside quality in creative generation systems. Bradley et al. [61] propose a quality–diversity framework in which language models provide feedback not only on the overall quality of generated texts but also on their diversity along specified dimensions. While such approaches demonstrate promising alignment between AI-based and human judgments, they also highlight risks of reward hacking and underspecification of evaluation axes, reinforcing the need for careful experimental design and human oversight.

Human evaluation, therefore, remains a central component of narrative generation research. Best practice guidelines emphasize the importance of clearly defined evaluation criteria, appropriate study design, and transparency in reporting [57]. Although costly and time-consuming, human judgments are currently indispensable for validating automatic metrics, uncovering qualitative failure modes, and assessing the practical usefulness of generated narratives in real-world applications. Surveys of evaluation practices consistently recommend combining multiple evaluation methods—automatic metrics, model-based evaluators, and human studies—to obtain a more reliable and nuanced picture of system performance [65,66]. Deriu et al. [71] provide a comprehensive review of evaluation methods for dialogue systems, highlighting similar challenges in assessing interactive and conversational AI, where context-dependence and user experience are critical.

In summary, evaluation of generated narrative content remains an open research problem, particularly in creative and interactive domains such as RPGs. Existing approaches exhibit trade-offs between scalability, interpretability, and alignment with human judgment. Consequently, recent work increasingly adopts mixed evaluation strategies that balance automatic measures with model-based and human evaluation, reflecting a broader shift toward methodological pluralism in narrative generation research.

Across the reviewed literature, a recurring limitation is the lack of systematic analysis of how concrete design parameters in generative pipelines affect the usability of narrative artifacts for human practitioners. While prior work explores planning depth, world modeling, knowledge integration, and narrative control largely in isolation, there is limited empirical evidence on how these factors interact in the context of scenario preparation for RPGs. In particular, the effects of prompt structure complexity, the scale and density of grounding information, and the use of compact versus elaborated narrative skeletons remain underexplored. The present study directly addresses this gap by experimentally evaluating how variations in prompt step count, knowledge compendium size, and structural design influence scenario quality across multiple user-relevant dimensions.

3. Materials and Methods

3.1. Game World Compendium Structure

The world compendia used in these experiments were modelled as a directed graph of entities and relations representing a single game world. This representation is based on the approach proposed in [55]. It is, however, extended by introducing entity descriptions to provide more context for prompt generation. Moreover, the distinction between Non-Player Characters and Creatures, introduced in [55], was eliminated, as even zoomorphic entities were assumed to possess goals and agency. Furthermore, the Resources category was generalized to the Objects category.

The compendium acts as the initial input of the pipeline and contains two primary data types:

• Entity: component of type: character, object, location, or event;
• Relation: directed connection between entities $(v_i,v_j)$, where $v_i$ denotes the source entity and $v_j$ the target entity.

The entities of type character serve as the foundation for defining motivations and strategies, which, in turn, shape the skeleton of the generated scenario. These strategies determine the narrative structure and the sequence of actions undertaken. The object entity type provides NPCs with the ability to perform actions on objects as well as requiring player characters to do so as part of the narrative. The location entity type provides essential spatial and contextual anchors for other world components such as objects, events, and interactions. Entities of type event are not used directly to populate the scenario skeleton. Instead, they are included only when related to other entities within the scenario. Consequently, events constitute the smallest entity group across all compendium variants.

Formally, the compendium is defined as a labelled directed graph $c=(V,E)$ where each vertex $v\in V$ is described as $v=(n_v,d_v,t_v)$ with $n_v$ denoting the name of the entity, $d_v$ the description of the entity, and $t_v$ its type. Each edge $e\in E$ is defined as $e=(v_i,v_j,n_e,d_e)$ where $n_e$ is the name of the relation and $d_e$ its description.

3.2. Generation Pipeline Design

The scenario generation is executed through a processing pipeline composed of:

• Skeletal scenario generation
• Skeletal scenario completion
• Scenario text generation

The pipeline was designed with a modular architecture to facilitate experimental flexibility, allowing the assessment of the contribution of its elements to the overall quality and coherence.

3.2.1. Skeletal Scenario Generation

The skeletal scenario generation is based on NPC motivations and strategies derived from an analysis of quests in cRPGs by Doran and Parberry [72], which are assumed to be transferable to RPG settings. The set of possible motivations M is defined as the motivations m introduced in [72], as summarized in

Table 1. NPC motivations as defined by Doran and Parberry [72] along with operationalized descriptions.

Motivation	Description	Operationalized Description
Knowledge	Information known to a character	Wants to gather information
Comfort	Physical comfort	Wants to have physical comfort
Reputation	How others perceive a character	Wants to be perceived well by others
Serenity	Peace of mind	Wants a peace of mind
Protection	Security against threats	Wants security against threats
Conquest	Desire to prevail over enemies	Wants to prevail over enemies
Wealth	Economic power	Wants economic power
Ability	Character skills	Wants to use their skills
Equipment	Usable assets	Wants physical assets

. In order to operationalize these motivations, they are represented in the framework as a textual description of characters’ desires. For each such $m\in M$, multiple strategies $s\in S_m$ exist that allow the fulfilment of this motivation. Each such strategy $s=(a_1,a_2,\dots ,a_n)$ is an ordered list of actions $a_i\in A$. The list of possible actions and templates for their textual descriptions are presented in

Table 2. Actions used in strategy definitions along with associated textual template and effect polarity. Note that the list of actions is derived from Doran and Parberry’s [72] definition of strategies and not the atomic action list.

Action	Template	Effect
capture	Capture {character}	Negative
damage	Damage {object}	Negative
defend	Defend {character}	Positive
escort	Escort {character} to {location}	Positive
exchange	Exchange {object} with {character}	Neutral
experiment	Experiment with {object}	Neutral
get	Obtain {object}	Neutral
give	Give {object} to {character}	Positive
goto	Go to {location}	Neutral
kill	Kill {character}	Negative
listen	Listen to {character}	Neutral
repair	Repair {object}	Positive
report	Report to {character}	Positive
spy	Spy on {character}	Negative
steal	Steal {object} from {character}	Negative
take	Take {object} from {character}	Neutral
use	Use {object}	Neutral

. Due to the extensive size of the lists of strategies proposed by Doran and Parberry, we refer the reader to [72].

Each scenario originates from a quest giver character—g, who is selected uniformly at random from the set of characters in the compendium. Due to the wide variety of possible entities in RPGs, it is necessary to estimate the probability of specific motivations and corresponding strategies for a given quest giver. Quest giver’s natural language-based description $d_g$ is used to select the motivation m and the strategy s. The operationalized motivation description (Table 1) can be compared against $d_g$. Having selected the motivation, the descriptions of strategies (see [72]) that correspond to this particular motivation are compared against $d_g$. The mutual comparison is estimated based on semantic similarity within a textual embedding space, where linguistic units are represented as numerical vectors preserving semantic relationships. To create the embeddings, the nomic-embed-text v1.5 [73] model was used. To measure the proximity of the embeddings cosine similarity method was employed. While widely used in text similarity research [74,75,76], this metric is susceptible to the hubness phenomenon [77]—where certain vectors appear artificially close to many others—and stylistic bias [78], which can obscure semantic diversity [61]. To mitigate these issues, the values were first standardized using the Z-score transformation. This step highlights the motivations and strategies that are relatively stronger for a given NPC, reducing the influence of common or stylistically uniform patterns. Finally, the scores were normalized using min–max scaling and clipped at a threshold $\epsilon =0.01$ to prevent zero weights, as formulated in Equation (1):

$${\widehat{z}}_{i,g}^{\left(t\right)}=max\left({\displaystyle \frac{z_{i,g}^{\left(t\right)}-{min}_j{z}_{j,g}^{\left(t\right)}}{{max}_j{z}_{j,g}^{\left(t\right)}-{min}_j{z}_{j,g}^{\left(t\right)}}},\epsilon \right),t\in \{t_m,t_s\},i=1,\dots ,n_t$$

(1)

where the superscript $t\in \{t_m,t_s\}$ acts as a categorical indicator for the motivation and strategy selection stages, respectively. The index i represents a candidate within the corresponding set, and g represents the quest giver. The number of candidates $n_t$ is determined by the stage: $n_{t_m}=\left|M\right|$ is the total number of motivations, while $n_{t_s}=\left|S_m\right|$ is the number of strategies available for the selected motivation m. Normalized scores ${\widehat{z}}_{i,g}^{\left(t\right)}$ were then used as weights to compute the probability of selecting a specific motivation or strategy, as shown in Equation (2):

$$P_g^{\left(m\right)}\left(i\right)={\displaystyle \frac{{\widehat{z}}_{i,g}^{\left(m\right)}}{{\sum }_{j=1}^{\left|M\right|}{\widehat{z}}_{j,g}^{\left(m\right)}}},P_{g,m}^{\left(s\right)}\left(k\right)={\displaystyle \frac{{\widehat{z}}_{k,g}^{\left(s\right)}}{{\sum }_{j=1}^{|S_m|}{\widehat{z}}_{j,g}^{\left(s\right)}}}$$

(2)

where $S_m$ represents the set of available strategies associated with motivation m. Once probabilities are calculated, the quest giver’s motivation $m_g$ and strategy $s_g$ are randomly selected according to those probabilities. The ordered set of actions associated with $s_g$ is added to the list of scenario actions $A_g$. If the number of required plot points defined by the prompt variant (see

Table 3. Overview of prompt variants and compendium characteristics.

Prompt Variants
Variant		Narrative Points		Description
$p_s$		4		Short
$p_m$		8		Medium
$p_l$		12		Long
Compendium Variants
Compendium	Characters	Events	Locations	Objects	Relations
$c_s$	5	1	3	1	10
$c_m$	6	4	6	4	20
$c_l$	18	9	12	11	50

) is not reached, the next strategy is drawn from the set excluding already selected strategies for the given $m_g$, and the list of actions is concatenated with previous strategies. If all strategies in $S_m$ have already been used, the set $S_m$ is reset to its full state, allowing strategies to be reused. This is repeated until the desired length is reached, with excess actions truncated.

The output of this stage of the pipeline is a skeletal scenario, consisting of a quest giver g, their motivation $m_q$, strategies, and a derivative series of actions $A_g$ with generic subjects.

3.2.2. Skeletal Scenario Completion

Following motivation and strategy assignment, the quest giver g is associated with an action sequence $A_g=(a_1,a_2,\dots ,a_n)$, where each action $a_i$ corresponds to an abstract operation listed in Table 2. The actions set, derived from [72], was augmented with natural language templates and effect descriptors specifying the semantic impact of each action on its target entity, enabling grammatically valid scenario text generation. To instantiate the scenario skeleton, each action template must be populated with specific entities of the required type $t\in \{\mathrm{character},\mathrm{object},\mathrm{location}\}$. Entity selection proceeds in two stages.

First, priority candidates are identified from entities directly connected to the quest giver g via any relation e in the compendium c.

For each candidate entity v, if its connecting relation description $d_e$ achieves cosine similarity threshold ${\tau }_c$ with the action’s effect descriptor, the entity is retained. The average value of the score is dependent on the compendium used. In this research, a constant value, residing in the middle of the positive scope, was chosen. This decision was made to avoid curating it towards any specific compendium. In practice, it is advised to select the threshold in an experimental process, as it controls the fraction of the descriptions to be treated with priority and therefore the sensitivity of this pipeline stage.

If no candidates meet this threshold, all entities of the required type t are considered, regardless of their relation to the quest giver g.

Final entity selection maximizes cosine similarity between two text embeddings:

• Candidate representation: The candidate entity description $d_v$, optionally prepended with its relation description $d_e$ if connected to g
• Context representation: The concatenation of the quest giver’s g description $d_g$ and the action textual template

To ensure narrative diversity, previously selected entities are excluded from subsequent selections unless no alternatives remain. The result is a structured scenario skeleton with actions populated by entity names $n_v$, ready for natural language generation.

3.2.3. Scenario Text Generation

Following the generation and completion of the skeletal scenario, two prompts are generated that allow for obtaining its natural language representation through the use of LLM. The system prompt defines the role and knowledge state of the model. The user prompt specifies the task the model is expected to perform. The generator combines scenario skeleton N, entity set E, and relation set R, serializing them into textual forms $E_t$ and $R_t$. The entity set and relation sets are serialized to textual form and concatenated with the system prompt, while the skeletal scenario is concatenated with the user prompt.

• System prompt

  You are an RPG gamemaster creating scenarios for tabletop RPG games. Base your work strictly on the provided context. Only use the listed entities and connections, and preserve their integrity and logic.

• User prompt

  Write an RPG scenario, realising the following structure.

3.3. Experimental Setup

This study aims to assess the quality of the generation variants with a research pipeline consisting of scenario generation under different compendium sizes and entity distribution, number of narrative points, and the language model used. The evaluation of the scenarios is performed under the set of qualitative criteria listed in

Table 4. Evaluation categories. The symbol ✓ indicates the presence of a given category in the respective publication or that it was added by the authors.

Category	[60]	[58]	Author	Description
Relevance	✓	✓		Does the scenario address the given prompt and appropriately incorporate the provided context?
Coherence	✓	✓		Are the events and character actions logical, causally consistent, and internally coherent?
Complexity	✓			Does the scenario feature a complex narrative structure, including multi-layered plots and interdependent subplots?
Informativeness		✓		Does the scenario provide sufficiently detailed and comprehensive information about the narrative and game world?
Interactivity			✓	Does the scenario offer players meaningful choices and opportunities for interaction that influence the course of events?
Structure			✓	Does the scenario follow an organizational structure appropriate for RPG scenarios, including clearly defined sections (e.g., locations, characters, items, and plot points)?

.

The set of compendia $C=\{c_s,c_m,c_l\}$ used in the experiments was composed of three compendium variants of increasing scale and complexity: small $c_s$, medium $c_m$, and large $c_l$, with entity and relation counts and distributions shown in Table 3. In all versions, characters are the largest group of entities, reflecting their central role in the generative pipeline.

Scenarios with different amounts of narrative steps were created. Table 3 presents the grouping of user prompts by the number of narrative points, denoted as $p\in P=\{p_s,p_m,p_l\}$.

3.3.1. Scenario Generation

The models presented in

Table 5. Sizes and release dates of the selected models.

Language Models
Model	Release Date	Size
Mistral: 7B	27 September 2023	4.1 GB
LLaMA3.1: 8B	23 July 2024	4.9 GB
StableBeluga: 7B	21 July 2023	4.1 GB
Gemma3: 4B	12 March 2025	3.3 GB

were selected as generators and evaluators for the experiment based on [58,60]. All selected models employed a decoder-only transformer architecture and fell within a similar parameter range from 4B to 8B parameters to control the size-related effects [58,60]. LLaMA models trained on datasets including literary fiction have demonstrated capability in generating short-form narratives (500–1000 words) with quality approaching human-written texts, with larger variants achieving scores comparable to human authors [60]. Stable Beluga exhibits system-level correlations with human judges approaching inter-human agreement [60], while maintaining high auto-consistency as measured by intraclass correlation coefficients [58]. These models support evaluation of multiple quality dimensions through natural language prompts without additional fine-tuning [58]. The inclusion of Gemma, trained with different instruction-tuning paradigms, was decided upon to increase the robustness of the evaluation. To ensure reproducibility of the experiments, the generation seed was set to correspond with the ordinal number of each scenario within its category.

3.3.2. Scenario Evaluation

The full dataset used in [60] for narrative text generation comprised 384 generated stories. Since the evaluation framework in this study was inspired by that work, maintaining a comparable dataset size was considered beneficial. Therefore, 40 scenarios were generated for each of the $4\times 3\times 3=36$ unique experimental configurations. Consequently, each prompt-length and compendium-size variant included 480 scenarios, while each model variant included 360.

During the evaluation stage, the generated scenarios were assessed by an automated procedure, relying mostly on LLMs. An additional small survey with human reviewers was conducted to compare the scores Section 4.2. This approach is inspired by research on overall [61,69] and category-specific [58,60] text quality assessment. Evaluation categories, chosen in this research, are summarised in Table 4.

Every generated scenario was evaluated twice by each verification model. For each category, a separate prompt was created, requesting an assessment of the scenario on a 0–5 scale. The scale is inspired by the Likert scale [79], expanded with a 0 to indicate an error state. Additionally, each prompt included the scenario content and the respective user prompt.

• Evaluation prompt

  You are an expert RPG scenario reviewer. Evaluate the following scenario based on the category:

  category with descriptions

  Provide a score from 0.0 to 5.0 and a one-sentence explanation. Respond ONLY in the following format:

  Score: number

  Note: brief explanation

  The following scenario was generated in response to this user prompt:

  Prompt for scenario generation:

  Write an RPG scenario, using the following structure:

  scenario_prompt

  Scenario: scenario

Most of the relevant criteria were selected among those proposed in [58,60]. In the approach proposed in these sources, it is possible to evaluate generated texts across custom quality dimensions, thus allowing the evaluation framework to be extended with domain-specific criteria. Two categories, Interactivity and Structure, specific to the RPG scenarios, were added, as shown in Table 4. The Interactivity category reflects the centrality of player agency in tabletop RPGs. Svan and Wuolo [49] identify player agency as a central factor for the quality of an RPG session, while [25] claims that RPG scenarios form through exchanges of interactions. The Structure category was chosen because RPGs are not only literary but also functional texts. Scenarios and other supporting materials for RPGs contain structured information on the history, geography and culture of the game world, often making them structurally similar to manuals or atlases [48]. Therefore, structural ease of use and understanding of the contained materials could benefit scenario utility, a defining characteristic of RPG systems, shaping how generated content must be structured to remain usable across different playstyles. Ono and Ogata [25] similarly identify player-driven scene development as a core property of RPG scenarios that resists full automation, precisely because meaningful choices cannot be specified in advance. Interactivity therefore operationalizes playability in the sense of GM usability: It assesses whether the generated scenario provides sufficient decision points and narrative affordances to support improvisation and player agency during actual play, rather than presenting a fixed, unidirectional plot.

Additionally, Table 4 lists all of the operational definitions, structured as questions and passed to the evaluation prompt. The purpose of adding these questions was to expand the set of category names and provide more context for the evaluation models, similarly to [58]. The resulting scores represent the degree to which the evaluator’s answer to the question is positive in the modified Likert scale.

Determining the judgment anchors was left to the LLM interpretation because, according to [60], explicitly providing the scoring anchors in the prompt results in worse correlation with human responses.

To assess the quality of the generated scenarios in comparison with other creative texts, three widely used evaluation metrics were applied: Bilingual Evaluation Understudy (BLEU) [62], Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [63], and Bidirectional Encoder Representations from Transformers (BERTScore) [67].

While BLEU and ROUGE were included for completeness, their applicability in this study is limited. These metrics were originally designed for evaluating text translations rather than creatively generated narratives based on structural templates, and their correlation with human judgments of scenario quality is weak [58]. In contrast, the BERT-based metric provides a more reliable measure in this context, as it evaluates semantic similarity between texts rather than surface-level overlap.

4. Results

The set of generated scenarios denoted by $\Sigma$ was created from the set of parameters $\Xi$. The set was defined as the Cartesian product of the compendium variants C, $\left|C\right|=3$, prompt-length variants P, $\left|P\right|=3$, LLM variants L, $\left|L\right|=4$ and Random Number Generator (RNG) seed variants $\mathcal{Z},\left|\mathcal{Z}\right|=40$. Z was equal to the index of each scenario, starting from 1 up to 40. Different RNG values were used to simulate the individual preferences of human evaluators. As calculated in Equation (3), 1440 scenarios were generated. Note that the parameter set $\Xi$ was also used in the ablation study.

$$|\Xi |=|C\times P\times M\times \mathcal{Z}|=1440$$

(3)

Let R be the set of criteria, ${\mathcal{Z}}^{\prime }=\{1,2\}$ the set of the RNG seeds used for scenario evaluation, and Equation (4) the set of single evaluations. A decision was made to use multiple values of $\mathcal{Z}$ and ${\mathcal{Z}}^{\prime }$ to ensure diversification of the results.

$${\Xi }^{\prime }=\Xi \times R\times L\times {\mathcal{Z}}^{\prime }$$

(4)

Let the evaluation outcomes be defined by Equation (5)

$$o_{{\xi }^{\prime }}=o\left({\xi }^{\prime }\right)$$

(5)

where ${\xi }^{\prime }\in {\Xi }^{\prime }$, and let the average score for a given criterion $r\in R$ be defined by Equation (6).

$${\overline{o}}_{\xi ,r}={\displaystyle \frac{1}{|\mathcal{Z}\times L\times {\mathcal{Z}}^{\prime }|}}\sum _{{\xi }^{\prime }\in {\Xi }^{\prime }|{\xi }_{\xi }^{\prime }=\xi ,{\xi }_r^{\prime }=r}{o}_{{\xi }^{\prime }}$$

(6)

As each scenario was evaluated twice by each verifier model, a total of 11,520 evaluation sets were generated for the non-ablation and ablation variants of the experimental pipeline.

Moreover, an additional survey on a small group of eight human respondents was conducted. The main goal of the survey was to determine if the patterns in verifier model preferences are similar to human preferences. It is acknowledged that this sample size was not large enough to wholly represent the population of GMs; therefore, this human-based study should be treated as preliminary in nature. The age of the respondents varied, ranging from 24 to 39 years old. Among the respondents, the average experience in years of actively running RPGs as a GM was $11.5$ with a standard deviation of $6.37$ years and ranging from 4 to 22 years. In total, they evaluated 44 scenarios, covering all of the compendium, prompt and model variants. Each respondent was asked to evaluate sets, consisting of 4 scenarios generated using the same prompt, compendium and seed. The order of presented scenarios within each set was randomized. The seed was equal to either 1 or 2 and was constant within each of the evaluated sets. Therefore, the only variable in each set was the generative model used for each of the scenarios. The human judges were instructed to reflect in their evaluations the degree to which they would answer positively to the respective questions from Table 4 on a modified Likert scale from 0 to 5 in six categories. They were instructed that 0 was reserved for potential erroneous generations. In practice, none of the candidates responded with a note lower than 1 to any scenario in any category. The results of the survey are further discussed in Section 4.2.

The values of ${\overline{o}}_{\xi ,r}$ are presented in Figure 1. It can be noticed that prompt $p_s$ achieved the highest averages in five of the six categories except for Complexity. The overall means were equal to 3.864 for $p_s$, 3.717 for $p_m$ and 3.682 for $p_l$. The quality drop between $p_s$ and $p_l$ variants is 0.182 points (3.64%). Categories most affected by the prompt lengthening were the Coherence (−0.15), (−0.065) and Complexity (−0.009) were relatively resistant to this negative impact. Among the chosen compendia, the $c_m$ achieved the best results in Coherence, $c_s$ in Relevance and $c_l$ in other categories.

Model-wise Mistral achieved the highest overall quality, scoring highest in Relevance, Coherence, and Informativeness. Out of all the models, it is the least sensitive to parameter changes. Overall mean of LLaMA was close to Mistral’s. It was observed that LLaMA specializes in Interactivity with an advantage of 0.119–0.342 points compared to other models. The lowest-scoring model in all categories was Stable Beluga.

It was observed that often high Coherence variants also had high Structure score. A similar pattern was discovered between Relevance and Informativeness and, to some degree, between Interactivity and Complexity. Sensitivity to compendium size changes tends to increase with prompt length. Among the chosen variants, Stable Beluga proved to be the least resistant to prompt extension, and Gemma—the most.

4.1. Ablation

To verify the validity of the algorithmic selection of elements, an ablation study was conducted. Instead of embedding-based selection, entities, motivations, and strategies were randomly chosen with equal probability.

The ablation results are presented in Figure 2. For each evaluation category, a decrease in the mean score relative to the baseline pipeline was observed across all assessed variants. Notably, Relevance and Coherence exhibited the strongest positive impact of algorithmic element selection, with mean improvements of $0.092$ and $0.103$ points. The smallest effects were recorded for Complexity ($0.023$) and Interactivity ($0.043$). Moderate differences were observed for Structure ($0.065$) and Informativeness ($0.068$).

The mean scores for individual variants are, in most cases, higher for the algorithmic approach. For Relevance, Informativeness, and Interactivity, the baseline pipeline achieved better results in $86.11\%$ of variants, for Coherence—in $88.89\%$, for Complexity—in $66.67\%$, and for Structure—in $83.33\%$. Among the variants that achieved higher scores for the ablation than the baseline, $43.24\%$ were variants using the Mistral model, $35.14\%$ used Gemma, $13.51\%$ used Stable Beluga, and $8.11\%$ used LLaMA.

In Relevance the largest increase related to algorithmic selection was observed for $(p_l,c_m,\mathrm{Mistral})$, amounting to $0.287$ points. For the category Coherence, the largest increase was achieved by the variant $(p_l,c_m,\mathrm{Stable}\mathrm{Beluga})$, with $0.296$. In Complexity, Informativeness, and Interactivity, $(p_l,c_m,\mathrm{LLaMA})$ achieved the largest increase—$0.15$, $0.165$, and $0.159$, respectively. In Structure, variant $(p_l,c_m,\mathrm{Mistral})$ also obtained the greatest benefit from algorithmic selection, although with Mistral.

Based on the ablation results, it can be deduced that employing the standard research pipeline is usually justified. The deterministic method exerts a particularly strong effect on Relevance and Coherence. Variants with a long prompt and a medium-sized compendium consistently show the greatest benefits. Regarding that evidence, it can be inferred that the integrity of the overall scenario depends on the design of their skeletons. The magnitude of the method’s positive effect depends on the specific language model used.

4.2. Self-Bias and Human Evaluation

The scoring of the generator LLMs by the verifier LLMs was analysed to detect any scoring patterns indicating the presence of self-bias. In 

Table 6. Average scores across the generator models by the verifier models and human judges.

Generator	Verifier
	Gemma	LLaMA	Mistral	Stable Beluga	Human
Relevance
Gemma	4.575	$\mathbf{4}.\mathbf{592}$	$4.755$	3.070	$3.227$
LLaMA	$4.794$	$4.508$	$4.769$	$\mathbf{3}.\mathbf{207}$	2.500
Mistral	$\mathbf{4}.\mathbf{935}$	$4.473$	$\mathbf{4}.\mathbf{789}$	$3.197$	$\mathbf{3}.\mathbf{364}$
Stable Beluga	$4.779$	4.019	4.450	$3.169$	$2.682$
Coherence
Gemma	3.213	$\mathbf{4}.\mathbf{475}$	$4.520$	$3.115$	$\mathbf{3}.\mathbf{364}$
LLaMA	$3.603$	$4.394$	$4.470$	$\mathbf{3}.\mathbf{156}$	2.818
Mistral	$\mathbf{3}.\mathbf{870}$	$4.250$	$\mathbf{4}.\mathbf{536}$	3.046	$3.136$
Stable Beluga	$3.551$	3.917	4.317	$3.049$	$\mathbf{3}.\mathbf{364}$
Complexity
Gemma	2.179	$\mathbf{3}.\mathbf{392}$	$3.774$	$3.109$	$\mathbf{3}.\mathbf{091}$
LLaMA	$2.215$	$3.377$	$3.725$	$\mathbf{3}.\mathbf{190}$	2.500
Mistral	$\mathbf{2}.\mathbf{435}$	$3.284$	$\mathbf{3}.\mathbf{807}$	$3.059$	2.500
Stable Beluga	$2.274$	3.163	3.724	3.050	$2.682$
Informativeness
Gemma	3.445	$\mathbf{4}.\mathbf{509}$	$4.332$	$3.138$	$\mathbf{3}.\mathbf{045}$
LLaMA	$3.697$	$4.502$	$4.335$	$\mathbf{3}.\mathbf{192}$	2.182
Mistral	$\mathbf{3}.\mathbf{976}$	$4.499$	$\mathbf{4}.\mathbf{455}$	3.039	$\mathbf{3}.\mathbf{045}$
Stable Beluga	$3.654$	4.258	4.257	$3.063$	$2.500$
Interactivity
Gemma	2.782	$3.804$	$\mathbf{4}.\mathbf{170}$	$\mathbf{3}.\mathbf{262}$	$\mathbf{2}.\mathbf{727}$
LLaMA	$3.106$	$\mathbf{4}.\mathbf{056}$	$4.154$	$3.243$	$2.318$
Mistral	$\mathbf{3}.\mathbf{137}$	$3.480$	$4.079$	3.075	$2.545$
Stable Beluga	$2.881$	3.312	3.873	$3.079$	2.273
Structure
Gemma	3.871	$\mathbf{4}.\mathbf{467}$	$\mathbf{4}.\mathbf{627}$	3.054	$3.000$
LLaMA	$4.092$	$4.270$	$4.565$	$3.136$	$2.773$
Mistral	$\mathbf{4}.\mathbf{299}$	$3.967$	$4.584$	$\mathbf{3}.\mathbf{156}$	$\mathbf{3}.\mathbf{182}$
Stable Beluga	$4.092$	3.310	4.197	$3.130$	2.409

Bold values indicate the highest scores in each evaluation category. Italic values indicate the lowest scores in each evaluation category.

, the highest average score for each category and verifier was shown in bold and the lowest in italics. Several trends were observed. The Gemma model scored its own generation the lowest in all of the categories, despite getting relatively high scores from most of the other LLMs. The LLaMA model rated Gemma’s scenarios especially favourably, giving them the highest average score in all categories. The Mistral model exposed preference in its own scenarios in Relevance, Coherence, Complexity and Informativeness, rating them highest on average. These notes might indicate some level of self-bias; however, the Mistral model was also, on average, rated the highest by Gemma in all categories. Moreover, Stable Beluga gave Mistral the highest average score in Structure and the second highest in Relevance. While no indisputable evidence of self-bias was found, it is apparent that some scoring preferences exist among the various models, likely resulting from differences in data sets and architectures.

Regarding the human-based evaluation, it is noticeable that human-granted scores are generally lower, on average by 0.95 point than the LLM-based ones. Especially in Relevance and Informativeness, the average LLM-based scores were overstated, by $1.32$ and $1.21$ points, respectively. The most accurate categories were Coherence and Complexity, with average differences of $0.68$ and $0.42$ points. Despite the disparity, it can be noticed that in general, human judgment has some similarities to the LLM-based one. In most categories, the highest-scored models were Gemma and Mistral with an average score of 3.08 and 2.96 across all of the categories, while LLaMA and Stable Beluga scored 2.52 and 2.65. Among the LLMs, Mistral got the highest average scores in both the human-based and LLM-based evaluations, while Stable Beluga was rated relatively low.

4.3. Statistical Significance

The statistical analysis in

Table 7. Results of Levene’s test and ANOVA. p-values $<0.005$ are shown in bold.

Factor	Source	Relevance	Coherence	Complex.	Inform.	Inter.	Structure
	l	$\mathbf{0}.\mathbf{0}\times {\mathbf{10}}^{-\mathbf{05}}$	$\mathbf{0}.\mathbf{0}\times {\mathbf{10}}^{-\mathbf{05}}$	$\mathbf{4}.\mathbf{3}\times {\mathbf{10}}^{-\mathbf{05}}$	$\mathbf{1}.\mathbf{9}\times {\mathbf{10}}^{-\mathbf{03}}$	$\mathbf{0}.\mathbf{0}\times {\mathbf{10}}^{-\mathbf{05}}$	$4.3\times {10}^{-01}$
Levene’s	p	$4.3\times {10}^{-01}$	$\mathbf{0}.\mathbf{0}\times {\mathbf{10}}^{-\mathbf{05}}$	$3.7\times {10}^{-02}$	$\mathbf{0}.\mathbf{1}\times {\mathbf{10}}^{-\mathbf{05}}$	$\mathbf{0}.\mathbf{0}\times {\mathbf{10}}^{-\mathbf{05}}$	$3.3\times {10}^{-01}$
p	c	$0.86761$	$0.76119$	$0.27223$	$0.57087$	$0.91147$	$0.60279$
	l	$\mathbf{1}.\mathbf{3}\times {\mathbf{10}}^{-\mathbf{18}}$	$\mathbf{1}.\mathbf{0}\times {\mathbf{10}}^{-\mathbf{16}}$	$1.7\times {10}^{-02}$	$\mathbf{3}.\mathbf{1}\times {\mathbf{10}}^{-\mathbf{13}}$	$\mathbf{2}.\mathbf{6}\times {\mathbf{10}}^{-\mathbf{43}}$	$\mathbf{1}.\mathbf{6}\times {\mathbf{10}}^{-\mathbf{54}}$
ANOVA	p	$\mathbf{9}.\mathbf{7}\times {\mathbf{10}}^{-\mathbf{09}}$	$\mathbf{7}.\mathbf{8}\times {\mathbf{10}}^{-\mathbf{45}}$	$7.3\times {10}^{-01}$	$\mathbf{3}.\mathbf{8}\times {\mathbf{10}}^{-\mathbf{20}}$	$\mathbf{1}.\mathbf{9}\times {\mathbf{10}}^{-\mathbf{25}}$	$\mathbf{2}.\mathbf{1}\times {\mathbf{10}}^{-\mathbf{27}}$
p	c	$0.49358$	$0.50784$	$0.12325$	$0.70186$	$0.59909$	$0.81185$
	l	$0.01539$	$0.01344$	$0.00184$	$0.01086$	$0.03498$	$0.04375$
ANOVA	p	$0.00655$	$0.03560$	$0.00011$	$0.01587$	$0.01965$	$0.02101$
${\eta }_p^2$	c	$0.00025$	$0.00023$	$0.00076$	$0.00012$	$0.00018$	$0.00007$
	l	$1.00000$	$1.00000$	$0.77062$	$1.00000$	$1.00000$	$1.00000$
ANOVA	p	$0.99992$	$1.00000$	$0.09941$	$1.00000$	$1.00000$	$1.00000$
$\pi$	c	$0.17013$	$0.16487$	$0.43239$	$0.10713$	$0.13487$	$0.08271$

,

Table 8. Tukey’s test results. p-values $<0.005$ are shown in bold. Abbreviations: Gem: Gemma; Lla: LLaMA; Mis: Mistral; Bel: Stable Beluga. Symbols: $\Delta$: mean difference; p: p-value; d: Cohen’s d.

Criterion	Value	Gem:Lla	Gem:Mis	Gem:Bel	Lla:Mis	Lla:Bel	Mis:Bel
Relevance	$\Delta$	$0.054$	$0.084$	$-0.163$	$0.031$	$-0.216$	$-0.247$
	p	$0.255$	$0.018$	$\mathbf{0}.\mathbf{000}$	$0.708$	$\mathbf{0}.\mathbf{000}$	$\mathbf{0}.\mathbf{000}$
	d	$-0.073$	$-0.113$	$0.208$	$-0.041$	$0.276$	$0.313$
Coherence	$\Delta$	$0.047$	$0.075$	$-0.138$	$0.028$	$-0.184$	$-0.212$
	p	$0.303$	$0.026$	$\mathbf{0}.\mathbf{000}$	$0.722$	$\mathbf{0}.\mathbf{000}$	$\mathbf{0}.\mathbf{000}$
	d	$-0.064$	$-0.104$	$0.193$	$-0.040$	$0.266$	$0.316$
Complex.	$\Delta$	$0.001$	$0.023$	$-0.060$	$0.022$	$-0.061$	$-0.083$
	p	$1.000$	$0.830$	$0.131$	$0.848$	$0.118$	$0.012$
	d	$-0.001$	$-0.033$	$0.082$	$-0.031$	$0.083$	$0.119$
Inform.	$\Delta$	$0.049$	$0.118$	$-0.064$	$0.069$	$-0.113$	$-0.182$
	p	$0.196$	$\mathbf{0}.\mathbf{000}$	$0.047$	$0.023$	$\mathbf{0}.\mathbf{000}$	$\mathbf{0}.\mathbf{000}$
	d	$-0.074$	$-0.181$	$0.100$	$-0.107$	$0.177$	$0.287$
Inter.	$\Delta$	$0.119$	$-0.073$	$-0.222$	$-0.192$	$-0.341$	$-0.149$
	p	$\mathbf{0}.\mathbf{000}$	$0.015$	$\mathbf{0}.\mathbf{000}$	$\mathbf{0}.\mathbf{000}$	$\mathbf{0}.\mathbf{000}$	$\mathbf{0}.\mathbf{000}$
	d	$-0.172$	$0.113$	$0.340$	$0.299$	$0.525$	$0.248$
Structure	$\Delta$	$-0.012$	$-0.022$	$-0.339$	$-0.010$	$-0.328$	$-0.318$
	p	$0.967$	$0.824$	$\mathbf{0}.\mathbf{000}$	$0.979$	$\mathbf{0}.\mathbf{000}$	$\mathbf{0}.\mathbf{000}$
	d	$0.018$	$0.032$	$0.517$	$0.015$	$0.496$	$0.477$

and

Table 9. Tukey’s test results. p-values $<0.005$ are shown in bold.

Criterion	Value	${\mathit{p}}_{\mathit{s}}:{\mathit{p}}_{\mathit{m}}$	${\mathit{p}}_{\mathit{s}}:{\mathit{p}}_{\mathit{l}}$	${\mathit{p}}_{\mathit{m}}:{\mathit{p}}_{\mathit{l}}$
Relevance	$\Delta$	$-0.126$	$-0.135$	$-0.008$
	p	$\mathbf{0}.\mathbf{000}$	$\mathbf{0}.\mathbf{000}$	$0.942$
	d	$0.165$	$0.176$	$0.011$
Coherence	$\Delta$	$-0.262$	$-0.298$	$-0.036$
	p	$\mathbf{0}.\mathbf{000}$	$\mathbf{0}.\mathbf{000}$	$0.266$
	d	$0.381$	$0.437$	$0.050$
Complex.	$\Delta$	$-0.010$	$-0.018$	$-0.008$
	p	$0.899$	$0.728$	$0.945$
	d	$0.015$	$0.025$	$0.011$
Inform.	$\Delta$	$-0.144$	$-0.190$	$-0.046$
	p	$\mathbf{0}.\mathbf{000}$	$\mathbf{0}.\mathbf{000}$	$0.078$
	d	$0.226$	$0.298$	$0.070$
Inter.	$\Delta$	$-0.162$	$-0.217$	$-0.055$
	p	$\mathbf{0}.\mathbf{000}$	$\mathbf{0}.\mathbf{000}$	$0.030$
	d	$0.253$	$0.339$	$0.081$
Structure	$\Delta$	$-0.180$	$-0.226$	$-0.046$
	p	$\mathbf{0}.\mathbf{000}$	$\mathbf{0}.\mathbf{000}$	$0.094$
	d	$0.270$	$0.335$	$0.070$

shows that significant effects were present in all criteria except Complexity, proving generator model (l) and instruction length (p) impactful on the scenario quality. Compendium size (c) did not produce significant effects in any criterion. Across Relevance, Coherence, Informativeness, Interactivity, and Structure, the dominant pattern was the inferior performance of Stable Beluga relative to the other three models. The largest between-model contrasts reached moderate effect sizes, for example, in Structure and Interactivity, where some differences reached over $0.3$ standard deviations. In criteria with weaker effects, such as Relevance and Informativeness, model comparisons still had significant differences but with smaller magnitudes. Instruction length exerted a similarly strong influence across five of the six criteria. Short prompts $p_s$ outperformed medium and long ones, with effect sizes reaching $d=0.34$ in Structure and Interactivity and $d=0.44$ in Coherence. Across all criteria where effects were present, longer prompts reduced clarity, organization, and interactivity, suggesting that more concise instructions enable models to generate more coherent scenarios. The only criterion that did not provide meaningful differences was Complexity. Taken together, the results indicate that model choice and prompt length are the primary determinants of scenario quality, with Stable Beluga consistently underperforming and short prompts yielding the best outcomes across nearly all evaluated dimensions.

4.4. Popular Metrics Analysis

In order to enhance the comparability, the generated scenarios were also evaluated using popular metrics for evaluating text. The selected metrics include the Bilingual Evaluation Understudy (BLEU) [62], the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [63], and the Bidirectional Encoder Representations from Transformers score (BERTScore) [67]. BLEU and ROUGE measure the degree of overlap of word sequences between the generated output and a reference text, while BERTScore measures the semantic overlap relative to the reference text. In the context of this work, the reference is the skeletal form of the scenario. For BLEU the minimum possible value is 0, and the maximum is 100, where 100 indicates that the generated output and the reference are identical. The resulting scores for ROUGE and BERTScore range from 0 to 1, with higher values meaning more word overlap for ROUGE and stronger semantic similarity for BERTScore.

In the context of the present study, BLEU and ROUGE are not particularly reliable. These metrics were primarily designed for the evaluation of machine translation rather than creative text generation. Their correlation with human evaluation of creative text is weak [58]. Nevertheless, due to their popularity, an analysis of the generated scenarios using these metrics was conducted.

A more suitable metric for the present task is BERTScore, as its ability to assess the semantic meaning of words allows for the determination of semantic similarity between two texts.

Table 10. Average scores in popular metrics.

Parameter	BLEU	ROUGE			BERTScore
		1F	2F	LF	P	R	F1
$c_l$	3.067	0.229	$0.086$	$0.137$	$0.807$	$0.811$	$0.809$
$c_m$	$2.791$	$0.220$	$0.082$	$0.133$	$0.807$	$0.807$	$0.807$
$c_s$	$2.741$	$0.217$	$0.083$	$0.135$	$0.807$	$0.807$	$0.807$
Stable Beluga	$3.201$	$0.206$	$0.088$	$0.143$	$0.803$	$0.794$	$0.798$
Gemma	$1.189$	$0.159$	$0.061$	$0.111$	$0.793$	$0.814$	$0.803$
LLaMA	$1.706$	$0.180$	$0.062$	$0.122$	$0.798$	$0.782$	$0.790$
Mistral	$2.998$	$0.223$	$0.090$	$0.156$	$0.810$	$0.832$	$0.820$
$p_l$	$3.368$	$0.252$	$0.098$	$0.155$	$0.810$	$0.798$	$0.804$
$p_m$	$2.972$	$0.232$	$0.089$	$0.141$	$0.808$	$0.803$	$0.805$
$p_s$	$1.964$	$0.181$	$0.063$	$0.109$	$0.802$	$0.824$	$0.812$

presents the evaluation metrics for different generation variants. BLEU scores are generally low, although the highest are for Stable Beluga and Mistral. ROUGE metrics follow a similar trend—Mistral reaches best results in all ROUGE markers among the models. BERTScore indicates that Mistral has the strongest semantic similarity, while LLaMA is the weakest. BLEU scores among the compendia are generally low, although higher for $c_l$, while the ROUGE score does not show significant distinctions in terms of lexical coverage. BERTScore shows a minor influence on semantic similarity. Variants with $p_l$ improve lexical coverage, achieving the highest BLEU and ROUGE scores, while shorter prompts show slightly higher semantic alignment, as indicated by BERTScore precision.

5. Discussion

The results indicate that the quality of the generated RPG scenarios depends on the number of steps in the prompt and the choice of an LLM. The influence of a knowledge compendium size is less important. The best outcomes were achieved for variants with shorter prompts, suggesting that more compact narrative structures output more coherent and relevant scenarios. Variants obtained the highest scores in Relevance and the lowest in Complexity, which may reflect the limitations of contemporary LLMs in producing complicated narratives. The finding that shorter prompts ($p_s$) yield higher quality results than longer, more detailed prompts ($p_l$) might initially seem counter-intuitive, especially when compared to complex analytical tasks where techniques like Chain-of-Thought reasoning thrive on expanded context. However, the root of this phenomenon in narrative generation lies not in context window noise or a suppression of creative capacity, but rather in the model’s struggle to maintain logical consequence and its lack of awareness regarding the relationships between consecutively joined elements. When a prompt strictly dictates a dense sequence of specific narrative constraints or beats ($p_l$), the LLM attempts to satisfy all conditions but sometimes fails to establish coherent causal links and transitions between them. Conversely, shorter prompts ($p_s$) provide a manageable structural skeleton, allowing the model to leverage its statistical language capabilities to naturally bridge the gaps and maintain a cohesive narrative flow.

A qualitative comparison illustrates this clearly. As shown in Appendix B, a failure case generated with a long prompt ($p_l$) is characterized by reduced narrative coherence: The model forces location changes without explicit transitions and assumes connections that were never established, simply to fulfill the prompt’s extensive requirements. In contrast, a success case generated with a short prompt ($p_s$), presented in Appendix A, demonstrates a noticeable, logical structure and maintains a clear, player-centric action.

Although there were evident preferences and trends in the ways the individual models judged the variants, they did not manifest any undisputable self-bias tendencies. It was also discovered that the human respondents tended to give the generated scenarios lower notes than their LLM counterparts. An exceptional disparity between the LLM and human judges scores in terms of Relevance and Informativeness was noticed. Perhaps the selected LLMs struggle to determine the purpose of an RPG scenario as a functional text. Understanding the precise source of this problem through research could improve the quality of the generated scenarios.

Low BLEU and ROUGE scores of the results might not imply poor scenario quality but rather a desirable departure from the schematic structure of the input. A high BERT score demonstrates that the semantic content of the skeleton scenario is preserved despite the use of different word sequences.

5.1. Limitations of LLM-Based Evaluation

The selected models were chosen with the understanding that other size variants might yield different results. Although examining additional modern models such as Gemini 2.5 Flash [80] or GPT-4 [81] could be beneficial, the large number of available architectures necessitated restricting the analysis to a curated subset.

It is acknowledged that LLM-Based evaluation does not perfectly reflect human preferences. However, to an extent, it might cover the most noticeable trends in humans, such as the preference of the generator model. Further research on similarities between human-based and LLM-based judgments of creative texts could help to improve this technique and its precision. Moreover, it could be beneficial to gather more human-based evaluations to achieve a better understanding of the similarities and differences between the human-based evaluation and the LLM-based evaluation.

Authors recognize that additional study using other game world compendia could be advantageous, but it would significantly broaden the scope of the research. In particular, studying worlds set in genres other than fantasy might be beneficial in further research.

5.2. Design Guidelines and Future Directions

As the research on this topic progresses a set of actionable design guidelines for developers of LLM-based narrative generation tools could be obtained:

G1: Minimise prompt complexity. Concise, structured prompts should be preferred over elaborate, open-ended instructions, as excessive complexity was found to negatively affect coherence, structure, and interactivity.

G2: Externalise world knowledge into a structured compendium. Generation should be grounded in an explicit, machine-readable representation of the game world rather than relying solely on the parametric memory of a language model. Even a compact knowledge structure was found sufficient to improve narrative consistency. For larger compendia, retrieval-based or staged injection strategies should be considered to avoid knowledge overload.

G3: Apply semantic matching when selecting narrative elements. Embedding-based selection of entities and strategies should be preferred over random sampling, as it was found to consistently produce more coherent and relevant outputs.

G4: Align model selection with target quality criteria. Different language models exhibit distinct strengths across quality dimensions; model selection should therefore reflect the quality criteria most relevant to the intended use case. In particular, human-in-the-loop validation is recommended when high interactivity is a priority.

G5: Complement automated evaluation with human judgment. Automated metrics were found to systematically overestimate scenario quality relative to human judges and should be treated as indicative rather than definitive.

Several directions for future research follow naturally from these guidelines. The generalisability of G1 and G4 could be assessed by extending the pipeline to incorporate larger models such as GPT-4 or Gemini 2.5 Flash. The scope of G2 could be broadened by expanding the compendium to cover alternative genres such as science fiction, horror, or historical settings. The validity of G5 would benefit from a larger and more diverse respondent pool, providing a more robust basis for validating automated evaluation metrics in narrative domains. Additionally, the construction of a GM-oriented usability metric capturing functional dimensions absent from current automated measures, such as the preparation effort required for a scenario, remains an important direction for future work.

6. Conclusions

This paper aimed to investigate the potential for LLM-based RPG scenario generation and factors affecting the quality of the results. A dedicated generative pipeline was designed, implemented and used to create a dataset.

The study found that both the number of steps in the prompt and the choice of an LLM significantly affect the quality of generated scenarios. More steps in the instruction prompt generally lead to worse results across criteria. This suggests that concise prompts produce a more effective basis for scenario generation than elaborate, open-ended instructions. LLMs show an inherent variety as some of them excel in particular scoring categories. Metrics based on lexical similarity provided low values, while semantic metrics indicated a high degree of meaning similarity between the scenario and the prompt, confirming that generated scenarios preserve the semantic content of the input skeleton despite surface-level difference. Furthermore, the results indicate that knowledge-grounded generation improves scenario consistency, and that even a compact compendium proved sufficient to achieve this effect, as compendium size did not produce statistically significant differences across evaluation criteria. This suggests that the overhead of maintaining a formalized world compendium need not be prohibitive for practical adoption. Ablation analysis confirmed the validity of using the baseline pipeline in comparison to a version with random choices instead of semantic similarity-based choices.

It was discovered that human judges generally rate the generated scenarios lower than the LLM judges. This finding highlights a methodological risk in relying solely on automated evaluation for creative, functional texts, and underscores the importance of incorporating human judgment in the assessment of generative tools for narrative domains. Despite that, it was noticed that LLM judges display similar preferences to human judges regarding the generative model, suggesting that automated evaluation may capture certain qualitative trends even when absolute scores diverge. Further research of the similarities and differences in human and machine ratings would be beneficial. The modular architecture of the proposed pipeline further enables its potential adaptation to scenario generation tasks beyond fantasy RPGs, including educational simulations, training scenarios, and interactive storytelling applications.