*2.2. microRTS*

There are many di fferent RTS game worlds in existence. Not all of them are openly available, but even some of the commercial ones have been opened up for research purposes (e.g., StarCraft ™ was opened through the programming interface). microRTS is a simple non-commercial simulation environment, which was created to test any theoretical ideas a game researcher might have.

This simulation environment follows standard RTS game genre game rules:


The microRTS environment includes the following features (seen in Figure 1):


**Figure 1.** Micro real-time strategy (microRTS) environment, with all features visible.

Workers are used to gather resources and build structures, and they also possess the ability to attacks with limited firepower. Light, heavy and ranged are the main battle units used for attacks on opponent structures and mobile units. Battle units have different initial properties (i.e., a heavy battle unit can sustain more damage before being destroyed versus a light battle unit, and a ranged unit can shoot farther). Bases produce workers. Barracks are used to create battle units. The wall is used as a physical barrier in the map.

microRTS allows configurable scenarios to be placed in the environment. Figure 1 presents one such scenario: an 8 × 8 cell map, with fixed positions for resources (in the top left and bottom right corners) and walls. The mobile units are not fixed and can be moved freely inside this environment.

Scenarios can be configured for varying map sizes (4 × 4, 8 × 8, 12 × 12, etc.) and with different starting positions for the unit types, structures, and resources (which can be placed anywhere on the map). The game can be played with visible features (graphical interface turned on for observations) or in the background (which allows for a faster execution of scenarios and quicker overall simulations, with less computer resources used).

microRTS also already includes many gameplaying agents that can be used in experiments.

#### **3. Proposal of a Metric for Game Feature Validation**

Our motivation to create a metric came from the need to be able to differentiate easily between different playtesting agents' performances, when multiple game features need to be validated. In order to propose a novel metric for comparing playtesting agents, the following steps were considered in our study:


All steps are described in detail in the following subsections and are presented graphically in Figure 2.

**Figure 2.** Pathway (steps taken) from the identification of the RTS game features to the novel metric.

#### *3.1. Identification of RTS Game Features*

Game features are mentioned in many RTS game research works, but they are scattered across different subdomains and research agendas. Our goal was to use the pool of research articles and dissertations and to identify the game features included in this research. The pool was reviewed with the help of a literature search. The ISI Web of Science and ProQuest research search engines were used. A search query with the following search string was made: "game features" and "real-time strategy games", which returned 88 hits for the ISI Web of Science and 34 hits for ProQuest.

The results (articles and dissertations) were filtered to exclude non-RTS game research works. A manual search was conducted through the research work for mentions of the "feature" string (note: 14 works from the ISI Web of Science and 0 for ProQuest were located after a manual search). The located text was extracted and analyzed for surrounding context, then transformed into a compact format that could act as a short game feature description. The surrounding context was used to transform the text, because not all research work has game features that can be used as-is. A short description was then made of the list of game feature descriptions. If a description was already on the list, and it was not adding additional information, it was omitted (note: seven works from the ISI Web of Science were omitted). Additionally, one research work can include more than just one mention of the string "feature" with the surrounding context.

Note: Future work could broaden the scope and include other search strings (like "aspect" or "feature") for a more in-depth survey of the general RTS features.

Table 1 includes a list of short game feature descriptions, which were produced after the completion of the first step. The short game feature descriptions are accompanied by a reference.


#### **Table 1.** Game feature descriptions derived from related work.

#### *3.2. Grouping the Game Features into Specific Groups*

The grouping of game features into specific groups has two benefits: a group consists of game features with similar modus operandi (i.e., correlated and in the same context), and groups can serve as a basis for sharing research with other game genres.

We already mentioned (Section 2.1) that the literature search revealed 18 groups, which are formed independently of the specific game genre, and which we will use for grouping. These groups are: adaptation, assessment/rewards/scores, challenge, conflict, control, exploration, fantasy/location, interaction/interactivity (equipment), interaction (interpersonal/social), language/communication, motivation, mystery, pieces or players, progress and surprise, representation, rules/goals, safety and sensory stimuli. A detailed description about the meaning of each of the groups can be found in the tabular presentation in [62].

Table 2 presents the results after the completion of both steps, with references to the source of the compact description. Our goal was to have at least one game feature representative for each of the groups. If there was no game feature available in Table 1 for an empty group, we tried to locate the research work for that group by searching via Google Scholar (STEP 3) using different search strings (e.g., "impassable terrain" for a conflict group) in regard to the context of the group. The research works found went through the procedure described in STEP 1, and a short game feature description was included in Table 1 and in the accordingly empty group in Table 2. (Observation: we noticed that many research works on game feature descriptions originated from the domain of player/opponen<sup>t</sup> modeling, RTS replay analysis, game balancing and strategy selection/prediction.) One game feature can belong to more than one group. For some groups, we could not find or create any viable game feature description that could be measured by the game mechanics. Such groups remained empty but were still included in the Table. The reason: future RTS research could produce game features for currently empty groups.


**Table 2.** Game definition groups and their game feature representatives.

**1** Representative of the group used for the experiment. **2** The interaction (Interpersonal/Social) group was left empty, because it would require the interaction of multiple players (a single gameplaying agen<sup>t</sup> modified for a playtesting agen<sup>t</sup> supports only single player operations).

#### *3.3. Classification of Feature Groups According to Their Correlation and Importance*

As game features tend to be correlated, so do groups. One group can be, context wise, closely related to some groups but only loosely related to others. Additionally, some contexts are more important than others with regard to RTS gameplay.

Table 3 presents the classification of feature groups into three importance classes:



**Table 3.** Classification of feature groups based on their correlation and importance.

The importance level of each of the groups is represented by a class. Regarding the game worlds, we allow for the possibility of different reconfigurations of the groups inside the classes. We also included the weight and mathematical description of the set. Weight is a numerical value that is set by the user of the metric. It represents how much the groups belonging to the specific class will count towards the metric score.

#### *3.4. Proposal of the Metric*

In this subchapter, we explain our metric for summarizing agents' performance while they validate game features in an RTS game space. The metric calculates its score based on how many times the playtesting agen<sup>t</sup> invalidated the game feature of a fixed number of repeats for a given scenario (the sum of validations and invalidations equals the number of scenario repeats).

If the playtesting agen<sup>t</sup> during the execution of the scenario could not test the game feature, because it does not come into a situation, or it is not programmed to deal with the situation where validation can take place, then such a game feature is valid from this point of view. The number of successful validations is, therefore, omitted from the game score, since it is biased.

For a set of groups Gi, 1 ≤ i ≤ 18, where each member of group Gi holds a set of Game features (GFs) (GFj ∈ Gi, 0 ≤ j), and each GFi holds a set of executable scenarios S (Sk ∈ GFi, 1 ≤ k), the number of unsuccessful validations per scenario is defined by numInvalidijk, and the number of times the scenario is repeated is defined by numOfScenRep = n, 1 ≤ n, the following formulas apply:

$$\text{inavalidPercPerSGen (i}\_{\overline{\mathbb{k}}} \text{ numOfSGenReps} ) = \text{numInvalid}\_{\overline{\mathbb{k}}} \text{[numOfScerRepp]} \tag{1}$$

calcSetScore (set, numOfScenReps) = i ∈ set *j*≥<sup>0</sup> *<sup>k</sup>*≥<sup>1</sup> invalidPercPerScen (ijk, numOfScenRep) (2)

$$\begin{array}{c} \text{agentPlayers} \text{Score} = \\ \text{W1 \* calcSetScore ((1, 3, 5, 13, 14, 16, 17), numOfSencRep)} \\ + \text{W2 \* calcSetScore ((2, 4, 6, 7, 8, 12), numOfSencRep)} \\ + \text{W3 \* calcSetScore ((9, 10, 11, 15, 18), numOfSencRep)} \end{array} \tag{3}$$

In Equation (1), the number of invalidations of a given group (index i), game feature (index j) and scenario (index k) is divided by the total number of scenario repeats. In Equation (2), the score is calculated for all the game features and scenarios that the set of groups holds. In Equation (3), the scores of the set of groups are multiplied by their respective weights.

#### **4. Experiments and Results**

In this chapter, we present the specifications of hardware and software used for the experimental environment, as well as the results of the experiments.
