*4.1. Experimental Environment*

Hardware: The experiment was carried out on an i7-3770k CPU computer @ 3.50 (turbo: 3.9) GHz, 4 cores (note: during the experimentation, only one core was used, since agents do not implement the multi-core support) and 16 GB RAM.

Software: OS Windows 10 Pro and Java Development Kit 13.0.2. The experiment was set in the latest version of the microRTS environment, acquired from an online source at the time of preparing this article [64]. The microRTS environment comes pre-loaded with the following gameplaying agents: RandomAI, RandomAIBiased, MonteCarlo, IDRTMinimax, IDRTMinimaxRandomized, IDABCD, UCT, PuppetSearchMCTS, and NaiveMCTS. TiamatBot was acquired from the online source [65]. MixedBot (which includes TiamatBot source files but an improved version) was acquired from the online source [66] and was included in the microRTS environment. Every gameplaying agen<sup>t</sup> is used in the experiment as it was acquired from the online source of original authors (i.e., no code or internal parameter was changed for experimental purposes).

Table 4 shows the hyper-parameters used for the validation of every game feature presented in Table 1.


**Table 4.** Hyper-parameters used in the experimentation.

These hyper-parameters are pre-set within the microRTS environment. The only parameter that we changed was iterations, which we set to 50 (before it was set to 10). The standard UnitTypeTable was used where necessary. Note: to validate the GF9\_PARI, we changed the environment from fully observable to partially observable.

The game feature descriptions presented in Table 1 were derived from related works and written independently of a specific game environment, i.e., they can be implemented in any RTS game engine. In Table 5, we present the same game features as those presented in Table 1, although the former are adapted to the microRTS environment and a specific scenario. All game features in Table 5 are written with the assumption that they are valid for the microRTS environment. If the playtesting agen<sup>t</sup> actually manages to invalidate a game feature from the list, it will add to its metric score.

#### *4.2. Adaptation of Gameplaying Agents as Playtesting Agents*

To adapt a gameplaying agen<sup>t</sup> to the playtesting task, we created a non-intrusive component. The component contains information about the scenario (map, position of units and the opponent) and controls the validation procedure by following the playtesting agents' progress (i.e., actions that it executes) and by accessing game environment information (e.g., current game state status). All the information is accessible through well-defined interfaces of the microRTS source code. One of the interface methods is the method that returns the best action for the given game state, and every gameplaying agen<sup>t</sup> operating in the microRTS environment implements it.


#### **Table 5.** microRTS game feature scenario.

When the actions are executed in a game state, it cycles to the next one (i.e., actions change the inner state). During such cycles, our component tests if the Game Feature is valid or invalid. A Game Feature is invalid if the condition that is written in the validation procedure of the game feature in question is not fulfilled. The condition is tested against the information provided from the agent's executed action and the environment's current game state. The validation procedure checks the validity of the game feature, until either the maximum number of cycles is reached, or the game is over (i.e., one of the players has no more units left on the field).

For example, the game feature, GF8\_FANT, is validated by checking if the resulting game state still holds this special unit after the agen<sup>t</sup> has given an order to fire on it. If in any cycle the unit is destroyed, the game feature is invalid.
