2. Materials and Methods
2.1. Application Framework
As mentioned earlier, this paper describes the implementation process of one of the ENFEN screening tests originally performed on paper, Sendero Gris (Grey Trail), on digital devices.
The chosen technological device to implement the solution was a tablet. Among other advantages, this type of device offers the benefit of being familiar to today’s children and has the capacity to collect relevant and diverse information parameters, such as the time between events and the number of touches on the screen [
36]. The solution was implemented using the Kodular platform, which is a programming environment that enables the development of block-based coding applications for Android devices [
37].
The developed app serves as a test container and currently only has
Sendero Gris fully implemented within the app (
Figure 1 and
Figure 2).
The operation of the application, which aligns with the design and protocol of the validated ENFEN paper-based test, is as follows. Once the application is started, the app logo is displayed, followed by the login screen (
Figure 1A). If it is the user’s first time accessing the app, they will be redirected to the registration screen (
Figure 1B). Returning users can simply enter their email and password to access the tests.
Figure 1A includes an option for an alias, allowing the same user (with the same registration email) to use different names for self-assessment. This eliminates the need for multiple user accounts for children within the same school or family; instead, using different aliases for each child is sufficient. Additionally, as shown in
Figure 1B, certain data and control questions (e.g., sex, gestation weeks, main previous diagnosis if any, etc.) are requested, which are crucial for establishing connections between collected information and ensuring accurate diagnosis by the specialist.
Once the user has completed the registration process, a verification email will be sent to the provided email address. After verification, the user will gain access to the main menu of the application (
Figure 2A). From this screen, the user can report errors and access the different tests, among other options. Finally,
Figure 2B displays the selection screen for the available tests.
Figure 3 illustrates the general functioning of the system. When the user interacts with the application, the personal data, control questions, and test results obtained are sent to a Firebase database [
38]. Access to this database is restricted to registered therapists, ensuring the privacy of all participants.
2.2. Test Design
The
Sendero Gris test is designed to assess the ability to connect consecutive numbers, ranging from 1 to 8 in the training version and from 1 to 20 in the evaluation version (
Figure 4).
The nodes, arranged randomly on the paper, need to be connected by a path or stroke in decreasing order to maximize the discriminative potential of the test. Among others, some of the sub-functions assessed in this test are sustained and selective attention, inhibitory control, working memory, and decision making.
The performance in the earlier sub-functions is measured based on the number of correct responses in the path of nodes drawn by the child (hits), errors in this path (substitutions), and omissions of nodes made by the child during the test, as well as the time required to complete it. The definition of these terms is detailed in the ENFEN manual [
12].
Although the graphical construction of the application may appear simple (with a minimalist appearance), there are several fundamental aspects in its construction, referring to the collection of information, that cannot be overlooked (
Table 1). In addition, to define solutions to the challenges presented in this table, several concepts related to the nodes had to be described:
Start node: the node number from which the path is initiated.
Target node: the node intentionally chosen as the next step in the path.
Pressed nodes: the set of nodes collected by the application and considered for calculating the player’s final score.
Final node: the node that contains the number 1. Once this node is considered a pressed node, the test is completed.
As the points listed in the previous table pertain to the data collection and calculation of the children’s final score, all these implementation aspects are crucial in the development of the solution, thus determining its effectiveness.
Regarding passing over a node while moving towards the target node, the strategy adopted was to consider a node as pressed only when the child lifts their finger from the screen. This proper execution technique is demonstrated through a GIF animation after the instructions have been provided (
Figure 5).
Regarding the measurement of the time taken by the child, the opportunity to communicate the rules was exploited. Therefore, once the individual has finished reading the guidelines, they should press the “understood” button to initiate the test, and the timer will start counting. Similarly, when the child presses the final node, the timer will stop.
In the case of the final node, the determination of whether it has been intentionally pressed is left to the player. That is, once the system recognizes that the number 1 has been pressed and the test can be considered complete, the timer stops, and a dialogue box appears asking if the child considers that the test has concluded. If the answer is no, the timer restarts, and the test continues as usual. On the other hand, if the individual responds in the affirmative, the test is over.
Concerning the explanation of the test rules, through a dialogue box and a voice message (auditory and visual aids) the child is informed about the required steps. This approach ensures that the test is accessible to a wider range of individuals [
34].
Finally, in relation to measuring inactivity, specific periods of inactivity were established in collaboration with the experts. Consequently, periods of 15 s during which the tablet does not detect any interaction are considered as inactive. If this occurs, an informational message will be displayed on the screen for the child, who will have the option to either “continue playing” or “end the test”. These notifications will be displayed a total of three times. If 45 s of inactivity are detected, the test will automatically end, and the child will be assigned a predetermined maximum time.
Once all the technical characteristics necessary for the proper collection of the relevant markers have been specified, the method for calculating the final score obtained in the test is determined. By maintaining the concepts of hit, substitution, and omission according to their original definition in the ENFEN, the formula by which the test scores the child’s performance is presented in [
12].
2.3. Hypothesis and Experimental Protocol
This study aims to test the hypothesis that there is no difference in the results between the paper-based Sendero Gris test, which is the original test, and its implementation as an e-health system (the tablet-based Sendero Gris test). If the hypothesis is accepted, it would indicate that the proposed e-health system has the same screening capability as the original test, under much higher conditions of comfort for the child, while eliminating bias in the assessor’s correction.
The experiments to test the hypothesis were undertaken in two scenarios: the first at the Amanecer school and the second at the Centro Municipal de Asociaciones de Salud de Alcorcón, both located in Madrid, Spain.
To assess the effectiveness of the proposed solution, a use case was designed to compare the results of the original paper-based test with those obtained using the tablet-based version. Thus, the same group of children participated in both versions of the test, including children with and without a diagnosis of ADHD.
Firstly, when taking the paper-based test, an expert was present to provide guidance to the children and supervise the test execution. The expert also recorded the time taken by the child to complete the test and calculated their final score according to the rules defined in the ENFEN protocol [
12]. Once the paper version of the test was completed, for the tablet-based test, the child was provided with a preconfigured tablet that had the application installed, the user account logged in, and the control questions answered by their family members beforehand. No additional explanation about the test procedure or task was given to the child, and they simply started interacting with the tool and performed the test.
The experiment and subsequent study were conducted in accordance with the principles outlined in the Declaration of Helsinki and received approval from the Ethics Committee of Rey Juan Carlos University, with informed consent obtained from the families of the children who participated.
2.4. Sample Summary
Due to the limited availability of eligible participants for the study, the initial sample size consisted of 28 individuals. Unfortunately, due to data collection issues related to technical issues on one of the hardware devices (the storage system failed), not all data from 4 players could be retrieved after the experiment was run, resulting in a final effective sample size of 24 children.
The initial sample consisted of 28 children of both sexes (12 girls and 16 boys), ranging in age from 6 to 12 years old (average 9.21, SD 2.01).
Table 2 provides an overview of the sample, including the number of children by age and sex. It can be observed that the sample covers all the age groups for which the ENFEN battery was developed.
Regarding the control questions collected in the initial application form (
Figure 1B), the average gestational weeks is 39.39, with a minimum of 34 weeks and a maximum of 42 weeks. Approximately 36% of the mothers of the children in the sample reported experiencing some delivery complications, with the majority being induced cesarean sections. Among the children born with birth complications, 60% received a diagnosis of ADHD. Moreover, all cases included in the sample were single births.
Table 3 presents the distribution of children according to sex and their ADHD diagnosis. Among the girls, a total of 4 had a diagnosis of ADHD (33.33%), while the majority of the sample consisted of undiagnosed girls (66.66%). In the case of males, 62.5% had a diagnosis of the deficit in question.
Lastly,
Table 4 summarizes some diagnostic parameters of interest for the experts. On average, for both sexes, children with ADHD are older than those undiagnosed. On the other hand, there does not seem to be a clear association between gestational weeks and the disorder, as the average gestational weeks for both groups coincide. Finally, regarding birth complications, it was observed that 3 out of 4 girls with ADHD were born without any complications (75%), while 50% of the boys with ADHD (5 out of 10 children) suffered birth complications.
2.5. Statistical Method
Two different statistical analyses were performed. On the one hand, a simple statistical comparison based on averages was made between the collected parameters of both the paper-based test and the tablet-based test. The parameters were described in the Materials and Methods section: success (hits), error (substitutions), omissions, time consumed, and final score obtained. On the other hand, the non-parametric Wilcoxon signed-rank test was employed to assess the statistical significance of the difference in mean scores obtained in each test format [
39]. This test is a non-parametric statistical hypothesis test used to compare the locations of two populations using two paired samples. Moreover, it is applicable for hypothesis testing with sample sizes of at least 16 individuals.
4. Discussion
ADHD is recognized as one of the neuropsychological disorders that significantly impact the development and integration of individuals into society, “with well-established diagnostic and treatment services available throughout most of Europe” [
40,
41,
42].
However, at present, screening related to this deficit, usually associated with a high number of children, lacks methods and tools that allow it to be carried out objectively and efficiently, which leads to an excessive burden both on specialists, who tend to assess in a biased way because they do not have the appropriate conditions, and on the children themselves, who must travel to specialized centers, which can affect their performance during the evaluation [
34,
43]. Fortunately, with advancements in technology and existing knowledge about ADHD, it is now possible to develop e-health-based solutions that facilitate the screening of the deficit for healthcare professionals. These solutions have the potential to improve the work of specialists and the quality of life for patients by providing accessible and efficient tools.
This article presents the findings of a comparative analysis aimed at assessing the discriminative efficacy of a validated ADHD test, known as Sendero Gris, and a component of the ENFEN test battery. The study involved administering the test in two formats: a traditional paper-based version, which is the format in which the test was designed, and a tablet-based version, both conducted within a school environment.
The results indicate that the tablet-based version of Sendero Gris has demonstrated promising discriminative power, comparable to the paper-based version. This encourages the research team to move forward with the implementation of all four ENFEN tests, aiming to create a comprehensive tool for screening ADHD in school-aged children.
One limitation of the present study is the relatively small sample size available. This limitation arose due to challenges in recruiting target individuals who were willing to participate in the study, as well as operational errors in implementing the proposed solution. As a result, the sample size was restricted to just 24 individuals. This constraint hinders our ability to generalize the findings on a broader scale, rendering this study a preliminary exploration of the discriminative potential of the proposed solution.
When comparing the scores obtained from the different test formats (supervised paper-based tests administered by specialists and unsupervised tablet-based tests), statistically similar results were observed. Furthermore, note that any learning that the children might have acquired from undertaking the test on two occasions is not a factor that warrants consideration as significant. In other words, if learning had occurred during the first instance of testing, improved performance could have been anticipated during the second iteration. As this is not the case and comparable outcomes were achieved in both instances, the exploration of the interaction stemming from the learning factor was deemed unnecessary.
Consequently, it can be inferred that the digitization of the ENFEN battery shows promise as a screening tool that is, at the very least, as effective as the original test while offering greater user convenience. It is essential to note that the Sendero Gris test exclusively assesses specific EFs. Therefore, not all individuals diagnosed with ADHD will necessarily display impairments in these particular EFs.
Table 5 and
Table 6 illustrate the test’s discriminative power in its two existing versions for identifying individuals with limitations in the assessed EFs. Furthermore, the manual calculation of scores by children has been removed, introducing an automated calculation process during the test’s execution.
In reference to statements S1 and S6, which assessed the intention of frequent usage of the developed system, it is noteworthy that the end users have exhibited a high degree of acceptance towards the solution. As for the statements addressing the test’s level of difficulty (S2 to S5), undoubtedly, overall, the participants found the test to be quite straightforward across both groups. This observation emphasizes that the presence of an adult during its execution was unnecessary.
Moreover, during the implementation of the presented study, other advantageous aspects of the digital format were discovered, such as the clear recording of the order in which the nodes are connected and the ability to collect a greater number of markers of interest compared to the current test.
However, the application did exhibit some functional errors and issues related to information collection. In certain instances, children had to be excluded from the analysis because of data collection failures while taking the test, which stemmed from errors in one of the hardware devices. Additionally, we identified instances of repeated traces, where the connection of two consecutive numbers occurred more than once resulting in error scores being added to their final score.
The gold standard for diagnosing ADHD is a specialized clinical evaluation that cannot be replaced. However, access to such assessments is limited in certain populations, especially without a proper screening process that allows for a more efficient allocation of existing resources [
20]. Additionally, tests based on EFs not only facilitate the evaluation process but can also provide important clues for a more efficient diagnostic process by providing indications of specific deficits. Therefore, the presented solution has the potential to implement this screening process more efficiently, which, in turn, will allow us to focus more on existing resources for an accurate clinical diagnosis. Based on the results obtained, the researchers involved in this study are confident that the implementation of the complete set of ENFEN battery tests as an e-health system would greatly contribute to the effective and user-friendly detection of ADHD at an early age in natural settings.
5. Conclusions
ADHD affects all aspects of a human being, from their social life to personal and school development. Its onset is marked in childhood and can be diagnosed from the age of six. Therefore, an early and accurate diagnosis, followed by appropriate treatment, can mitigate and/or eliminate future challenges associated with the condition. This study, to our knowledge, is the first to introduce and validate the discriminatory ability of a digital screening tool to assess ADHD in children compared to a previously validated paper-based test. The tablet-based test addresses several significant limitations of the paper-based format. It can be conducted in a natural environment without the necessity for specialized supervision. Additionally, it enables objective tracking of a child’s progress throughout the test, thereby eliminating the need for mandatory specialist presence and the potential for observation bias. By addressing these limitations of paper-based tests, the developed solution offers an innovative approach to ADHD screening, leveraging e-health system technologies to enhance the whole process.
Finally, the potential for future enhancements includes the addition of more markers of interest, incorporating new tests, and exploring novel methods of analysis and result measurement. These advancements have the potential to further enhance the discriminatory power of the system.