3.2.4. Results

In the first round of voting (Table 3), 43 participants voted electronically; 37 were citizens, three were politicians and there were three from associations. This represented 2.17% of the total census and 14.96% of the weighted participation. In this round, the participants opted for the cultural criteria (52.99%) over sports (47.01%) (Table 4). The priorities of both rounds can be seen in [16].

**Table 3.** The two rounds of voting.


\* Citizens over 18 in the 2008 Census (data from the Aragon Statistics Institute) (source: the authors)

**Table 4.** Priorities of the criteria of each group of actors.


(Source: the authors).

Between the first round and the second round, a forum of debate was opened on the Internet, participants expressed their concerns and preferences (Table 5). A total of 61 messages were posted, 37 belonged to the cultural criteria and 24 to sports. There were 195 comments about these messages, 114 belonged to the cultural criteria and 81 to sports. In the second round of voting (Table 3), there were 41 participants, 35 were citizens, four were politicians and two were from associations, a weighted participation of 17.60% (2.08% of the total census).

**Table 5.** Messages and comments on the Cadrete forum.


(Source: the authors).

In the two rounds, the variations in absolute terms were minimal, although, due to the small number of voters, the relative variation, at least with respect to politicians and associations, shows significant modifications. The percentage of politicians voting increased (from the first to the second round) by 33.3% and that of associations fell by the same figure (33.3%). The voters again opted for cultural criteria (56.58%) over sports (43.42%), by an increased margin (Table 4).

The low citizen participation in the electronic experience contrasts with the municipal (2007) and the general (2004 and 2008) elections whose levels of participation were 69.9% and 76.6%, respectively (Table 6).


**Table 6.** Participation in electoral processes in Cadrete.

(Source: the authors).

#### *3.3. Estimated Structural Model*

After the e-participation experience in Cadrete, participants were asked to complete an online questionnaire; 24 residents responded though only 20 replies were valid. The questionnaire was considered as invalid if less than 80% of the questions were answered and/or there was zero variability with regards to the total number of questions.

The measurement scale was from 0 to 10 (0 = total disagreement, 10 = total agreement). A total of 51 questions were grouped into seven sections: (i) The System of Citizen Participation; (ii) The Creation of a Better Society; (iii) Motivation; (iv) Evaluation of the Technological Support and Applications; (v) Evaluation of the Information; (vi) Evaluation of the Support Personnel; and, (vii) Overall Evaluation.

The theoretical EF3-framework was first evaluated through a survey in Cadrete using the SEM or the Covariance Structure Analysis approach [49–52]. The first analysis was a conjoint descriptive analysis of the variables, using the measurements of their position, dispersion and correlation. These measurements led to the identification of groups of interrelated indicators that could be expected to define the utilised constructs; the analysis was complemented by an examination of the main components. The study was completed with structural equation models with latent variables, or covariance structure analysis [51].

This methodological approach was chosen as it allows the researcher to formulate and evaluate the existence of latent variables from the reflected indicators [50], that is to say, variables that are not susceptible to direct observation. The software used was EQS 6.1 [52]. Table 7 shows the constructs and indicators used to describe the effectiveness, efficacy and efficiency in the estimation of the structural model for an EF3-evaluation of e-Cognocracy.

The indicators concerning *information*, *communication* and *decision*, revealed evaluations that were just above or just below an acceptable level of satisfaction (5.0, 4.7, 5.5, 5.0, 5.2 and 5.1, respectively); *expectations* reached a high level (7.5 and 7.2), which suggests that citizens are not convinced about actively participating in the system. Therefore, efficacy is defined by the indicators *information, communication, decision* and *expectations*.


**Table 7.** Constructs and Indicators.

\* Average indicator evaluation. (Source: the authors).

In general, the average valuations of the indicators of efficiency were relatively high. Participants gave a more positive valuation to *service quality* (8.50) than to *information* and *system quality* (7.0 and 6.0, respectively). The correlation was over 0.7, which indicates that the structure should be maintained. Therefore, efficiency is defined by the participants' satisfaction with the *quality of information and of the system.* In our empirical study, the third construct (personal support) considered by DeLone and McLean was centred on informing about the AHP methodology, thus it was integrated into the Information Quality.

E-Cognocracy was given a more positive evaluation than the current system of citizen participation (*current situation*), both in terms of its ability to improve the current system (*future situation*) and to achieve its ultimate goal-the creation of a better society (7.9 and 7.7 compared to 5.5). It is, therefore, clear that the improvement of the current system and the creation of a better society require the introduction of more dynamic and participative mechanisms, such as those employed by e-Cognocracy. All of the above can be defined as effectiveness. The role of the citizen is perceived as being as relevant as that of the associations. The relationships between the indicators were coherent with the constructs.

The confirmatory factor analysis (Table 8) offered sufficient evidence to maintain a first order structure of four correlated factors (Information, Communication, Decision, Expectation) from the eight observed variables (X1, ... , X8). The structures of the constructs, in theoretical terms, were empirically corroborated at both an exploratory and a confirmatory level. The eight indicators reflected a structure of four interrelated latent variables.


**Table 8.** Confirmatory factor analysis model.

χ2(25): 26.97, p-value: 0.52; SRMR: 0.07; GFI: 0.90; CFI: 0.99. \* significant to 10%, \*\* significant to 5% and \*\*\* significant to 1%. (Source: the authors).

Table 8 also shows that the relationships between the dimensions of *information, communication* and *decision* were positive and this implies that a higher perception of information signifies a higher perception of *communication* and *decision*, and vice versa. The relationship between *expectations* and *decision* was negative, and this implies that citizen disappointment with regards to the existence of *decision* is greater if they have greater *expectations* of participation. Reliability indices for both these observed variables and their respective latent variables were more than acceptable (R2, omega and C-FL).

Figure 6 depicts the estimated structural model. Table 9 shows the estimated structural model; there are three determinants of the citizens' perception of the current system: *information*, *communication* and *<sup>e</sup>*ffi*ciency*. Determinants of the *future situation* are *decision* (negative) and *expectations*. The *current and future situations* affect the perception of the creation of a better society, though the effect of the *future situation* (e-Cognocracy) is greater.

**Figure 6.** Estimated structural model for an EF3-evaluation of e-Cognocracy. (Source: the authors).



χ2(46): 43.16, *p*-value: 0.71; SRMR: 0.07; GFI: 0.90; CFI: 0.99. \* significant to 10%, \*\* significant to 5% and \*\*\* significant to 1%. (Source: the authors)

If citizens perceive that information exists, that is to say, that the administration informs society of the participation mechanisms and the decisions that are taken (top-down unidirectional flow), then they have a positive perception of the current system of representation.

If communication exists, that is to say, information flows in both directions (feedback), it is also a determinant, but this is not the case with *decision* and *expectations*. Moreover, if citizens feel that they have no influence on the taking of political decisions, irrespective of their perceptions of the existence of information and communication, they will favour a change in the participation system. If the citizens have higher expectations of involvement in the design and formulation of public policies, they will also favour change.

Due to the limited number of responses, it was not possible to validate a general framework for the conjoint evaluation of all the aspects outlined in the theoretical EF3-framework. Nevertheless, the results obtained from the 20 valid responses identified a series of relationships that contribute to the formulation of a general framework.

The small sample size means that the evaluation and selection of the models is governed by goodness of fit indicators (SRMR, GFI and CFI) that do not directly depend on the number of observations [49]. For all the measured and/or structural models, the estimated parameters were presented in their completely standardised version, norm 0–1, and, in addition, all the equations were given their corresponding coefficients of explained variance.

The assessment of the construct is based on the methodology proposed by Bagozzi [55] for the validation of multidimensional constructs and the covariance structure analysis of observed variables (McDonald's omega coefficient [56] and Fornell and Larcher's coefficient, C-FL [57]. The stability of the parameters of the models was estimated and evaluated sequentially.

Although the simplified analysis of the theoretical EF3-framework has not allowed significant statistical conclusions, it has meant that, in conjunction with a review of the existing literature, this framework may be extended to evaluate any e-Participation experience, not only e-Cognocracy.

## **4. Conclusions**

The traditional democratic system finds it difficult to efficaciously react in the context of a dynamic, complex and uncertain environment. The democratic legitimacy of public institutions is being questioned by a citizenry that is more and more educated, reflexive, critical and interconnected. Citizens are demanding more open and receptive governments that are prepared to listen, share, and co-decide.

In the search for an appropriate response to the needs of democracy in the epoch of the Knowledge Society, new models of e-participation and systems are being advanced; the validity of these models needs to be analysed by taking into account their effectiveness, efficacy and efficiency (EF3-approach). The integral evaluation of the three perspectives in citizen participation models is still pending; in this work, the EF3-approach has been applied to a real-life experience resolved by means of the cognitive democracy model known as e-Cognocracy. Through the use of structural equations, the relevant factors extracted for the EF3 evaluation of this model will help to analyse the performance of the remaining citizen participation models.

The theoretical model (EF3) was applied to the design of cultural and sporting policies in the Spanish municipality of Cadrete. Due to the limited number of observations, it was not possible to validate a generalised model for the integral evaluation of e-participation experiences. Nevertheless, it was possible to extract a series of relationships and relevant factors that will be considered when dealing with similar cases in the future.

Despite the small number of responses to the evaluation questionnaire, it appears that the experience was seen as very positive by the people of Cadrete. These kinds of projects should be continued; they make a contribution to the conjoint construction of a better society.

Among the relevant aspects derived from the study we can highlight the followings: (i) The efficiency is determined by information and system quality; (ii) The efficacy is explained by information, communication, decision and expectations; and, the effectiveness (creation of a better society) is studied by analysing the change (impact) between the current and future situations.

More specifically, the high value given for the evaluation of expectation is a reflection of the uncertainty that citizens feel with regards to actively participating in the system. The role of the citizen was perceived to be of the same importance as that of the local associations.

The confirmatory factor analysis offered sufficient evidence to maintain a structure of four, interrelated, first order factors based on the eight observed variables. The relationships between the dimensions of information, communication and decision imply that a greater perception of levels of Information will mean that perceptions of communication and decision are higher, and vice versa.

The relationship between participation expectation and decision was negative. This implies that when citizen disappointment regarding the existence of decision increases, there is also an increase in participation expectation. The sample gave a more positive evaluation of e-Cognocracy than the traditional democratic model, both in terms of its ability to improve the present system and to achieve its ultimate aim-the construction of a better society.

As already mentioned, a limitation of the study (that could be resolved with further research) is the small number of responses to the evaluation questionnaire. It is well known that a higher number of responses improves statistical robustness so it is difficult to validate more generalised models. However, one of the strengths of the study is that the analysis was of a real-life situation, and this enhances the internal validity of the research.

Finally, it should be emphasised that the improvement of the system and the creation of a better society require the introduction of more dynamic participation mechanisms such as e-Cognocracy. The ideas extracted in this project for the evaluation of e-Cognocracy could be used for the evaluation of any democracy model that combines, as Michels and De Graff [9] assert, participatory processes and formal decision-making.

**Author Contributions:** Conceptualization, J.M.M.-J. and P.R.-T.; Formal analysis, J.M.M.-J., C.P.-E. and P.R.-T.; Investigation, J.M.M.-J., C.P.-E. and P.R.-T.; Methodology, J.M.M.-J., C.P.-E. and P.R.-T.; Project administration, J.M.M.-J.; Writing—original draft, J.M.M.-J. and C.P.-E.; Writing—review & editing, J.M.M.-J and C.P.-E. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the Spanish Ministry of Economy and Competitiveness and FEDER funding, project ECO2015-66673-R.

**Acknowledgments:** The authors would like to acknowledge the work of English translation professional David Jones in preparing the final text.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A. The Cadrete Questionnaire**

After finishing the real-life e-Participation experience based on e-Cognocracy (Cadrete), participants were asked to complete an online questionnaire; 24 residents responded though only 20 replies were valid. The questionnaire was considered as invalid if: (i) less than 80% of the questions were answered; and/or (ii) there was zero variability with regards to the total number of questions. The measurement scale was from 0 to 10 (0 = total disagreement, 10 = total agreement). A total of 51 questions were grouped into seven sections:

(i) The System of Citizen Participation:

	- 1. Participation is limited to Citizen consultation by the Administration
	- 2. Participation includes Debate/Discussion with the Citizen, but the Decision is taken by the Administration
	- 3. Participation allows the joint decision between the Administration and the Citizen
	- 1. I cannot miss the opportunity to be part of a citizen participation initiative like this one
	- 2. I think it is a very important opportunity to express my opinions
	- 3. I believe that this initiative will allow me to enrich myself as a person
	- 4. I am interested in participating in the planning of cultural/sports activities
	- 5. I do not agree with the current managemen<sup>t</sup> of cultural and sports activities
	- 1. The computer equipment was adequate
	- 2. The presentation structure of the program was simple and understandable
	- 3. It was easy and comfortable to move from screen to screen (navigate)
	- 4. There were too many errors/incidents in the computer application \*
	- 5. The number of screens was not excessive \*
	- 6. The voting system was easy to use
	- 7. The discussion system for the incorporation of arguments was adequate
	- 8. The discussion system allowed me to know and share opinions
	- 9. I consider that my anonymity was assured throughout the entire process
	- 10. In general, I liked the design of the software application
	- 11. In general, I am satisfied with the computer application used
	- 1. It was easy to understand
	- 2. It was suitable
	- 3. It was received on time
	- 4. There were virtually no errors
	- 5. In general, I am satisfied with the information that I have received
	- 1. I really enjoyed participating in this initiative
	- 2. I have learned a lot from the experience
	- 3. I feel that my participation has improved my ingenuity and creativity
	- 4. The experience allowed me to feel involved in political decision making
	- 5. My perception of social belonging in my municipality has increased (identity)
	- 6. The discussions in the forum influenced my decisions
	- 7. Participating in this experience was not a waste of time
	- 8. I would participate again in a similar experience
	- 9. Other municipalities should incorporate this type of citizen participation
	- 10. E-Cognocracy improves the current democratic system
	- 11. I feel satisfied with my participation in this initiative
