Defining a Successful Project in Sustainable Project Management through Simulation—A Case Study

Abstract

Modern approaches to project management link project success both to sustainability and with the value delivered to project stakeholders. In the final analysis, it is the satisfaction of the identified stakeholders based on the sustainability principle which decides whether a project is successful or not. The value delivered to appropriate stakeholders has to be incorporated in the earliest phases of the project defining process. Otherwise, it may be too late to introduce changes which would allow the right value to be delivered to the right stakeholders. In this paper, we propose that a simulation is used in the phase of project definition to choose such implementation forms for individual project phases or tasks which would facilitate the delivery of the value expected by the stakeholders. Our approach also supports the process of agreeing (among the stakeholders) on the value expected from the project. We show, using a post-factum real-world project case, that the application of a simulation in the predictive project stage may considerably increase the probability of the project’s success.

1. Introduction

One of the principles of sustainability is balancing social, environmental, and economic interests. This principle should be incorporated in the whole project management process including, among others, the definition of the project, the management of project stakeholders, and the control of the [1]. During the definition of the project, the success criteria should be defined considering the sustainability principle, and the management of the project stakeholders should take this principle into account while selecting stakeholders, analyzing them, and undertaking actions with respect to them. The control of the project should verify whether the sustainability-related success criteria are likely to be fulfilled.

A tight relationship between the success of the project and the satisfaction of project stakeholders is nowadays generally accepted [2]. Here we adopt the following definition of a project stakeholder [3]: “individual, group, or organisation who may affect or be affected by, or perceive itself to be affected by a decision, activity, or outcome of a project”. The sustainability principle requires that all stakeholders representing or having social, environmental, and economic interests have to be taken into account. A modern approach to project management [4] defines project success as the situation when the right value has been delivered to the selected stakeholders at a sufficient level. Moreover, the stakeholders have to agree on the definition of the value to be delivered by the project and its required minimum level.

As Kerzner has expressed [4], the future of project management is metrics-based project management. A metric is “any type of measurement to assess some quantifiable element of project success or performance” [5]. Project metrics are values, measured in various dimensions, whose role is to assess the probability of the project’s success. Metrics help us to see where we are going in the projects. They should be related to project success criteria. If possible, they should be oriented to the future and play the role of warning signals if the ultimate success of the project is in danger. They should be integrated into all project management stages and in the earliest stages, such as project defining, and they should be used not only for reactive management but above all for proactive management. Specifically, they should be integrated into project management predictive tools [6]. The use of metrics in predictive tools will turn them into efficient warning signals because the predictive values will be available earlier than if the metrics were calculated based only on the actual project course.

Metrics related to the value expected by project stakeholders (who will be selected under the sustainability principle) can help us to assess the probability of project success, as defined above. There has been some research on the selection of the right metric for each project stakeholder [6,7,8,9], but the selection of metrics has to be performed for each project individually.

Simulation can be used as a predictive tool in project management to simulate the values of the selected success metrics and to decide the best scenario for project implementation. One position within the literature [10] has been identified which applies simulation to the selection of success factors for the whole project, but usually, simulation in project management is used to predict certain project features. It can be applied either before the start of the project or during project implementation. Simulation is used both at the strategic and at the operational management levels and supports both project planning and monitoring functions. In the planning context, simulation is used to estimate the project outcome and to identify planning alternatives. In the monitoring process, simulation models are used to evaluate project status based on selected metrics [11,12].

System dynamics-based project simulation is seen as an alternative and in many cases an important compensation tool with respect to traditional project management methods. Contrary to the latter, it allows the consideration of projects for which no WBS or specific task definitions have been prepared and many undefined elements still exist—which depend on external factors and feedback that will be available only during project implementation. It allows a holistic view of a project and its environment, taking into account both soft and hard internal and external factors that may manifest themselves in unpredictable moments [13]. The attitudes and behaviours of project stakeholders are among the factors which are often easier to be taken into account by a system dynamics-based simulation than by traditional project management methods [11].

The simulation models, to be effective, should be tailor-made to specific project types. We have identified simulation models dedicated to the following project types: construction projects [14,15], IT projects [16,17,18,19], agriculture projects [20], new product development projects [21,22], or research projects [23]. Here, a social project is used as a case study.

Although it is theoretically possible to elaborate models encompassing all project features and aspects, they are bound to be too general and, consequently, not particularly informative. Thus, numerous models proposed in the literature simulate various project aspects, e.g., the impact of introducing requirement changes [11], the rework generated during project implementation [12], project sustainability [20,24], the impact of legislative changes [14], interactions with the customer [11], project team communication [25] a specific project outcome [23], project scope [21,22], the choice of project control moments [26], and using PMBOK standards [27]. Given the above project success definition, the research by the authors of [28] is especially interesting, because it concentrates on the value delivered by the project. It was assumed that the value expected from the project itself was defined and a simulation was used to identify the most appropriate types of responses to the predicted deviations from the expected value and the most appropriate reporting types which guarantee a high probability of delivering the value expected from the project.

Of course, project simulation has its limitations [11]. The main problem is the calibration and validation of the models, which may be difficult because of the uniqueness of projects. It is not always possible to have enough similar projects or similar elements of the same project for the repeated use and the stepwise improvement of the model [14]. However, project simulation may be helpful especially in predictive models, when the project has not been fully defined and a variety of factors, especially the unpredictable behavior and the reaction of stakeholders, have yet to be grasped. Such predictive models can be used before the application of traditional project management methods to identify scenarios that can then be analysed and introduced into traditional project management structures. Various implementation forms and various behaviours and expectations of stakeholders can be tried out so that the project definition, elaborated possibly according to traditional methods, has the highest possible probability of delivering to the right stakeholders the value expected by them.

The objective of this paper is thus to propose an approach that applies simulation to sustainable project management, where project success is defined as above (thus based on the value delivered to project stakeholders identified and analysed in accordance with the sustainability principles), in an early stage, prior to project full definition, to adjust the project success definition among stakeholders and to choose one implementation from from a collection of several possible ones. The objective is original because, as the above literature review shows, there is only one project simulation model that addresses the problem of the value delivered to the stakeholders [28], and this model is not destined for the early project stages preceding the project definition: it is proposed for the phase after the project has been completely defined, together with its success criteria and implementation forms, and the questions to be answered refer to responses and reporting during project implementation. Our approach can be applied to a whole project, but can also be selected at the project stages.

The proposed approach will be especially useful for projects with at least one of the following features:

• a number of project stakeholders of a varied nature with varied interests, especially political ones;
• methods to be used in the project, unknown or ambiguous;
• a project objective that may be ambiguous or disputed among stakeholders.

For example, scholars claim that public projects often possess the first and the third feature to a higher degree than private projects [29]. Public institutions have to juggle between the often-conflicting interests of various parties, politicians, local community, and media, and the proposed approach will help to reconcile all parties or at least will help to clearly articulate and accept the differences. Turner and Cochrane [30] have considered the problem of “ill-defined goals and/or methods of achieving them”. They distinguish projects in which the goals and methods are well defined before the project start (mostly engineering projects)—for such projects they suggested that the proposed approach would probably be similar to “using a sledgehammer to crack a nut”. However, they also distinguish projects where the methods are defined and the objectives are not fully formulated (e.g., software development projects), where the objectives are defined but the methods are not (product development projects) and projects where none of the two aspects are well defined before the project start (research projects, organisational change projects). In all three project types, the proposed approach can potentially be helpful. Even if it does not lead to a definitive choice of methods or a definitive objective formulation, it may bring the project stakeholders closer to an exact project definition and in this way it can help to increase the probability of the project’s success.

The proposed approach will be illustrated using a case study: a real-world project which has been terminated and whose success (in the above sense of the word) may be a posteriori judged as questionable. One of the co-authors of this paper took part in the project as an expert. In the context of the whole project, its stakeholders, and success, we concentrate on one project stage and post factum, using our approach, we analyse how this stage could have been planned and performed more effectively to better contribute to the actual project’s success.

The outline of the paper is as follows: in Section 2, we introduce the necessary notions and terminology relating to the success, metrics, and stakeholders of the project, present the proposed approach in a general way, describe the simulation tool we use, and present the project in question, together with the proposed (post factum) way of defining its success. In Section 3, we apply the proposed approach to one project stage. The paper terminates with some conclusions and research perspectives.

2. Materials and Methods

2.1. Project Success Definition and Project Metrics

We assumed, according to the author of [4], that project success is measured by the value delivered by the project to project stakeholders, and this value will be measured by the weighted sum of the values of n success criteria $C_i, i=1,\dots ,n$. The criteria weights will be a consequence of stakeholder decisions and weights.

The success criteria may be of a quantitative or qualitative nature, but we assumed that in each case their value can be assessed on a 6-value scale. Thus, the criteria will take on values from the set $V=\left\{0, 1, 2, 3, 4, 5\right\}$, where 0 represents the situation when the respective criterion is completely unfulfilled and 5 when it is completely fulfilled. Let $\wp$ denote the specific project or project phase we are dealing with. $C_i (\wp )\in V, i=1,\dots ,n$, will denote the value of the i-th criterion taken at the end of the project.

Each criterion will be linked to a weight $w_i, i=1,\dots ,n, 0\le w_i\le 1$, such that ${\displaystyle \sum }_{i=1}^N{w}_i=1$. Value $\mho (\wp )$

$$\mho (\wp )={{\displaystyle \sum }}_{i=1}^N{w}_i{C}_i (\wp )$$

(1)

is the representation of the value delivered by the project. A threshold $\mathsf{{\rm T}}$ for $\mho (\wp )$ should be set by the stakeholders, which defines project success as the situation when $\mho (\wp )\ge \mathsf{{\rm T}}$.

Formula (1) represents the ultimate value delivered by the project. Its predicted value in time moments t prior to the project end, and also those before the project is fully defined ($t\in \left[0,T\right]$, where 0 is the beginning of the period when the project is under consideration—project idea birth moment included—and T is the end of the project life cycle), will be given by formula (2):

$$\mho \left(\wp ,t\right)={{\displaystyle \sum }}_{i=1}^N{w}_i{C}_i \left(\wp ,t\right)$$

(2)

where $C_i \left(\wp ,t\right)$ are the predicted values of the respective criteria in moment t.

For each project success criterion $C_i, i=1,\dots ,n$ a set ${ℳ}_i={\left\{m_j^i\right\}}_{j=1}^{L_i}$ of the metrics will be defined. Their values may be measured in an arbitrary moment. $M_j^i=\left\{m_j^i\left(t\right), t\in \left[0,T\right]\right\}$ is the set of all possible values of the metric $m_j^i$. We assumed that for $i=1,\dots ,n$, $j=1,\dots ,L_i$ there exists a function $f_j^i$ defined on $M_j^i$ such that $f_j^i\left(m_j^i\left(t\right)\right)\in \left\{0, 1, 2, 3, 4, 5\right\}$. The value $f_j^i \left(m_{j }^i\left(t\right)\right)$ is an estimation attempt of the eventual value of the i-th criterion, based on the value of metrics $m_j^i$ in moment t. Its accuracy will vary, depending on the project stage and on the metrics nature, but values $f_j^i \left(m_j^i \left(t\right)\right)$ $i=1,\dots ,n$, $j=1,\dots ,L_i$ observed over time (in groups, within the sets ${ℳ}_i$ for individual $i=1,\dots ,n$) will be a valuable indication of the ultimate value of the respective criterion, if the metrics are selected carefully, in cooperation with the sustainably chosen stakeholders.

By tracking the metrics values at any moment, and also by simulating them before the project has been even defined, we can see which metric values would have to be changed to increase the respective criterion value (2), and consequently, with a high probability, also (1). We would be able to concentrate on those metrics whose contribution is the highest and of course on the criteria whose weight is the highest.

The metrics have to be selected in cooperation with the respective stakeholders [4].

2.2. General Approach (Project Simulation in the Initial Stage of Project Definition)

The goal was to use metrics simulation in the predictive project phase to decide on the implementation form of $\wp$—whcih may stand for individual project stages or the whole project. We used the following general approach for the project or a selected project stage:

Step 1: Define the success criteria with their weights, thus the formula for the value delivered by the project (1), threshold $\mathsf{{\rm T}}$ and the respective metrics ${ℳ}_i={\left\{m_j^i\right\}}_{j=1}^{L_i}$;

Step 2: Define possible implementation forms $F_l, l=1,\dots ,L;$

Step 3: Build the simulation model;

Step 4: For each implementation form $F_l, l=1,\dots ,L$ perform the simulation and determine the values of the metrics and value (2) for the end of the stage being simulated or for the whole project. Compare it with $\mathsf{{\rm T}}$;

Step 5: Check the sensitivity of weights from (1) and of $\mathsf{{\rm T}}$;

Step 6: Select the optimal implementation form.

2.3. Methods and Tools Used for Simulation in the Case Study

In the simulation in Steps 3 and 4, we used the theory of Systems Dynamics (SD) [31]. Projects are seen as systems there. SD models capture the simultaneity in systems by updating all variables in small time increments with positive and negative feedbacks and time delays, taking into account the interactions between them. An SD model is composed of several elements (see Figure 1):

• streams (flows)—which indicate the movements of objects: e.g., materials, orders, staff, etc. (marked with double-line arrows);
• levels—places in which the inflow is compared with the outflow and if the former exceeds the latter, an accumulation of objects occurs, e.g., inventory level, staff available for employment, activities still to complete, etc. (will be marked with text strings in rectangles);
• decision points—regulating the flows as a function of information about the system state (will be marked as small triangles);
• variables—storing the values of parameters and auxiliary variables: e.g., duration of a project, the definition of the scope of work, etc. (will be marked as text strings without any framing or brackets);
• input variables—variables whose values are imported from other simulation models, treated here as parameters (will be marked as text strings in triangular brackets);
• information flow direction—will be indicated by single-line arrows.

The use of SD in practice is supported by software with a graphical user-friendly interface. Here we used the application Vensim: its free version Vensim PLE 7.3.5 [23].

2.4. The Project Case

2.4.1. Description of the Project

The project aimed to monitor the implementation of the rights of people with disabilities and to adapt the existing measures to the provisions resulting from the ratification by Poland of the UN Convention on the Rights of Persons with Disabilities (CRPD) in selected Polish government and local government units. The project was financed under the Knowledge Education Development Operational Program (EU Program) and was implemented from April 2017 to March 2019 in Lower Silesia (a district in Poland). Its minimum assumptions and course were specified in the Project Methodology prepared by the Ministry of Labor and Social Policy (acting as an institution commissioning project implementation through an open competition, supervising the correct implementation of project recommendations, and monitoring the achievement of indicators adopted in the project). Its actual implementation in Lower Silesia was undertaken by a non-governmental organization selected in a competition (hereinafter referred to as organization X), experienced in activities for the benefit of disabled people. A total of 30 randomly selected institutions were selected for monitoring, with a threshold of 13 as the minimum requirement: it was required by the Ministry that at least 13 institutions signed the contract and were subject to monitoring.

The main aim of the project was to increase the capacity of monitored institutions to implement the provisions of the CRPD by analyzing the activities of these institutions, formulating recommendations regarding these activities, and verifying their use. First, the selected institutions had to be persuaded to participate and sign a contract. Subsequently, in the institutions that signed the contract, several monitoring activities were carried out in line with the methodology. It defined the following stages of the study: (1) analysis of existing data for each of the monitored institutions; (2) in-depth interviews with representatives of people with disabilities in the environment of each of the monitored institutions; (3) in-depth interviews with representatives of each of the monitored institutions; (4) initial reviews in each of the monitored institutions to identify problem areas; (5) monitoring visits to each of the monitored institutions to establish and refine specific recommendations for changes; (6) in-depth interviews with representatives of each institution and from people with disabilities from the environment, to support the implementation of selected recommendations; (7) visits to verify the implementation of the recommendations, along with the preparation of reports verifying the implementation process of the recommendations.

Monitoring was carried out with the use of various methods, with particular emphasis on the credibility of the information sources, access to relevant information sources, and the possibility of documenting monitoring tasks. The conducted analysis covered a wide range of areas, including, in particular:

• architectural adaptation of buildings;
• adapting forms of informing citizens about the needs of disabled people;
• actions for the employment of disabled people in the institution;
• information activities aimed at combating stereotypes, raising awareness of the rights and dignity of people with disabilities, and promoting the employment of people with disabilities;
• social consultation;
• competencies of the institution’s employees;
• fulfillment of the general obligations under the CRPD.

However, first of all, the institutions had to be persuaded to cooperate and sign the contract. Here, much resistance was encountered, or at least a lack of enthusiasm on the part of the selected institutions. Moreover, the mere fact of obtaining the contract signature was not a guarantee that the institution in question would participate with due involvement. Thus, among all the project phases, Phase 1 was of crucial importance for the whole project. Below we present all the project phases.

Phase I: getting to know the monitored institution and signing an agreement between Organization X and the institution regarding participation in the project;

Phase II: planning of the monitoring, covering the development of tools, research techniques, analysis of existing data, and collection of important information;

Phase III: implementation of monitoring in institutions;

Phase IV: verification of the implementation of monitoring recommendations.

Each of these stages was associated with certain difficulties, some of which turned out to be crucial for the course and success of the whole project. At the same time, each of the stages was implemented without a systematic operational and contingency plan. After the project end, the team of Organisation X saw numerous aspects of the project where they could have proceeded more efficiently if the project had been better planned. That is why, post factum, the project team saw potential benefits from applying simulation in or even before the definition stage of the project.

In this paper, we concentrate on Phase I and treat the other stages only superfluously. That is why in what follows the problems related to this phase are described more in detail.

Phase I: In this phase, the aim was to sign a contract for participation in the project by at least a certain number of public institutions (13 at the minimum, and they were to be selected from a pool of 30 institutions drawn at random). The phase turned out to be surprisingly difficult. Institutions (beneficiaries of the project), despite repeated attempts to negotiate and persuade, did not decide to join the project, or resigned from participation, mainly at the initial stage of the project, although there were also resignations in the last stages of the project. The refusals were due to political instability, additional unpaid work, low awareness of the needs of people with disabilities, the need for lengthy administrative procedures for signing contracts, etc. This significantly extended the recruitment process and led to excessive involvement and stress among the project staff from organization X. It was necessary, often even several times, to renew invitations in writing, to contact the institution in other ways, to look for substitutes in case an institution was not willing to cooperate, to intervene in the formal issues related to signing the contract, and to repeatedly explain the purpose of joining the project. For these reasons, the recruitment of institutions for the project (Phase 1), which was planned for 2 months, was extended to the whole duration of the project, intensively for the first half of the year, and less intensively for the next half year, and occasionally, in the event of later withdrawal, even longer. In this phase, as with in the other phases, no conscious choice of implementation form and no systematic analysis of the potential problems was performed: the project team reacted spontaneously each time a problem arose. Regardless of the reasons for refusing to participate in the project, the recruitment process, and the ongoing possibility of refusal, constituted a significant difficulty and risk factor for achieving project success.

However, it is important to place the problem in the context of the whole project, and to also describe the challenges and problems linked to the other phases. In Phase II, there were problems related to the lack of regulations regarding the involvement, remuneration, and delegation of the personnel of the monitored units to work in the project.

These problems were related to the nature of the project itself—the long duration of monitoring, its meticulousness, and the need to involve numerous employees because of the intensive work required. In fact, the project implementation process required the monitoring teams to establish other monitoring teams, which unfortunately was not possible. As a result, the persons appointed to contact the monitoring teams often perceived participation in the project as additional work and an additional burden, which resulted in the need to considerably extend the implementation of individual stages of the project. In Phase III, problems related to specific procedures and conditions on the part of the monitored institutions were encountered. They were related to the lack of, in a large part of the monitored institutions, employees who would be disability-oriented and who would feel empowered to make decisions at the same time. As a result, the monitored institutions gave a relatively low priority to the project. Additional problems were related to the issues regarding the legal and administrative conditions in the monitored institutions that were related, among others, to the problems of planning expenses, the administration of historic buildings, or buildings not owned by the monitored institution. In Phase IV, difficulties related to the fulfillment of recommendations, caused by the lack of interest or the nonavailability of technical or financial means for implementing the recommendations, occurred. On the whole, it has to be stated that the whole project was far from delivering the right value to its most important stakeholders, although it fulfilled all ministry-set indicators.

2.4.2. Stakeholders, Success Criteria and Metrics for the Case Project

There were four basic stakeholder groups in the project: the government, organization X, public units which should be monitored, and finally, disabled people. The sustainability principle requires subgroups of those stakeholders, e.g., individual members of the project team from organization X or various groups (with various disability types and degrees) of disabled people to be taken into account.

In the original project success definition, there were two criteria: as mentioned above, the number of institutions participating in the project (i.e., with which a contract was signed), the number of recommendations planned to be implemented, and the number of recommendations that were confirmed to be implemented by the monitored institutions. For both numbers, thresholds were set. Phase I of the project, which is the object of the study here, influenced merely the first criterion. A threshold of 13 was set for it. The institutions were selected from a pool of 30 institutions that were previously picked at random.

After project completion, it was understood that such a project success definition was not coherent with sustainable project management or even with the basic principles of stakeholder management, apart from the value-based project success definition considered in this paper. The negative consequences of such an approach were diverse. For example, low-cost recommendations were preferred by the monitored institutions, independently of their actual (non-financial) value for the disabled people. Moreover, different groups of disabled people were not explicitly identified with their specific needs and some of them did not feel that an acceptable value had been delivered to them. The need to obtain a certain number of institutions to sign the contract exercised pressure on the project team, which, as it was understood afterward, could have been alleviated if the project team and the values expected from them had been taken into account in the project success definition. Additionally, if a public institution was forced in some way to sign the contract, they were often not very involved in the project and tended to cooperate with the project team without due care and enthusiasm, concentrating on the pure number of recommendations linked with low cost and low effort and refusing to cooperate after a long period of having done so, creating problems connected with deadlines, etc.

Hence, post factum, based on the opinion of a project expert (one of the co-authors of this paper) and the recommendations formulated by the author of [4], it was proposed that project success was defined using the following criteria:

$C_1:$ satisfaction with the number of confirmed recommendations (this criterion takes into account the expectations of the ministry but should also consider other stakeholders). For the Lower Silesia region, to which we limited our case study, 30 potential institutions were randomly selected as candidates and the ministry set the threshold at 13—thus, the ministry was satisfied with this number, but disabled people, for example, were not necessarily content.

$C_2:$ satisfaction with the total of the value (not economic value, but subjective value from the point of view of disabled people) of the confirmed recommendations for individual groups of disabled people, weighted by the assumed “importance” of individual groups of disabled people (a deeper analysis of this criterion will be included in further research plans);

$C_3:$ satisfaction with the involvement of the participating institutions (this criterion takes into account the satisfaction of two groups of stakeholders: the disabled people, who would desire to be the focus of interest of the involved public institutions (instead of pure numbers and economic aspects), and the project team, for whom working with the involved institutions is easier;

$C_4:$ acceptable workload of the project team in organisation X during project realisation;

$C_5:$ acceptable level of stress (due among others to a high risk that the threshold of signed contracts will not be met in due time) of the project team from organisation X during project realisation;

Weights $w_i, i=1,\dots ,5$ should be agreed upon among project stakeholders.

Of course, the above proposal is merely tentative to formulate a project success definition which would be the best compromise for all stakeholders involved.

Then, for each criteria $C_i,i=1,\dots ,5,$ metrics ${ℳ}_i={\left\{m_j^i\right\}}_{j=1}^{L_i}$ would have to be defined. The metrics, controlled at certain intervals of time, should allow a prediction of the final value of the respective success criterion. Obviously, they will not be a very accurate estimate (especially at the end of Phase I), but they still would give an orientation of what we are heading at. Often a combination of several metrics would be most informative.

Below we present a far from exhaustive list of metrics for the selected success criteria. More metrics would have to be defined if we considered the whole project. Here, we concentrate on its first phase. We thus defined the following metrics:

• for criterion $C_1$: $m_1^{1 }\left(t\right)$: the number of institutions which have agreed to participate by moment t; $m_{2 }^1\left(t\right)$: the number of institutions which have been contacted by moment t;
• for criterion $C_2$ we will not define any metrics here, as in the first phase of the project no recommendations were identified yet;
• for criterion $C_3$: $m_1^{3 }\left(t\right)$: the number of institutions which have agreed to participate by moment t without the need for additional persuasion;
• for criterion $C_4$: $m_{1 }^4\left(t\right)$—number of interventions to persuade a hesitant or unwilling institution (phone calls, emails, meetings, letters, appeals), $m_2^4 \left(t\right)$—number of invitations sent ( $m_2^4 \left(t\right)$ will take on the same value as $m_2^{1 }\left(t\right)$, but it will be interpreted in another way: in the case of criterion $C_1$ the higher the metric value, the better, here the opposite is true);
• for criterion $C_5$: $m_1^5 \left(t\right)$: number of institutions which have refused to participate by moment t; $m_{2 }^5\left(t\right)$: number of institutions contacted by moment t which exceeds the minimum set by the ministry (reserve), $m_3^5 \left(t\right)$: number of institutions which refused to participate in the second round, after negotiations or appeals, $m_4^{5 }\left(t\right)$: number of institutions that refused to participate after the contract was signed (late refusal).

Some of the metrics will point directly to the ultimate value of the respective criterion, some will do it less directly:

• criterion $C_1$: $m_1^1 \left(t\right)$ will be closer to the ultimate value of the criterion, the later moment we consider, but even at the end of the considered phase it does not allow its ultimate value to be determined, because the institutions can quit the project even in later stages; $m_1^{2 }\left(t\right)$ is a very rough indicator, which does not take into account the refusal rate. It is of the highest importance at the beginning and loses its importance with time;
• criterion $C_3$: $m_1^{3 }\left(t\right)$ will be a very imperfect indicator of the ultimate value of the criterion at the end of the considered stage, because the actual involvement of the institutions will be known only in the later stages of the project;
• criterion $C_4$: $m_1^{4 }\left(t\right)$ and $m_2^4 \left(t\right)$ at the end of the considered project stage will represent the workload only in this stage, thus they will be very indirect indicators of the final criterion value; the value of the criterion diminishes with the values of the metrics (contrary to the metrics for $C_1$ and $C_3$);
• criterion $C_5$: this criterion is of an exceptional nature: it is not enough to look at the final values of the metrics, but also at their distribution over time (stress during the whole project duration counts, not only that in the final moment). It is thus proposed that the integral of the value of the stress predicted in subsequent moments is taken. The predicted value of stress in individual moments will increase with $m_{1 }^5\left(t\right)$, $m_{3 }^5\left(t\right)$, and $m_4^{5 }\left(t\right)$ (number of refusals in various stages) and decrease with $m_2^5 \left(t\right)$ (indicator concerning the reserve with respect to the minimal requirement)$.$

As defined above, we assume that the criteria express satisfaction with a certain project aspect in the given scale. The corresponding metrics have to be defined accordingly, so that they represent (individually or combined with other metrics) an estimation of the final criterion value. The following thresholds $m_{j,s}^i$, $i=1,\dots ,5$, $j=1,\dots ,L_i, s=0, 1, 2, 3, 4, 5$ can thus be defined such that:

• for criterion $C_1$: $f_r^1 \left(m_{r }^1\left(t\right)\right)=s$ if and only if $m_1^1 \left(t\right)\in \left[m_{r,s}^1,m_{r,s+1}^1\right]$, s = 1, …, 4, $f_r^{1 }\left(m_r^1 \left(t\right)\right)=0$ if and only if $m_{r }^1\left(t\right)<m_{r,1}^1$, $f_r^1 \left(m_r^1 \left(t\right)\right)=5$ if and only if $m_r^1 \left(t\right)>m_{r,5}^1$, for r = 1, 2;

We have then:

$$C_{1 }\left(\wp ,t\right)=R \left(0,5 \left(f_1^1 \left(m_1^1 \left(t\right)\right)+m_2^1 \left(m_2^1\left(t\right)\right)\right)\right)$$

(3)

where symbol R stands for $ROUND$—the closest integer number, choosing the minimal one in case of ambiguity. Of course, formula (3) (such as formulae (4)–(6)) is merely a proposal. The formulae should be agreed upon among stakeholders.

• for criterion $C_3$:

$$C_{3 }\left(\wp ,t\right)=f_1^3 \left(m_1^3 \left(t\right)\right)$$

(4)

where $f_1^3 \left(m_1^3 \left(t\right)\right)$ is defined analogously as the respective values for criterion $C_1$;

• for criterion $C_4$: $f_r^{4 }\left(m_r^{4 }\left(t\right)\right)=5-s$ if and only if $m_r^4 \left(t\right)\in \left[m_{r,s}^4,m_{r,s+1}^4\right]$, s = 1, …, 4, $f_r^{4 }\left(m_{r }^4\left(t\right)\right)=5$ if and only if $m_r^{4 }\left(t\right)<m_{r,1}^4$, $f_r^4 \left(m_r^{4 }\left(t\right)\right)=0$ if and only if $m_r^4 \left(t\right)>m_{r,5}^4$, for r = 1, 2

$$C_{4 }\left(\wp ,t\right)=R \left(0,5 \left(f_1^4 \left(m_1^4 \left(t\right)\right)+f_2^4 \left(m_2^4\left(t\right)\right)\right)\right)$$

(5)

• for criterion $C_5$: Function $f_r^5 \left(m_{r }^5\left(t\right)\right)$ will be defined as the respective functions for criterion $C_4$ for r = 1, 3, 4 and as the respective functions for $C_1$ for r = 2. Then we will have (we assume that the satisfaction with the reserve has the highest weight as far as the stress is concerned):

$$C_4 \left(\wp ,t\right)=R \left(\frac{{{\displaystyle \int }}_{0 }^t\left(0.4f_r^5 \left(m_2^5\left(w\right)\right)+0.2 \left(f_r^5 \left(m_1^5 \left(w\right)\right)+f_{r }^5\left(m_3^{5 }\left(w\right)\right)+f_r^{5 }\left(m_{4 }^5\left(w\right)\right)\right)\right)dw}{t}\right)$$

(6)

2.4.3. Model and Possible Implementation Forms for Phase I of the Project

As explained above, in Phase I, whose planned duration was expected to be 2 months, but was extended to almost the whole project, numerous problems occurred. As a result, it was felt that the project was not quite successful according to the sustainable stakeholder management and the value-based project definition presented above. Post factum, it was realised that if this phase had been implemented differently, the result may have been more satisfactory. It was considered that the simulation of this project phase would have allowed the project team to plan and prepare for it much better.

In Figure 1, we propose a model, worked out in the Vensima software, which we used for the simulation of the implementation of Phase 1 of the project to help the project team to choose its best implementation form and to help the stakeholders agree on the required value to be delivered by the project.

The model in Figure 1, with its two loops, represents various modes of implementing Phase I of the project. Work to complete stands for the number of institutions that still have to be contacted and refers initially to the threshold set by the ministry for the minimal number of institutions with which a contract had to be signed (13 in our case). During the simulation it will be diminished by the number of units which have responded positively, denoted as Work completed.

The inner loop represents the problem of negative answers, in which case new invitations have to be sent. Level of Returning Works is changed from 0 to 1 if such a situation occurs, and the number of negative answers feeds the Work to complete. Work postponed represents the answers which are delayed and require several activities whose aim is to urge an institution to respond and to agree to sign the contract. The number of negative answers feeds the value of Work to complete.

The outer loop represents refusals that come after the initial acceptance of the invitation (late refusals). The number of the late refusals feeds Work to complete (new invitations have to be sent). The exit Level of Losses stands for the late refusals when it is too late to send a new invitation and the refusal is permanent (no substitute can be found).

The following implementation modes were identified:

• Natural variant: invitations to 13 selected institutions are sent. If an institution refuses, another one from the pool of 30 is selected at random. No further persuasion is used;
• Negotiation variant: same as the natural variant, but persuasion (negotiations) and appeals are used in case of negative answers;
• Intermediate variant: invitations sent to 23 institutions selected from the pool of 30. The number of invited institutions is increased from 13 to 23 to create a reserve (over the threshold required by the ministry, which was 13), with the aim of reducing the stress of the project team. In case of the late refusal of one institution, an invitation to the other two is sent;
• Power variant: invitations to all potential 30 candidates are sent right at the beginning.

Each variant represents a certain way of implementing the project phase in question. The simulation model can help us to decide, using the control of simulated metrics values, which variant to choose for the actual implementation. In reality, the phase was implemented according to a scenario close to the negotiation variant, but this happened rather accidentally, with a conscious decision-making process.

3. Results

3.1. Application of the Approach to the First Stage of the Project

The general assumptions for the model are as follows: Initial time: 0; Final time: 54 weeks (we assume that all contracts should be signed in the first year). Moreover, in the example, we assumed that the probability that an institution agrees to participate without additional negotiations is 55% and the probability that it agrees after refusal (as a result of appeal or further negotiations) is 30%. For the other parameters, we assumed (by way of example) deterministic values: the speed of sending invitations: three pieces/week, waiting for the response: 10 weeks in the first round of invitations (without additional persuasion), 5 weeks after appeals or negotiations etc. The simulation tool allows the introduction of probability distributions for all parameters, but we chose to use deterministic values for the sake of simplicity.

We assumed the following thresholds for the criteria calculation:

The first two metrics should indicate the chances of fulfilling criterion 1 (satisfaction of the ministry with the number of confirmed recommendations) to a sufficient degree. They represent the number of institutions which have agreed to participate and those which have been contacted. The second metric has higher thresholds for individual criterion values because it refers to institutions that have not confirmed their participation.

The third metric refers to the number of institutions that participate with involvement. For the moment, its value and thresholds are equal to those of $m_1^1$, as at the end of Phase I the involvement cannot be evaluated yet. We might consider the introduction of higher thresholds, assuming the rate of involved institutions is smaller than 100%.

The fourth and fifth metrics refer to the workload, measured by the number of reactions after refusal (negations, appeals, etc.) and invitations sent. The nature of the two metrics imposes a considerable difference of thresholds (the number of interventions is per se much lower than the number of invitations sent). However, in the future, these metrics should be modified, based on the effort linked to individual invitations and interventions (conducting negotiations may be much more effort consuming than sending an invitation).

The last four metrics are related to the stress of the project team in organization X. Metric $m_2^5$ represents the reserve over the minimal ministry requirement. It was assumed that the satisfaction with this reserve is 0 if there is no reserve, but already with one institution over the threshold 13, the satisfaction begins to be positive. The other three metrics for the fifth criterion represent the refusals. The last two metrics refer to the refusals in later stages: this is the reason for the lower thresholds than for metric $m_1^5$, which shows the refusals in the first stage.

Obviously, the thresholds in

Table 1. Thresholds for the metrics to be used in the calculation of the criteria values.

$\mathbf{Metrics} {\mathit{m}}_{\mathit{j}}^{\mathit{i}}$	${\mathit{m}}_{\mathit{j},\mathit{s}}^{\mathit{i}}, \mathit{s} = 1, 2, 3, 4, 5$
$m_1^1$	10, 12, 14, 15, 18
$m_2^1$	14, 16, 18, 20, 22
$m_1^3$	10, 12, 14, 16, 18
$m_1^4$	1, 2, 3, 4, 5
$m_2^4$	16, 18, 20, 22, 24
$m_1^5$	5, 6, 8, 10, 12
$m_2^5$	1, 6, 9, 10, 12
$m_3^5$	1, 2, 3, 4, 5
$m_4^5$	1, 2, 3, 4, 5

should be agreed upon among stakeholders. Table 1 represents merely a proposal.

In

Table 2. Simulation of the natural variant.

Period (Weeks)	Activity	Values of the Metrics
0–5	Sending invitations to 13 selected units	$m_1^1 \left(5\right)=0, m_2^1 \left(5\right)=13$ $m_1^3 \left(5\right)=0$ $m_1^4 \left(5\right)=0$, $m_2^4 \left(5\right)=13$ $m_1^5 \left(5\right)=0$,$m_2^5 \left(5\right)=0$, $m_3^5\left(5\right)=0$, $m_4^5\left(5\right)=0$
5–15	Waiting for responses. Result: 7 Yes, 6 No or no response.	$m_1^1 \left(15\right)=7, m_2^1 \left(15\right)=13$ $m_1^3 \left(15\right)=7$ $m_1^4 \left(15\right)=0$, $m_{2 }^4\left(15\right)=13$ $m_1^5 \left(15\right)=6$,$m_2^5 \left(15\right)=0$, $m_3^5 \left(15\right)=0$, $m_4^5 \left(15\right)=0$
15–17	Dispatching invitations to 6 more units	$m_1^1 \left(17\right)=7, m_2^1 \left(17\right)=19$ $m_1^3 \left(17\right)=7$ $m_1^4 \left(17\right)=0$, $m_2^4 \left(17\right)=19$ $m_1^5 \left(17\right)=6$,$m_2^5 \left(17\right)=6$, $m_3^5 \left(17\right)=0$, $m_4^5 \left(17\right)=0$
17–27	Awaiting responses. Result 3 Yes, 3 No	$m_1^1 \left(27\right)=10, m_2^1 \left(27\right)=19$ $m_1^3 \left(27\right)=10$ $m_1^4 \left(27\right)=0$, $m_2^4 \left(27\right)=19$ $m_1^5 \left(27\right)=9$,$m_2^5 \left(27\right)=6$, $m_3^5 \left(27\right)=0$, $m_4^5 \left(27\right)=0$
27–28	Mailing invitations to 3 more units	$m_1^1 \left(28\right)=10, m_2^1 \left(28\right)=22$ $m_1^3 \left(28\right)=10$ $m_{1 }^4\left(28\right)=0$, $m_2^4 \left(28\right)=22$ $m_1^5 \left(28\right)=9$, $m_2^5 \left(28\right)=9$, $m_3^5 \left(28\right)=0$, $m_4^5 \left(28\right)=0$
28–38	Awaiting responses. Result: 2 Yes, 1 No	$m_1^1 \left(38\right)=12, m_2^1 \left(38\right)=22$ $m_{1 }^3\left(38\right)=12$ $m_1^4 \left(38\right)=0$, $m_2^4 \left(38\right)=22$ $m_1^5 \left(38\right)=10$, $m_2^5 \left(38\right)=9$, $m_3^5 \left(38\right)=0$
38–39	Invitation sent to another unit	$m_1^1 \left(39\right)=12, m_2^{1 }\left(39\right)=23$ $m_1^3 \left(39\right)=12$ $m_1^4 \left(39\right)=0$, $m_2^4 \left(39\right)=23$ $m_1^5 \left(39\right)=10$,$m_2^5 \left(39\right)=10$, $m_3^5\left(39\right)=0$, $m_4^5 \left(39\right)=0$
39–49	Awaiting responses. Result: 1 Yes, 0 No	$m_1^1 \left(49\right)=13, m_2^1 \left(49\right)=23$ $m_1^3 \left(49\right)=13$ $m_1^4 \left(49\right)=0$, $m_{2 }^4\left(49\right)=23$ $m_1^5 \left(49\right)=10$,$m_2^5 \left(49\right)=10$, $m_3^5 \left(49\right)=0$,$m_4^5 \left(49\right)=0$
49–52	We sign agreements with 13 units	As above

, we present the simulation results for variant A (natural variant).

In the simulation here, we assumed no late refusals (after negotiations or after signing the agreement). The simulation model could be changed, to introduce a certain late refusal probability. Figure 2 presents the values of Work to complete and Work completed during the duration of Phase 1 under the considered variant:

The changes of the slopes in both curves represent the moments (weeks 15, 27, 28) when negative answers arrived, and it was found that new institutions had to be contacted to replace the ones that had refused.

Using formulae (3)–(6) for the end of the stage, thus t = 52, we obtained the following (

Table 3. Predictive values of the criteria for the end of Phase 1 in the natural variant.

Criterion	Value
$C_{1 }\left(\wp ,52\right)$	2
$C_{2 }\left(\wp ,52\right)$	Not available (NA)
$C_3 \left(\wp ,52\right)$	2
$C_4 \left(\wp ,52\right)$	3
$C_5 \left(\wp ,52\right)$	3

) predictive values of the criteria (criterion $C_2$ cannot be evaluated yet):

We can see that even though we assumed no late refusals, the fulfillment of the criteria was not highly satisfactory, especially in relation to the estimation of the number of institutions that will merely participate, and of those which will do so with certain enthusiasm (criteria $C_1$ and $C_3$) are low. This is because only a minimal number of institutions were contacted at the beginning. On the other hand, thanks to this fact, the workload was fairly satisfactory. Stress was acceptable too, because of no late refusals and of the fact that additional contacts were established in the case of initial refusals.

In the negotiation variant (

Table 4. Simulation of the negotiation variant.

Period (Weeks)	Activity	Values of the Metrics
0–5	Sending invitations to 13 selected units	$m_{1 }^1\left(5\right)=0, m_2^1 \left(5\right)=13$ $m_1^3 \left(5\right)=0$ $m_1^4 \left(5\right)=0$, $m_{2 }^4\left(5\right)=13$ $m_1^5 \left(5\right)=0$,$m_{2 }^5\left(5\right)=0$, $m_3^{5 }\left(5\right)=0$, $m_4^5 \left(5\right)=0$
5–15	Waiting for responses. Result: 7 Yes, 6 No or no response.	$m_1^1 \left(15\right)=7, m_2^1 \left(15\right)=13$ $m_{1 }^3\left(15\right)=7$ $m_1^4 \left(15\right)=0$, $m_2^4 \left(15\right)=13$ $m_1^5 \left(15\right)=6$,$m_2^5\left(15\right)=0$, $m_{3 }^5\left(15\right)=0$, $m_4^{5 }\left(15\right)=0$
15–17	Negotiations 6 units which refused	$m_{1 }^1\left(17\right)=7, m_2^1 \left(17\right)=13$ $m_1^3 \left(17\right)=7$ $m_{1 }^4\left(17\right)=6$, $m_2^4 \left(17\right)=13$ $m_1^5 \left(17\right)=6$,$m_{2 }^5\left(17\right)=0$, $m_3^5 \left(17\right)=0$, $m_4^5 \left(17\right)=0$
17–22	Awaiting responses. Result 2 Yes, 4 No	$m_1^1 \left(22\right)=9, m_{2 }^1\left(22\right)=13$ $m_1^3 \left(22\right)=7$ $m_1^4 \left(22\right)=6$, $m_2^4 \left(22\right)=13$ $m_1^5 \left(22\right)=4, m_2^5 \left(22\right)=0$, $m_3^5 \left(22\right)=4$, $m_4^5 \left(22\right)=0$
22–24	Mailing to 4 more units	$m_1^1 \left(24\right)=9, m_2^1 \left(24\right)=17$ $m_1^{3 }\left(24\right)=7$ $m_{1 }^4\left(24\right)=6$, $m_2^4 \left(24\right)=17$ $m_1^5 \left(24\right)=4, m_{2 }^5\left(24\right)=4$, $m_3^5 \left(24\right)=4$, $m_4^5 \left(24\right)=0$
24–34	Awaiting responses. Result: 2 Yes, 2 No	$m_1^1 \left(34\right)=11, m_2^1 \left(34\right)=17$ $m_1^3 \left(34\right)=9$ $m_1^4 \left(34\right)=6$, $m_2^4 \left(34\right)=17$ $m_1^5 \left(34\right)=6, m_2^{5 }\left(34\right)=4$, $m_3^5 \left(34\right)=4$, $m_4^5 \left(34\right)=0$
34–35	Negotiations with 2 units which have refused	$m_1^1 \left(35\right)=11, m_2^1 \left(35\right)=17$ $m_1^3 \left(35\right)=9$ $m_1^4 \left(35\right)=8$, $m_{2 }^4\left(35\right)=17$ $m_1^5 \left(35\right)=6, m_2^{5 }\left(35\right)=4$, $m_{3 }^5\left(35\right)=4, m_4^5 \left(35\right)=0$
35–40	Awaiting responses. Result: 1 Yes, 1 No	$m_1^{1 }\left(40\right)=12, m_2^1 \left(40\right)=17$ $m_1^3 \left(40\right)=9$ $m_1^4 \left(40\right)=8$, $m_{2 }^4\left(40\right)=17$ $m_1^5 \left(40\right)=5, m_{2 }^5\left(40\right)=4$, $m_3^5 \left(40\right)=5$, $m_4^5 \left(40\right)=0$
40–41	Sending invitation to another unit	$m_1^1 \left(41\right)=12, m_2^1 \left(41\right)=18$ $m_1^3 \left(41\right)=9$ $m_1^4 \left(41\right)=8$, $m_2^4 \left(41\right)=18$ $m_1^5 \left(41\right)=5, m_2^{5 }\left(41\right)=5$, $m_3^5 \left(41\right)=5$ $m_4^5 \left(41\right)=0$
41–51	Awaiting response: yes	$m_1^1 \left(51\right)=13, m_2^{1 }\left(51\right)=18$ $m_1^3 \left(51\right)=10$ $m_1^4 \left(51\right)=8$, $m_{2 }^4\left(51\right)=18$ $m_1^5 \left(51\right)=5, m_2^5 \left(51\right)=5$, $m_{3 }^5\left(51\right)=5, m_4^5 \left(51\right)=0$
51–54	We sign agreements with 13 units	As above

), we proceeded in a similar way to the natural variant: at the beginning, we sent invitations to 13 selected districts. The difference was that in the case of initial refusal, we undertook the effort of negotiating or raising appeals (weeks 15–17 and 34–35).

In this case, we had refusals after negotiations $(m_4^5$), but a lower number of refusals generally $\left(m_1^5\right)$. Figure 3 presents the values of Work to complete and Work completed during the duration of the phase under the negotiation variant:

We observed a quicker increase in Work completed around the 18th week than in the natural variant, thanks to the negotiations with the institution that initially refused.

Table 5. Predictive values of the criteria for the end of Stage 1 in the negotiation variant.

Criterion	Value
$C_1 \left(\wp ,54\right)$	2
$C_2 \left(\wp ,54\right)$	NA
$C_3 \left(\wp ,54\right)$	1
$C_4 \left(\wp ,54\right)$	1
$C_5 \left(\wp ,54\right)$	2

presents the estimation of the final criteria values according to the selected formulae.

We can see that this variant turned out to be worse than the natural one. Of course, this observation is strongly dependent on the choice of formulae (3)–(6), but in the present situation the refusals after negotiations and the mere need to negotiate decreased criteria $C_3, C_4, C_5.$ It is important to note that in reality the phase was implemented similarly to this variant.

In the intermediate variant (

Table 6. Simulation of the intermediate variant.

Period (Weeks)	Activity	Values of the Metrics
0–8	Sending invitations to 23 units	$m_1^1 \left(8\right)=0, m_2^1 \left(8\right)=23$ $m_1^3 \left(8\right)=0$ $m_1^4 \left(8\right)=0$, $m_{2 }^4\left(8\right)=23$ $m_1^5 \left(8\right)=0$,$m_2^5 \left(8\right)=10$, $m_3^{5 }\left(8\right)=0$, $m_4^5 \left(8\right)=0$
8–18	Awaiting responses: 13 Yes, 10 No or no answer	$m_1^1 \left(18\right)=13, m_2^1 \left(18\right)=23$ $m_1^3 \left(18\right)=13$ $m_1^4 \left(18\right)=0$, $m_2^4 \left(18\right)=23$ $m_1^5 \left(18\right)=10$,$m_2^5 \left(18\right)=10$, $m_3^5 \left(18\right)=0$, $m_4^5 \left(18\right)=0$
18–23	No activity, but at the end of the period unexpected late refusal of one unit	$m_1^1 \left(23\right)=12, m_{2 }^1\left(23\right)=23$ $m_1^3 \left(23\right)=12$ $m_1^4 \left(23\right)=0$, $m_2^4 \left(23\right)=23$ $m_1^5 \left(23\right)=11$,$m_{2 }^5\left(23\right)=10$, $m_3^{5 }\left(23\right)=0$, $m_4^5 \left(23\right)=1$
23–24	Sending invitation to 2 more units	$m_1^1 \left(24\right)=12, m_2^1 \left(24\right)=25$ $m_1^3 \left(24\right)=12$ $m_1^4 \left(24\right)=0$, $m_2^{4 }\left(24\right)=25$ $m_1^5 \left(24\right)=11$,$m_2^5 \left(24\right)=12$, $m_{3 }^5\left(24\right)=0$, $m_4^5 \left(24\right)=1$
24–34	Awaiting responses, 1 yes, 1 no	$m_1^1 \left(34\right)=13, m_2^1 \left(34\right)=25$ $m_1^3 \left(34\right)=13$ $m_1^4 \left(34\right)=0$, $m_2^4 \left(34\right)=25$ $m_1^5 \left(34\right)=12$,$m_2^5 \left(34\right)=12$, $m_3^5 \left(34\right)=0$, $m_4^5 \left(34\right)=1$
34–53	No activity, but at the end of the period unexpected late refusal of one unit	$m_1^1 \left(54\right)=12, m_2^1 \left(54\right)=25$ $m_1^3 \left(54\right)=12$ $m_1^4 \left(54\right)=0$, $m_{2 }^4\left(54\right)=25$ $m_1^5 \left(54\right)=13$,$m_2^{5 }\left(54\right)=12$, $m_{3 }^5\left(54\right)=0$, $m_4^5 \left(54\right)=2$
53–54	We sign an agreement with 12 units	As above

) a reserve concerning the requirement of the ministry was built from the beginning (we sent invitations to 23 institutions instead of 13). Apart from that, we considered here the possibility of late refusal of institutions that had initially agreed to participate (weeks 23 and 53). Such situations occurred in reality and should be considered in the model. The latter refusal came so late it was not possible to find a replacement (Stream of Losses in Figure 1).

Figure 4 presents the values of Work to complete and Work completed during the duration of the phase under the intermediate variant:

We observed a rapid increase in Work completed (because of a higher than 13 number of invitations sent), but also two drops of its value, due to late refusals. The predictive values of the criteria are given in

Table 7. Predictive values of the criteria for the end of Stage 1 in the intermediate variant.

Criterion	Value
$C_{1 }\left(\wp ,54\right)$	3
$C_{2 }\left(\wp ,54\right)$	NA
$C_3 \left(\wp ,54\right)$	2
$C_4 \left(\wp ,54\right)$	2
$C_5 \left(\wp ,54\right)$	4

.

Despite the late refusals, the value of the criterium related to stress was better than in the case of the preceding variants. This is due to a reserve that was created right at the beginning.

In the power variant (

Table 8. Simulation of the power variant.

Period (Weeks)	Activity	Values of the Metrics
0–10	Sending invitations to all 30 units	$m_1^1 \left(10\right)=0, m_2^1 \left(10\right)=30$ $m_1^3 \left(10\right)=0$ $m_1^4 \left(10\right)=0$, $m_2^4 \left(10\right)=30$ $m_1^5 \left(10\right)=0$,$m_2^5 \left(10\right)=17$, $m_3^5 \left(10\right)=0$, $m_4^5\left(10\right)=0$
10–20	Awaiting responses: 17 Yes, 13 No or no answer	$m_1^1 \left(20\right)=17, m_2^1 \left(20\right)=30$ $m_1^3 \left(20\right)=17$ $m_1^4 \left(20\right)=0$, $m_2^4 \left(20\right)=30$ $m_1^5 \left(20\right)=13$,$m_2^5 \left(20\right)=17$, $m_3^5\left(20\right)=0$, $m_4^5 \left(20\right)=0$
20–23	No activity, but at the end of the period unexpected late refusal of one unit	$m_1^1 \left(23\right)=16, m_2^1 \left(23\right)=30$ $m_1^3 \left(23\right)=16$ $m_1^4 \left(23\right)=0$, $m_2^4\left( 23\right)=30$ $m_1^5 \left(23\right)=14$,$m_2^5 \left(23\right)=17$, $m_3^5 \left(23\right)=0$, $m_4^5 \left(23\right)=1$
23–24	Sending invitation to 2 more units	$m_1^1 \left(24\right)=16, m_2^1 \left(24\right)=32$ $m_1^3 \left(24\right)=16$ $m_1^4 \left(24\right)=0$, $m_2^4 \left(24\right)=32$ $m_1^5 \left(24\right)=14$,$m_2^5 \left(24\right)=19$, $m_3^{5 }\left(24\right)=0$, $m_4^{5 }\left(24\right)=1$
24–34	Awaiting responses, 1 yes, 1 no	$m_1^1 \left(34\right)=17, m_2^1 \left(34\right)=32$ $m_1^3 \left(34\right)=17$ $m_1^4 \left(34\right)=0$, $m_2^4 \left(34\right)=32$ $m_1^5 \left(34\right)=15$,$m_2^5 \left(34\right)=19$, $m_3^{5 }\left(34\right)=0$, $m_4^5 \left(34\right)=1$
34–53	No activity, but at the end of the period unexpected late refusal of one unit	$m_1^1 \left(54\right)=16, m_2^1 \left(54\right)=32$ $m_1^3 \left(54\right)=16$ $m_1^4 \left(54\right)=0$, $m_{2 }^4\left(54\right)=32$ $m_1^5 \left(54\right)=16$,$m_2^5 \left(54\right)=19$, $m_3^5 \left(54\right)=0$, $m_4^5 \left(54\right)=1$
54–54	We sign an agreement with 16 units	As above

), we created at the beginning the highest possible reserve, sending the invitation to all institutions from the pool. Here too, we assumed the occurrence of late refusals. In such a case, we sent an invitation to an institution from outside the initial pool of 30 units. The search for such institutions might mean an additional workload, not taken into account in the present definition of the respective criterion.

As we can see in Figure 5, despite the late refusals, Work completed rapidly attained a high level because of the reserve created immediately at the beginning. Apart from a high number of institutions that agreed to participate, this reserve assured the feeling of security, but on the other hand, it created a high workload (because of a high number of invitations). All this is visible in

Table 9. Predictive values of the criteria for the end of Stage 1 in the power variant.

Criterion	Value
$C_{1 }\left(\wp ,54\right)$	4
$C_{2 }\left(\wp ,54\right)$	NA
$C_{3 }\left(\wp ,54\right)$	4
$C_{4 }\left(\wp ,54\right)$	2
$C_{5 }\left(\wp ,54\right)$	4

.

This variant has the best values concerning all criteria, apart from the workload. The need to send 32 invitations altogether meant much more work than in the case of the other variants. However, thanks to reducing the value of this criterion, the other criteria attained higher values.

3.2. Conclusions from the Model Application and Further Application Possibilities for the Case Project

In the previous subsection, we presented the simulation results for four different implementation forms of Phase I of the project in question. We took into account four of the five proposed criteria—the value of the omitted one could not be estimated in the considered project phase. It was possible to observe the criteria values and to determine that their mutual relationships varied from one implementation form to another. In this subsection, we illustrate the procedure of choosing one implementation form to be put into practice.

Table 10. Predictive values of the criteria for the end of Stage 1 in all considered variants.

Criterion	Natural	Negotiation	Intermediate	Power
$C_1 \left(\wp ,54\right)$	2	2	3	4
$C_3 \left(\wp ,54\right)$	2	1	2	4
$C_4 \left(\wp ,54\right)$	3	1	2	2
$C_5 \left(\wp ,54\right)$	3	2	4	4

presents the values of the four criteria for the four considered implementation variants.

The adopted procedure of defining project success requires the adoption of weights $w_i, i=1,\dots ,5, 0\le w_i\le 1$. For simplicity reasons, we assumed $w_2=0$. Let us then consider three different sets of criteria weights. In practice, the weights have to be agreed on among the stakeholders (see [4] for details). The weights are bound to be a controversial point and it is not possible to impose them without a debate among the stakeholders.

The first question which has to be answered is the value assigned to $\mathsf{{\rm T}}$. If $\mathsf{{\rm T}}\ge 3$, which is a natural decision (it seems logical to expect at least a medium degree of project success), then only the power variant will assure an acceptable predicted value of project success, and this is only under the assumption that criterion $C_4$(workload) does not have a too high weight. If the workload has a high weight (e.g., 0,7), the natural variant seems to be better: the predicted value of success will be closer to 3 than in case of the power variant. The intermediate variant would be fairly acceptable if the first criterion (the number of conformed recommendations) had a high weight (e.g., 0,7). The negotiation variant, close to the one selected in reality, is clearly the worst one.

The weights, together with the threshold $\mathsf{{\rm T}}$, have to be agreed upon among the stakeholders. Some of the stakeholders (above all, organization X) will also have a say as far as the choice of the implementation form is concerned. If, for example, the power variant will be excluded for some reason, then the project team, together with the other stakeholders, have to be aware that the predicted value delivered by the project is likely to be under 3. As the proposed procedure is applied in the predictive phase, even before the project definition is fully determined, there is still time to introduce other implementation forms which would ensure higher values of $\mho \left(\wp ,54\right)$ and then possibly, of the final criteria values.

Of course, the results in

Table 11. Predicted value $\mho \left(\wp ,54\right)$ delivered by the project for various criteria weights.

$$\mathbf{Weights} {\mathit{w}}_1,{\mathit{w}}_3,{\mathit{w}}_4,{\mathit{w}}_5$$	Natural	Negotiation	Intermediate	Power
0.1, 0.2, 0.4, 0.3	2.7	1.4	2.7	3.2
0.1, 0.1, 0.7, 0.1	2.8	1.2	2.3	2.6
0.7, 0.1, 0.1, 0.1	2.2	1.8	2.9	3.8

refer only to the first stage of the project and disregard one of the criteria. Thus, in the final analysis, all project stages and all criteria should be taken into account. However, even such a reduced application of the proposed approach would have supported the project team in the selection of a better implementation form than the one that was used in reality.

What is more, the discussion about the criteria and weights would have forced the stakeholders to communicate the value all of them expected from the project, which would have improved the communication in the project altogether.

4. Conclusions

In this paper, we proposed an approach to simulate projects or their stages in an early phase, prior to the end of the project definition process. The simulation model should encompass metrics, which should indicate the predictive value delivered by the project to the stakeholders. The stakeholders should be identified following the sustainability principle—thus, the less important ones should be taken into account, too. The formula for the success degree (formula 1), as well as the link between the metrics and the predictive criteria values (formulae 3, 4, 5, 6) should be agreed upon among the stakeholders.

The proposed approach allows:

• the project success definition to be verified and accepted by individual stakeholders;
• the option to “try out” various project implementation forms and to select the one which best suits the adopted project success definition;
• the introduction of changes in the project definition and implementation before the project has been started, and, in this way, to avoid the (usually high) cost of late changes or rework.
• to help all stakeholders to become acquainted with the project and among themselves and to clarify all misunderstandings.

An important limitation of the study is the fact that the proposed approach was applied to only one stage of one project. Further research will comprise a continuation of the simulation of the same project, where such important aspects as weights or values of various disability groups or recommendations will be modelled. Further on, other case studies will be covered too.

Additionally, it has to be underlined that the project considered here is a social project, implemented with the support of a ministry for a specific social group (disabled people) with the participation (not always voluntary and enthusiastic) of public institutions, which were selected at random. The results, criteria, and metrics used here could be easily adopted to projects with similar stakeholders, but in the case of other projects with varied stakeholders, methods that are not fully defined, or objectives that are not fully defined (e.g., software development, new product development or research projects) would lead to different criteria and metrics. However, the authors believe that some criteria (such as the stress of the project team) considered here may be important in most projects but are usually omitted. In this sense, the present paper may be useful with respect to any project type where the methods, objectives, and/or different expectations as to the value delivered by the project need clarification and balancing.

An important limitation and at the same time an important aspect that will have to be included in further research is the randomness. The simulation tool used here allows the introduction of random variables. Thus, the long-term research objective is to elaborate a comprehensive model for the project in question in which all phenomena (such as late refusals, lack of interest of the monitored institutions, the tendency to prefer low-cost recommendations, etc.) will be modelled as random variables. Such a model will make it possible to simulate the whole project and will serve as the basis for simulation models of other similar projects in the future.

The reference to “other projects” in the above sentences is important because simulation models need several objects for calibration and verification. We have to mention that similar projects were implemented in other Polish districts; thus, as part of future research, the model will be verified using several similar projects. A long term result we intend to aim at in our future research is thus not a case-based model, but rather a stylized model [21]; thus, one which offers the possibility to simulate similar, but still diverse projects. Such a model will be useful for defining and planning future projects implemented by organization X and similar ones.

However, the most important result of the paper is the general approach from Section 2.2, which can be applied to any project and will support all its stakeholders in obtaining the desired level of satisfaction with the value delivered to them. As projects which do not deliver the expected value have in fact no raison d’être, we believe that the approach will be helpful to numerous project managers and stakeholders.