Value Perception Analysis in the Brazilian Company of Research and Industrial Innovation

Abstract

This study aims to analyze the perceived value of services provided by the Brazilian Company of Research and Industrial Innovation (EMBRAPII) to its contracting ministries and institutional partners. It utilizes the theory of value perception analysis and Constructivist Multi-criteria Decision Analysis to identify critical elements for evaluating EMBRAPII’s contracting organizations. Brainstorming sessions with experts led to the identification of five criteria and 14 sub-criteria. These criteria include a relationship with EMBRAPII, a signed agreement, EMBRAPII’s reputation, technical capacity, and the ability to adapt to changes. Data were entered into the second version of the MyMCDA-C software for value perception analysis. The findings showed a positive perceived value, with the best-performing sub-criteria relating to the organization’s reputation and the agreement signed. The study concludes that EMBRAPII needs to improve in areas such as adapting to change, the adequacy of its proposals for distinct types of partnership, and social media positioning. However, the contracting organizations generally support EMBRAPII’s direction and proposed solutions.

1. Introduction

In order to generate value for the customer according to what they want from the service or product, it is necessary to know how they analyze what is delivered [1]. This notion can guide the decisions to be made by companies in favor of satisfying the customer and adding value to the company. Companies with a business strategy emphasis on marketing and value creation are able to meet stakeholder needs, respond better to social demands [2], and, consequently, have a positive market performance [3].

A significant challenge for companies is to understand what the customer and the stakeholders expect from a product or service and what is the basis for value definition [4]—for example, if it is based only on the price and quality of the product or service. One of the ways to achieve this definition is through a value perception analysis that consults customers and stakeholders about which benefits and costs are involved in buying a product or hiring a service. Being aware of the value delivered “[…] is important to a company’s clarification and communication of its own interests. In order to be considered a trustworthy cooperation partner, a company must have a ‘‘face’’ that identifies it as such.” [5] (p. 270).

The company’s understanding of value is important but difficult to define [6]. Value can mean different things to different people, and each company’s market segment has different perceptions of value. Therefore, the customer’s assessment of the value of a product or service must be subjective, and the company needs to adapt to conform to the different perceptions of value in each customer segment [7].

In situations in which the company needs to understand the perceived value of the organizations that invest in their services, it faces a challenging task, and a method of analysis is welcome. In this context, this work conducted a case study in the Brazilian Company of Research and Industrial Innovation (EMBRAPII), a non-profit legal entity qualified as a social organization by the Federal Government of Brazil, which promotes opportunities for industrial companies to strengthen their innovation through projects in cooperation with 96 technological research institutions, credited as EMBRAPII Units by the organization. This is possible due to the contracts that EMBRAPII maintains with government ministries and the partnerships signed with actors that support innovation, such as the Brazilian Development Bank (BNDES), business associations, industry federations, the Brazilian Micro and Small Business Support Service (SEBRAE), the National Service for Industrial Training (SENAI), and international development institutions. Thus, EMBRAPII offers services for the promotion of national and international projects, the management of public resources for innovation policies, and technical cooperation with development institutions.

After ten years of operation, EMBRAPII’s portfolio includes 2206 projects supported, 1500 companies supported, and more than BRL 3.24 billion invested in corporate research, development, and innovation (R,D&I) projects [8]. Part of this amount comes from four government ministries—including Science, Technology, and Innovation; Education, Health and Development; and Industry, Commerce, and Services—and also from Brazilian public policies. Within the scope of technical cooperation, EMBRAPII has partnerships with 35 Brazilian institutions and 17 international institutions from 12 countries, the European Union, and the United Kingdom.

Because EMBRAPII has played a crucial role in the Brazilian innovation scenario, it is considered essential to analyze the value perception of the services provided from the point of view of the partners, that is, from the point of view of the Brazilian ministries and national and international institutions that support EMBRAPII’s work, both with resource contributions and with essential institutional relationships for EMBRAPII’s performance. The goal was to analyze whether the services correspond to the expectations of institutions and public agencies regarding EMBRAPII’s performance in the Brazilian innovation system and the activities involved in the partnership.

Besides that, the projects demanded by industrial companies and the EMBRAPII Units, the technological research institutions responsible for executing the projects, are already evaluated both by the contracting companies and by the EMBRAPII team. In the first semester of 2023, the results of the evaluation showed that 94% of the companies were delighted with the quality of project delivery, and all the EMBRAPII Units’ performances were approved by the technical team [9] (p. 19).

However, the perceived value of its institutional partners is unknown to EMBRAPII’s senior management. With this definition, the opinion of all stakeholders of the social organization is identified. It enables the development of a more proactive approach regarding the services provided, aiming to enhance the client’s perceived value.

In this matter, the present study presented the organization with the need to measure the perceived value by supervisory government agencies and institutional partners and to determine which criteria would be fundamental. In order to deal with the search, the work set the goal of analyzing the value perception of the services provided by EMBRAPII by the stakeholders who support the development of its activities. With that, it intends to mitigate or eliminate the risks inherent to the relationship and explore or improve existing opportunities.

To achieve the results proposed by the case study, Constructivist Multi-criteria Decision Analysis (MCDA-C) was applied to encompass the multiple factors involved in the value perceived by the customer. After applying and validating the method, the analysis of the perceived value across EMBRAPII’s service was positive, which infers that the contracting ministries and institutional partners see value in the organization when analyzing the criteria for the relationship with EMBRAPII, the agreement signed, EMBRAPII’s reputation, technical capacity, and ability to adapt to change. Within each criterion, some factors need EMBRAPII’s attention to improve them, because of the impact on the overall perceived value of the organization.

The structure of the paper is organized as follows. Section 2 presents the theoretical foundation of perceived value and Constructivist Multi-criteria Decision Analysis. Section 3 deals with research methodology. Section 4 details the construction of the self-evaluation model of the case study based on Constructivist Multi-criteria Decision Analysis. Section 5 discusses the study’s results. Lastly, in Section 6, the relevant conclusions and future directions are presented.

2. Literature Review

2.1. Perceived Value

The concept of perceived value available in the literature differs according to the perspective of each author and the line of research, but parts of the arguments corroborate with each other. Therefore, it is essential to bring different conceptualizations from the beginning of the discussion to the present.

Zeithaml’s research about the quality and value of a product in the beverages market from the perspective of the client defined perceived value as “the consumer’s overall assessment of the utility of a product based on perceptions of what is received and what is given” [10] (p. 14).

To Woodruff, customer value is about preference and the evaluation of product attributes and consequences from use that facilitate, or not, the achievement of the customer’s goals and purposes. The author concludes by pointing out that knowing the customer’s perceived value is an important management tool that must be understood by the company and implemented through sales strategies [11].

Sheth, Newman, and Gross establish forms of perceived value that are analyzed by the client and can assess multiple characteristics of a product or service [12]:

• Functional value: capacity to deliver functional performance;
• Social value: positive or negative association with one or more specific social groups;
• Emotional value: the capacity to arouse feelings or affective states;
• Epistemic value: the capacity to arouse curiosity, offer novelty, and/or satisfy the desire for knowledge;
• Conditional value: value perceived as a result of a specific situation or set of circumstances that enhances the functional or social value.

In more recent studies, Kotler and Keller determine perceived value as the difference between the consumer evaluation of all the benefits and costs and the known alternatives. They further determine that the benefits can be economic, functional, and psychological, and the costs can be monetary, time-related, physical, and psychological. Thus, for the authors, perceived value can be defined by the ratio of consumers’ benefits and costs without the limitation of a singular analysis of price, quality, or value. They add that knowledge of the perceived value generates a competitive advantage because the consumer also evaluates the perceived alternatives [13].

Grönroos and Ravald assert that the customer should be responsible for assigning value to what he is buying. They place value creation in the context of services, understanding that co-creating value with customer participation can be confusing. With this, the authors suggest that companies conform to the values created by customers, instead of making them follow the processes they have determined [14].

There is an interrelationship between the authors regarding the creation of perceived value, which must come from the customer and, consequently, generate an adaptation of companies to the different perceptions of value raised from an analysis. The objective of this work is to analyze the perceived value of the stakeholders involved in order to understand the values attributed to EMBRAPII’s service.

Both a qualitative and a quantitative analysis are essential for analyzing perceived value beyond the common factors of price and quality. For the case study in this work, it is essential to analyze other factors, since the context of the company does not involve the purchase of a product or a service but rather the experience of using the service and the benefits generated.

In order to fulfill the needs of a value perception analysis, the authors chose the Constructivist Multi-criteria Decision Analysis method, seeking to develop a self-evaluation model for EMBRAPII by identifying different criteria of analysis according to the opinions of decision-makers and what they sought to understand about stakeholders’ and institutional partners’ opinions of the services provided.

2.2. Constructivist Multi-Criteria Decision Analysis

One way to accomplish a value perception analysis in order to address several factors is to use Multi-Criteria Decision Analysis (MCDA). Ishizaka and Nemery define MCDA as “a discipline that encompasses mathematics, management, informatics, psychology, social science and economics. Its application is even wider as it can be used to solve any problem where a crucial decision needs to be made. These decisions can be either tactical or strategic, depending on the time perspective of the consequences.” [15] (p. 2).

In recent studies about decision aid, the approaches intended to propose methods that aim to reduce the probability of error in the subjective process of determining the relevance and weights of the criteria considered. According to Więckowski, Kizielewicz, Shekhovtsov, and Sałabun, to perform correctly, the method should be repeatable and exact and incite the experts not to hesitate. Therefore, the authors proposed the novel RANking COMparison (RANCOM) method that transforms the experts’ assessments into a score value [16].

Another mechanism for decision aid that has similar objectives to those that recent studies show is Constructivist Multi-criteria Decision Analysis, which is defined as the “ability to provide conditions for the identification, implementation, and measurement of the criteria that represent the perception of the decision makers about the possibilities of a self-evaluation performance model. It also enables the incorporation of improvement suggestions for alternatives, with a performance profile incompatible with expectations.” [17] (p. 5). Thus, this use for MCDA was chosen with the objective of involving both stakeholders and partners, as well as EMBRAPII’s decision-makers. The proximity of the researchers of this work and the researchers of the method was also considered when choosing the MCDA-C approach, which is detailed below.

The protocol applied was based on the constructivist methodology developed by [18], which is divided into three phases—structuring, evaluation, and recommendations—all detailed below.

The Structuring Phase incorporates, first, the contextualization of the problem, the survey label, and the definition of the actors involved in the decision-making. Then, it is sought to develop the Tree of Points of View and define with the actors the Primary Elements of Evaluation that, subsequently, are transformed into Fundamental Points of View and Elementary Points of View. Each EPV must be assigned a rating related to its impact level.

The Evaluation Phase encompasses the phases of constructing the self-evaluation model and determining contribution rates and effort levels. The overall evaluation of the points of view presented by the decision-makers is carried out using graphs that illustrate the impacts of each decision-maker’s point of view.

With the graphs, it is possible to move on to the Recommendations Phase, in which the strategies to be followed are determined by what was found with the application of the constructivist multi-criteria analysis method. At this stage, decision-makers rely on the results of the impacts of each Elementary Point of View to improve the performance of the question—for example, the value perception analysis.

Considering this literature review, this work will use the theories of perceived value and Constructivist Multi-criteria Decision Analysis to carry out a case study about the analysis of the perceived value by stakeholders involved with EMBRAPII. The value perception analysis must guide the value given to contracting ministries and Brazilian and international institutional partnerships.

3. Research Methodology

3.1. Classification of the Research

This study sought to understand EMBRAPII’s relations with its stakeholders, considering their multiple and subjective realities and their perception of value in the face of the partnerships signed. These aspects were understood through research with a qualitative approach, one in which the researcher uses the participants’ point of view to understand the phenomena of the problem [19].

Regarding its objectives, this research is classified as exploratory, as it seeks to provide a greater familiarity with the problem, making it more explicit and considering several aspects related to the phenomenon studied [20]. As for its purpose, it is classified as applied research, based on the concept that this type of research is directed to the solution of specific problems and involves local truths and interests [21].

The present work is considered a case study, since it deals with a contemporary phenomenon in its real-world context from the perspective of multiple sources of evidence [22]. The case study was carried out at the Brazilian Company of Research and Industrial Innovation located in Brasília, Brazil, from February 2021 to May 2021.

3.2. Company Characterization

The Brazilian Company of Research and Industrial Innovation (EMBRAPII) is a non-profit legal entity qualified as a social organization by the Federal Government of Brazil. EMBRAPII has the mission of contributing to the development of innovation in the Brazilian industry by strengthening its collaboration with research institutes and universities. It is the case study of this research.

Due to the management contract EMBRAPII holds with the Brazilian government ministries of Science, Technology and Innovation, Education, Health and Development, Industry, Commerce, and Services, the results achieved by the company are monitored every six months and evaluated annually by the Monitoring and Evaluation Committee of the contract, concerning the goals and performance indicators established [23].

Regarding the organization’s management and operation, the evaluation and monitoring are periodically undertaken by different actors. According to EMBRAPII’s bylaws, four bodies make up the organization’s administration: a General Assembly, Administrative Council, Board of Directors, and Fiscal Council. Among them, two are responsible for evaluation and inspection: the Administrative Council, EMBRAPII’s highest guidance and deliberation body that oversees compliance with the defined guidelines and goals, and the Fiscal Council, the financial and accounting supervisory body [24]. Figure 1 shows the relationship of the stakeholders based on the knowledge acquired by the authors during the research and brainstorming carried out in the case study.

This study focused on analyzing the perception of value of the institutional partners and supervisory bodies regarding the service provided by EMBRAPII. These two groups were selected because they are part of the company’s institutional relationship and, as said before, are not included in the other evaluations conducted by EMBRAPII.

The following section lists the methodology stages in providing the perceived value of EMBRAPII, considering the points of view of the institutional partners and the supervisory bodies.

3.3. Methodology Stages

The first two stages of the research dealt with the literature review and documentary research. A literature review was carried out on perceived value and Constructivist Multi-criteria Decision Analysis. Due to the approximation of one of the researchers with EMBRAPII, access to relevant documents, support materials, and sources was ensured. The documentary research had the role of assisting in the definition of which stakeholders and partnerships would participate in the research.

The third stage consisted of planning the case study for application in the chosen multi-criteria analysis method, described in the next section. At this point in the research, essential elements of the Structuring Phase of the model were defined, such as Primary Elements of Evaluation (PEE), Fundamental Points of View (FPVs), and Elementary Points of View (EPVs) with the support of actors who participate in brainstorming and a focus group formed by experts and decision-making agents within EMBRAPII.

In the fourth stage, data collection was carried out based on the formulation of the questionnaire, a vital instrument to understand the perception of value of the research participants regarding their relationships with the social organization. Through the Google Forms platform, the questionnaire was supplied to the contracting ministries and partners chosen to collect data on the perception of each Elementary Point of View, which were inputs for the MyMCDA-C software. This program was conceived by two professors from the Department of Business Administration at the University of Brasília (UnB) and developed by a computer engineering student, also from UnB. The software was created to simplify and increase the use of the constructivist multi-criteria analysis method in academic research and companies [25].

The fifth stage resulted in the analysis of value perception based on the use of the Constructivist Decision Support Methodology as a way to obtain the perspectives of the contracting ministries and partners about the relationship with EMBRAPII.

The sixth and final stage consisted of validating, with the participants of the research focus group, the study carried out and presenting the results obtained with the application of MCDA-C for the analysis of value perception. The application of the chosen methodology allowed us to obtain the perception of the value of the partnerships signed by the company and the parties involved. With the second version of MyMCDA-C, it was possible to analyze each criterion (FPV) and sub-criterion (EPV) of the value perception analysis in relation to the impacts they have on the perceived value of the service offered by EMBRAPII.

4. Self-Evaluation Model

The constructivist multi-criteria analysis methodology chosen has stages that involve the participation of the researchers, people involved in the decision-making process, and the survey respondents—in this case, the questionnaire for the analysis of the perception of value about EMBRAPII. In this case study, initially, essential elements of the Structuring Phase of the model were defined through brainstorming meetings, with the participation of professionals from the partner entities and EMBRAPII. These professionals chose five criteria and 14 sub-criteria for the analysis. Next, data collection via questionnaire was carried out to measure the value perception of the research participants regarding their relationship with EMBRAPII. In turn, the value perception analysis based on MCDA-C was used to obtain the perspectives of the ministries and national and international partners concerning the services of EMBRAPII. Finally, the results were validated for the value perception analysis with focus group participants. The details of these steps will be listed below.

4.1. Research Label and Actors

Based on the research problem and the objective, the research label of this work was defined as identifying the perceived value of institutional partners and contracting ministries regarding the service provided by EMBRAPII. The research label guides the research actors and the researchers on the objective of applying the method, so that the steps carried out to define the criteria and sub-criteria comply with the label and deliver what is necessary to answer the problem.

As for the research actors, those who intervene in the decision-making process in some way are divided into involved, decision-making, and moderator. According to the professor responsible for the software used in this work, the involved actors have “passive participation in the decision-making process, but receive the results or consequences of the decisions,” and the decision-making actors “formally hold power to decide.” The moderator “assumes the role of intermediary” [26] (p. 103).

In this case, the involved actors are the respondents of the value perception analysis survey about EMBRAPII, the decision-making actors are the employees responsible for managing the contracts and partnerships of the organization, and the moderators are the researchers. Figure 2 illustrates the relationship of the actors with the research and presents the strategic entities that supported the decision-making process with knowledge about the topic studied.

4.2. Brainstorming

In order to define the criteria and sub-criteria for analysis by the survey respondents, brainstorming meetings were held with experts in marketing and risk and quality management, represented by professors from the Production Engineering course at UnB and an EMBRAPII employee who specialized in public policies, with years of experience in the relationship with government ministries and institutional partners. The main question of the brainstorming was, “With what criteria do you think it is possible to understand the perceived value of EMBRAPII?”

Prior to these meetings, one of the researchers of this case study gathered the Primary Evaluation Elements, which are the first version of the criteria and sub-criteria that have the goal of establishing what is intended and what is the minimum acceptable in relation to the problem for the decision-makers [27]. They were decided based on her experience as an intern at the company and an investigation of reports and interviews of partners giving their opinions on EMBRAPII.

The PEAs were fundamental, so that the participants of the brainstorming could be familiar with the problem and express their opinions about the items presented, being able to agree, disagree, add, combine, or remove elements according to their experiences and opinions. After the meetings, proceeding from the PEAs, the criteria, or Fundamental Points of View, and sub-criteria, or Elementary Points of View, were determined and are listed in

Table 1. Result of the brainstorming meetings.

Criterion (FPV)	Sub-Criterion (EPV)
1. Relationship with EMBRAPII	1.1 Credibility
	1.2 Transparency
	1.3 Communication
	1.4 Meeting deadlines
2. Agreement signed	2.1 Meeting the objectives of the agreement and alignment with your institution’s strategies
	2.2 Relevance of the results achieved with the agreement
	2.3 Multiplier effect of resources offered by EMBRAPII
	2.4 Cost–benefit ratio
3. EMBRAPII’s reputation	3.1 Recognition as a relevant institution
	3.2 EMBRAPII’s image as a complement to public innovation policies/international programs
	3.3 Media positioning
4. Technical capacity	4.1 Experience of the technical staff
	4.2 Staff skills
	4.3 Technical staff efficiency
5. Ability to adapt to change	5.1 Responding to changes in partnership requirements
	5.2 Resilience to changes in the innovation environment
	5.3 Ability to present solutions in the face of changes in the innovation environment

.

4.3. Focus Group

After the brainstorming meetings, the researchers reunited the focus group to discuss four specific matters: to validate the FPVs and EPVs raised in the brainstorming; define the contribution rates of each FPV and EPV; define the effort levels of each FPV and EPV; and validate the description and response options in the questionnaire.

The participants were four employees from EMBRAPII with at least five years in the company and with a specialization in the positions they occupy. The choice was based on the function they perform, considering that they interact with the actors of the research.

Table 2. Profile of the focus group participants.

#	Job and Position	Time in Position	Qualification
1	Coordinator in the Planning and Institutional Relations department and responsible for the national institutional partnerships of EMBRAPII	6 years and 6 months	PhD in Science and Technology Policy at University of Sussex, England, and Post-doctorate in International Relations from the Universities of Oxford, England, and Princeton, USA
2	Executive assistant of EMBRAPII’s Board of Directors and responsible for the international institutional partnerships of EMBRAPII	5 years and 10 months	Master’s Degree in Policy Studies in Education and PhD in Sociology of Knowledge, both from the Institute of Education of the University of London, England
3	Executive assistant of EMBRAPII’s Board of Directors and responsible for the agreement signed with the Brazilian Micro and Small Business Support Service -SEBRAE	7 years and 1 month	PhD in Economics from the Fluminense Federal University, Brazil, and specialist in innovation policy and public policy management
4	Specialist in Industrial Innovation in the Planning and Institutional Relations department and responsible for the Priority Programs coordinated by EMBRAPII	6 years and 4 months	PhD student in Science and Technology Policy at University of Campinas, Brazil, and specialist in public policy management

lists the profile of the focus group participants.

The first activity consisted of the validation of the FPVs and EPVs determined during the brainstorming and the removal or addition of other pertinent ones. The group agreed with all the criteria presented and chose to change two and exclude three sub-criteria, which resulted in five criteria and fourteen sub-criteria.

EPV 1.4—Meeting Deadlines was substituted with “Agility”, because the group concluded that compliance with the deadline is EMBRAPII’s obligation. However, it is possible to work in an agile way, an aspect that the respondent can evaluate. EPV 4.2—Staff Skills was altered to “Quality of the materials delivered and methods applied by EMBRAPII”, because the respondents did not have enough content to conclude the level of skill of the technical staff. However, they were able to give their opinion about the materials received and prepared by EMBRAPII’s employees.

The EPVs excluded were 2.4—Cost–Benefit Ratio, 4.1—Experience of the Technical Staff, and 5.1—Responding to Changes in Partnership Requirements. The first was subject to different interpretations of its definition, the second was similar to another sub-criterion, and the third was considered a mandatory point for EMBRAPII to comply with, as well as EPV 1.4.

The final result is presented in

Table 3. Result of the focus group.

Criterion (FPV)	FPV Description	Sub-Criterion (EPV)	EPV Description
1. Relationship with EMBRAPII	Relationship that your institution maintains with EMBRAPII from the partnership negotiations to the present day	1.1 Credibility	The quality of inspiring belief
		1.2 Transparency	Availability of full information required and open communication regarding the progress of the agreement
		1.3 Communication	Effective exchange of information between EMBRAPII and your institution
		1.4 Agility	Ability to act in fast cycles of learning and decision-making
2. Agreement signed	EMBRAPII’s compliance with the scope of the agreement and/or Memorandum of Understanding and its requirements	2.1 Meeting the objectives of the agreement and alignment with your institution’s strategies	Fulfilment of the objectives set out in the contract
		2.2 Relevance of the results achieved with the agreement	Importance of the results achieved with EMBRAPII
		2.3 Multiplier effect of resources offered by EMBRAPII	Model of leverage of financial resources provided in projects
3. EMBRAPII’s reputation	Opinion of your institution about EMBRAPII’s image in the media and in the innovation scenario	3.1 Recognition of EMBRAPII as a relevant institution in RD&I	Seeing EMBRAPII as a relevant institution in the innovation scenario
		3.2 EMBRAPII’s image as a complement to public innovation policies/international programs	Perception of EMBRAPII as an important instrument to complement international innovation programs
		3.3 Media positioning	EMBRAPII’s image in the media, including social media
4. Technical capacity	Technical capacity of the team located at EMBRAPII’s head office in Brasília-DF	4.1 Quality of the materials delivered and the methods applied by EMBRAPII	Qualification and adequacy of what your institution receives
		4.2 Technical staff efficiency	Ability of the technical staff to perform what was planned in a productive way when applying the available resources
5. Ability to adapt to change	EMBRAPII’s positioning in the face of changes that occur in Brazil and in the world in the field of innovation	5.1 Resilience to changes in the innovation environment	EMBRAPII’s ability to adapt to new realities and new markets, while maintaining its operations and evolution
		5.2 Ability to present solutions in the face of changes in the innovation environment	How EMBRAPII shapes its business proposals to changes in the innovation environment and meets the needs of the moment

, which describes each Fundamental Point of View and Elementary Point of View.

The second stage was the definition of the contribution rates of the FPVs and EPVs, values that represent the relative importance of each point of view in the evaluation and are obtained from the judgment of the decision-makers [17]. The contribution rates of the criteria and sub-criteria must sum 100%, and the value should represent how much that point of view contributes to the research.

The third stage sought to define the effort levels of the FPVs and EPVs, which consist of the energy expended to positively reverse the user’s response to the criterion or sub-criterion. The focus group determines, from highest to lowest, the points of view that require the most effort. The values of the contribution rates and effort levels are shown in

Table 4. Contribution rates and effort levels of the criteria and sub-criteria.

Criterion (FPV)	FPV Contribution Rate	Sub-Criterion (EPV)	EPV Contribution Rate	EPV Effort Level	FPV Effort Level	Overall Effort Level
1. Relationship with EMBRAPII	20%	1.1 Credibility	25%	3rd	4th	13th
		1.2 Transparency	15%	4th		14th
		1.3 Communication	40%	1st		4th
		1.4 Agility	20%	2nd		9th
2. Agreement signed	25%	2.1 Meeting the objectives of the agreement and alignment with your institution’s strategies	30%	3rd	2nd	11th
		2.2 Relevance of the results achieved with the agreement	35%	2nd		7th
		2.3 Multiplier effect of resources offered by EMBRAPII	35%	1st		2nd
3. EMBRAPII’s reputation	20%	3.1 Recognition of EMBRAPII as a relevant institution in RD&I	40%	2nd	3rd	8th
		3.2 EMBRAPII’s image as a complement to public innovation policies/international programs	35%	1st		3rd
		3.3 Media positioning	25%	3rd		12th
4. Technical capacity	20%	4.1 Quality of the materials delivered and the methods applied by EMBRAPII	50%	2nd	5th	10th
		4.2 Technical staff efficiency	50%	1st		5th
5. Ability to adapt to change	15%	5.1 Resilience to changes in the innovation environment	45%	2nd	1st	6th
		5.2 Ability to present solutions in the face of changes in the innovation environment	55%	1st		1st

.

The final stage was to validate the main question and the response options (impact levels) for the questionnaire sent to the participants and incorporate the formalization and the best representation of the performance of the sub-criterion analyzed [19]. The guiding question of the questionnaire supplied, and the defined answer options, are presented in

Table 5. Guiding question and answer options.

Guiding Question	Answer Option	Impact Level	Reference Level
What is your perceived value of the service delivered by EMBRAPII regarding the following items?	Excellent	L5	Great levels
	Very good	L4
	Good	L3
	Neutral	L2	Neutral
	Bad	L1	Negative

.

The scale used in the constructivist multi-criteria analysis method should contain a zone above the decision-makers’ expectations, represented by the impact level L5; a zone within the decision-makers’ expectations, between L4 and L2; and a zone below expectations, represented by L1 [17].

At the end of the Structuring Phase of the method, it was possible to conclude the evaluation stage with the data collection, addressed below, through the application of a questionnaire to the participants in order to analyze the perception of the value of the institutional partners about the service provided by EMBRAPII. With the answers, the MyMCDA-C software generated the results covered in the Discussion section.

4.4. Data Collection

With the results of the focus group, the data collection stage was initiated to generate the necessary inputs for the value perception analysis.

4.4.1. Construction and Application of the Questionnaire

The questionnaire was constructed according to the results of the focus group and included the criteria and sub-criteria, as well as their descriptions and response options. It was applied remotely using the Google Forms platform, and two versions of the questionnaire were sent, one in Portuguese and the other in English.

The focus group participants collected the contacts of potential respondents representing business associations, industry federations, development banks, SEBRAE, SENAI, international institutions, and contracting ministries. This gathering resulted in 61 e-mail addresses, and the questionnaire received 21 responses. Therefore, it reached 34.4% of the recipients.

4.4.2. Data Tabulation and Application

After collecting the responses, the data were tabulated and applied to MyMCDA-C. The FPVs and EPVs were inserted into the software with the contribution rates and effort levels and were ordered according to the overall effort level presented in Table 3. After entering these data, it was necessary to indicate the reference position of the impact levels answered by the participants for each EPV using the median of the number of responses. Appendix A shows the number of responses for each level and the position of the defined median.

With all the required data added to the software, graphs were generated indicating the current, maximum, and minimum value levels of each FPV and EPV. They also indicate the steps of managerial effort that the manager must undertake to reach the maximum value, and, thus, allow for an analysis of which criteria and sub-criteria require a greater effort, providing input for decision-making. The results obtained by the software and the value perception analysis are presented below.

5. Results and Discussion

This section analyses the profile of the survey respondents, the results of the questionnaire responses, and the graphs generated by the MyMCDA-C software, which simultaneously translates the decisions of the focus group and the opinions of the respondents.

The questionnaire was sent to people who work as intermediates in the partnership with EMBRAPII within their institution. The majority of respondents occupy senior positions: 38% are coordinators, 19% are managers, and 10% are directors. Analysts and one assistant represent the other 34%. A total of 48% of these employees have worked in their organizations since EMBRAPII’s foundation in September 2013, but most of them, 76%, have been interacting in the partnership for 0 to 2 years and 2 to 4 years. Those who have participated for 4 to 6 years and six years or more represent 24%.

This difference is due to the fact that EMBRAPII has become better known recently, as the pursuit of strategic partnerships began in 2014, and, since 2016, the organization has sought to expand its partnerships, both national and international. EMBRAPII’s Multiannual Report [28] reveals that, mainly from 2017 onwards, the number of partnerships began to be significant. By the time of the research, EMBRAPII counted 35 Brazilian partners [29] (p. 23) and 14 international partners [30] (p. 1). These numbers portray the importance of carrying out the present study, given that the tendency for EMBRAPII’s institutional relationship is growth in the number of partnerships.

5.1. Criteria and Sub-Criteria Analysis

With the definition of the medians of each sub-criterion, available in Appendix A, the MyMCDA-C software has the necessary data to generate the graphs presented in this section. For a better understanding of the analyses made on the graphs, the following should be considered.

Each step of the graph is an impact level. The blue line represents the maximum value (Excellent), the pink line scores the median value, and the orange line represents the minimum value (Bad) that can be achieved by each sub-criterion, according to the survey participants’ responses. Values above zero are the optimal points and expected results, while values below zero are the negative points and, therefore, the unexpected results.

The distance between the orange and blue line points depicts the effort required by the sub-criterion to achieve peak performance and illustrates the effort levels defined in the focus group. Therefore, the more steps, the more energy must be expended for the criterion to reach the blue line.

The “Total” column uses the contribution rates and values for each sub-criterion to define the criterion values using the weighted average. The totals for each criterion are grouped together in the overall performance chart. Next, the graphs of each criterion will be analyzed according to the points explained above.

5.1.1. Relationship with EMBRAPII

The criterion “Relationship with EMBRAPII” covers the relations maintained between EMBRAPII and the partner institution from the time of hiring until the date of the response in the survey. The respondents evaluated none of the sub-criteria as “Bad,” and the one that obtained the best evaluation was criterion 1.1—Credibility, followed by 1.3—Communication, 1.2—Transparency, and, finally, 1.4—Agility.

Figure 3 shows that all sub-criteria reached the same value (100). Hence, the median defined was the impact level L4—Very good. However, each of them has different values in the maximum and minimum lines, and thus, the efforts to reach the maximum level differ.

Sub-criteria 1.2 and 1.3 stand out for having, respectively, the lowest and highest effort level. The “Transparency” sub-criterion also stands out for the fact that the median line (pink) is five steps away from the maximum line (blue). This result means it is the closest sub-criterion to achieving excellence due to its low effort level. Given that it is the item with the lowest effort and occupies third place in terms of the perception of value, it is interesting to invest in increasing the value of transparency because it impacts the overall value of the relationship with EMBRAPII.

On the other hand, sub-criterion 1.3—Communication, displays an effort level of 210 steps (−55 to 155), since the focus group pointed it out as an item that would be difficult to convert positively if the answer regarding its value was negative. However, the perception of the value of this sub-criterion was satisfactory. Therefore, because it requires the greatest effort and has presented satisfactory results, it is not necessary, from a managerial point of view, for EMBRAPII to invest, at this moment, in communication. It is recommended to be alert in keeping the current or high level because, when reaching unexpected points, the energy spent to reach the optimal points is high.

Sub-criterion 1.1—Credibility also has a low effort level. However, there is no need to invest in this sub-criterion because it was the best-evaluated by the respondents. It is crucial to observe the good performance of this item and the position achieved by the “Transparency” sub-criterion, as they are complementary factors when analyzing that, if the company is transparent with its customers, its credibility will increase. In that case, it can be concluded that the respondents are relating credibility to different factors or that other factors are more important than transparency, to conclude that the quality of EMBRAPII is trustworthy.

Sub-criterion 1.4—Agility ranked fourth in the respondents’ evaluation and has the second highest effort level, but with a difference of 40 (210–160) steps from the highest effort level in the criterion, “Communication.” Therefore, during decision-making, it is more worthwhile to invest in improving agility when compared to “Communication,” the second best-rated by respondents.

5.1.2. Agreement Signed

The sub-criteria discussed below are related to the contracts signed between EMBRAPII and the partner institution. Sub-criterion 2.3—Multiplier Effect of EMBRAPII’s Resources was in first place, 2.1—Meeting the Objectives of the Partnership and Alignment with Strategies in second place, and 2.2—Relevance of the Results Achieved with the Partnership took third place. Considering that the multiplier effect of resources is EMBRAPII’s significant differential, the result of this sub-criterion is essential to understanding the perceived value.

Figure 4 shows that the sub-criterion with the greatest effort was 2.3, the one with the best evaluation by the respondents. Therefore, there is no need for investments to improve the resource multiplier effect when compared to the other two sub-criteria. However, as it is EMBRAPII’s operating model, it is essential to manage this item in order to promote adjustments and improvements as innovation advances and in the face of the shortcomings of the RD&I environment.

In this sense, the organization has made changes in its operating model to meet the different demands of Brazilian innovation participants, increasing, for example, the contribution of projects from startups and small and medium-sized companies to 50% of the total value of the project. In view of this fact, it was noticed that the senior management and the EMBRAPII team are aware of the need to manage this sub-criterion actively. Thus, this study confirmed the importance of this management on the multiplier effect and validated the actions taken to improve the financing model.

The answer to the sub-criterion 2.2—Relevance of the Results Achieved with the Agreement occupied third place and was the second highest effort. This result provoked thinking about the hiring model for each type of partnership, since, for some respondents, the results achieved from the partnership with EMBRAPII needed to be more relevant. On the other hand, criterion 2.1—Meeting the Objectives of the Agreement and Alignment with the Institution’s Strategies was the second best-evaluated and the one with the lowest effort. This position demonstrates that the contracts signed are aligned with the strategies of the partner institutions and, according to the focus group, are easier to reverse positively. Therefore, they are molded for each of them. In view of this counterpoint, in order to understand how EMBRAPII can achieve excellence in terms of the relevance of its results, it is necessary to invest in understanding which aspects of each agreement can be more relevant to the institution.

5.1.3. EMBRAPII’s Reputation

This criterion was questioned to investigate how the partnerships see EMBRAPII’s reputation in the public and private RD&I scenario. Regarding its perceived value according to the respondents, 3.1—Recognition of EMBRAPII as a Relevant Institution in RD&I ranked first, and then 3.2—EMBRAPII’s Image as a Complement to Brazilian Public Innovation Policies and International Programs and 3.3—Media Positioning.

In addition to first place in this criterion, item 3.1 was the sub-criterion with the best evaluation of all the 14 sub-criteria questioned by the respondents. This result is important since EMBRAPII is the only Brazilian institution with this operational model of leveraging public and private resources. In addition, it has a few years of experience and has achieved valuable positions in the field. The ministries and partners consider it a relevant institution for Brazilian research, development, and innovation.

The good performance of sub-criteria 3.1 and 3.2 was reflected in Figure 5 in the points of the pink line superimposed on the blue line, implicating that the current value of the performance of these items reached the maximum level. They also represent the highest levels of effort in the criterion. Because the respondents already consider them excellent, they do not need energy from EMBRAPII to reverse them, only to keep them at the current stage.

Sub-criterion 3.3—Media Positioning was not evaluated, and neither were the others. However, it is the one with the least effort, and therefore, it is suggested that the 15 steps to increase the perceived value be invested. EMBRAPII’s result of the indicator of positive/neutral citations in the media has always been 100% [24] (p. 20). Therefore, it is necessary to understand how to improve the value of positioning in the media beyond published articles with positive or neutral content, seeking to manage EMBRAPII’s image.

5.1.4. Technical Capacity

Technical capacity is precious to EMBRAPII’s lean internal team, and for this reason, it was chosen as a criterion for evaluating the value seen by the respondents. The answers established 4.2—Technical Staff Efficiency in the first place and 4.1—Quality of the Materials Delivered and Methods Applied in the second place. It should be noted that, in accordance with the other sub-criteria of the survey, the results were very positive, and it can be stated in advance that the ministries and partners also see technical capacity as of great value.

In this case, the median was the impact level L4—Very Good, as Figure 6 illustrates. The efforts defined in the focus group elected sub-criterion 4.2 as the one that demands most energy for change. Since the quality of the materials delivered and the methods applied by EMBRAPII require less effort and were the second best-evaluated, it was concluded that this item deserves investment by the team.

An important fact is that both sub-criteria have a contribution rate equal to 50%, thus contributing equally to the criterion of Technical Capacity. Consequently, even with less energy spent on item 4.1 to improve the perception of value, the impact caused on the Technical Capacity item will be equivalent to investing in the efficiency of the technical staff.

5.1.5. Ability to Adapt to Change

By being inserted in the innovation environment, EMBRAPII must always be aware of the changes that occur in this scenario. The organization works with the technological demands of the industry, which is constantly advancing. Therefore, it is necessary to accompany these systems’ needs to present solutions to companies that seek EMBRAPII to conduct RD&I projects with accredited science and technology institutions. With the application of the questionnaire, it was found that the respondents see more value in item 5.2 than in item 5.1. Still, as with the other criteria, the result was positive.

Figure 7 shows that the focus group defined the highest effort level for sub-criterion 5.2 and the lowest level for sub-criterion 5.1. Therefore, due to the result of the questionnaire and the distance of 45 steps between the pink line (current) and the blue line (maximum) to reach the highest value, it is interesting to invest resources in being more resilient to changes in the innovation environment to keep up with the growth of the industry and the advancement in innovation.

During 2020, close to the year that this research took place, the COVID-19 pandemic was decreed. EMBRAPII’s resilience was put to the test to quickly start promoting health projects to solve problems such as the lack of ventilators. Institutional partners expect this type of movement, so the organization must maintain its investments in adapting its operations, in order to be in tune with the needs of the Brazilian industry.

Sub-criterion 5.2 is related to sub-criterion 2.2—Relevance of the Results Achieved with the Agreement, which was the least well evaluated in the criterion of Agreement Signed. The performance of sub-criterion EMBRAPII’s Ability to Present Solutions in the Face of Changes was very positive, and, therefore, it reports that, in relation to this type of result of the signed agreement, the respondents are satisfied. So, for item 2.2, it is worth investing in improving the results of other factors, such as the number of projects supported.

5.1.6. Overall Performance of the Criteria

With the “Total” performance columns of the graphs presented above, the software generated the graph in Figure 8 of the overall performance of the criteria. For this, the weighted average of the contribution rates and the minimum, current, and maximum values of each sub-criterion were used. The criterion that obtained the best evaluation was 3—EMBRAPII’s Reputation, and it reached impact level L5—Excellent. The other four achieved L4 impact level—Very Good.

By improving one of the sub-criteria, the performance of the criterion is improved. Thus, the value perception analysis from the Constructivist Multi-criteria Decision Analysis is essential to be aware of which factor to invest more energy in, but also to understand that it is not necessary to invest in all sub-criteria to improve performance. Therefore, criteria 1—Relationship with EMBRAPII, 2—Signed Contract, 4—Technical Capacity, and 5—Ability to Adapt to Changes can still reach the maximum line by making an effort to climb the steps of one of its sub-criteria.

The “Total” column in Figure 8 represents the research label: Identify the perceived value of institutional partners and contracting ministries about the service provided by EMBRAPII. The current value reached 107, standing beyond the L4 impact level—Very Good. This performance means that, in general, the value perceived by institutional partners and contracting ministries about the service provided by EMBRAPII is between “Very Good” and “Excellent.”

Table 6. Positions of the sub-criteria.

Sub-Criterion (EPV)	Overall Effort Level	Contribution Rate	Position
3.1 Recognition of EMBRAPII as a relevant institution in RD&I	8th	40%	1st
3.2 EMBRAPII’s image as a complement to public innovation policies/international programs	3rd	35%	2nd
2.3 Multiplier effect of resources offered by EMBRAPII	2nd	35%	3rd
4.2 Technical staff efficiency	5th	50%	4th
1.1 Credibility	13th	25%	5th
1.3 Communication	4th	40%	6th
1.4 Agility	9th	20%	7th
2.2 Relevance of the results achieved with the agreement	7th	35%	8th
2.1 Meeting the objectives of the agreement and alignment with your institution’s strategies	11th	30%	9th
1.2 Transparency	14th	15%	10th
5.2 Ability to present solutions in the face of changes in the innovation environment	1st	55%	11th
3.3 Media positioning	12th	25%	12th
5.1 Resilience to changes in the innovation environment	6th	45%	13th
4.1 Quality of the materials delivered and the methods applied by EMBRAPII	10th	50%	14th

shows the positions of the sub-criteria according to the answers to the questionnaire. They were determined by the number of answers to “Excellent”, and the tiebreaker was determined by the quantity of “Very Good”; and if the tie remained, the quantity of “Good” was verified. With it, it is possible to analyze the relationship between effort levels, contribution rates, and position.

With the ranking of positions, the importance of sub-criteria 3.1 and 3.2 for criterion 3 was observed, since sub-criterion 3.3 occupied the 12th place and, still, the criterion obtained the best evaluation. Even with all the sub-criteria of criteria 1 and 2 occupying the top ten places, they did not achieve excellence as number 3.

Sub-criterion 5.2—The Ability to Present Solutions in the Face of Changes in the Innovation Environment requires the greatest effort among all and is ranked 11th. Therefore, major investments are needed to improve the perceived value of the contracting ministries and institutional partners. The bottom three have low effort levels, so they also deserve attention from decision-makers to increase their values. Therefore, with the two sub-criteria of criterion 5—Ability to Adapt to Change positioned in 11th and 13th places, this was the criterion least well evaluated by the respondents.

To synthesize the constructivist multi-criteria analysis, one of the focus group participants had access to the results to validate them and give her opinion. For the participant, the primary learning from the research was that it is necessary to learn how to translate the results achieved in order to communicate with the contracting ministries and institutional partners effectively.

This learning impacts not only sub-criteria 1.2—Transparency and 1.3—Communication, but also sub-criterion 2.2—Relevance of the Results Achieved with the Agreement, so that stakeholders have access to the results of the partnership in order to understand better the impacts caused. Sub-criterion 1.4—Agility was also commented on as a factor to invest in, as agility is one of the pillars of EMBRAPII’s service.

Regarding item 3.3—Media Positioning, it was considered an investment capable of improving the perceived value as a whole. For sub-criterion 4.1—Quality of Materials Delivered and Methods Applied, the participant of the focus group noted that, by investing in transparency and agility, materials and methods would also be improved. The participant judged sub-criterion 5.1—Resilience to Changes in the Innovation Environment as one that is already being improved by EMBRAPII and agreed to continue investing in it.

The following conclusions are presented, considering the content previously seen and the possibilities of future work to complement the reported results and extend the research to other related fields.

6. Conclusions

It is essential to align the service provided with what they want to receive in order to maintain the partnership with the contracting organizations. For EMBRAPII, the contracting ministries and Brazilian and international institutional partners are important actors in its operation. In addition, they guarantee both financial resources and the expansion of the organization’s scope of action. To understand the partners’ needs and the perceived value of the service delivered by EMBRAPII, the concepts of perceived value and value perception analysis were used in this work. The researchers sought to understand the value the contracting ministries and partners attribute to EMBRAPII, considering aspects of the relationship established between them based on the application of Constructivist Multi-criteria Decision Analysis.

Based on the partner companies’ responses, it was found that the service provided by EMBRAPII was evaluated as positive. The highlight factors were the recognition of EMBRAPII as a relevant institution in RD&I and as a complement to public policies or international programs, and for the multiplier effect of resources. Therefore, we can conclude that the contracting ministries and institutional partners support the paths that EMBRAPII is following and the solutions proposed during its seven years of operation.

On the other hand, factors relating to media positioning, material quality, and resilience to changes in the innovation environment earned a lower value. As part of the results’ validation, the relationship between sub-criteria was also identified; that is, by improving one of them, it is possible to increase the value of a different criterion. As an example, the improvement of the sub-criteria of transparency, communication, and agility can affect the values of media positioning, the relevance of the agreement’s results, and the quality of the materials delivered and methods applied.

This work contributes to the value perception analysis literature, given that it uses a method to measure the subjective opinion of stakeholders regarding the services provided by an organization. The methodology also allows the decision-makers to visualize the perceived value with graphs illustrating the research results, and these can be replicated for other problem-solving needs.

From a managerial point of view, this article is important for facilitating the identification of criteria to be improved, by defining partner satisfaction strategies aligned with the results of the study. In this way, the aim is to improve partners’ perception of value and develop a greater competitive advantage.

From a theoretical point of view, the present study intends to serve as a model for the other 96 EMBRAPII units expanded in the five regions of Brazil. Within this context, the research can be characterized as an innovation, being the first study within the organization to analyze the perceived value of the service provided by EMBRAPII.

As a research study limitation, it is important to mention that the results reflect the reality of a single EMBRAPII Unit. Nevertheless, future studies can address an analysis of the other units by region, for example, and then overall. Also, the following studies are suggestions for future research: study benefits management based on value perception analysis and select third-level criteria to understand the perceived value of other factors within the sub-criteria. With this, it will be possible to act more assertively on the sub-criteria that requires more attention from EMBRAPII managers. It is also suggested that the research be applied to other stakeholders, such as EMBRAPII Units and Science and Technology Institutions accredited to conduct RD&I projects with contracting companies. This recommendation can ensure a broader view of the perceived value for each stakeholder. Also, as a way of constantly evaluating whether the service provided by EMBRAPII agrees with the stakeholders’ expectations, it is suggested that the collected data be transformed into an indicator that can demonstrate the perceived value of EMBRAPII in percentages. In addition to that, it is suggested that the RANCOM methodology be applied to the criteria considered, with the aim of analyzing possible errors in the expert’s assessments within the focus group.

In order to enable control of the perceived value on the part of EMBRAPII, it is recommended that the risks that can generate positive or negative effects and decrease or increase the value attributed by the ministries and partners be identified. This would serve as input for the development of a proactive posture regarding the deliveries made to improve or maintain the perceived value disclosed in this case study.