Integrating Analytic Hierarchy Process (AHP) and Balanced Scorecard (BSC) Framework for Sustainable Business in a Software Factory in the Financial Sector

Abstract

A balanced scorecard (BSC) framework for a factory that develops software for banking was proposed by us at the end of 2015 to ensure its sustainability, and was focused on improving its productivity and cost. Based on this framework, the aim of this study is to construct an approach using the analytic hierarchy process (AHP) and BSC for evaluating a factory’s performance in order for it to become a sustainable business. In this study, AHP is proposed to prioritise and determine weights for the perspectives and indicators included in the BSC for a financial software factory (FSF). The combination of these weights with different indicator measures produces a model that provides an effective assessment tool for FSF managers. The results of the study, which are shown both globally and disaggregated according to the different roles of FSF stakeholders, show that user satisfaction is the main pillar for making decisions. In addition, the result considering roles shows differences according to the relationship of each stakeholder with the software factory. The current study has been validated in a Spanish factory that develops software for several financial entities.

1. Introduction

Since the advent of the internet, and especially since the emergence of smartphones, which coincided with the last financial crisis that significantly hit the banking sector, banks are trying to reposition in order to retain their dominant position in the financial sector. However, it seems that traditional banking is not well adapted to keeping up with the constantly changing technological landscape, according to SAP and IBM [1]. Banks are facing challenges in several areas, but there are four that stand out which affect banking information technology (IT). These are:

• Not making enough money. Despite all of the headlines about banking profitability, banks and financial institutions are still not making enough return on investment, or return on equity, that shareholders require.
• Consumer expectations. These days, it’s all about the customer experience, and many banks are feeling pressure because they are not delivering the level of service that consumers are demanding, especially in regards to technology.
• Increasing competition from financial technology companies. Financial technology (FinTech) companies are usually start-up companies based on using software to provide financial services. The increasing popularity of FinTech companies is disrupting the way in which traditional banking is being run. This creates a big challenge for traditional banks because they are not able to adjust quickly to the changes—not just in technology, but also in operations, culture, and other facets of the industry.
• Regulatory pressure. Regulatory and legal requirements continue to increase, and banks need to spend a large part of their discretionary budget on being compliant, and on building systems and processes, in order to keep up with the escalating requirements.

These challenges continue to grow, so traditional banks need to constantly evaluate and improve their operations and systems in order to keep up with the fast pace of change in the current banking and financial industry. Tackling these challenges requires new software developments, since much of the existing software dates from the beginning of the century, and is therefore not able to deal with present day issues and, of course, the future of the sector. That is why financial entities need to act in different critical aspects, and the companies that develop their software are possibly the main pillar of this action. The banking sector is one of the sectors requiring the most amount of software for its operation [2]. This is why the financial software industry needs to transform itself into an industrialised software manufacturer, able to provide software projects related to the efficiency, fast delivery, and quality that the financial markets expect today [3]. This industry not only needs to increase production capabilities, but also needs to produce more with the resources available for production, in order to increase productivity [4].

Our paper [5] published at the end of 2015 describes a case in which an FSF in Spain decided to review the concept of industrialisation of the software, to implement the principles and elements of the software factory approach [6], suggesting a strategic management approach using the BSC [7,8] that would increase the productivity of the different teams. This initial paper proposes a specific model for FSF as a way to improve sustainability and validates the model in a Spanish firm specialising in financial software. These FSF have special features that make them different from other software factories: they work almost exclusively for one financial group, which is both the owner and the principal customer, although revenue to a non-corporate group is increasingly important to financial groups; they have a greater demand for software development; and they have secured the payment of deliveries. These features make it necessary to include specific indicators to assess the financial perspective of the BSC, because the typical indicators for this perspective (Return on investment (ROI), Return on equity (ROE), or Economic value added (EVA)) are not suitable for these software companies in which shareholders and customers are both part of the same financial group. One problem with this initial proposal of a BSC framework is that it does not establish the importance of the perspectives and indicators in the BSC. The integration of BSC with multi-criteria decision-making techniques (MCDM), as AHP, has been adopted in this current paper as a useful method for calculating weights, and can help to establish an evaluation performance system [9]. In this study, AHP is proposed for determining the weights for the selected key performance indicators (KPIs) included in each of the four hierarchical perspectives of the BSC for an FSF, with the aim of ensuring business sustainability. This is related to achieving a viable business over time in terms of productivity and profitability, through appropriate decision-making. In addition, the results will be shown not only globally, but also by stakeholder roles, which represents a novel approach compared to other studies that apply a similar methodology to the one used in this paper.

The remainder of this paper is organised as follows. Section 2 reviews the literature and introduces the concepts of BSC, BSC for an FSF, and the AHP method as an integrated performance evaluation framework. Section 3 describes the research methodology. Section 4 shows the results of the study. Section 5 presents the analysis and discussion of the results and Section 6 presents the conclusions, implications, limitations, and future research directions related to this paper.

2. Performance Evaluation Framework for an FSF Based on BSC and AHP

In today’s competitive world, only the companies that observe the needs of their customers and provide them with satisfaction can compete with others and gain benefits. Every organisation needs a complete management system that includes all or nearly all of the elements or aspects of the company and permits an awareness of the productivity with which it is working, in order to protect its competitive advantages. However, the productivity measure in software development is more difficult than in other industries, due to the intangible nature of software and the difficulty in measuring the output [10]. Nevertheless, it is necessary for a management system to be able to monitor production. A framework that integrates BSC, a multiple perspective framework for performance assessment, and AHP, a decision-making tool used to prioritise multiple performance perspectives, can generate a metric [11] that helps achieve this objective.

2.1. The BSC Framework

In general, performance can be defined as the output or operating result at the end of a given period. This output is an indicator measuring the extent to which the companies reach their goals and, consequently, performance can be described as the general evaluation of all the efforts to accomplish their goals. Business performance is the degree of fulfilment of managerial goals in business practices and the realized outputs of these goals by the end of a certain period [12]. A performance measurement system can be defined as a system by which a company monitors its daily operations and evaluates whether the company is attaining its objectives [13]. To fully utilise the function of performance measurement, it is suggested that businesses set up a series of indexes, which properly reflect the performance of a company.

The BSC approach is probably the best-known management tool for a company. In 1992 and the years that followed, Kaplan and Norton, of Harvard University, presented The Balanced Scorecard [7] as a corporate performance tool that allows managers to look at the business from four important perspectives: financial, customer, internal business process, and learning and growth. In the following years, Kaplan and Norton presented new views and ways to improve the initial BSC approach, and linked it with measures, objectives, and business strategy [8,14]. The BSC is applied by many businesses to assess their performance in diverse aspects of their organisation, but the BSC is only a template and must be customised for the specific elements of an organisation. In addition, studies such as Bourguignon et al. [15] suggest that local ideology and culture should be taken into consideration, especially when the system has been born and developed in another culture. Since the BSC was introduced, many authors have proposed modifications for adapting the initial BSC to other scorecards that are specific to different areas or industrial environments. The studies about the IT BSC [16,17] in the information technologies sector, the Sustainability BSC (SBSC) [18,19] for a sustainable business and [20] for sustainable competitive strategies, those such as [21,22] that expose the benefits of using the BSC in the financial sector, and finally, the specific BSC for an FSF [5,23], are some previously published examples in relation to this study.

Figure 1 shows the BSC for an FSF [5] and includes the chosen KPIs with various fields assigned to each: Code, KPI Name, KPI Owner, Frequency Measurement, Value Measurement, Target Tendency, and the Upper and Lower Control Limits to manage the stability of processes. This BSC was established by the FSF as a control tool for the top management of the organization, according to the strategic objectives and the Mission and Vision of the FSF defined in its strategy map [5]. The indicators were derived from the strategic goals of the FSF through four different strategic lines:

• Improve the performance and productivity of the FSF.
• Delivery of software development must be of a good quality and the level of use of the delivered software by the customers should be increased.
• Optimize and decrease the cost and expense of the software factory.
• Sell the developments and services to companies outside of the owning financial group, to recover the development cost with external revenue.

The KPIs were defined during several work sessions with the participation of different roles of employees. Some of the KPIs were simple to create and easy to obtain, but others were difficult to obtain because they were a compendium of different indicators and measures. The top management and consultants annually review the validity of the KPIs, and KPIs are modified or adjusted if necessary.

Table 1. Descriptions and explanations of the selection of KPIs for FSF performance.

No.	KPIs	Description and Explanation
1	(F1) Cost Structure	Cost evolution according to financial entity size. The costs of the software factory and the size of the financial entity are compared in such way that when the size of the matrix financial group decreases, the costs of the software factory should also decrease in a similar proportion. This indicator is in connexion with the third strategic line (optimization of cost).
2	(F2) Reduction of Cost	The ratio of cost that is covered by sales to companies outside the group. The purpose of this indicator is to assess the percentage of the structural cost of the software factory that is covered by income over the last year, whose source is the sale of software developments and services to companies outside the corporate group. Because of the huge cost of software development, sales revenue outside the financial group owner is generally seen as a reduction of costs. The indicator is derived from the fourth strategic line (sell the developments and services to other companies different from the owning financial group).
3	(F3) Useful Developments	Degree of use of the delivered software by the customers. In this particular case, the degree of use of the developments is indicated by the number of software executions and the indicator is calculated as the cumulative number of these executions in relation to the size of the financial institution over the last year. In this framework is understood that the greater use of the developments, the higher income should be achieved. The indicator is derived from the second strategic line (the level of use of the delivered software by the customers should be increased).
4	(C1) User Satisfaction	Degree of customer satisfaction concerning software developments and services given by the software factory. Customers are the users of these software developments and services. The indicator is connected to the second strategic line.
5	(C2) Cost per Use	The proportion between the cost paid by the customer and the degree of use of the delivered software, measured by means of the cumulative number of executions as in (F3). The indicator is connected to the third strategic line.
6	(C3) SLA	Level of service expected from the software factory as a service provider. The proposed SLA indicator to use in the BSC is a multi-indicator that joins and unifies all the agreements reached with the financial group, and more specifically between the financial institution and the FSF. The indicator is connected to the first and second strategic lines.
7	(I1) Work Performance	Efficiency level in terms of improving rates of software built. The indicator is calculated as ratio between budgeted hours and the performed hours and it is derived from the strategic line of improve the performance and productivity of the FSF.
8	(I2) Employee Productivity	Amount of software that an employee produces for each hour on the job. The ratio is connected with the improvement of performance and productivity of the FSF.
9	(I3) Delay	Average waiting time for delivery of software. This indicator shows the delay in hours when the software factory is delivering software developments. The indicator is connected with the improvement of performance and productivity of the FSF.
10	(I4) Software Quality	Aggregated indicator that assess the software quality, in connection with the second strategic line (delivery of software development must be good quality).
11	(I5) Budgeting Error	Accuracy of the estimates linked to software development. The indicator shows how good the estimations are over the last year and it is related with the third strategic line.
12	(L1) Employer Branding	Reputation of the software factory as an employer. Employees, and especially high-quality employees as a very important part of intellectual capital, must be attracted and retained by firms, and employer branding can help them to address the different needs and expectations of potential and existing employees. This indicator contributes to support all the strategic lines.
13	(L2) Intellectual Capital	Aggregated indicator that assess the intellectual capital as a compendium of human, structural and relational capital. This indicator contributes to support all the strategic lines.

shows a short description and explanation of the KPIs included in the BSC for an FSF, which are extensively explained in our prior paper [5].

2.2. AHP

The AHP was initially developed by Saaty in the 1970’s [24,25], to solve the allocation of scarce resources and the planning needs of the military. Since its introduction, the AHP has become one of the most widely used MCDM methods, and has been used to solve unstructured problems in different areas of human needs and interests, such as political, economic, social, and management sciences [26]. The AHP is a useful approach for solving complex problems. In this approach, selected stakeholders prioritise the relative importance of criteria by making pairwise comparisons of the factors using a nine-point scale.

The most creative part of decision-making is modelling the problem. The identification of the decision hierarchy is the key to success in using AHP. The process of building a hierarchy structure is to more accurately identify all of the elements of the decision and recognise the interrelationships between them [27]. This hierarchical structure has at least three levels: the overall goal of the problem is located at the top, the multiple criteria that define the alternatives is found in the middle, and the competing alternatives (decision alternatives) are situated at the bottom [28]. In particular, AHP is useful for prioritising decision alternatives [29] and is effective in addressing many types of problems that involve multiple criteria [30], including the analysis of performance in business [31], which is the main objective of the present study.

Saaty [32] argues that five options (equal, weak, strong, very strong, and absolute) represent a manager’s ability to make qualitative distinctions. The deployment of AHP in real-life decision-making involves successive comparisons between each alternative, criterion by criterion, according to the fundamental scale of Saaty [33], as presented in

Table 2. Scale to use in making expert judgments [33].

Intensity of Importance on an Absolute Scale	Definition	Explanation
1	Equally important	Two activities contribute equally to the objective
3	Weakly important	Experience and judgment strongly favour one activity over another
5	Essentially important	Experience and judgment strongly favour one activity over another
7	Very strongly important	An activity is strongly favoured and its dominance demonstrated in practice
9	Absolutely important	The evidence favouring one activity over another is of the highest possible order of affirmation
2, 4, 6, 8	Intermediate value between two adjacent judgments	When compromise is needed
Reciprocals	If activity i has one of the above numbers assigned to it when compared with activity j, then j has the reciprocal value when compared with i

.

Different studies based on Saaty’s AHP theory, such as [9,34], explain that, if an alternative $A_i$ is preferable to an alternative $A_j$, then the value of the comparison scale $P_c\left(A_i,A_j\right)=a_{ij}$ indicates the intensity of relative importance of $A_i$ over $A_j$, assigned by the decision maker. The scale allows, in a pairwise comparison, the investigator to establish which alternative is better. Higher values of $a_{ij}$ indicate a stronger preference of alternative $A_i$ over $A_j$. Finally, the matrix $A$ is the result of all of the comparisons, and represents the relative importance $a_{ij}$ of each element. Given that an element is as important as itself, and taking into account the theorem of reciprocity, then $a_{ij}=1$ if i = j and $a_{ij}=$ 1/$a_{ji}$ if i ≠ j. The calculation of weights relies on an iterative process in which matrix $A$ is successively multiplied by itself, resulting in normalised weights, $w_i$, and these weights represent the importance of alternative $A_i$ relative to all other alternatives. The judgment of decision makers in pairwise comparisons may present inconsistencies when all of the alternatives are taken into consideration simultaneously. The consistency index (CI) and the consistency ratio (CR) measure the degree to which judgments are not coherent [35]. If CR < 0.10, then the degree of consistency is satisfactory [36].

2.3. Integration of BSC Framework and AHP to Evaluate Performance

Several studies have proven that the BSC framework (see Section 2.1) is effective in helping organisations to evaluate performance [37]. However, the BSC, a tool which describes the perspectives of performance, does not determine perspective and indicator weights [38]. AHP is useful in a BSC analysis because it can explain several relevant dimensions of organisational performance [9] and their weight of importance in a comprehensive framework. In practice, however, perspectives and indicators seldom have equal importance. As a valuable tool for prioritising and consolidating performance metrics based on multiple criteria, AHP is a promising mechanism to help overcome the limitations of BSC. It has been adopted in several cases as a method for calculating weights in an evaluation performance system [9,39].

Although there are many studies that use the BSC to improve and control organisation performance, there are far fewer cases in the software industry that have also chosen to study the relative weight of each indicator included in the BSC, with the intention to establish a metric to assess performance. In the IT sector, there are several studies [9,39] that propose these methods to create performance evaluations systems. In the financial sector, some studies [40,41] present different proposals using the BSC and AHP in order to evaluate performance for banking. Other studies combine BSC with AHP with the purpose of achieving the sustainability of companies [42,43]. Other variants of the AHP, such as the analytical network process (ANP), have also been used with the BSC for sustainable management [44]. This study proposes a combination between AHP and BSC for an FSF to evaluate performance in a factory that develops software for banking, with the aim of maintaining the sustainability of this business.

3. Methodology

This paper lies in the area of applied research because it focuses on a methodology used to solve a problem in the real-world and the results can be applied in real decision-making situations. In this research, the four perspectives of the BSC were taken as the framework for establishing the performance evaluation indexes. The analytical structure of this research is explained in Figure 2. We first used the BSC model for an FSF to study the different KPIs proposed. Next, we interviewed several managers in a Spanish financial software factory. We then ascertained the views of software factory experts and designed a pairwise comparisons questionnaire based on the hierarchical structure of the BSC. Finally, AHP was applied to obtain the weights of the KPIs included in the specific BSC for an FSF.

The steps taken to achieve this purpose are:

• Analyse the BSC of the FSF (the studied organization).
• Define the hierarchical framework related to this BSC.
• Prepare a questionnaire in AHP format based on this hierarchy to obtain the opinion of experts regarding the indicators and perspectives of the BSC, with the aim of business sustainability.
• Prepare the answers obtained from the experts and give them the necessary format for computer treatment with AHP software.
• Calculate the weights of the global and different stakeholder roles of all the indicators and perspectives included in the BSC, according to the performance aim of the FSF.
• Show the results and analyse them to discuss the result and obtain conclusions.

3.1. Hierarchical Framework of the BSC Performance Evaluation Criteria in an FSF

Based on the concept of the BSC for an FSF [5] and the literature review, this BSC is now adopted as a hierarchical research model of the BSC performance evaluation criteria (i.e., four perspectives and 13 KPIs) for this study, which is shown in Figure 3. To construct the hierarchy, we have used the four perspectives of the BSC for FSF and the KPIs included in each one of them. The 13 KPIs are grouped into the four BSC dimensions: “F: Financial (F1–F3)”, “C: Customer (C1–C3)”, “I: Internal Business Process (I1–I5)”, and “L: Learning and Growth (L1–L2)”. The four perspectives of the BSC provide a robust structure expressing the organisation’s needs and its strategic objectives [45]. Each KPI and its value directly support the achievement of the mission and vision of the organisation [19].

3.2. Data Collection

Based on the BSC for an FSF hierarchy, a questionnaire was designed to ascertain the judgements of the participants with a conventional AHP questionnaire format, five-point scale, and pairwise comparison. The questionnaire included the definition and meaning of each criteria or sub-criteria included in every question. After the questionnaire was designed, it was given to experts, shareholders, and managers of an FSF, to test its reliability and validity. Their corrective views were taken into consideration in the final design and examples about the use of every KPI were included, to facilitate understanding and to avoid potential interpretation errors when the different stakeholders answered the questionnaire. An instruction sheet for completing the questionnaire was created to assist respondents and an example of how the questionnaire is filled in was given to the respondents. A total of 83 questionnaires were distributed (by email and an online form) to different stakeholder roles related to FSF. These were grouped, according to the proposal of roles made by the top management of the FSF and with the purpose of considering different types of points of view, into shareholders, top management, middle managers, other employees, customers, and experts in the field of software factories. The latter were included to seek their professional opinions and the purpose of weighting criteria under the FSF sustainability aim.

Table 3. Different stakeholder roles and their relation with an FSF.

Stakeholder Roles	Relation with the FSF
Shareholders	The shareholders are the owners of the company. They are usually top management of the financial entities involved in the same financial group.
Top Management	Top management are the highest ranking executives responsible for the entire FSF. They translate policy into goals and strategies for the future. Top management make decisions affecting everyone in the software factory.
Middle Managers	The middle managers are the employees of the FSF who manage at least one subordinate level of managers, and report to the top management in the organisation to ensure smooth functioning of the software factory.
Other Employees	Other employees are the lowest level of managers and the rest of the employees of the software factory, including analysts and developers.
Customers	Customers are the financial entities for which the software factory develops software and offers services to them.
Experts in Software Factories	Experts are a group of managers of other software factories from different sectors, professors and researchers in relation to software factories.

shows the different stakeholder roles and their relation with an FSF.

Finally, 61 questionnaires were received (73.5% of questionnaires sent); three more were received late but were discarded for this reason (3.6% of questionnaires sent). Of the remaining 19 (22.9%) sent questionnaires for which we did not receive a response, the time required to answer the questionnaire (between 25 and 30 min), the deadline for filling in the questionnaire (one week), or the lack of interest in this study (as in the case of Other Employees role, only with 46.1% of received questionnaires), represent the three possible reasons why no response was received. The questionnaire responses were revised and adapted to the necessary format, in order to enter this information into the AHP system (the software used in this study was the FuzzyAHP package of R [46]), generating the pairwise comparison matrixes. The questionnaires that did not pass the consistency test (used to directly estimate the consistency of pairwise comparisons [47]), were revised and the values of the comparison matrix were considered and checked. In the cases in which the adjustments were minor or not significant, the procedure was repeated [48]. The definitive number of valid questionnaires was 44 (72.1% of questionnaires received and 53.0% of questionnaires sent).

Table 4. Number and percentage of sent, received, and valid questionnaires, by role.

Stakeholder Roles	Number of Sent Questionnaires	Number of Received Questionnaires		Number of Valid Received Questionnaires (Received/Sent)
Shareholders	13	9	69.2%	6	66.7%/46.2%
Top Management	12	8	66.7%	6	75.0%/50.0%
Middle Managers	15	13	86.7%	8	61.5%/53.3%
Other Employees	13	6	46.1%	5	83.3%/38.5%
Customers	15	14	93.3%	10	71.4%/66.7%
Experts in Software Factories	15	11	73.3%	9	81.8%/60.0%
Total	83	61	73.5%	44	72.1%/53.0%

shows the number and percentage of questionnaires sent, received, and validated, by stakeholder role. Finally, the AHP system calculated the comparison weights of KPIs and BSC perspectives, and generated the result with the weights of the perspectives and the KPIs.

4. Results: A Case Study for FSF

The results of the four BSC dimensions for an FSF performance evaluation show that the Customer Perspective, with a priority weight of 0.4586, is the most important perspective. This is followed by the Finance Perspective, with a priority weight of 0.2035. The priority weight of the Internal Business Process Perspective is 0.1712, and the Learning and Growth Perspective has the lowest priority weight, of 0.1667.

The priority order of the 13 KPIs shows that User Satisfaction, with a priority weight of 0.2482, and SLA, with a value of 0.1319 (both included in the Customer Perspective), are the most important KPIs among all of those included in the BSC for an FSF. The third most important indicator is Intellectual Capital, with a weight of 0.1049 (included in the Learning and Growth Perspective), followed by Cost Structure, with a weight of 0.0841 (included in the Financial Perspective). The seventh indicator is Software Quality, with a weight of 0.0633 (included in the Internal Business Process Perspective). Finally, Figure 4 shows the initial hierarchy based on the BSC for an FSF (see Figure 3) and the global results ranked by weights.

5. Discussion

As shown in

Table 5. Weights of perspectives and KPIs of BSC for an FSF performance evaluation index by AHP, according to the roles of different stakeholders of FSF.

Criteria and Sub-Criteria/Roles of Stakeholders	Total/Global	Share-Holders	Top Management	Middle Managers	Other Employees	Customers	Experts
(F) Finance	0.2035	0.2172	0.1536	0.2090	0.2914	0.1482	0.2407
(F1) Cost Structure	0.0841	0.1430	0.0446	0.0859	0.1051	0.0632	0.0825
(F2) Reduction of Cost	0.0496	0.0409	0.0430	0.0588	0.0522	0.0305	0.0684
(F3) Useful Developments	0.0698	0.0333	0.0659	0.0643	0.1340	0.0545	0.0898
(C) Customer	0.4586	0.5352	0.4933	0.4417	0.2829	0.4735	0.4628
(C1) User Satisfaction	0.2482	0.2263	0.2495	0.2284	0.1704	0.2508	0.2739
(C2) Cost per Use	0.0785	0.2041	0.0917	0.0446	0.0487	0.0646	0.0797
(C3) SLA	0.1319	0.1048	0.1522	0.1686	0.0638	0.1581	0.1093
(I) Internal Business Processes	0.1712	0.1403	0.2287	0.1992	0.2214	0.1510	0.1264
(I1) Work Performance	0.0363	0.0252	0.0381	0.0341	0.0508	0.0339	0.0320
(I2) Employee Productivity	0.0260	0.0178	0.0169	0.0219	0.0342	0.0364	0.0230
(I3) Delay	0.0272	0.0205	0.0455	0.0340	0.0348	0.0181	0.0198
(I4) Software Quality	0.0633	0.0580	0.0998	0.0906	0.0865	0.0527	0.0282
(I5) Budgeting Error	0.0185	0.0187	0.0285	0.0185	0.0151	0.0099	0.0234
(L) Learning and Growth	0.1667	0.1073	0.1244	0.1502	0.2043	0.2272	0.1700
(L1) Employer Branding	0.0618	0.0241	0.0459	0.0640	0.0473	0.1218	0.0591
(L2) Intellectual Capital	0.1049	0.0832	0.0785	0.0861	0.1570	0.1054	0.1110

and Figure 5 and Figure 6, the results by role in the AHP analysis reveal that the Customer Perspective is the primary focus of the BSC, and that User Satisfaction is the most important evaluation KPI. The Customer Perspective is the most important by role (Other Employees being the only exception), but User Satisfaction is the best ranking KPI in all cases. The SLA indicator (also included in the Customer Perspective) is globally the second most important KPI, and is, in most cases, one of the four most important KPIs for the different roles (Other Employees is again the only exception). This means that the importance given to the Customer Perspective, which is twice as important as the Financial Perspective, which is the next important perspective, clearly demonstrates the relationship and mutual dependence between the FSF and its customers. All of this implies that the main objective of the FSF should be focused on the continuous improvement of the services offered to their customers, to cover their needs in the most efficient way.

The third most important KPI is Intellectual Capital. Intellectual Capital is a compendium of human, structural, and relational capital indicators relating to the value of the organisation. The importance given to this indicator (included in the Learning and Growth Perspective) is that it provides evidence about the meaning of human factors in software development [49], where human capital plays a very important role [50]. The very nature of this kind of industry, when absolutely linked to the banking sector, means that one of the most important indicators for banks today is the Cost Structure (included in the Financial Perspective), which is the fourth most important KPI in this study. This indicator, like the Financial Perspective, holds a special meaning for shareholders and is the second most important indicator for them. The Software Quality KPI is the most important factor in the Internal Business Process Perspective. This indicator has a special relationship with the User Satisfaction KPI, because the degree to which customer needs are satisfied is part of Software Quality KPI, according to ISO/IEC 25010:2011 [51]. The relation between Software Quality with User Satisfaction, the most important KPI, is probably one of the causes that gives this indicator a much greater weight than the other indicators included in the Internal Business Processes Perspective.

On the other hand, and although it is not the main objective of this study, it can be concluded from the analysis of the results obtained for the different roles of participants, that each role places more importance to those indicators or perspectives that, in theory, have a greater relation to its own role. In this way, it can be observed that the shareholders give greater weight to the Financial Perspective. The employees (Top Management, Middle Managers and Other Employees) of the software factory place a greater value on the Internal Business Process Perspective. Customers are those that give less importance to the Financial Perspective, but instead, they attribute great importance to the assets and image of the company that offers them services. Although, to confirm this appreciation, User Satisfaction and SLA are the two main indicators for Customers. Furthermore, the role of Other Employees substantially differs from the rest of the employees, which could indicate the lack of organizational communication, related to the importance of customers for the FSF and satisfying their needs, from a higher level of employees to lower levels. The importance given to Intellectual Capital, an indicator that includes human capital value, by the Other Employees, is the highest of all the stakeholders.

6. Conclusions

The banking sector requires huge amounts of new software developments to satisfy its needs, which makes it necessary to increase both the production and the productivity of their developments. With the intention of achieving this aim, this study proposes a framework based on the BSC of an FSF and AHP incorporating the weights of each KPI, which allows a more efficient evaluation of the decisions that must be taken, depending on the different needs. To achieve this purpose, we have analysed the BSC of an FSF and then used its hierarchical structure to design the performance evaluation framework. According to this hierarchical structure of the BSC, we have designed a pairwise comparisons questionnaire in an AHP format to obtain the opinion of different stakeholder roles. Finally, we have treated their responses with AHP software to calculate the weights of all indicators and perspectives of the BSC of an FSF for business sustainability analysis of the results. In view of the results obtained, this study confirms that the combination of BSC with AHP provides a valid methodology for obtaining the necessary criteria and indicator weights to evaluate the performance of an FSF, in order to achieve a sustainable business. The results of this study are potentially useful to software factory managers. This is because, when viewed through the limits and characteristics of their own financial software factories, it can assist them in quantifying the magnitude of the changes required to increase the performance of the organisation. Knowing the ranking of importance of the perspectives and KPIs could be very useful for strategic plans and establishing priorities. This could be even more remarkable if the priorities of the different stakeholders are known. It could also be useful for reinforcing the organizational communication in order to avoid the gaps detected, for example, the different priorities identified for the employees. Although each organization will still have its own idiosyncrasy, the results achieved here (for example the different view that employees have on some aspects) can be taken as illustrative of what can happen in many organizations.

The results presented in this paper clearly demonstrate the preponderant role of the Customer Perspective, and that satisfying customer needs should be the primary reference for decision-making in an FSF. User Satisfaction and SLA are the two most important indicators for an FSF, which is why the managers of the software factory must strive for continuous improvement in these indicators. Improving the Intellectual Capital (which is the main asset of a software development company), and achieving a viable cost structure for the organisation, are the two most important internal challenges facing the managers of an FSF. In practice, the results discussed here, and shown in Figure 4, will allow managers to justify their decisions and will allow an assessment of the impact of measures on FSF performance, focused on the organisation’s sustainability.

It is also necessary to highlight the different behaviours observed according to the different stakeholder roles (see Table 5), which places a greater importance on those indicators or perspectives that have a closer relation to their role or function in the software factory. According to Figure 5, we can conclude that, regarding the perspectives, all of the different profiles are quite well aligned, except for the employees, and to a lesser extent, but as expected, the customers. Regarding the indicators, the differences are more remarkable. The profiles exhibiting the greatest differences are those of the employees and customers, but also the shareholders, who place more importance on the cost structure and cost per use. This fact implies that, in order to have a global vision that improves the performance and achieves the sustainability of this kind of organisation, it is necessary to take into account the perspectives of all the roles related to the FSF.

Future research will aim to improve the proposed framework in this study, integrating the strategy map and the BSC of an FSF with Fuzzy AHP. This integration will allow us to assess the impact that each indicator included in the BSC has on the different objectives defined in the strategy map of the organization. This knowledge should be the basis for making decisions based on the Mission and Vision of the organization. Furthermore, the use of methods such as fuzzy AHP, avoids the uncertainty and the ambiguity associated with human preferences. Fuzzy AHP, which could be incorporated with the pairwise comparison as an extension on AHP, allows for a more accurate description of the decision making process.