A Decision Support System for Public Funding of Experimental Development in Energy Research
Abstract
:1. Introduction
- Full industry-scale implementations require considerable resources for scale-ups and real-world adaptations and involve major risks (e.g., [14,15]). Given that public budgets are limited, funds can only be allocated to the most promising projects and these need to be identified among numerous funding opportunities.
- Funding agencies have some leeway in how they spend their money (e.g., [16]). The use of public resources is governed by political factors and is subject to the influence of stakeholders in R&D, e.g., from academia, industry, policy making, or the general population. Yet, the allocation of funds is expected to be fair and open [17]. An increasing pressure for accountability has been observed, which requires public institutions to present the results of their programs and to justify their existence (e.g., [13]). There is therefore a need to document the coherency of funding decisions transparently and comprehensively.
- It has been pointed out that public sector financers have a different perspective on R&D than private investors [11]. This concerns the strategic and long-term character of public funding, which intends to encourage private investments in potentially attractive areas. As a consequence, traditional financial justification techniques are seen as probably unsuitable [17]. Furthermore, research project evaluation itself is considered complex since it usually involves the consideration of multiple criteria and the views of multiple experts [18]. This makes it necessary to judge proposals from different angles. However, obtaining the required information can also be difficult.
- Public funding is subject to legal frameworks, which also define the limits for state interventions. It is therefore necessary to consider these.
2. Background and Methodological Approach to Developing the DSS
2.1. Proposal Evaluation Process in Germany’s Applied Energy Research Funding
2.2. Methodological Approach to Developing the DSS
3. Literature and Selected Practical Applications for Public R&D Funding
3.1. Literature on Public R&D Funding
3.2. Selected Practical Approaches to Proposal Selection
4. Requirements and Methods for the DSS
4.1. Requirements for the DSS for Public Funding Decisions in Germany
- Universal: In view of multiple energy technologies and varying implementations of specific funding lines, the DSS should be broadly applicable and not be tailored to specific energy technologies or limited to either draft or full proposals. Furthermore, it should allow both the scoring of individual proposals, e.g., in open-ended submission formats, and their comparison, e.g., in the case of fixed submission deadlines.
- Simple: While advanced methods for decision-support are attractive from a methodological perspective, there seem to be empirical limits to their adoption (e.g., [27]). As implied by Figure 2, different and varying groups are involved in proposal evaluations for energy research. They are not necessarily proficient day-to-day users of a DSS (e.g., experts on dedicated topics or personnel only occasionally involved in evaluations). The DSS should therefore be an intuitive non-expert system.
- Supportive: The workshops highlighted that proposal evaluation is a process that is requiring experience. The DSS should therefore help in the evaluation process, but should not be a mechanical scoring mechanism or algorithm that seeks to replace skills and knowledge. Otherwise, acceptance might be low and a ‘mechanistic’ view of proposals might affect the quality of the evaluation.
- Documenting: Due to the increasing demand for transparent records of public funding decisions, the DSS should also allow for documenting decisions.
- Specific: The need for clear criteria definitions has been pointed out as a strategy in [29] to harmonize results across several evaluators. Correspondingly, the DSS should provide specific guidance in case of potentially ambiguous terms and concepts (e.g., ‘excellence’, ‘innovation’). One observation from the workshop was that there is a general understanding of terms like ‘demonstration project’ or ‘model project’, which were selected as default project types for the DSS. Yet, there is no formalized operational definition of these terms. The DSS should provide specific guidance for identifying such project types.
- Linked to TRL: The use of TRL is established in the context of energy research, but experience from practice shows that linking specific TRL levels to the content of proposals is challenging. The DSS should facilitate that link.
- Non-compensatory: Proposals may vary in their quality. The DSS should ensure that good scores in some areas do not compensate for bad scores in other areas. This means that the DSS should allow for minimum aspiration levels to be set.
- Related to funding ceilings: Discussions during the workshops pointed out that funding must be a compromise between minimizing public spending and giving sufficient incentives to ensure private engagement. Finding this balance was perceived as difficult by the participants, but they thought it would be helpful to the DSS to include information on the maximum admissible ceilings for public funding without a formalized notification process, according to EU legislation.
4.2. Selection of a Multi-Criteria Decision Aid (MCDA) Method for Proposal Evaluation
- User group: Methods of varying complexity are suitable for different user groups and purposes. In line with the general requirements, the DSS should be suitable to non-MCDA experts.
- Perspective: The DSS should document decisions transparently, but as it is primarily intended as an operational tool for funding bodies, it should consider proposals from their perspective.
- Decision-makers: Though decisions on proposals are rarely taken by individuals, the use of a method for individuals or a group treated as an individual ensures a simpler, straight-forward DSS.
- Alternatives: Proposals are given as is and thus belong to the group of explicitly defined alternatives in MCDA terminology.
- Problem statement: The DSS may address several different types of problems that are distinguished in MCDA theory (e.g., [42]). The R&D proposal evaluation could concern the description problem (i.e., documentation of performance parameters without aggregation), the ranking problem (i.e., ranking the best to the worst proposal), or the sorting problem (i.e., sorting into classes such as excellent, adequate, and insufficient proposals).
- Type of information: Some of the information in the proposal evaluation could be quantitative (e.g., [18]) and the analysis of existing methods shows that some use weights to differentiate criteria. This suggests that the DSS should be able to process cardinal information, i.e., numerical values.
- Uncertainties: Real-world decision problems are fraught with uncertainty due to non-quantifiable, incomplete, unavailable information, or ignorance (e.g., [43]). Despite numerous methods to deal with uncertain information, the need for simplicity suggests that the DSS should avoid an explicit procedural consideration of uncertainty.
- Weights: An important component of a multi-criteria procedure are the weights assigned to decision criteria. A flexible DSS should allow for individual weights to be selected.
5. Description of the DSS
5.1. Filtering Stage
5.1.1. Review of Fulfilling the Formal Criteria
5.1.2. Review of Innovative Parts
5.1.3. Verification of the Proposal Type
5.1.4. Review of Existing Market Failures
5.1.5. Review of the Relevance for the Funding Program
5.2. Evaluation Stage
5.3. Review Stage
5.3.1. Minimizing Deadweight Losses
5.3.2. Level and Extent of Funding
6. Discussion Concerning the DSS and the Funding Process
7. Conclusions and Outlook
Author Contributions
Acknowledgments
Conflicts of Interest
Acronyms
AHP | Analytical Hierarchy Process |
ARPA-E | US-American Advanced Research Agency-Energy |
BMWi | German Federal Ministry for Economic Affairs and Energy |
CCS | Carbon Capture and Storage |
DSS | Decision Support System |
ELECTRE | Elimination and Choice Expressing the Reality |
EU | European Union |
EUDP | Danish Energy Technology Development and Demonstration Program |
ETS | Emissions Trading System |
FFG | Austrian Research Promotion Agency |
H2020 | Horizon 2020 |
IEA | International Energy Agency |
LINMAP | Linear Programming Technique for Multidimensional Analysis of Preference |
MCDA | Multi-Criteria Decision Aid |
MADM | Multi-Attribute Decision-Making |
MAUT | Multi-Attribute Utility Theory |
MDS | Multidimensional Scaling |
MODM | Multi-Objective Decision-Making |
NER 300 | New Entrants Reserve 300 |
ORESTE | Organization, Ranking and Synthesis of Relational Data |
PROMETHEE | Preference Ranking Organization Method for Enrichment Evaluations |
R&D | Research & Development |
SAW | Simple Additive Weighting |
TOPSIS | Technique for Order Performance by Similarity to Ideal Solution |
TRL | Technology Readiness Level |
USD | US-Dollar |
References and Notes
- United Nations. Paris Agreement. 2015. Available online: http://unfccc.int/files/essential_background/convention/application/pdf/english_paris_agreement.pdf (accessed on 17 April 2018).
- European Commission. A Roadmap for Moving to a Competitive Low Carbon Economy in 2050. Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions. COM (2011) 112 Final. 2011. Available online: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM:2011:0112:FIN (accessed on 17 April 2018).
- Mankins, J.C. Technology readiness assessments: A retrospective. Acta Astronaut. 2009, 65, 1216–1223. [Google Scholar] [CrossRef]
- Lepori, B.; van den Besselaar, P.; Dinges, M.; van der Meulen, B.; Potì, B.; Reale, E.; Slipersaeter, S.; Theves, J. Indicators for comparative analysis of public project funding: Concepts, implementation and evaluation. Res. Eval. 2007, 16, 243–255. [Google Scholar] [CrossRef]
- OECD/IEA. Energy Technology RD&D Budgets. Overview: Paris, 2017. Available online: http://wds.iea.org/WDS/tableviewer/document.aspx?FileId=1578 (accessed on 17 April 2018).
- Roper, S.; Hewitt-Dundas, N.; Love, J.H. An ex ante evaluation framework for the regional benefits of publicly supported R&D projects. Res. Policy 2004, 33, 487–509. [Google Scholar] [CrossRef]
- Brezis, E.S. Focal randomisation: An optimal mechanism for the evaluation of R&D projects. Sci. Public Policy 2007, 34, 691–698. [Google Scholar] [CrossRef]
- Bointner, R. Innovation in the energy sector: Lessons learnt from R&D expenditures and patents in selected IEA countries. Energy Policy 2014, 73, 733–747. [Google Scholar] [CrossRef]
- OECD. Frascati Manual 2015. Guidelines for Collecting and Reporting Data on Research and Experimental Development; The Measurement of Scientific, Technological and Innovation Activities; OECD Publishing: Paris, France, 2015. [Google Scholar]
- European Commission. Commission Regulation (EU) No 651/2014 of 17 June 2014 declaring certain categories of aid compatible with the internal market in application of Articles 107 and 108 of the Treaty. Official Journal of the European Union, 2014; L187/1-78.
- Murphy, L.M.; Edwards, P.L. Bridging the Valley of Death: Transitioning from Public to Private Sector Financing. NREL/MP-720-34036. Available online: https://www.nrel.gov/docs/gen/fy03/34036.pdf (accessed on 29 March 2018).
- Markham, S.K.; Ward, S.J.; Aiman-Smith, L.; Kingon, A.I. The Valley of Death as Context for Role Theory in Product Innovation. J. Product Innov. Manag. 2010, 27, 402–417. [Google Scholar] [CrossRef]
- Bulathsinhala, N.A. Ex-ante evaluation of publicly funded R&D projects: Searching for exploration. Sci. Public Policy 2015, 42, 162–175. [Google Scholar] [CrossRef]
- Åhman, M.; Skjærseth, J.B.; Eikeland, P.O. Demonstrating climate mitigation technologies: An early assessment of the NER 300 programme. Energy Policy 2018, 117, 100–107. [Google Scholar] [CrossRef]
- European Commission. A European Strategy for Key Enabling Technologies—A Bridge to Growth and Jobs. Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions. COM (2012) 341 Final. 2012. Available online: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM:2012:0341:FIN (accessed on 17 April 2018).
- Braun, D. The role of funding agencies in the cognitive development of science. Res. Policy 1998, 27, 807–821. [Google Scholar] [CrossRef]
- Hsu, Y.-G.; Tzeng, G.-H.; Shyu, J.Z. Fuzzy multiple criteria selection of government-sponsored frontier technology R&D projects. R&D Manag. 2003, 33, 539–551. [Google Scholar] [CrossRef]
- Zhu, W.-D.; Liu, F.; Chen, Y.-W.; Yang, J.-B.; Xu, D.-L.; Wang, D.-P. Research project evaluation and selection: An evidential reasoning rule-based method for aggregating peer review information with reliabilities. Scientometrics 2015, 105, 1469–1490. [Google Scholar] [CrossRef]
- Federal Ministry of Economics and Technology. Research for an Environmentally Sound, Reliable and Affordable Energy Supply. 6th Energy Research Programme of the Federal Government: Berlin, Germany. 2011. Available online: https://www.bmwi.de/Redaktion/EN/Publikationen/research-for-an-environmentally-sound-reliable-and-affordable-energy-supply.pdf?__blob=publicationFile&v=3 (accessed on 17 April 2018).
- Projektträger Jülich; Forschungszentrum Jülich. Konsultationsprozess für das 7. Energieforschungsprogramm. Available online: https://www.energieforschung.de/ (accessed on 17 April 2018).
- Hake, J.-F.; Fischer, W.; Venghaus, S.; Weckenbrock, C. The German Energiewende—History and status quo. Energy 2015, 92, 532–546. [Google Scholar] [CrossRef]
- Bundesministerium für Wirtschaft und Technologie; Bundesministerium für Umwelt, Naturschutz und Reaktorsicherheit. Energiekonzept für Eine Umweltschonende, Zuverlässige und Bezahlbare Energieversorgung. 28 September 2010: Berlin, Germany, 2010. Available online: https://www.bmwi.de/Redaktion/DE/Downloads/E/energiekonzept2010.pdf?__blob=publicationFile&v=3; see also: https://www.bmwi.de/Redaktion/EN/Dossier/energy-transition.html (accessed on 17 April 2018).
- Linton, J.D.; Walsh, S.T.; Morabito, J. Analysis, ranking and selection of R&D projects in a portfolio. R&D Manag. 2002, 32, 139–148. [Google Scholar] [CrossRef]
- Solak, S.; Clarke, J.-P.B.; Johnson, E.L.; Barnes, E.R. Optimization of R&D project portfolios under endogenous uncertainty. Eur. J. Oper. Res. 2010, 207, 420–433. [Google Scholar] [CrossRef]
- Verbano, C.; Nosella, A. Addressing R&D investment decisions: A cross analysis of R&D project selection methods. Eur. J. Oper. Res. 2010, 13, 355–379. [Google Scholar] [CrossRef]
- Vandaele, N.J.; Decouttere, C.J. Sustainable R&D portfolio assessment. Decis. Support Syst. 2013, 54, 1521–1532. [Google Scholar] [CrossRef]
- Henriksen, A.D.; Traynor, A.J. A Practical R&D Project-Selection Scoring Tool. IEEE Trans. Eng. Manag. 1999, 46, 158–170. [Google Scholar] [CrossRef]
- Duch-Brown, N.; Garcia-Quevedo, J.; Montolio, D. Assessing the Assignation of Public Subsidies: Do the Experts Choose the Most Efficient R&D Projects? Res. Appl. Econ. Netw. 2008, 12. [Google Scholar] [CrossRef]
- Biegelbauer, P.; Palfinger, T. Verschiedene Verfahren der Auswahl von Forschungsprojekten: Ein Vergleich von neun angewandten Forschungsförderungsorganisationen. Forsch. Polit. Strateg. Manag. Fo 2014, 7, 45–51. [Google Scholar]
- Biegelbauer, P.; Palfinger, T. Auswahlverfahren von Forschungsförderungsorganisationen im internationalen Vergleich. Fteval J. Res. Technol. Policy Eval. 2016, 41, 63–68. [Google Scholar]
- Biegelbauer, P.; Mayer, S.; Palfinger, T. Task Force Select. Final Report. 12 May 2016. Available online: http://www.taftie.org/sites/default/files/Taskforce_SELECT_final_report_0.pdf (accessed on 17 April 2018).
- Biegelbauer, P.; Palfinger, T.; Mayer, S. How Do Innovation Agencies Evaluate and Select Projects? A Comparison of 12 European Agencies. Fteval J. Res. Technol. Policy Eval. 2017, 43, 21–23. [Google Scholar]
- European Commission. Commission Decision of 3 November 2010 laying down criteria and measures for the financing of commercial demonstration projects that aim at the environmentally safe capture and geological storage of CO2 as well as demonstration projects of innovative renewable energy technologies und the scheme for greenhouse gas emission allowance trading within the Community established by Directive 2003/87/EC of the European Parliament and of the Council. Official Journal of the European Union, 2010; L290/39-48.
- ENS. Indkaldelse af Ansøgninger til Energiteknologisk Udviklings- og Demonstrationsprogram (EUDP) 2012-I. Available online: https://ens.dk/sites/ens.dk/files/info/nyheder/nyhedsarkiv/indkaldes-ansoegninger-eudp-2012/Indkaldelse_Januar_2012_DK.pdf (accessed on 19 April 2018).
- Bonvillian, W.B.; Atta, R. ARPA-E and DARPA: Applying the DARPA model to energy innovation. J. Technol. Transf. 2011, 36, 469–513. [Google Scholar] [CrossRef]
- ARPA-E. Facsimile Appearance to Create Energy Savings (FACES): Financial Assistance Funding Opportunity Announcement. Advanced Research Projects Agency—Energy (ARPA-E) U.S. Department of Energy. Available online: https://arpa-e-foa.energy.gov/FileContent.aspx?FileID=74a4a1bc-8f23-495e-8225-899144a24605 (accessed on 17 May 2017).
- European Commission. Horizon 2020. Work Programme 2016–2017. 20. General Annexes. European Commission Decision C (2017) 2468 of 24 April 2017. Available online: http://ec.europa.eu/research/participants/data/ref/h2020/other/wp/2016-2017/annexes/h2020-wp1617-annex-ga_en.pdf (accessed on 17 April 2018).
- Österreichische Forschungsförderungsgesellschaft. Leitfaden für Leitprojekte. Version 2.3. Gültig ab 1 September 2017. Available online: https://www.ffg.at/sites/default/files/dok/il_leitprojekte_v23.pdf (accessed on 17 April 2018).
- Zhang, K.; Kluck, C.; Achari, G. A Comparative Approach for Ranking Contaminated Sites Based on the Risk Assessment Paradigm Using Fuzzy PROMETHEE. Environ. Manag. 2009, 44, 952–967. [Google Scholar] [CrossRef] [PubMed]
- Klein, R.; Scholl, A. Planung und Entscheidung. Konzepte, Modelle und Methoden Einer Modernen Betriebswirtschaftlichen Entscheidungsanalyse, 2nd ed.; Vahlen: München, Germany, 2011. [Google Scholar]
- Zimmermann, H.-J.; Gutsche, L. Multi-Criteria Analyse. Einführung in die Theorie der Entscheidungen bei Mehrfachzielsetzungen; Springer: Berlin, Germany, 1991. [Google Scholar]
- Belton, V.; Stewart, T.J. Multiple Criteria Decision Analysis. An Integrated Approach; Kluwer: Boston, MA, USA, 2002. [Google Scholar]
- Chen, S.-J.; Hwang, C.-L. Fuzzy Multiple Attribute Decision Making. Methods and Applications; Springer: Berlin Germany, 1992. [Google Scholar]
- Hwang, C.-L.; Yoon, K. Multiple Attribute Decision Making. Methods and Applications A State-of-the-Art Survey; Springer: Berlin/Heidelberg, Germany, 1981. [Google Scholar]
- Götze, U. Investitionsrechnung. Modelle und Analysen zur Beurteilung von Investitionsvorhaben, 6th ed.; Springer: Berlin/Heidelberg, Germany, 2008. [Google Scholar]
- Jellema, J.; Mulder, H.A.J. Public Engagement in Energy Research. Energies 2016, 9. [Google Scholar] [CrossRef]
Program | NER 300 | EUDP | ARPA-E | H2020 | FFG |
---|---|---|---|---|---|
Target | Demonstration projects for renewable energies/CCS | Demonstration and development projects for new energy technologies | Early high-potential, high-impact energy technologies | Innovation, Research and Innovation, Coordination and Support Action in energy domain | Lead projects |
Region | Europe | Denmark | USA | Europe | Austria |
Criteria | economic performance | 4 criteria supported by sub-items | 4 criteria supported by sub-items | 3 criteria supported by sub-items | 4 criteria with 3 to 4 sub-criteria supported by sub-items |
Weights | - | - | by criterion | equal (default) by criterion | by sub-criterion |
Threshold | - | - | - | per criterion and for entire proposal | per criterion |
Aggregation | - | - | weighted sum | weighted sum | weighted sum |
Remarks | requirements for knowledge sharing apply | refers to specific call from 2012 | refers to full applications in specific call ‘FACES’, additional criteria may apply | additional criteria apply in case of identical scoring | - |
Does the Project Foresee Broad Transfer Activities Regarding Its Innovative Parts? | ☐ Yes ☐ No |
---|---|
e.g.,
|
Demonstration Project… | Model Project… | |
---|---|---|
Specificcharacteristics | … contributes to enhancing norms, standards or approval processes … contributes to research on or improvement of acceptance … is based on single-location implementation or complementary parts in several locations … has realistic remaining lead for development of less than five years | … largely follows existing regulatory framework … contributes to improving public acceptance … is based on multiple implementations of similar concepts and technologies … provides realistic suggestion for broad utilization after conclusion |
Commoncharacteristics | … foresees broad set of transfer activities for its innovative parts … is carried out by a team with proficient competence on knowledge transfer … is first-of-its-kind activity … can yield economically viable solutions after project conclusion |
Definition | Transparency and quality of the procedure for achieving the objectives of the overall project and its parts |
Premises | The higher the quality and efficiency of the working plan, the more attractive the project |
Main criterion | Quality of the project |
Perspective | Perspective of the funding body |
Supporting Questions |
|
Verification | How reliable, justified and transparent is the judgement of this criterion? |
© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Hirzel, S.; Hettesheimer, T.; Viebahn, P.; Fischedick, M. A Decision Support System for Public Funding of Experimental Development in Energy Research. Energies 2018, 11, 1357. https://doi.org/10.3390/en11061357
Hirzel S, Hettesheimer T, Viebahn P, Fischedick M. A Decision Support System for Public Funding of Experimental Development in Energy Research. Energies. 2018; 11(6):1357. https://doi.org/10.3390/en11061357
Chicago/Turabian StyleHirzel, Simon, Tim Hettesheimer, Peter Viebahn, and Manfred Fischedick. 2018. "A Decision Support System for Public Funding of Experimental Development in Energy Research" Energies 11, no. 6: 1357. https://doi.org/10.3390/en11061357