Enhancing Agile Story Point Estimation: Integrating Deep Learning, Machine Learning, and Natural Language Processing with SBERT and Gradient Boosted Trees
Abstract
:1. Introduction
2. Related Work
3. Background
3.1. Scrum
3.2. Sentence Transformers
3.3. Estimation Model
- Categorical Feature Handling: They can naturally handle categorical features without the need for one-hot encoding, simplifying the preprocessing steps for structured data and outperforming other algorithms [117];
- Robust Performance: They often outperform other algorithms when faced with structured data due to their adaptability and ability to capture intricate patterns [117];
- Predictive Accuracy: They excel at achieving high predictive accuracy, making them particularly valuable in scenarios in which precise predictions are crucial [117]; and
- Interpretability: While not as inherently interpretable as simpler models, GBTs provide insights into feature importance, aiding in understanding the underlying patterns in tabular datasets [117].
4. The Proposed Model
4.1. Text Preprocessing
4.2. Feature Extraction
4.3. Estimation Models
- 1.
- Catboost: Known for efficiently handling categorical features, CatBoost excels in automatic categorical feature conversion and provides robust handling without the need for manual preprocessing [121]. Given the dataset’s inclusion of categorical variables, CatBoost is a fitting choice.
- 2.
- Xgboost: Another GBT model with improved speed and performance, XGBoost incorporates regularization techniques to prevent overfitting [122]. Its flexibility in defining custom optimization objectives makes it a valuable addition to SP estimation models.
- 3.
- LightGBM: Utilizing a histogram-based learning approach, LightGBM proved effective in handling large datasets. Its capability to handle categorical features, coupled with outstanding parallelization, significantly improved performance and reduced memory usage [123].These models were implemented and trained in Python 3.8 using the CatBoostRegressor 1.2.5 (https://catboost.ai/, accessed on 18 August 2024), XGBRegressor 2.1.1 (https://xgboost.readthedocs.io/en/stable/, accessed on 18 August 2024), and LGBMRegressor 4.5.0 (https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRegressor.html, accessed on 18 August 2024) libraries, respectively.
5. Empirical Study Design
5.1. Research Questions
- 1.
- In the first variation, only ‘issue type’, such as bugs, tasks, enhancements, or user stories, is added to the feature extraction process. ‘Issue type’ serves as the categorization or classification of a task within a system or project management framework. It aids in organizing and prioritizing work, providing a means to distinguish between various types of tasks within a project or system.
- 2.
- In the second variation, only ‘components’, such as user interface (UI), runtime, etc., referring to modular and reusable units of code or software that perform specific functions, are added to the feature extraction process.
- 3.
- In the last variation, both ‘issue type’ and ‘components’ are included in the feature extraction process.
5.2. Dataset
5.3. Evaluation Criteria
- n is the number of data points;
- is the actual value for the ith data point;
- is the predicted value for the ith data point.
- is the average performance metric (e.g., MAE) for the proposed model;
- is the average performance metric for the baseline or comparison model;
- is the pooled standard deviation calculated as
6. Results
7. Discussion
8. Threats to Validity
9. Conclusions and Future Work
- Integration of Contextual and Qualitative Data: Future studies could explore incorporating additional contextual information, such as team experience, project complexity, and technological novelty. Qualitative data, such as stakeholder feedback and team dynamics, might also provide valuable insights that could refine the estimation process, echoing the complexities and challenges of implementing Scrum in a global software development context discussed in the paper.
- Cross-Industry Data Integration for Agile Estimation Models: Incorporating datasets from various software development sectors, such as healthcare, finance, and entertainment, could significantly enhance the ability of the SBERT-LGBMTC model with Bayesian optimization to generalize across industry-specific requirements and terminologies. This integration would help refine the model’s accuracy by adapting to different sectors’ unique challenges and practices, promoting a more versatile application of Agile estimation techniques.
- Enhancing Data Collection through Global Collaboration: Promoting global collaborative efforts to gather a more comprehensive dataset including Agile projects from diverse geographical locations could provide invaluable insights into regional differences in project management and execution. The contribution of researchers to this approach could help develop more universally applicable estimation models, reflecting a broader spectrum of Agile implementation contexts and cultural influences on software development practices.
- Temporal Expansion of Data for Agile Practices: Including datasets that cover extended periods is crucial for capturing the evolution of Agile practices and technologies. This temporal data inclusion helps analyze trends and shifts in development practices over time, providing a deeper understanding of how Agile methodologies adapt and evolve, which is instrumental in refining effort estimation models.
- Incorporating Team Dynamics into Estimation Models: Expanding the dataset to include metrics related to team dynamics and individual performance is not just a suggestion but a necessary step to incorporate human factors into the estimation process. By understanding the influence of team interactions and individual contributions, estimation models could be enhanced to predict effort more accurately, acknowledging the human elements that significantly impact project outcomes.
- Real-Time Application and Feedback: Implementing the model in real-world Agile projects and gathering continuous feedback could lead to iterative improvements, making the model more robust and aligned with industry needs.
- Advancements in NLP and ML Techniques: As NLP and ML fields evolve, integrating newer algorithms could further enhance the accuracy and efficiency of the SEE process.
- Human Factors in Estimation: Investigating the interplay between automated estimations and human judgment could provide insights into how to balance these approaches for optimal project management.
- Scalability and Efficiency: Assessing the model’s scalability for large-scale projects and its efficiency in different project environments is essential for broader adoption.
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
ASD | Agile Software Development |
FSM | Functional Size Measurement |
SP | SP |
FL | Fuzz Logic |
SEE | Software Effort Estimation |
SI | Swarm Intelligence |
DL | Deep Learning |
BN | Bayesian Network |
SBERT | Sentence-BERT |
PCA | Principle Component Analysis |
GBT | Gradient Boosted Tree |
XP | Extreme Programming |
MAE | Mean Absolute Error |
LR | Linear Regression |
MdAE | Median Absolute Error |
RBF | Radial Basis Function |
SA | Standardized Accuracy |
LOOV | Leave-One-Out |
NLP | Natural Language Processing |
MRE | Magnitude of Relative Error |
ML | Machine Learning |
PRED(k) | Prediction at Level k |
LGBM | Light Gradient Boosting Machine |
MER | Magnitude of Error Relative |
KNN | k-Nearest Neighbor |
SD | Standard Deviation |
RF | Random Forest |
TF-IDF | Term Frequency-Inverse Document Frequency |
DT | Decision Tree |
MMRE | Mean Magnitude of Relative Error |
SVM | Support Vector Machine |
LSTM | Long Short-Term Memory |
SGB | Stochastic Gradient Boosting |
RHWN | Recurrent Highway Network |
NB | Naive Bayes |
ATLM | Automatically Transformed Linear Model |
NN | Neural Network |
LRM | Linear regression Model |
ANN | Artificial Neural Network |
USE | Universal Sentence Encoder |
DBN | Deep Belief Network |
LaBSE | Language-agnostic BERT Sentence Embedding |
FNN | Fuzzy Neural Network |
BERT | Bidirectional Encoder Representations from Transformers |
MLP | Multilayer Perceptron |
XGBoost | Extreme Gradient Boosting |
RMSE | Root Mean Square Error |
Appendix A. Detailed Flowchart of the Proposed Model
References
- Fowler, M.; Highsmith, J. The agile manifesto. Softw. Dev. 2001, 9, 28–35. [Google Scholar]
- Cerpa, N.; Verner, J.M. Why did your project fail? Commun. ACM 2009, 52, 130–134. [Google Scholar] [CrossRef]
- Gurung, G.; Shah, R.; Jaiswal, D.P. Software Development Life Cycle Models-A Comparative Study. Int. J. Sci. Res. Comput. Sci. Eng. Inf. Technol. March 2020, 6, 30–37. [Google Scholar] [CrossRef]
- Foschini, L. Project Management in the Consultancy Sector: Comparing Waterfall and Agile Approaches. Ph.D. Thesis, Politecnico di Torino, Torino, Italy, 2021. [Google Scholar]
- Pargaonkar, S. A Comprehensive Research Analysis of Software Development Life Cycle (SDLC) Agile & Waterfall Model Advantages, Disadvantages, and Application Suitability in Software Quality Engineering. Int. J. Sci. Res. Publ. (IJSRP) 2023, 13, 345–358. [Google Scholar]
- Bloch, M.; Blumberg, S.; Laartz, J. Delivering large-scale IT projects on time, on budget, and on value. Harv. Bus. Rev. 2012, 5, 2–7. [Google Scholar]
- Flyvbjerg, B.; Budzier, A. Why your IT project might be riskier than you think. arXiv 2013, arXiv:1304.0265. [Google Scholar]
- Venkataraman, R.R.; Pinto, J.K. Cost and Value Management in Projects; John Wiley & Sons: Hoboken, NJ, USA, 2023. [Google Scholar]
- Shepperd, M.; Schofield, C. Estimating software project effort using analogies. IEEE Trans. Softw. Eng. 1997, 23, 736–743. [Google Scholar] [CrossRef]
- Chulani, S.; Boehm, B.; Steece, B. Bayesian analysis of empirical software engineering cost models. IEEE Trans. Softw. Eng. 1999, 25, 573–583. [Google Scholar] [CrossRef]
- Angelis, L.; Stamelos, I. A simulation tool for efficient analogy based cost estimation. Empir. Softw. Eng. 2000, 5, 35–68. [Google Scholar] [CrossRef]
- Boehm, B. Cost Estimation with COCOMO II. 2002. Available online: https://www.researchgate.net/publication/228600814_Cost_estimation_with_COCOMO_II (accessed on 14 August 2024).
- Sentas, P.; Angelis, L.; Stamelos, I. Multinomial logistic regression applied on software productivity prediction. In Proceedings of the 9th Panhellenic Conference in Informatics, Thessaloniki, Greece, 21–23 November 2003; pp. 1–12. [Google Scholar]
- Bibi, S.; Stamelos, I.; Angelis, L. Software cost prediction with predefined interval estimates. Proc. SMEF 2004, 4, 237–246. [Google Scholar]
- Sentas, P.; Angelis, L.; Stamelos, I.; Bleris, G. Software productivity and effort prediction with ordinal regression. Inf. Softw. Technol. 2005, 47, 17–29. [Google Scholar] [CrossRef]
- Kanmani, S.; Kathiravan, J.; Kumar, S.S.; Shanmugam, M. Neural network based effort estimation using class points for oo systems. In Proceedings of the 2007 International Conference on Computing: Theory and Applications (ICCTA’07), Kolkata, India, 5–7 March 2007; IEEE: Piscataway, NJ, USA, 2007; pp. 261–266. [Google Scholar]
- Kanmani, S.; Kathiravan, J.; Kumar, S.S.; Shanmugam, M. Class point based effort estimation of oo systems using fuzzy subtractive clustering and artificial neural networks. In Proceedings of the 1st India Software Engineering Conference, Hyderabad, India, 19–22 February 2008; pp. 141–142. [Google Scholar]
- Kocaguneli, E.; Menzies, T.; Keung, J.W. On the value of ensemble effort estimation. IEEE Trans. Softw. Eng. 2011, 38, 1403–1416. [Google Scholar] [CrossRef]
- Sarro, F.; Petrozziello, A.; Harman, M. Multi-objective software effort estimation. In Proceedings of the 38th International Conference on Software Engineering, Austin, TX, USA, 14–22 May 2016; pp. 619–630. [Google Scholar]
- Qi, F.; Jing, X.Y.; Zhu, X.; Xie, X.; Xu, B.; Ying, S. Software effort estimation based on open source projects: Case study of Github. Inf. Softw. Technol. 2017, 92, 145–157. [Google Scholar] [CrossRef]
- BaniMustafa, A. Predicting software effort estimation using machine learning techniques. In Proceedings of the 2018 8th International Conference on Computer Science and Information Technology (CSIT), Amman, Jordan, 11–12 July 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 249–256. [Google Scholar]
- Usman, M.; Britto, R.; Damm, L.O.; Börstler, J. Effort estimation in large-scale software development: An industrial case study. Inf. Softw. Technol. 2018, 99, 21–40. [Google Scholar] [CrossRef]
- Mustapha, H.; Abdelwahed, N. Investigating the use of random forest in software effort estimation. Procedia Comput. Sci. 2019, 148, 343–352. [Google Scholar]
- Phannachitta, P. On an optimal analogy-based software effort estimation. Inf. Softw. Technol. 2020, 125, 106330. [Google Scholar] [CrossRef]
- Usman, M.; Börstler, J.; Petersen, K. An effort estimation taxonomy for agile software development. Int. J. Softw. Eng. Knowl. Eng. 2017, 27, 641–674. [Google Scholar] [CrossRef]
- Abrahamsson, P.; Fronza, I.; Moser, R.; Vlasenko, J.; Pedrycz, W. Predicting development effort from user stories. In Proceedings of the 2011 International Symposium on Empirical Software Engineering and Measurement, Banff, AB, Canada, 22–23 September 2011; IEEE: Piscataway, NJ, USA, 2011; pp. 400–403. [Google Scholar]
- Porru, S.; Murgia, A.; Demeyer, S.; Marchesi, M.; Tonelli, R. Estimating story points from issue reports. In Proceedings of the the 12th International Conference on Predictive Models and Data Analytics in Software Engineering, Ciudad Real, Spain, 7 September 2016; pp. 1–10. [Google Scholar]
- Scott, E.; Pfahl, D. Using developers’ features to estimate story points. In Proceedings of the 2018 International Conference on Software and System Process, Gothenburg, Sweden, 26–27 May 2018; pp. 106–110. [Google Scholar]
- Soares, R.G. Effort estimation via text classification and autoencoders. In Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil, 8–13 July 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 1–8. [Google Scholar]
- Choetkiertikul, M.; Dam, H.K.; Tran, T.; Pham, T.; Ghose, A.; Menzies, T. A deep learning model for estimating story points. IEEE Trans. Softw. Eng. 2018, 45, 637–656. [Google Scholar] [CrossRef]
- Tawosi, V.; Al-Subaihin, A.; Sarro, F. Investigating the effectiveness of clustering for story point estimation. In Proceedings of the 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), Honolulu, HI, USA, 15–18 March 2022; IEEE: Piscataway, NJ, USA, 2022; pp. 827–838. [Google Scholar]
- Bettenburg, N.; Nagappan, M.; Hassan, A.E. Think locally, act globally: Improving defect and effort prediction models. In Proceedings of the 2012 9th IEEE Working Conference on Mining Software Repositories (MSR), Zurich, Switzerland, 2–3 June 2012; IEEE: Piscataway, NJ, USA, 2012; pp. 60–69. [Google Scholar]
- Nagpal, G.; Uddin, M.; Kaur, A. Analyzing software effort estimation using k means clustered regression approach. ACM SIGSOFT Softw. Eng. Notes 2013, 38, 1–9. [Google Scholar] [CrossRef]
- Minku, L.L.; Yao, X. Ensembles and locality: Insight on improving software effort estimation. Inf. Softw. Technol. 2013, 55, 1512–1528. [Google Scholar] [CrossRef]
- Tawosi, V.; Al-Subaihin, A.; Moussa, R.; Sarro, F. A versatile dataset of agile open source software projects. In Proceedings of the 19th International Conference on Mining Software Repositories, Pittsburgh, PA, USA, 23–24 May 2022; pp. 707–711. [Google Scholar]
- Bishop, C.M. Neural Networks for Pattern Recognition; Oxford University Press: Oxford, UK, 1995. [Google Scholar]
- Gandomani, T.J.; Wei, K.T.; Binhamid, A.K. A case study research on software cost estimation using experts’ estimates, wideband delphi, and planning poker technique. Int. J. Softw. Eng. Its Appl. 2014, 8, 173–182. [Google Scholar]
- Pozenel, M.; Hovelja, T. A comparison of the planning poker and team estimation game: A case study in software development capstone project course. Int. J. Eng. Educ. 2019, 35, 195–208. [Google Scholar]
- SI, I.; Tanveer, B.; Vollmer, A.M.; Braun, S.; Ali, N.b. An evaluation of effort estimation supported by change impact analysis in agile software development. J. Software: Evol. Process 2019, 31, e2165. [Google Scholar]
- Mas’ad, R.; Ñanculef, R.; Astudillo, H. BlackSheep: Dynamic effort estimation in agile software development using machine learning. In Proceedings of the 22nd Ibero-American Conference on Software Engineering, CIbSE, La Habana, Cuba, 22–26 April 2019. [Google Scholar]
- Ungan, E.; Cizmeli, N.; Demirörs, O. Comparison of functional size based estimation and story points, based on effort estimation effectiveness in SCRUM projects. In Proceedings of the 2014 40th EUROMICRO Conference on Software Engineering and Advanced Applications, Verona, Italy, 27–29 August 2014; IEEE: Piscataway, NJ, USA, 2014; pp. 77–80. [Google Scholar]
- Moharreri, K.; Sapre, A.V.; Ramanathan, J.; Ramnath, R. Cost-effective supervised learning models for software effort estimation in agile environments. In Proceedings of the 2016 IEEE 40th Annual computer software and applications conference (COMPSAC), Atlanta, GA, USA, 10–14 June 2016; IEEE: Piscataway, NJ, USA, 2016; Volume 2, pp. 135–140. [Google Scholar]
- Tanveer, B.; Guzmán, L.; Engel, U.M. Effort estimation in agile software development: Case study and improvement framework. J. Softw. Evol. Process 2017, 29, e1862. [Google Scholar] [CrossRef]
- Commeyne, C.; Abran, A.; Djouab, R. Effort estimation with story points and cosmic function points-an industry case study. Softw. Meas. News 2016, 21, 25–36. [Google Scholar]
- Torrecilla-Salinas, C.J.; Sedeño, J.; Escalona, M.; Mejías, M. Estimating, planning and managing Agile Web development projects under a value-based perspective. Inf. Softw. Technol. 2015, 61, 124–144. [Google Scholar] [CrossRef]
- Lenarduzzi, V.; Lunesu, I.; Matta, M.; Taibi, D. Functional size measures and effort estimation in agile development: A replicated study. In Agile Processes in Software Engineering and Extreme Programming: Proceedings of the 16th International Conference, XP 2015, Helsinki, Finland, 25–29 May 2015; Proceedings 16; Springer: Berlin/Heidelberg, Germany, 2015; pp. 105–116. [Google Scholar]
- Ahmed, A.R.; Tayyab, M.; Bhatti, S.N.; Alzahrani, A.J.; Babar, M.I. Impact of story point estimation on product using metrics in scrum development process. Int. J. Adv. Comput. Sci. Appl. 2017, 8, 385–391. [Google Scholar]
- Adnan, M.; Afzal, M. Ontology based multiagent effort estimation system for scrum agile method. IEEE Access 2017, 5, 25993–26005. [Google Scholar] [CrossRef]
- Adnan, M.; Afzal, M.; Asif, K.H. Ontology-oriented software effort estimation system for e-commerce applications based on extreme programming and Scrum methodologies. Comput. J. 2019, 62, 1605–1624. [Google Scholar] [CrossRef]
- Taibi, D.; Lenarduzzi, V.; Diebold, P.; Lunesu, I. Operationalizing the experience factory for effort estimation in agile processes. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, Salerno, Italy, 21–24 June 2017; pp. 31–40. [Google Scholar]
- Gandomani, T.J.; Faraji, H.; Radnejad, M. Planning Poker in cost estimation in Agile methods: Averaging vs. Consensus. In Proceedings of the 2019 5th Conference on Knowledge Based Engineering and Innovation (KBEI), Tehran, Iran, 28 February–1 March 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 66–71. [Google Scholar]
- Grapenthin, S.; Book, M.; Richter, T.; Gruhn, V. Supporting feature estimation with risk and effort annotations. In Proceedings of the 2016 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Limassol, Cyprus, 31 August–2 September 2016; IEEE: Piscataway, NJ, USA, 2016; pp. 17–24. [Google Scholar]
- Huijgens, H.; Solingen, R.v. A replicated study on correlating agile team velocity measured in function and story points. In Proceedings of the 5th International Workshop on Emerging Trends in Software Metrics, Hyderabad, India, 3 June 2014; pp. 30–36. [Google Scholar]
- Dantas, E.; Costa, A.A.M.; Vinicius, M.; Perkusich, M.B.; de Almeida, H.O.; Perkusich, A. An Effort Estimation Support Tool for Agile Software Development: An Empirical Evaluation. In Proceedings of the SEKE, Lisbon, Portugal, 10–12 July 2019; pp. 82–116. [Google Scholar]
- Arifin, H.H.; Daengdej, J.; Khanh, N.T. An empirical study of effort-size and effort-time in expert-based estimations. In Proceedings of the 2017 8th International Workshop on Empirical Software Engineering in Practice (IWESEP), Tokyo, Japan, 13 March 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 35–40. [Google Scholar]
- Usman, M.; Petersen, K.; Börstler, J.; Neto, P.S. Developing and using checklists to improve software effort estimation: A multi-case study. J. Syst. Softw. 2018, 146, 286–309. [Google Scholar] [CrossRef]
- Basri, S.; Kama, N.; Haneem, F.; Ismail, S.A. Predicting effort for requirement changes during software development. In Proceedings of the 7th Symposium on Information and Communication Technology, Ho Chi Minh City, Vietnam, 8–9 December 2016; pp. 380–387. [Google Scholar]
- Dumas-Monette, J.F.; Trudel, S. Requirements engineering quality revealed through functional size measurement: An empirical study in an agile context. In Proceedings of the 2014 Joint Conference of the International Workshop on Software Measurement and the International Conference on Software Process and Product Measurement, Rotterdam, The Netherlands, 6–8 October 2014; IEEE: Piscataway, NJ, USA, 2014; pp. 222–232. [Google Scholar]
- Lusky, M.; Powilat, C.; Böhm, S. Software cost estimation for user-centered mobile app development in large enterprises. In Proceedings of the Advances in Human Factors, Software, and Systems Engineering: Proceedings of the AHFE 2017 International Conference on Human Factors, Software, and Systems Engineering, The Westin Bonaventure Hotel, Los Angeles, CA, USA, 17–21 July 2017; Springer: Berlin/Heidelberg, Germany, 2018; pp. 51–62. [Google Scholar]
- Malgonde, O.; Chari, K. An ensemble-based model for predicting agile software development effort. Empir. Softw. Eng. 2019, 24, 1017–1055. [Google Scholar] [CrossRef]
- Ochodek, M. Approximation of COSMIC functional size of scenario-based requirements in Agile based on syntactic linguistic features—A replication study. In Proceedings of the 2016 Joint Conference of the International Workshop on Software Measurement and the International Conference on Software Process and Product Measurement (IWSM-MENSURA), Berlin, Germany, 5–7 October 2016; IEEE: Piscataway, NJ, USA, 2016; pp. 201–211. [Google Scholar]
- Vetrò, A.; Dürre, R.; Conoscenti, M.; Fernández, D.M.; Jørgensen, M. Combining data analytics with team feedback to improve the estimation process in agile software development. Found. Comput. Decis. Sci. 2018, 43, 305–334. [Google Scholar] [CrossRef]
- Satapathy, S.M.; Rath, S.K. Empirical assessment of machine learning models for agile software development effort estimation using story points. Innov. Syst. Softw. Eng. 2017, 13, 191–200. [Google Scholar] [CrossRef]
- Chongpakdee, P.; Vatanawood, W. Estimating user story points using document fingerprints. In Proceedings of the 2017 8th IEEE International Conference on Software Engineering and Service Science (ICSESS), Beijing, China, 24–26 November 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 149–152. [Google Scholar]
- Satapathy, S.M.; Rath, S.K. Class point approach for software effort estimation using various support vector regression kernel methods. In Proceedings of the 7th India Software Engineering Conference, Chennai, India, 19–21 February 2014; pp. 1–10. [Google Scholar]
- Zakrani, A.; Najm, A.; Marzak, A. Support vector regression based on grid-search method for agile software effort prediction. In Proceedings of the 2018 IEEE 5th International Congress on Information Science and Technology (CiSt), Marrakech, Morocco, 21–27 October 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 1–6. [Google Scholar]
- Kaushik, A.; Tayal, D.K.; Yadav, K. A comparative analysis on effort estimation for agile and non-agile software projects using DBN-ALO. Arab. J. Sci. Eng. 2020, 45, 2605–2618. [Google Scholar] [CrossRef]
- Prasada Rao, C.; Siva Kumar, P.; Rama Sree, S.; Devi, J. An agile effort estimation based on story points using machine learning techniques. In Proceedings of the Second International Conference on Computational Intelligence and Informatics: ICCII 2017, Telangana, India, 25–27 September 2017; Springer: Singapore, 2018; pp. 209–219. [Google Scholar]
- Khuat, T.T.; Le, M.H. An effort estimation approach for agile software development using fireworks algorithm optimized neural network. Int. J. Comput. Sci. Inf. Secur. (IJCSIS) 2016, 14, 122–130. [Google Scholar]
- Premalatha, H.M.; Srikrishna, C.V. Effort estimation in agile software development using evolutionary cost-sensitive deep belief network. Int. J. Intell. Eng. Syst. 2019, 12, 261–269. [Google Scholar]
- Bilgaiyan, S.; Mishra, S.; Das, M. Effort estimation in agile software development using experimental validation of neural network models. Int. J. Inf. Technol. 2019, 11, 569–573. [Google Scholar] [CrossRef]
- Panda, A.; Satapathy, S.M.; Rath, S.K. Empirical validation of neural network models for agile software effort estimation based on story points. Procedia Comput. Sci. 2015, 57, 772–781. [Google Scholar] [CrossRef]
- de Campos Souza, P.V.; Guimaraes, A.J.; Araujo, V.S.; Rezende, T.S.; Araujo, V.J.S. Incremental regularized data density-based clustering neural networks to aid in the construction of effort forecasting systems in software development. Appl. Intell. 2019, 49, 3221–3234. [Google Scholar] [CrossRef]
- Štrba, R.; Štolfa, J.; Štolfa, S.; Košinár, M. Intelligent software support of the SCRUM process. Front. Artif. Intell. Appl. 2014, 272, 408–416. [Google Scholar]
- Septian, W.; Gata, W. Software development framework on small team using Agile Framework for Small Projects (AFSP) with neural network estimation. In Proceedings of the 2017 11th International Conference on Information & Communication Technology and System (ICTS), Surabaya, Indonesia, 31 October 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 259–264. [Google Scholar]
- Hamouda, A.E.D. Using agile story points as an estimation technique in cmmi organizations. In Proceedings of the 2014 Agile Conference, Kissimmee, FL, USA, 28 July–1 August 2014; IEEE: Piscataway, NJ, USA, 2014; pp. 16–23. [Google Scholar]
- Karunakaran, E.; Sreenath, N. A method to effort estimation for XP projects in clients perspective. Int. J. Appl. Eng. Res. 2015, 10, 18529–18550. [Google Scholar]
- Dhir, S.; Kumar, D.; Singh, V. An estimation technique in agile archetype using story points and function point analysis. Int. J. Process Manag. Benchmark. 2017, 7, 518–539. [Google Scholar] [CrossRef]
- Lenarduzzi, V.; Taibi, D. Can Functional Size Measures Improve Effort Estimation in SCRUM? In Proceedings of the ICSEA-International Conference on Software Engineering and Advances, Nice, France, 12–16 October 2014. [Google Scholar]
- Salmanoglu, M.; Hacaloglu, T.; Demirors, O. Effort estimation for agile software development: Comparative case studies using COSMIC functional size measurement and story points. In Proceedings of the 27th International Workshop on Software Measurement and 12th International Conference on Software Process and Product Measurement, Gothenburg, Sweden, 25–27 October 2017; pp. 41–49. [Google Scholar]
- Hacaloglu, T.; Demirors, O. Measureability of functional size in Agile software projects: Multiple case studies with COSMIC FSM. In Proceedings of the 2019 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Kallithea-Chalkidiki, Greece, 28–30 August 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 204–211. [Google Scholar]
- Angara, J.; Prasad, S.; Sridevi, G. Towards Benchmarking User Stories Estimation with COSMIC Function Points-A Case Example of Participant Observation. Int. J. Electr. Comput. Eng. 2018, 8, 3076–3083. [Google Scholar]
- Wanderley, E.G.; Vasconcelos, A.; Avila, B.T. Using function points in agile projects: A comparative analysis between existing approaches. In Proceedings of the Agile Methods: 8th Brazilian Workshop, WBMA 2017, Belém, Brazil, 13–14 September 2017; Revised Selected Papers 8. Springer: Berlin/Heidelberg, Germany, 2018; pp. 47–59. [Google Scholar]
- Prykhodko, N.; Prykhodko, S. A multiple non-linear regression model to estimate the agile testing efforts for small web projects. Radio Electron. Comput. Sci. 2019, 2, 158–166. [Google Scholar] [CrossRef]
- Popli, R.; Chauhan, N. An agile software estimation technique based on regression testing efforts. In Proceedings of the 13th Annual International Software Testing Conference in India, Bangalore, India, 4 December 2013; pp. 1–9. [Google Scholar]
- Owais, M.; Ramakishore, R. Effort, duration and cost estimation in agile software development. In Proceedings of the 2016 Ninth International Conference on Contemporary Computing (IC3), Noida, India, 11–13 August 2016; IEEE: Piscataway, NJ, USA, 2016; pp. 1–5. [Google Scholar]
- Popli, R.; Chauhan, N. Estimation in agile environment using resistance factors. In Proceedings of the 2014 International Conference on Information Systems and Computer Networks (ISCON), Mathura, India, 1–2 March 2014; IEEE: Piscataway, NJ, USA, 2014; pp. 60–65. [Google Scholar]
- Aslam, W.; Ijaz, F.; Lali, M.I.U.; Mehmood, W. Risk Aware and Quality Enriched Effort Estimation for Mobile Applications in Distributed Agile Software Development. J. Inf. Sci. Eng. 2017, 33, 1481–1500. [Google Scholar]
- Alostad, J.M.; Abdullah, L.R.; Aali, L.S. A fuzzy based model for effort estimation in scrum projects. Int. J. Adv. Comput. Sci. Appl. 2017, 8. [Google Scholar]
- Raslan, A.T.; Darwish, N.R. An enhanced framework for effort estimation of agile projects. Int. J. Intell. Eng. Syst. 2018, 11, 205–214. [Google Scholar] [CrossRef]
- Rola, P.; Kuchta, D. Application of fuzzy sets to the expert estimation of Scrum-based projects. Symmetry 2019, 11, 1032. [Google Scholar] [CrossRef]
- Khuat, T.T.; Le, M.H. A novel hybrid ABC-PSO algorithm for effort estimation of software projects using agile methodologies. J. Intell. Syst. 2018, 27, 489–506. [Google Scholar] [CrossRef]
- Manga, I.; Blamah, N. A particle swarm optimization-based framework for agile software effort estimation. Int. J. Eng. Sci. (IJES) 2014, 3, 30–36. [Google Scholar]
- Kumar, C.S.; Kumari, A.A.; Perumal, R.S. An optimized agile estimation plan using harmony search algorithm. Int. J. Eng. Technol. (IJET) 2014, 6, 1994–2001. [Google Scholar]
- Dragicevic, S.; Celar, S.; Turic, M. Bayesian network model for task effort estimation in agile software development. J. Syst. Softw. 2017, 127, 109–119. [Google Scholar] [CrossRef]
- Radu, L.D. Effort Prediction in Agile Software Development with Bayesian Networks. In Proceedings of the ICSOFT, Porto, Portugal, 26–28 July 2019; pp. 238–245. [Google Scholar]
- Garg, S.; Gupta, D. PCA based cost estimation model for agile software development projects. In Proceedings of the 2015 International Conference on Industrial Engineering and Operations Management (IEOM), Dubai, United Arab Emirates, 3–5 March 2015; IEEE: Piscataway, NJ, USA, 2015; pp. 1–7. [Google Scholar]
- Bhaskaran, N.; Jayaraj, V. A hybrid effort estimation technique for agile software development (HEETAD). Int. J. Eng. Adv. Technol. 2019, 9, 1078–1087. [Google Scholar] [CrossRef]
- Basri, S.; Kama, N.; Sarkan, H.M.; Adli, S.; Haneem, F. An algorithmic-based change effort estimation model for software development. In Proceedings of the 2016 23rd Asia-Pacific Software Engineering Conference (APSEC), Hamilton, New Zealand, 6–9 December 2016; IEEE: Piscataway, NJ, USA, 2016; pp. 177–184. [Google Scholar]
- Popli, R.; Chauhan, N.; Sharma, H. Prioritising user stories in agile environment. In Proceedings of the 2014 International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT), Ghaziabad, India, 7–8 February 2014; IEEE: Piscataway, NJ, USA, 2014; pp. 515–519. [Google Scholar]
- Khatri, S.K.; Malhotra, S.; Johri, P. Use case point estimation technique in software development. In Proceedings of the 2016 5th international conference on reliability, infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, India, 7–9 September 2016; IEEE: Piscataway, NJ, USA, 2016; pp. 123–128. [Google Scholar]
- Foss, T.; Stensrud, E.; Kitchenham, B.; Myrtveit, I. A simulation study of the model evaluation criterion MMRE. IEEE Trans. Softw. Eng. 2003, 29, 985–995. [Google Scholar] [CrossRef]
- Moser, R.; Pedrycz, W.; Succi, G. Incremental Effort Prediction Models in Agile Development Using Radial Basis Functions. In Proceedings of the SEKE, Citeseer. 2007, pp. 519–522. Available online: https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=a094027803f6fc090c35caef958b33924789c960#page=539 (accessed on 14 August 2024).
- Abadeer, M.; Sabetzadeh, M. Machine learning-based estimation of story points in agile development: Industrial experience and lessons learned. In Proceedings of the 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW), Notre Dame, IN, USA, 20–24 September 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 106–115. [Google Scholar]
- Tawosi, V.; Moussa, R.; Sarro, F. Agile Effort Estimation: Have We Solved the Problem Yet? Insights From a Replication Study. IEEE Trans. Softw. Eng. 2022, 49, 2677–2697. [Google Scholar] [CrossRef]
- Shepperd, M.; MacDonell, S. Evaluating prediction systems in software project estimation. Inf. Softw. Technol. 2012, 54, 820–827. [Google Scholar] [CrossRef]
- Mittas, N.; Mamalikidis, I.; Angelis, L. A framework for comparing multiple cost estimation methods using an automated visualization toolkit. Inf. Softw. Technol. 2015, 57, 310–328. [Google Scholar] [CrossRef]
- Whigham, P.A.; Owen, C.A.; Macdonell, S.G. A baseline model for software effort estimation. ACM Trans. Softw. Eng. Methodol. (TOSEM) 2015, 24, 1–11. [Google Scholar] [CrossRef]
- Fu, M.; Tantithamthavorn, C. GPT2SP: A transformer-based agile story point estimation approach. IEEE Trans. Softw. Eng. 2022, 49, 611–625. [Google Scholar] [CrossRef]
- Sousa, A.O.; Veloso, D.T.; Gonçalves, H.M.; Faria, J.P.; Mendes-Moreira, J.; Graça, R.; Gomes, D.; Castro, R.N.; Henriques, P.C. Applying Machine Learning to Estimate the Effort and Duration of Individual Tasks in Software Projects. IEEE Access 2023, 11, 89933–89946. [Google Scholar] [CrossRef]
- Anand, A.; Kaur, J.; Singh, O.; Alhazmi, O.H. Optimal Sprint Length Determination for Agile-Based Software Development. Comput. Mater. Contin. 2021, 68, 3693–3712. [Google Scholar] [CrossRef]
- Vlaanderen, K.; Jansen, S.; Brinkkemper, S.; Jaspers, E. The agile requirements refinery: Applying SCRUM principles to software product management. Inf. Softw. Technol. 2011, 53, 58–70. [Google Scholar] [CrossRef]
- Kittlaus, H.B. Software product management and agile software development: Conflicts and solutions. In Software for People: Fundamentals, Trends and Best Practices; Springer: Berlin/Heidelberg, Germany, 2012; pp. 83–96. [Google Scholar]
- Wang, C.; Li, M.; Smola, A.J. Language models with transformers. arXiv 2019, arXiv:1904.09408. [Google Scholar]
- Reimers, N.; Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv 2019, arXiv:1908.10084. [Google Scholar]
- Natekin, A.; Knoll, A. Gradient boosting machines, a tutorial. Front. Neurorobotics 2013, 7, 21. [Google Scholar] [CrossRef]
- Grinsztajn, L.; Oyallon, E.; Varoquaux, G. Why do tree-based models still outperform deep learning on typical tabular data? Adv. Neural Inf. Process. Syst. 2022, 35, 507–520. [Google Scholar]
- Konstantinov, A.V.; Utkin, L.V. Interpretable machine learning with an ensemble of gradient boosting machines. Knowl.-Based Syst. 2021, 222, 106993. [Google Scholar] [CrossRef]
- Bentéjac, C.; Csörgő, A.; Martínez-Muñoz, G. A comparative analysis of gradient boosting algorithms. Artif. Intell. Rev. 2021, 54, 1937–1967. [Google Scholar] [CrossRef]
- Ebrahimi, F.; Tushev, M.; Mahmoud, A. Classifying mobile applications using word embeddings. ACM Trans. Softw. Eng. Methodol. (TOSEM) 2021, 31, 1–30. [Google Scholar] [CrossRef]
- Dorogush, A.V.; Ershov, V.; Gulin, A. CatBoost: Gradient boosting with categorical features support. arXiv 2018, arXiv:1810.11363. [Google Scholar]
- Chen, T.; Guestrin, C. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016; pp. 785–794. [Google Scholar]
- Ke, G.; Meng, Q.; Finley, T.; Wang, T.; Chen, W.; Ma, W.; Ye, Q.; Liu, T.Y. Lightgbm: A highly efficient gradient boosting decision tree. Adv. Neural Inf. Process. Syst. 2017, 30, 1–9. [Google Scholar]
- Bergstra, J.; Yamins, D.; Cox, D. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of the International Conference on Machine Learning, PMLR, Atlanta, GA, USA, 16–21 June 2013; pp. 115–123. [Google Scholar]
- Tawosi, V.; Sarro, F.; Petrozziello, A.; Harman, M. Multi-objective software effort estimation: A replication study. IEEE Trans. Softw. Eng. 2021, 48, 3185–3205. [Google Scholar] [CrossRef]
- Langdon, W.B.; Dolado, J.; Sarro, F.; Harman, M. Exact mean absolute error of baseline predictor, MARP0. Inf. Softw. Technol. 2016, 73, 16–18. [Google Scholar] [CrossRef]
- Gignac, G.E.; Szodorai, E.T. Effect size guidelines for individual differences researchers. Personal. Individ. Differ. 2016, 102, 74–78. [Google Scholar] [CrossRef]
Repository | Project Key | Project | #Issues | Mean | Std | Min | 25% | 50% | 75% | Max |
---|---|---|---|---|---|---|---|---|---|---|
Apache | MESOS | Mesos | 1513 | 3.15 | 2.14 | 0 | 2 | 3 | 5 | 13 |
Appcelerator | ALOY | Alloy | 251 | 3.71 | 2.32 | 0 | 3 | 3 | 5 | 13 |
Appcelerator | TISTUD | Appcelerator Studio | 2794 | 5.48 | 3.23 | 0 | 3 | 5 | 8 | 40 |
Appcelerator | APSTUD | Aptana Studio | 476 | 7.93 | 7.19 | 0 | 5 | 8 | 8 | 100 |
Appcelerator | CLI | Command–Line Interface | 293 | 3.18 | 2.30 | 0 | 1 | 3 | 5 | 13 |
Appcelerator | DAEMON | Daemon | 205 | 5.58 | 3.76 | 1 | 3 | 5 | 8 | 13 |
Appcelerator | TIDOC | Documentation | 1005 | 3.58 | 3.68 | 0 | 1 | 2 | 5 | 40 |
Appcelerator | TIMOB | Titanium | 3915 | 4.68 | 3.32 | 0 | 3 | 5 | 5 | 20 |
Atlassian | CLOV | Clover | 336 | 5.33 | 11.03 | 0 | 1 | 2 | 5 | 100 |
Atlassian | CONFCLOUD | Confluence Cloud | 234 | 2.91 | 2.24 | 0 | 2 | 2 | 3 | 13 |
Atlassian | CONFSERVER | Confluence Server and Data Center | 456 | 3.12 | 1.93 | 0 | 2 | 3 | 3 | 13 |
DNNSoftware | DNN | DotNetNuke Platform | 234 | 2.91 | 2.24 | 0 | 2 | 2 | 3 | 13 |
DuraSpace | DURACLOUD | Duracloud | 310 | 1.72 | 1.7 | 0 | 1 | 1 | 2 | 20 |
Hyperledger | FAB | Fabric | 303 | 2.69 | 3.2 | 0 | 1 | 2 | 3 | 40 |
Hyperledger | STL | Sawtooth | 206 | 2.09 | 1.19 | 0 | 1 | 2 | 3 | 5 |
Lsstcorp | DM | Data Management | 5381 | 3.05 | 6.87 | 0 | 1 | 2 | 2 | 100 |
MongoDB | COMPASS | Compass | 260 | 3.55 | 1.85 | 1 | 2.75 | 3 | 5 | 8 |
MongoDB | SERVER | Server | 519 | 2.58 | 2.4 | 0 | 1 | 2 | 3 | 20 |
MongoDB | EVG | Evergreen | 2824 | 1.43 | 0.86 | 0 | 1 | 1 | 2 | 8 |
Moodle | MDL | Moodle | 1394 | 11.8 | 18.81 | 0 | 2 | 5 | 13 | 100 |
MuleSoft | MULE | Mule | 2935 | 3.88 | 3.46 | 0 | 1 | 3 | 8 | 13 |
Sonatype | Sonatype’s Nexus | NEXUS | 1425 | 1.7 | 1.82 | 0 | 1 | 1 | 2 | 40 |
Spring | XD | SpringXD | 2811 | 3.16 | 2.56 | 0 | 1 | 3 | 5 | 20 |
Talendforge | TDP | Talend Data Preparation | 471 | 2.31 | 1.84 | 0 | 1 | 2 | 3 | 13 |
Talendforge | TDQ | Talend Data Quality | 859 | 6.01 | 4.66 | 0 | 3 | 5 | 8 | 40 |
Talendforge | TESB | Talend ESB | 730 | 2.13 | 1.45 | 0 | 1 | 2 | 3 | 13 |
Cohen’s d Value | Interpretation |
---|---|
d = 0.2 | Small Effect |
d = 0.5 | Medium Effect |
d = 0.8 | Large Effect |
Manual Tuning | Bayesian Optimization | Random Search | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
OPTUNA | Hyperopt | Randomizedcv | ||||||||||||
Feature Extraction | SP Estimation | Method | MAE | MdAE | SA | MAE | MdAE | SA | MAE | MdAE | SA | MAE | MdAE | SA |
Title and description | Extracted features | SBERT-CAT | 2.603 | 1.963 | 94.487 | 2.457 | 1.902 | 94.775 | 2.615 | 1.978 | 94.438 | 2.461 | 1.782 | 94.711 |
SBERT-XG | 2.530 | 1.769 | 94.556 | 2.456 | 1.737 | 94.695 | 2.339 | 1.714 | 94.967 | 2.548 | 1.826 | 94.518 | ||
SBERT-LGBM | 2.451 | 1.829 | 94.719 | 2.528 | 1.794 | 94.600 | 2.459 | 1.816 | 94.791 | 2.541 | 1.815 | 94.530 | ||
Extracted features and issue type | SBERT-CAT | 4.639 | 2.609 | 94.432 | 2.452 | 1.876 | 94.775 | 2.628 | 1.977 | 94.401 | 2.463 | 1.733 | 94.751 | |
SBERT-XG | 4.725 | 2.548 | 94.640 | 2.512 | 1.792 | 94.602 | 2.463 | 1.832 | 94.706 | 2.515 | 1.815 | 94.670 | ||
SBERT-LGBM | 4.563 | 2.442 | 94.726 | 2.480 | 1.742 | 94.643 | 2.438 | 1.763 | 94.748 | 2.498 | 1.800 | 94.707 | ||
Extracted features and components | SBERT-CAT | 3.315 | 3.196 | 92.932 | 2.938 | 2.728 | 93.673 | 2.809 | 2.491 | 93.948 | 2.932 | 2.451 | 93.684 | |
SBERT-XG | 2.536 | 1.794 | 94.582 | 2.527 | 1.883 | 94.634 | 2.606 | 2.064 | 94.446 | 2.556 | 1.914 | 94.554 | ||
SBERT-LGBM | 2.433 | 1.850 | 94.814 | 2.492 | 1.847 | 94.720 | 2.506 | 1.767 | 94.663 | 2.534 | 1.896 | 94.640 | ||
Extracted features, issue type, and components | SBERT-CAT | 2.889 | 2.618 | 93.803 | 2.893 | 2.620 | 93.774 | 2.882 | 2.559 | 93.775 | 2.583 | 2.093 | 94.436 | |
SBERT-XG | 2.591 | 1.721 | 94.441 | 2.591 | 1.721 | 94.424 | 2.420 | 1.533 | 94.843 | 2.534 | 1.658 | 94.569 | ||
SBERT-LGBM | 2.402 | 1.794 | 94.882 | 2.535 | 1.748 | 94.586 | 2.608 | 1.730 | 94.474 | 2.637 | 1.949 | 94.420 | ||
Title, description, and issue type | Extracted features | SBERT-CAT | 2.617 | 1.937 | 94.401 | 2.466 | 1.863 | 94.745 | 2.588 | 1.920 | 94.484 | 2.433 | 1.754 | 94.815 |
SBERT-XG | 2.518 | 1.774 | 94.625 | 2.489 | 1.750 | 94.649 | 2.427 | 1.819 | 94.783 | 2.478 | 1.781 | 94.686 | ||
SBERT-LGBM | 2.448 | 1.826 | 94.713 | 2.481 | 1.745 | 94.641 | 2.461 | 1.782 | 94.699 | 2.532 | 1.801 | 94.635 | ||
Extracted features and components | SBERT-CAT | 3.956 | 3.759 | 91.450 | 2.588 | 1.818 | 94.418 | 2.707 | 2.136 | 94.153 | 3.179 | 2.760 | 93.151 | |
SBERT-XG | 2.532 | 1.748 | 94.569 | 2.533 | 1.981 | 94.603 | 2.625 | 1.894 | 94.407 | 2.563 | 1.874 | 94.526 | ||
SBERT-LGBM | 2.431 | 1.888 | 94.820 | 2.451 | 1.866 | 94.806 | 2.460 | 1.803 | 94.728 | 2.530 | 1.888 | 94.648 | ||
Title, description, and components | Extracted features | SBERT-CAT | 2.591 | 1.942 | 94.478 | 2.459 | 1.913 | 94.760 | 2.599 | 1.906 | 94.462 | 2.460 | 1.777 | 94.757 |
SBERT-XG | 2.522 | 1.763 | 94.567 | 2.452 | 1.726 | 94.730 | 2.393 | 1.760 | 94.856 | 2.456 | 1.749 | 94.718 | ||
SBERT-LGBM | 2.435 | 1.820 | 94.740 | 2.498 | 1.778 | 94.619 | 2.443 | 1.770 | 94.737 | 2.513 | 1.790 | 94.675 | ||
Extracted features and issue type | SBERT-CAT | 2.598 | 1.930 | 94.448 | 2.432 | 1.806 | 94.817 | 2.599 | 1.928 | 94.462 | 2.475 | 1.770 | 94.726 | |
SBERT-XG | 2.506 | 1.760 | 94.675 | 2.404 | 1.691 | 94.833 | 2.461 | 1.713 | 94.706 | 2.457 | 1.721 | 94.738 | ||
SBERT-LGBM | 2.427 | 1.799 | 94.758 | 2.459 | 1.739 | 94.703 | 2.412 | 1.741 | 94.803 | 2.497 | 1.780 | 94.709 | ||
Title, description, issue type, and components | Extracted features | SBERT-CAT | 2.602 | 1.936 | 94.445 | 2.445 | 1.780 | 94.790 | 2.631 | 1.935 | 94.393 | 2.431 | 1.769 | 94.820 |
SBERT-XG | 2.526 | 1.794 | 94.603 | 2.471 | 1.795 | 94.690 | 2.484 | 1.764 | 94.660 | 2.494 | 1.784 | 94.650 | ||
SBERT-LGBM | 2.438 | 1.814 | 94.734 | 2.442 | 1.729 | 94.726 | 2.466 | 1.751 | 94.688 | 2.505 | 1.787 | 94.692 |
Project | Method | MAE | MdAE | SA | Project | Method | MAE | MdAE | SA | Project | Method | MAE | MdAE | SA |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ALOY | SBERT-XG | 2.07 | 1.75 | 19.10 | DNN | SBERT-XG | 1.91 | 1.75 | 61.07 | STL | SBERT-XG | 1.03 | 0.84 | 21.04 |
SBERT-LGBMTC | 2.07 | 1.86 | 19.30 | SBERT-LGBMTC | 1.40 | 1.23 | 71.25 | SBERT-LGBMTC | 0.98 | 0.66 | 44.42 | |||
LHC-SE | 1.84 | 2.00 | 26.57 | LHC-SE | 0.71 | 1.00 | 42.60 | LHC-SE | 1.28 | 1.00 | −6.77 | |||
LHCTC-SE | 2.28 | 2.00 | 9.01 | LHCTC-SE | 0.71 | 1.00 | 42.60 | LHCTC-SE | 0.95 | 1.00 | 20.41 | |||
Deep-SE | 1.51 | 1.28 | 39.07 | Deep-SE | 0.72 | 0.69 | 41.69 | Deep-SE | 1.18 | 1.12 | 1.91 | |||
TF-IDF-SE | 1.44 | 2.00 | 42.53 | TF-IDF-SE | 0.79 | 1.00 | 36.13 | TF-IDF-SE | 0.84 | 0.00 | 30.12 | |||
Mean | 2.23 | 2.17 | 10.84 | Mean | 0.80 | 0.88 | 35.28 | Mean | 0.97 | 1.02 | 19.32 | |||
Median | 1.44 | 2.00 | 42.53 | Median | 0.71 | 1.00 | 42.60 | Median | 0.95 | 1.00 | 20.41 | |||
APSTUD | SBERT-XG | 5.81 | 4.49 | 63.86 | DURACLOUD | SBERT-XG | 2.15 | 1.90 | −19.61 | TDP | SBERT-XG | 2.39 | 2.02 | 51.47 |
SBERT-LGBMTC | 5.70 | 4.50 | 62.75 | SBERT-LGBMTC | 1.82 | 1.71 | 12.26 | SBERT-LGBMTC | 1.77 | 1.59 | 64.05 | |||
LHC-SE | 4.14 | 3.00 | 30.20 | LHC-SE | 1.25 | 1.00 | −9.65 | LHC-SE | 1.00 | 1.00 | 37.08 | |||
LHCTC-SE | 3.99 | 3.00 | 32.81 | LHCTC-SE | 0.68 | 1.00 | 39.94 | LHCTC-SE | 1.03 | 1.00 | 35.44 | |||
Deep-SE | 4.31 | 2.70 | 27.37 | Deep-SE | 0.68 | 0.58 | 39.90 | Deep-SE | 0.99 | 0.81 | 37.69 | |||
TF-IDF-SE | 3.99 | 3.00 | 32.81 | TF-IDF-SE | 0.68 | 1.00 | 39.94 | TF-IDF-SE | 0.99 | 1.00 | 37.74 | |||
Mean | 4.00 | 2.49 | 32.72 | Mean | 0.67 | 0.85 | 41.13 | Mean | 1.17 | 1.38 | 26.26 | |||
Median | 3.99 | 3.00 | 32.81 | Median | 0.68 | 1.00 | 39.94 | Median | 0.99 | 1.00 | 37.74 | |||
CLI | SBERT-XG | 1.37 | 1.23 | 20.00 | EVG | SBERT-XG | 2.09 | 1.85 | −107.72 | TDQ | SBERT-XG | 3.63 | 2.31 | 48.83 |
SBERT-LGBMTC | 1.97 | 1.88 | −19.21 | SBERT-LGBMTC | 1.56 | 1.40 | −64.83 | SBERT-LGBMTC | 3.85 | 2.35 | 44.02 | |||
LHC-SE | 1.87 | 2.00 | 29.32 | LHC-SE | 0.60 | 1.00 | 22.69 | LHC-SE | 3.52 | 3.00 | 27.60 | |||
LHCTC-SE | 1.76 | 2.00 | 33.35 | LHCTC-SE | 0.62 | 1.00 | 19.97 | LHCTC-SE | 2.92 | 3.00 | 40.01 | |||
Deep-SE | 1.76 | 1.30 | 33.44 | Deep-SE | 0.63 | 0.54 | 19.39 | Deep-SE | 2.47 | 2.23 | 49.14 | |||
TF-IDF-SE | 2.98 | 3.00 | −12.84 | TF-IDF-SE | 0.69 | 1.00 | 10.67 | TF-IDF-SE | 5.05 | 5.00 | −3.95 | |||
Mean | 2.14 | 2.61 | 18.93 | Mean | 0.68 | 0.56 | 12.98 | Mean | 4.20 | 3.82 | 13.65 | |||
Median | 1.77 | 2.00 | 33.04 | Median | 0.69 | 1.00 | 10.67 | Median | 2.88 | 3.00 | 40.72 | |||
CLOV | SBERT-XG | 2.51 | 1.69 | 85.21 | FAB | SBERT-XG | 2.13 | 1.76 | 63.98 | TESB | SBERT-XG | 1.91 | 1.72 | 63.31 |
SBERT-LGBMTC | 3.28 | 3.06 | 81.75 | SBERT-LGBMTC | 1.60 | 1.41 | 67.15 | SBERT-LGBMTC | 1.37 | 1.29 | 74.41 | |||
LHC-SE | 3.88 | 2.00 | 46.35 | LHC-SE | 0.67 | 1.00 | 69.75 | LHC-SE | 0.99 | 1.00 | 32.56 | |||
LHCTC-SE | 4.23 | 1.50 | 41.55 | LHCTC-SE | 0.65 | 1.00 | 70.47 | LHCTC-SE | 1.04 | 1.00 | 29.31 | |||
Deep-SE | 3.78 | 1.05 | 47.73 | Deep-SE | 0.86 | 0.71 | 61.06 | Deep-SE | 1.15 | 0.73 | 21.36 | |||
TF-IDF-SE | 4.04 | 1.00 | 44.15 | TF-IDF-SE | 1.10 | 1.00 | 50.31 | TF-IDF-SE | 0.97 | 1.00 | 33.95 | |||
Mean | 5.93 | 5.30 | 18.06 | Mean | 1.19 | 1.10 | 46.21 | Mean | 0.99 | 0.99 | 32.71 | |||
Median | 4.01 | 2.00 | 44.55 | Median | 0.97 | 1.00 | 69.75 | Median | 0.98 | 1.00 | 33.02 | |||
COMPASS | SBERT-XG | 1.58 | 1.16 | 42.97 | MDL | SBERT-XG | 7.21 | 3.25 | 83.00 | TIDOC | SBERT-XG | 2.65 | 2.34 | 84.49 |
SBERT-LGBMTC | 1.76 | 1.75 | 20.78 | SBERT-LGBMTC | 6.26 | 2.77 | 86.32 | SBERT-LGBMTC | 2.07 | 1.41 | 88.93 | |||
LHC-SE | 1.38 | 2.00 | 28.54 | LHC-SE | 6.31 | 7.00 | 57.30 | LHC-SE | 2.79 | 1.00 | 23.48 | |||
LHCTC-SE | 1.30 | 1.00 | 32.46 | LHCTC-SE | 6.31 | 7.00 | 57.30 | LHCTC-SE | 3.65 | 2.00 | −0.30 | |||
Deep-SE | 1.63 | 1.34 | 15.25 | Deep-SE | 3.55 | 2.77 | 76.00 | Deep-SE | 2.72 | 1.19 | 25.35 | |||
TF-IDF-SE | 1.38 | 2.00 | 28.54 | TF-IDF-SE | 6.31 | 7.00 | 57.30 | TF-IDF-SE | 3.03 | 1.00 | 16.69 | |||
Mean | 1.48 | 1.63 | 23.05 | Mean | 14.54 | 15.23 | 1.58 | Mean | 2.99 | 2.59 | 18.00 | |||
Median | 1.38 | 2.00 | 28.54 | Median | 6.31 | 7.00 | 57.30 | Median | 2.77 | 1.00 | 24.03 | |||
CONFCLOUD | SBERT-XG | 1.74 | 1.41 | -50.68 | MESOS | SBERT-XG | 1.64 | 1.25 | 63.06 | TIMOB | SBERT-XG | 2.10 | 1.57 | 51.53 |
SBERT-LGBMTC | 2.56 | 2.57 | −140.46 | SBERT-LGBMTC | 1.48 | 0.92 | 66.32 | SBERT-LGBMTC | 2.26 | 2.09 | 47.62 | |||
LHC-SE | 1.34 | 1.00 | 40.41 | LHC-SE | 1.34 | 1.00 | 34.38 | LHC-SE | 2.53 | 2.00 | 30.70 | |||
LHCTC-SE | 1.37 | 1.00 | 39.04 | LHCTC-SE | 1.33 | 1.00 | 34.63 | LHCTC-SE | 2.48 | 2.00 | 32.12 | |||
Deep-SE | 1.48 | 0.93 | 33.89 | Deep-SE | 1.34 | 1.12 | 34.07 | Deep-SE | 2.41 | 1.81 | 33.90 | |||
TF-IDF-SE | 1.33 | 1.00 | 40.86 | TF-IDF-SE | 1.34 | 1.00 | 34.38 | TF-IDF-SE | 2.53 | 2.00 | 30.70 | |||
Mean | 1.49 | 1.23 | 33.65 | Mean | 1.37 | 1.08 | 32.72 | Mean | 2.55 | 1.81 | 30.23 | |||
Median | 1.33 | 1.00 | 40.87 | Median | 1.34 | 1.00 | 34.38 | Median | 2.53 | 2.00 | 30.70 | |||
CONFSERVER | SBERT-XG | 1.74 | 1.49 | −2.47 | MULE | SBERT-XG | 2.67 | 2.37 | 42.00 | TISTUD | SBERT-XG | 1.71 | 1.29 | 54.62 |
SBERT-LGBMTC | 2.00 | 1.78 | −9.49 | SBERT-LGBMTC | 2.58 | 2.02 | 43.89 | SBERT-LGBMTC | 2.51 | 2.48 | 32.08 | |||
LHC-SE | 0.96 | 1.00 | 49.64 | LHC-SE | 2.27 | 2.00 | 37.11 | LHC-SE | 1.51 | 2.00 | 51.89 | |||
LHCTC-SE | 0.96 | 1.00 | 49.64 | LHCTC-SE | 2.60 | 3.00 | 28.16 | LHCTC-SE | 1.51 | 2.00 | 51.89 | |||
Deep-SE | 0.91 | 0.64 | 52.28 | Deep-SE | 2.24 | 1.68 | 37.95 | Deep-SE | 1.63 | 1.38 | 48.08 | |||
TF-IDF-SE | 0.96 | 1.00 | 49.64 | TF-IDF-SE | 3.58 | 2.00 | 0.81 | TF-IDF-SE | 1.51 | 2.00 | 51.89 | |||
Mean | 1.35 | 1.45 | 29.17 | Mean | 2.79 | 3.18 | 22.68 | Mean | 2.01 | 2.16 | 35.93 | |||
Median | 0.96 | 1.00 | 49.64 | Median | 2.24 | 2.00 | 38.05 | Median | 1.51 | 2.00 | 51.89 | |||
DAEMON | SBERT-XG | 3.06 | 2.25 | 12.06 | NEXUS | SBERT-XG | 1.46 | 1.27 | 45.92 | XD | SBERT-XG | 1.91 | 1.57 | 77.61 |
SBERT-LGBMTC | 2.80 | 1.84 | 42.16 | SBERT-LGBMTC | 1.57 | 1.06 | 39.85 | SBERT-LGBMTC | 1.71 | 1.21 | 78.91 | |||
LHC-SE | 2.81 | 3.00 | 32.09 | LHC-SE | 1.14 | 1.00 | 22.52 | LHC-SE | 1.54 | 1.00 | 39.53 | |||
LHCTC-SE | 2.74 | 3.00 | 33.81 | LHCTC-SE | 1.22 | 1.00 | 16.88 | LHCTC-SE | 1.50 | 1.00 | 40.85 | |||
Deep-SE | 3.29 | 2.00 | 20.55 | Deep-SE | 1.08 | 0.88 | 26.56 | Deep-SE | 1.45 | 1.16 | 43.06 | |||
TF-IDF-SE | 2.74 | 3.00 | 33.81 | TF-IDF-SE | 1.17 | 1.00 | 20.68 | TF-IDF-SE | 2.01 | 2.00 | 20.82 | |||
Mean | 2.75 | 2.75 | 33.53 | Mean | 1.11 | 0.58 | 54.69 | Mean | 1.65 | 1.71 | 34.89 | |||
Median | 2.74 | 3.00 | 33.81 | Median | 1.17 | 1.00 | 20.68 | Median | 1.55 | 1.00 | 39.05 | |||
DM | SBERT-XG | 2.79 | 2.06 | 84.23 | SERVER | SBERT-XG | 1.77 | 1.52 | −19.02 | |||||
SBERT-LGBMTC | 2.81 | 2.32 | 84.82 | SBERT-LGBMTC | 1.29 | 1.27 | 28.03 | |||||||
LHC-SE | 1.56 | 1.00 | 53.87 | LHC-SE | 0.85 | 1.00 | 59.47 | |||||||
LHCTC-SE | 1.52 | 1.00 | 54.94 | LHCTC-SE | 0.85 | 1.00 | 59.47 | |||||||
Deep-SE | 1.61 | 0.89 | 52.41 | Deep-SE | 0.89 | 0.71 | 57.60 | |||||||
TF-IDF-SE | 1.49 | 1.00 | 55.71 | TF-IDF-SE | 0.93 | 1.00 | 55.88 | |||||||
Mean | 2.60 | 2.43 | 22.83 | Mean | 1.56 | 1.86 | 25.99 | |||||||
Median | 1.61 | 1.00 | 52.19 | Median | 0.85 | 1.00 | 59.46 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Yalçıner, B.; Dinçer, K.; Karaçor, A.G.; Efe, M.Ö. Enhancing Agile Story Point Estimation: Integrating Deep Learning, Machine Learning, and Natural Language Processing with SBERT and Gradient Boosted Trees. Appl. Sci. 2024, 14, 7305. https://doi.org/10.3390/app14167305
Yalçıner B, Dinçer K, Karaçor AG, Efe MÖ. Enhancing Agile Story Point Estimation: Integrating Deep Learning, Machine Learning, and Natural Language Processing with SBERT and Gradient Boosted Trees. Applied Sciences. 2024; 14(16):7305. https://doi.org/10.3390/app14167305
Chicago/Turabian StyleYalçıner, Burcu, Kıvanç Dinçer, Adil Gürsel Karaçor, and Mehmet Önder Efe. 2024. "Enhancing Agile Story Point Estimation: Integrating Deep Learning, Machine Learning, and Natural Language Processing with SBERT and Gradient Boosted Trees" Applied Sciences 14, no. 16: 7305. https://doi.org/10.3390/app14167305