Methodology for Constructing and Using a Mathematical Model for Assessing Labor Costs for the Development of Distance Learning Courses

Abstract

Distance learning plays a significant role in solving the problem of the sustainable development of society. Unfortunately, the development and implementation of distance courses are still carried out intuitively, which does not allow practical implementation of effective mathematics methods and slows down the introduction of breakthrough technologies. The aim of the research was to develop a new model and a methodology for assessing the labor costs of designing distance learning courses by university teachers based on a comprehensive assessment of the courses’ quality. The suggested model originally extends the constructive cost evaluation model already used in IT industry. The developed criteria make it possible to assess labor costs in relation to the quality of course development. The paper provides and analyzes such calculations on the example of courses in the theory of algorithms and programming.

1. Introduction

There is no doubt that distance learning focuses on two of the biggest challenges of our time: sustainability and disruptive technology [1]. Sheriff and Aliyu [2] have shown that distance learning if properly planned and managed, can lead to the sustainable development of education institutions because of its flexibility and conveniences. It is demonstrated that alternative educational services of distance learning can make a significant contribution to achieving sustainable development of society.

Analyzing and considering the process of rapid development and improvement of information and communication technologies, modern university (higher) education is gradually transforming its own existing traditional (face-to-face) forms of education. These forms are being supplemented or replaced by blended and/or distance learning forms. The role of electronic teaching tools (educational resources) and electronic educational materials is constantly growing mainly because they are being actively used by the younger generation of students, which is both a challenge and a reason that higher education certainly is responding to.

In the 21st century, it is obvious that the updating speed of information used in universities for educational purposes requires a continuous change in the content of academic courses, the reviewing of teaching materials and the designing of completely new courses. This fact means a significant increase in the labor costs of the university academic and auxiliary staff to maintain the relevance of the educational process in its form and content. Since the work of designing distance learning courses lies mainly with the university staff, it has resulted in the appearance of an urgent task to evaluate the labor costs of designing new courses and modernizing existing ones while ensuring the proper quality of the educational process.

The Transport and Telecommunication Institute (TTI) and European Humanities University (EHU) have accumulated many years of experience in the use of distance learning (DL) in the educational process. The authors of the study have summarized this experience in the assessment of labor costs and the quality of distance learning courses in the form of a technique that was brought to specific calculations and recommendations.

Currently, for virtually any institution of higher education that uses DL and designs distance learning courses (DC) on their own, there is an acute task of keeping the quality of the developed DCs at a high level, which would allow them to successfully compete with “massive open online courses” (MOOC) or transfer their best developments to well-known platforms for open use [3]. Today, such open online platforms as Coursera, edX, Udacity offers free DCs of high quality.

The university practice for assessing the quality of designed DCs is not always in open access. Further, we present potential numerical criteria for assessing the quality of a university DC development. The criteria have become the result of more than 10 years of experience accumulated by two higher educational institutions that are in close cooperation. The numerical criteria are presented based on the qualitative analysis of more than a hundred distance learning courses. This analysis has been carried out by specialists supporting distance learning, its results have been discussed with teachers at conferences, seminars, and continuing education courses in both higher educational institutions. The article presents the original model and methodology for assessing the teacher’s labor costs for a DC development. As an example of the methodology implementation for undergraduate students in the theory of algorithms and programming and related courses, the paper discusses course characteristics used for practical labor cost assessment.

2. Motivation and Foundations for Creating a Methodology for Assessing Labor Costs for the Development of DC’s

2.1. Brief Historical Retrospective

From the very beginning, the emergence of distance learning courses took place on the individual initiative of individual teachers [4]. These courses reflected an individual pedagogical experience and were created by enthusiastic teachers for themselves and their university. In this case, the exact measurement of labor costs for the development of a specific distance course did not have much value, and the courses themselves were not mandatory. At present, the situation has changed fundamentally, which has shown us once again the effects of the COVID-19 coronavirus pandemic—read the message [5] from The President of Harvard College. Moving to distance education has become a vital problem for all universities in the world.

In this regard, all the already developed distance learning technologies have been in demand and the development of new distance learning courses have become widespread and mandatory for universities as one of the main competitive advantages.

Additional significant funds have been allocated for the development of distance education, which has required the organization of planning and accounting of labor costs for the development of new distance courses.

The most known attempt to model the cost of distance education was presented by Bates [6] in early 1995 when today’s possibilities were not imaginable yet. The model is oriented to the development of the whole program of distance education. The program development costs include the following items: (1) Course Developer; (2) Contract Instructional Designer; (3) Graphic Designer; (4) Program Coordinator; (5) Project Manager; (6) Production Assistant; and (7) Learning Support Specialist. The model amortizes development costs over three years and specifies the minimum number of students per course that is required for the program to meet its development costs over this period. However, the model does not account for course update costs. University of the West Indies [7] uses this model in its framework for the development of sustainable e-learning.

Moseley and Valverde [8] suggested the following mathematical model for the cost estimation for an e-learning project: Y = 32.13 − 0.7 × Number of Screens + 0.26 × Still Images + 0.13 × Audio Clips − 0.03 × Interactions + 0.86 ×Animations − 7.81 × Video Clips. The value of Y defines the number of human-days required to develop the e-learning project. The results of the comparison are provided with the COCOMO model. The presented results show that the predicted cost according to the proposed model is closer to the real value than according to the COCOMO model.

Unfortunately, the publicly available literature does not provide any reasonable instruction for DC’s development labor cost assessment, which has its own specifics. There are a lot of books on pedagogical aspects of distance learning [9,10]. But the question of development resources planning, and consumption formal assessment is still open. There may be various reasons for this.

Firstly, the modern implementation of a distance learning course is possible only in a digital form, which requires the appropriate qualifications of the developer and assessor with meaningful intellectual input.

Secondly, the requirements for the quality of such a distance course are significantly increased due to the presence of global competition and must be assessed explicitly.

These two specific points make the assessment process untrivial and knowledge-consuming.

That is why this paper discusses these specific features in detail to create a model and methodology for assessing the labor costs of developing distance courses. As a basis for the methodology, the ideas of the new model were borrowed from the constructive cost estimation model—COCOMO [11]—which has been successfully used in the IT industry for decades when evaluating software development projects. The second important aspect of the proposed assessment methodology is directly related to the quality characteristics of the distance course being developed.

The proposed methodology is not only a generalization of the practical experience of distance learning but also a step forward towards further digitalization of education. The widespread introduction of the developed methodology will not only facilitate the planning and management of the development of specific distance courses with a given level of quality but will also ensure the sustainable development of training programs and educational institutions in general.

2.2. Criteria for Assessing the Quality of DC Development

Assessing the DL quality is a complex and time-consuming process, which includes:

• A comprehensive analysis of the organizational structure of an educational institution, its technical and information resources;
• Assessment of university syllabi, the quality of the developed and used electronic educational resources, and course materials;
• Analysis of the effectiveness of the teacher/student feedback system and the system for assessing the process of DL;
• Determination of the effectiveness of DC didactics and teaching process techniques;
• Assessment of the level of literacy of teachers and students in computer technology.

Further, we consider the quality of DL, related only to the designing of distance learning courses and the development of electronic training resources and materials for them.

By assessment of the DC quality, we mean a system for assessing the course compliance with modern requirements of pedagogical design: a DC strategy, electronic learning resources and materials, pedagogical activities, and a student achievement evaluation system.

Khan, an authoritative scientist, and researcher in the field of DL, notes the need to consider the following aspects of the institutional direction of DL development: financial assessment; infrastructure assessment; and assessment of the cultural readiness of educational institutions [12].

Considering the issue of quality of a DC development based on infrastructure assessment, one should first agree on:

• A strategy model that is proposed for a DC development;
• An organization form of the learning process in an educational institution where electronic learning materials are created for teaching students in a distance format;
• The type of evaluated DCs;
• The characteristics of teaching methods, depending on areas of knowledge.

2.3. Strategy Model

There are a few publications on DL development strategies (e.g., Distance Learning Strategic Plan [13], Strategic Plan for Distance Education [14,15]) with no resource allocation. Currently, these are two most popular main models for the organization of teachers in DL.

In the first model, each teacher prepares distance learning courses mainly on his/her own, seeking advice from IT specialists and pedagogical design specialists. Experts admit that such courses are usually characterized by the simplest pedagogical design, inefficiently managed technological infrastructure, and a significant overload of teachers. These distance courses are mediocre from the point of view of electronic pedagogy and are not distinguished by the high quality of electronic educational materials.

The second model is based on a team approach, in which employees with the necessary level of competence in each of the areas of design, development, and maintenance of distance learning courses participate in both the development and the teaching of distance learning courses. These are the developers of educational textual content, educational multimedia and video materials, pedagogical designers, and others. The second model is more expensive, but it is also more efficient in terms of the quality and competitiveness of educational services in a distance format [16].

The experience of the center for distance learning at EHU has shown that the second model is the most efficient one and it dramatically affects the quality of the developed DCs. This is because the majority of the university DC teachers deliver courses in humanities and social sciences. That allows them, for example, to well understand and use the Internet technology in teaching, but they do not have a sufficient level of competence for the independent development of high-quality, and sometimes quite complex electronic resources or materials for a DC implementing the requirements of pedagogical design.

2.4. Organization Form of the Educational Process

Assessing the DC quality on the rubric’s basis, it is necessary to determine the form of organization of the educational process in an educational institution, since at present, in all forms of education, even in traditional ones, students are offered to learn electronic materials and various pedagogical activities in a distance format. In this case, we single out the three key independent forms of organization of the educational process: high-residence, low-residence, distance learning. It should be noted that new forms have now begun to be widely applied, for example, the low-residency form, which seems to be a combination of educational didactic approaches used once for correspondence courses, distance learning, and a high-residence form of education with face-to-face classes.

Further, we apply the criteria for assessing a DC development for the distance learning form as an independent form of organization of the educational process at the university.

One should not ignore the life-long organizational forms of the educational process; such forms are actively used in the adult continuing education system. According to one of the authors of the article, the distance learning form for adults is divided into several types: network, case-network, hybrid, and distance self-study [16]. The stringent assessment criteria of a developed DC are applicable, first, to distance self-study, but in some cases, they can be used for adult network education.

The correct assessment of labor costs for the development of distance courses should be based on the specific form of using digital material in the educational process and the required quality, which is essential for achieving sustainable development goals [1]. The methodology proposed by the authors is based on a process approach that involves continuous improvement of distance learning courses.

2.5. Evaluated Type after the DC Development

The assessing rubrics for the quality of DCs may also vary depending on the typology of DCs. The type, of course, is selected depending on the conceptual directions of the design and development of the distance course adopted by the educational institution, as well as on the goals and objectives of the course and the methodological features of its teaching.

We list the classification of the four typologies of courses, which are determined by:

• The form of use in the educational process: full-fledged distance learning courses, hybrid courses, supporting courses [17];
• The nature of communication: asynchronous and synchronous distance learning courses;
• The principle of placement of the main educational material [18];
• Type of educational activity [16].

The criteria for assessing the quality of development will be applied to full-fledged distance learning courses.

2.6. Brief Description of the Research Methodology

The study on the creation of a mathematical model for assessing the labor costs for the development of DC was carried out according to the following methodology [17]:

• The analysis of information sources in the field of assessing the resources necessary for the development of distance learning courses as digital (software) products was carried out;
• A hypothesis about the possibility of using a model such as the COCOMO model was put forward;
• Empirical material on the real labor costs for the development of DC was collected;
• A mathematical model was developed for assessing labor costs;
• The necessary coefficients for the model were selected iteratively;
• Labor costs and corrected the values of the coefficients were calculated;
• The applicability of performing calculations on selected DCs was assessed.

3. Implementation of the Methodology for DC’s Development Labor Costs Assessment

3.1. Teaching Methods Peculiarities and Areas of Knowledge

A retrospective analysis of the quality of the DC development shows that the quality of the course largely depends on the peculiarities of the methodology applied to various fields of knowledge that students major in. For example, to learn a foreign language, an entire set of educational elements is required to develop not only grammar skills but what is more important to master listening and speaking skills. A DC in social sciences and humanities also has certain peculiarities, although the requirements in pedagogical design and didactics are very similar. For example, in social sciences, computer technologies are more actively used for the analysis of statistical data, modeling of social processes, etc. Quite different ones are the criteria for assessing the quality of courses in design, the training materials of which cannot do without high-quality instructional videos and the teacher/student feedback.

When building a model for calculating the cost of developing a distance course, the goal is to preliminarily forecast the efforts of developing a course and the duration of work.

Based on the existing classifications of models [18,19] by the nature of the ongoing processes, we classified our model as a deterministic model, since the analytically presented criteria for the laws of management and pedagogical design, expressed in quantitative values, gave us a specific numerical result for each set of input data.

By the way of implementation, we referred this model to an abstract mathematical analytical model, since the description of functional relationships between quantities is made in it using formulas.

When calculating the cost of developing a modern distance course, we further considered the criteria for assessing the quality of the development of a modern course (pedagogical design) and criteria for assessing the qualifications of developer personnel (management), which were quantitatively expressed.

At the same time, we understood that indicators from both areas of knowledge may be partially dependent on each other. But such dependence cannot be absolute, only relative. For example, the availability of highly qualified personnel to develop a course is not an absolute guarantee of achieving the required quality.

As for courses in mathematics and engineering, their structure of the rubrics was similar, but their numerical values K should be empirically adapted considering the higher labor costs for preparing electronic learning resources for organizing the online access to professional software and remote laboratory equipment.

Here are the criteria for evaluating the results of the DC development for:

• The second strategic model (work in a group project, teacher + specialists in pedagogical design and IT);
• Distance self-study;
• A full-fledged distance course;
• Without a focus on courses, whose teaching methods can significantly affect the rubrics for evaluating the quality of the development of distance learning courses and the ratio for different criteria.

The methodology of criteria for evaluating the results of the DC development was formed by us on the basis of the recommendations and research of scientists [20,21,22], as well as on the basis of our own research [23].

Further, to calculate the required labor costs, we also used another indicator E—as a dimensionless characteristic of the team and development conditions.

We present

Table 1. Standard labor costs ratios.

Name of the Group Criteria	Criteria for Quality Assessment of a Developed DC	The Maximum Sum of Weight Criteria in a Group	Standard Labor Costs Ratio Ki
1. Course strategy	Course summary	0.25 (max 0.25)	0.05
	Course syllabus designed in accordance with the requirements of the university		0.15
	Simple and clear statement of the goals, objectives and expected results of the course		0.05
2. Compliance with the didactic principles of a DC: apprehensibility, accessibility, interactivity	Clear structure of the course as a whole and its individual modules	0.13 (max 0.15)	0.01
	Clear definition of course certification policy		0.01
	Correspondence of the number of working weeks to the number of credits		0.01
	Clear and detailed instructions and recommendations for working with each module/topic of the course		0.04
	Clear instructions or recommendations for completing (final) tests/course activities and an exam		0.05
	Interactive teaching alerts and teacher/student feedback		0.03
3. E-learning resources (materials)	Teacher’s copyright materials in the course	0.24 (max 0.3)	0.1
	Attribution of authorship for all study materials		0.02
	Additional electronic training materials		0.02
	Weblinks to the resources that are freely available on the Internet		0.02
	Learning materials with modern multimedia technologies (multimedia presentations, videos, computer models, mind maps, etc.)		0.1
	Compliance of electronic educational materials with recommendations for their design		0.04
4. Learning didactic resources and activities	Correspondence of the number of tests/tasks per one credit according to the requirements of the university	0.16 (max 0.3)	0.02
	Variety of didactic resources created using the DL system: instructional glossaries, instructional Wikis, interactive lectures, educational forums, computer tests, and other educational tools imbedded in the DL system		0.1
	Tasks/activities for students’ self-study in accordance with the requirement of the university		0.1
	Activities aimed at collaborative learning		0.03
	Modern DL pedagogical technologies and methods		0.05

, which contains the procedure for calculating the coefficient K—standard labor costs in human-weeks for 1 ECTS, the value of which depended on the weight of the criteria for assessing the quality of a developed DC. The value of the coefficient K (for our example in human-weeks) was the sum of the values K_i of the coefficients for each group of criteria, i.e.,

$$\mathrm{K}={\sum }_{i=1}^n{K}_i, \mathrm{where} n=20, \mathrm{K} \le 1$$

(1)

The ratios of standard labor costs in human-weeks for 1 ECTS, depending on the criteria for assessing the quality of a developed DC, are presented in Table 1.

The value of each ratio was determined by the teachers themselves at seminars and continuing education courses. The presented ratios can be supplemented or changed depending on the groups of academic courses. Accordingly, each university, considering its features and financial and technological capabilities, can perform this work of calculating the sum of the ratios independently based on surveys of teachers or questionnaires.

The results of the quality evaluation of developed DCs must necessarily be the basis for calculating the ratio of labor costs for their design since everything that is developed for DCs should maximally reflect the quality of development.

In order to take into account the characteristics of a developer’s team for the labor costs Q_j, it was proposed to use

Table 2. Characteristics for assessing the skill level of a group of DC developers.

Name of the Characteristic Groups	Characteristics of the Development Team	Qj
Language proficiency	Language of instruction in the course	0.95
	English	0.9
	DC design language	0.95
DL pedagogy and pedagogical design	Skills in DL didactics	0.95
	Expertise in methods and ways of organizing communications in DL and intercultural environment	0.9
	Expertise in the requirements of pedagogical design	0.9
Mastery of the course material	Experience in conducting research in the subject area of the course	0.8
	Copyright publications to be used in the course	0.95
	Professional (by education)	0.95
	According to sources of literature (self-education)	0.95
	General vision (short courses)	0.95
	Experience in teaching the course	0.8
Team communication	Experience in collaborative development of similar courses	0.9
Experience in ICT and media technologies for the development of ELR and materials (electronic learning resources)	Development of multimedia presentations	0.95
	Scripting and filming learning media materials	0.95
	Designing and creating learning animations	0.9
	Using video conferencing in distance learning	0.9
	Using online educational tools, such as a Google application	0.7
	Experience in creating educational computer models for DL	0.9
Number of developers	More than 4	1.05
Quality of development management	Full-fledged project	0.9
	Resource allocation development plan	0.95
	The task for development only	0.95
Staff turnover	With a turnover rate of more than 25%	1.1

, through which we could calculate the product of the values of the characteristics to assess the level of qualification and the readiness of the group (team) for development. The product of all characteristic values in quantitative terms was indicated by the letter Q (it should be in the range 1.01–1.26). The idea of using just such constant limits was borrowed from the COCOMO model [11]. In the absence of any j-th characteristic for the development team, the value Q_j = 1.

For a qualitative assessment of the qualification level and readiness of specialists in a group, it is necessary to proceed from its possible composition: teacher–author of the course, manager for coordination of work, instructor in pedagogical design, a specialist in the development of educational video materials, and a specialist in the creation of educational animated materials, if animation is present in an updated version of the course [16] (Table 1).

The value of Q_j() in the Table 2 may also be greater than 1—they are the subject of expert evaluations based on the experience gained in developing distance learning courses.

At the same time, assessing the level of readiness of the group for the DC development, we could present general requirements for all members of the group or orient these requirements depending on the professional qualities of each specialist j.

3.2. Labor Costs Calculation for the DC Development

The study used the structure of the COCOMO II model [11] and proposed an approach to assessing efforts in designing a distance course based on the size and complexity of the developmental course. The approach was based on the similarity of software development processes (SE) and distance learning courses.

We considered this distance learning course as a specialized software development project that would result in the creation of online digital educational materials in accordance with specific requirements. The distance course was measured in credit points (ECTS).

By analogy with the COCOMO model, the basic formula was first used, which involved calculating the first rough estimate of the labor costs involved in developing a distance course only based on knowledge of the course volume in ECTS credit points (the main driver of development costs). As practice has shown, a preliminary calculation using the basic formula allows the ratios shown in Table 1 and Table 2 to be correctly evaluated and adjusted (adapted) for their further use in practical calculations in the organization.

In this case, the basic calculation formula for labor costs (in man-weeks) had the following form:

Labor costs = M × (NC) ^E1,

(2)

where M and E1 were selected from

Table 3. The table of ratios for the basic formula (labor cost).

NC (DL Course Size in ECTS)	M (Man-Weeks/ECTS)	E1
1–2	2.5	1.01
3–6	3.0	1.10
7–10	3.5	1.26

, NC (DL course size in credits).

As the next step we can use the basic calculation formula for schedule (in weeks) estimation (like COCOMO recommendations). The result would be the following:

Schedule = N × (Labor cost) ^E2,

(3)

where N and E2 are selected from

Table 4. The table of ratios for the basic formula (schedule).

DL Course Size (ECTS)	N (Weeks/Labor Cost)	E2
1–2	2.5	0.38
3–6	2.5	0.35
7–10	2.5	0.32

.

The next step in increasing the accuracy of calculating the required labor costs was the use of other existing drivers of cost and labor, which were associated with ensuring the quality of the developed distance course. For this, a list of drivers was used, which we display above in Table 1 and Table 2.

In this case, the general formula for calculating labor costs was modified and took the following form:

$$\mathrm{Labor} \mathrm{costs}=\mathrm{NC} \times {\sum }_{i=1}^n{K}_i \times { \mathrm{NC}}^{(1.01+0.24\times {\prod }_{j=1}^m{Q}_j)},$$

(4)

that is, in accordance with the formula (2), where M = NC ×${\sum }_{i=1}^n{K}_i \mathrm{and}$ E1 =$(1.01+0.24\times {\prod }_{j=1}^m{Q}_j),$ here for $K_i$ i varies from 1 to n (n is the number of ratios from Table 1, $n=20$), and for Q_j, j varies from 1 to m (m is the number of characteristics from Table 2, m = 24). Constants 1.01 and 0.24 were borrowed from the COCOMO model [11].

Our recommendations for evaluating labor costs can be used for the following situations:

• Making an investment or other financial decisions related to the development of a distance course;
• Setting budgets and project schedules as the basis for planning and control;
• Selection of specialists with necessary qualifications in a group of developers;
• Making decisions or agreeing on trade-offs between software costs, schedule, functionality, and performance or quality metrics.

The considered approach to the assessment of labor costs during the development of a distance course was experimentally implemented as part of the development of the educational resource for the portal of assessing competencies in software engineering [24].

The numerical values Q_j given in the tables were checked by the authors only for the indicated group of courses. For other groups of courses, their empirical refinement was required, the algorithm of which is of independent scientific interest and is not considered in this paper. We only note that this algorithm was based on expert evaluation of the labor costs of the process of a distance course development.

As an example, consider the calculation of labor costs for the distance course “Modern Information Technologies” (MIT, 3 ECTS credits), which is mandatory and is studied at all educational programs at EHU.

The course was intended to systematize students’ knowledge and skills in the field of using computer technologies for processing text and graphic information, techniques for working with spreadsheets, and Internet technologies for humanitarian educational programs.

Filling out tables with coefficients of normative labor costs was carried out by specialists in the field of pedagogical design of EHU University for the course of MIT.

Using the first basic calculation formula for labor costs (in man-weeks) for ECTS = 3 and date from the Table 3 (course size 3–6 ECTS, M = 3, NC = 3 and E = 1.1) we had the following result:

Labor costs = 3.0 × 3^1.1 = 10 (man × weeks)

(5)

Schedule calculation (duration in weeks) can be done using data from Table 4 for the same DL course size 3–6 ECTS with already calculated Labor cost = 6.7. The schedule duration result is as follows:

Schedule = 2.5 × 10^0.35 = 5.6 (weeks)

(6)

As the result of using the basic formulas, we could assume that for the development of one 3 ECTS size distance learning course we needed a nominal 5.6-week development for two full-time employees (10/5.6 = 1.8).

The second calculation was performed using a general formula that allows one to consider the quality criteria of the developed distance course. In other words, we calculated the values of M and E1 based on the data from Table 1 and Table 2 but did not take them from Table 3 and Table 4.

For a calculation example, we chose from Table 1 the maximum number of course quality criteria—all values from all groups of criteria. In this case, the value of M = 3.

We also proceeded with the calculation of the exponent value E2. To do this, we multiplied the values of all the criteria from Table 2, except for one staff turnover, that is, suppose that we had all the desirable characteristics of the development team, and the composition of the team would not change during development. Then the exponent for the formula for calculating labor costs E1 would be determined by the formula:

$$\mathrm{E}1=(1.01+0.24\times {\prod }_{j=1}^m{Q}_j)$$

(7)

where j is from 1 to 24.

For the data taken from Table 2 we have:

E1 = 1.01 + 0.24 × 0.006 = 1.011

(8)

In this case (for the course MIT), the adjusted value of the projected labor cost would be:

$$\mathrm{M}=\mathrm{NC}\times {\sum }_{i=1}^n{K}_i=3\times 0.78=2.34$$

Labor costs = 2.34 × 3^1.011 = 7.1 (man × weeks)

In turn, the assessment of the duration of the development of the course would differ:

Schedule = 2.5 × 7.1^0.35 = 4.9 (weeks)

As can be easily seen from the calculation example, the required quality indicators of the developed course can noticeably affect the development time.

3.3. The Limitations of the Proposed Evaluation Methodology

A serious practical limitation of the proposed methodology application for assessing labor costs in the development of distance courses is the use of a significant number of empirical coefficients included in the mathematical model.

To solve this replication problem [25], the authors proposed to use an iterative process that would allow each organization to form its own metric basis (set of coefficients) by sequentially comparing real labor costs with the estimates obtained. It is recommended that one uses the values provided in this article as the default values for starting the coefficient refinement process [26].

The method of how to optimize this process was not considered in this work.

4. Conclusions

This study offered a methodology for calculating the labor costs of DL courses development with an example for a group of courses in the theory of algorithms and programming. The authors formulated the feature groups and criteria for the DL course quality assessment as well as the characteristics of the team of developers, which were considered in the calculation of labor costs.

The authors proposed to expand the study to other groups of courses that are likely to require a conversion of the ratios proposed for use in the developed methodology. In broad terms, the given formula, as the authors consider, may be of a universal character for calculating the labor costs of any DL courses development.

The study of the degree of relativity and mutual influence of the factors used in the proposed model requires further research. From these studies, it is be possible to justify the magnitude of the error in the calculations. In the future, following the provisions of a systematic approach to the problem under study, the calculation formulas presented by us could be supplemented, taking into account a number of additional factors and indicators, for example, hardware and software support for the course, which cannot be provided by the capabilities of specific distance learning systems.

The developed methodology will not only facilitate the development planning and management of specific distance courses with a given level of quality but will also ensure the sustainable development of training programs and educational institutions in general.