Evaluating 3D Printers Using Data Envelopment Analysis

Abstract

Data Envelopment Analysis (DEA) is an established powerful mathematical programming technique, which has been employed quite extensively for assessing the efficiency/performance of various physical or virtual and simple or complex production systems, as well as of consumer and industrial products and technologies. The purpose of the present study is to investigate whether DEA may be employed for evaluating the technical efficiency/performance of 3D printers, an advanced manufacturing technology of increasing importance for the manufacturing sector. For this purpose, a representative sample of 3D printers based on Fused Deposition Modeling technology is examined. The technical factors/parameters of 3D printers, which are incorporated in the DEA, are investigated and discussed in detail. DEA evaluation results compare favorably with relevant benchmarks from experts, indicating that the suggested DEA technique in conjunction with technical and expert evaluation could be employed for evaluating the performance of a highly technological system, such as the 3D printer.

1. Introduction

In the 21st century global competition for manufacturing enterprises has become even harder; manufacturers of every size are trying to develop and sustain competitive advantages that will enable them to protect or increase their market share and enter new markets. The increased competitive pressure translates to higher requirements for both the production and product development processes, where process speed, flexibility, and continuous innovation are of increasing importance, besides the fundamental goals of high cost efficiency and quality. Additive Manufacturing and 3D Printing technologies are considered as key enablers for achieving the above goals at product design and development, as prototyping and visualization methods, as well as in the production floor, where they are commonly employed for the construction of production jigs and fixtures, and increasingly as the basic manufacturing method of the actual final parts and components [1,2,3].

The field of Additive Manufacturing (AM), encompasses an expanding and rather versatile group of manufacturing technologies where parts are gradually formed/fabricated through the successive stacking/joining of thin layers. Associated technologies, such as Stereolithography (SLA), Fused Deposition Modeling (FDM) or Fused Filament Fabrication (FFF), Material Jetting (MJ), Selective Laser Sintering (SLS), Selective or Electron Laser Melting (SLM/EBM), etc., usually process a limited set of materials that possess certain technology-specific characteristics (e.g., photocurable resins for SLA) and are supplied in a specific form (powder for SLS, filament for FDM/FFF etc.). The term “3D Printing” was initially (early 1990s) associated with the binder jetting AM process developed at the Massachusetts Institute of Technology. In the following years, its meaning has expanded, and it is often used as a synonym to Additive Manufacturing, especially in mainstream media and non-technical context, but this usage is quite inaccurate. In the present study, we employ the term “3D Printing” to refer to the fabrication of objects through the deposition of a material using a print head, nozzle, or another printer technology, as defined in recent international standards concerning AM terminology (ISO/ASTM 52900:2015). According to these standards, 3D printing is, also, commonly associated with systems (3D printers) that are low end in price and/or overall capability.

Since all AM technologies are relatively young, there is still undergoing and significant development of almost all their attributes (cost, speed, quality, repeatability, material variety, etc.). As a result of this, the corresponding market is still quite immature and volatile, with several competitors/manufacturers, of almost every size, competing at several market segments. This is particularly evident at the field of FDM/FFF, a material extrusion method, where an explosion in the number of available systems and materials has been observed during the last ten years. Beginning with the expiration of several patents regarding basic components of the technology, several researchers, amateur hobbyists, and enterprises started developing their own versions of FDM/FFF-based printers. Some of these systems began and continue to be developed as ‘open-hardware’ projects following the ‘open-source’ software paradigm (characteristic example is the Rep-Rap project and its spinoffs [4], while others incorporate proprietary technology. Both variants are available in the market, which due to the relatively low-cost of the systems has expanded beyond the manufacturing sector to those of equipment for home/hobby usage and small businesses. As a result of the above, there is currently a plethora of consumer FDM/FFF 3D printers in the world market, covering various demands and applications. Low-cost printers, initially, required a relatively high level of technical knowledge and were not very user-friendly [5], but these aspects are continuously improving. Furthermore, the assessment of the efficiency of the available 3D Printers is by no means a trivial task and requires a high level of technological expertise and the evaluation of several factors. There is, therefore, from a buyer’s/consumer’s perspective, the need of having reliable tools for assessing the relative efficiency/value of 3D printers, which can be considered as a production and/or design tool. Assessing the efficiency of 3D Printers can also be useful for 3D printer manufacturers/developers, for constant benchmarking of their products against the competition.

For evaluating a product’s performance, several models, methods, and tools have been proposed. Indicatively, one could refer to the EVR (eco-cost value ratio) model [6], the Multiattribute Value Analysis method [7,8], the Gestalt-Minimalism-based decision-making model [9], the Analytical Target Cascading method [10], and the Analytical Hierarchy Process method [11]. A basic characteristic of the above-mentioned methods is that they demand, as input, several claims and judgments that are based on subjective evaluations and complicated calculations [12]. Data Envelopment Analysis (DEA), first introduced by Charnes, Cooper, and Rhodes [13], is a technique that can overcome such concerns. In DEA, the technical efficiency of a set of Decision-Making Units (DMUs) is evaluated by considering the amount of resources that have been employed by each DMU (inputs) for the production of certain products or outcomes (outputs) [14]. The analysis results are useful not only for comparing DMUs, but also for directing inefficient DMUs towards more efficient ways of production, defined in this context as transformation of inputs to outputs [15]. The method employs a linear non-parametric programming procedure in order to identify the most efficient DMUs, which collectively define the so-called efficient frontier. All other DMUs, i.e., those located below the frontier, are considered as inefficient. The method has the basic advantage of being a multi-dimensional method that can handle a large number of parameters, which may be functionally independent and/or evaluated in quite different measures and units. Moreover, DEA as a non-parametric comparative method, does not require all DMUs to balance the input and output parameters in a similar way. On the other hand, a disadvantage of the method is that it is quite sensitive to extreme data points and outliers.

Despite the fact that DEA has been primarily developed and used as a benchmarking method for business and non-profit units and organizations [16,17], it has also been extensively used for measuring the technical efficiency of consumer products, such as printers [18], automobiles [19,20], and cell phones [21,22], as well as of advanced manufacturing technological systems and equipment, such as robots [23,24,25,26,27,28,29], CNC machines [30,31,32], expert systems [33], and flexible manufacturing systems [34,35,36,37]. Based on the above, DEA seems a suitable method for assessing the relative efficiency of currently available 3D printers. The corresponding results and information could prove useful both for potential users of the technology, in order to navigate the rapidly changing and evolving landscape of available systems and manufacturers, as well as for developers and manufacturers of systems, in order to assess the technical efficiency of their systems compared with that of competitors and aptly adjust their market and product development strategy. In this context the application of DEA for the assessment of 3D printers is investigated in the present paper.

Since it is difficult to compare 3D printing systems that are based on different technologies, mostly due to the associated material and technical differences, the study focuses on FDM/FFF technology, which is one of the most widely adopted and marketed 3D Printing methods. Materials used in FDM/FFF are typically based on thermoplastic polymers, such as ABS, PLA, and PETG, which are processed either in pure form or serve as the thermoplastic matrix for composites with several types of reinforcement (wood/metal particles, carbon fibers, glass fibers, etc.). The material is supplied in the form of filaments which is extruded from a heated moving nozzle, constructing thereby the thin layers that compose the part. Compared with other AM technologies, FDM/FFF is relatively simple from a technical point of view and allows the use of a relatively wide variety of materials (some metals and ceramics can also be processed), reasons which partly explain the popularity of the method and the high number of systems available in the market.

The rest of the paper is organized as follows. For the sake of completeness and integrity of presentation, a brief presentation of the theoretical background regarding the DEA method and its variations/models that are employed in the present study, is provided in the following section. Subsequently, the results from the investigation of a set of 38 3D printers based on FDM technology are presented and discussed in detail. In order to check the reliability of the proposed methodology the analysis results are compared with those of relevant benchmarks on the Internet. Finally, the basic concluding remarks and directions for further research are briefly discussed in the last section.

2. Theoretical Framework

DEA is one of the most popular and best documented techniques of mathematical programming, and several variations of the technique have been developed [38,39]. In the context of the present study, three DEA models/variations were examined: the CCR (Charnes–Cooper–Rhodes) model [13], the BCC (Banker–Charnes–Cooper) model [40] and the context-dependent DEA model [41]. The CCR or ‘constant returns to scale’ model is the oldest and most commonly employed variation of DEA [42]. In the CCR formulation of DEA, the technical efficiency ${\theta }^{*}$ of each DMU (DMU₀) is evaluated by solving the following minimization problem [43].

$$\left\{\begin{array}{c}{\theta }^{*}=min\theta \\ \mathrm{Subject} \mathrm{to}\\ {\displaystyle \sum _{j=1}^n}{x}_{ij}{\lambda }_j\le \theta x_{i0},\hspace{1em}i=1, 2, \dots , m\\ {\displaystyle \sum _{j=1}^n}{y}_{rj}{\lambda }_j\ge y_{r0},\hspace{1em}r=1, 2, \dots , s\\ {\lambda }_j\ge 0,\hspace{1em}\hspace{1em}j=1, 2, \dots , n\end{array}\right.$$

(1)

where n is the number of DMUs, m is the number of inputs, s the number of outputs and ${\lambda }_j$ are the corresponding input/output coefficients (nonnegative scalars) assigned to each DMU. The above model is, also, commonly referred as the Farrell model [44]. The above formulation of the problem implies that the value of ${\theta }^{*}$ cannot exceed 1. Accordingly, all DMUs for which ${\theta }^{*}=1$ are considered efficient, while all other DMUs are considered inefficient. An efficiency score of 1 does not guarantee ‘full efficiency’ in the sense that some inputs could be reduced or some outputs could be increased without affecting the need for other inputs or the production of other outputs [45]. To assess this possibility slack values for DMUs are usually evaluated. The slack values s− and s+ of DMU₀ represent the amount that resource consumption or output production should be improved for the specific unit to reach the efficient frontier, and are evaluated by solving the following maximization problem.

$$\left\{\begin{array}{c}max{\displaystyle \sum _{i=1}^m}{s}_i^{-}+{\displaystyle \sum _{r=1}^s}{s}_r^{+}\\ \mathrm{Subject} \mathrm{to}\\ {\displaystyle \sum _{j=1}^n}{x}_{ij}{\lambda }_i+s_i^{-}={\theta }^{*}{x}_{i0},\hspace{1em}i=1, 2, \dots , m\\ {\displaystyle \sum _{j=1}^n}{y}_{rj}{\lambda }_j-s_r^{+}=y_{r0},\hspace{1em}r=1, 2, \dots , s\\ {\lambda }_j, s_i^{-}, s_r^{+}\ge 0\hspace{1em}\forall i,j,r\end{array}\right.$$

(2)

Following slack evaluation, the identification of fully efficient DMUs is possible, for which both efficiency ${\theta }^{*}=1$ and all slack values are equal to zero ($s_i^{-*}=s_r^{+*}=0$). Τhe BCC or ‘variable returns to scale’ model is formulated as the CCR model described above, but incorporates the following additional constraint:

$${\displaystyle \sum _{j=1}^n}{\lambda }_j=1$$

(3)

Efficient units, in this model also, are those that have the maximum efficiency score and zero slack values.

Classic DEA models help in evaluating the efficiency of DMUs and provide an indication of the distance to the best practice frontier, but do not provide specific guidance regarding how inefficient units may become efficient, i.e., whether outputs should be increased, inputs be reduced or both, and to what extent. Furthermore, as has been stressed in several studies [46,47], the performance ranking is often influenced by the context, i.e., the set of DMUs a specific unit is rated against. Especially in the case of product selection it has been argued that the choice context, i.e., what other options are available, is especially important for determining the “attractiveness” of an option [48]. For addressing the above considerations, the method of context-dependent DEA has been proposed [41]. In context-dependent DEA DMUs are classified in several subgroups/-sets which are considered efficient in specific contexts. This is achieved through successive optimization runs (levels), where efficient DMUs are assigned to a context-specific group and are subsequently removed from further DEA evaluation. The process is repeated until all DMUs are assigned to a specific group. The mathematical formulation of the corresponding linear programming model is the following [49].

$$\left\{\begin{array}{c}{\theta }^{*}\left(l,k\right)=\underset{{\lambda }_j,\theta \left(l,k\right)}{min}\theta \left(\lambda ,k\right)\\ \mathrm{Subject} \mathrm{to}\\ {\displaystyle \sum _{j\in F\left(J^l\right)}}{\lambda }_j{x}_{ij}\le \theta \left(l,k\right)x_{ik}\\ {\displaystyle \sum _{j\in F\left(J^l\right)}}{\lambda }_j{y}_{rj}\ge y_{rk}\\ {\lambda }_j\ge 0,\hspace{1em}j\in F\left(J^l\right)\end{array}\right.$$

(4)

where ${\theta }^{*}\left(l,k\right)$ is the optimum efficiency of DMU_k at level l, $J^l$ is the set of all DMUs at level l ($J^1$ is the initial set of all n DMUs), and $x_{ij}$, $y_{rj}$ and ${\lambda }_j$ represent the inputs, outputs coefficient of DMU j respectively, following the previously adopted notation. Since the set of DMUs under evaluation at each level changes, the notation $j\in F\left(J^l\right)$ is employed for defining the relevant DMUs.

As described earlier the subsequent categorization of DMUs in context-specific DMUs is a repetitive process which begins at level l = 1 by evaluating the initial set $J^1$, which is comprised of all n DMUs. Employing the standard DEA methodology described above, the subset E¹ of efficient DMUs at the first evaluation level, is defined. Subset E¹ is then removed from the initial evaluation set $J^1$ in order to obtain the second-level evaluation set $J^2$ and a new DEA evaluation is performed. The process is repeated until all DMUs are assigned to a level-specific efficient set, i.e., the set of DMUs for further evaluation is empty.

With context-dependent DEA it is, also, possible to further compare the efficiency of DMUs within each level-specific efficient set, by evaluating their attractiveness. A high attractiveness score indicates that a DMU is compared favorably with most other units, while a low attractiveness score will indicate a DMU with more referent DMUs from the inefficient stratum [50]. Attractiveness is, therefore, a measure of a DMU’s relative efficiency within an efficient group, and is evaluated according to the following linear programming model [49].

$$\left\{\begin{array}{c}{H}_q^{*}\left(d\right)=min{H}_q\left(d\right),\hspace{1em}d=1,\dots , L-lo\\ \mathrm{Subject} \mathrm{to}\\ {\displaystyle \sum _{j\in F\left(E^{l_o+d}\right)}}{\lambda }_j{x}_j\le H_q\left(d\right)x_q\\ {\displaystyle \sum _{j\in F\left(E^{l_0+d}\right)}}{\lambda }_j{x}_j\ge y_q\\ {\lambda }_j\ge 0,\hspace{1em}j\in F\left(E^{l_o+d}\right)\end{array}\right.$$

(5)

The $H_q^{*}\left(d\right)$ is called the d-degree attractiveness of DMU_q for a specific level $E^{l_o}$. As noted earlier, this measure helps the decision-maker in comparing alternatives to others that belong to the same efficient frontier, as well as within various evaluation contexts, i.e., evaluate the attractiveness of an alternative for different peer sets. Another measure that can be evaluated in context-dependent DEA is that of progress. The progress score indicates the substantial improvement that a DMU needs to acquire to reach the frontier; hence the lower, the better. Progress for a specific DMU_q is evaluated in a similar fashion, according to the following linear programming model [49].

$$\left\{\begin{array}{c}{G}_q^{*}\left(g\right)=min{G}_q\left(g\right),\hspace{1em}g=1,\dots ,L-1\\ \mathrm{Subject} \mathrm{to}\\ {\displaystyle \sum _{j\in F\left(E^{l_o-g}\right)}}{\lambda }_j{x}_j\le G_q\left(\beta \right)x_q\\ {\displaystyle \sum _{j\in F\left(E^{l_0-g}\right)}}{\lambda }_j{x}_j\ge y_q\\ {\lambda }_j\ge 0,\hspace{1em}j\in F\left(E^{l_o-g}\right)\end{array}\right.$$

(6)

The corresponding g-degree progress of a DMU_q from a specific level $E^{l_o}$ is evaluated in this case as follows.

$$M_q^{*}\left(g\right)\equiv 1/G_q^{*}\left(g\right)$$

(7)

Progress values are computed at each level and for all DMUs, except those belonging in the first-level frontier, and represent the incremental improvement a DMU must achieve in order to ascend to higher levels of efficiency [51].

3. Data Collection and Implementation

There is currently a plethora of 3D printers based on FDM/FFF technology that are available on the market. A rough estimation of available systems and machine vendors according to several specialized and general industrial web sites and sources (directindustry.com, 3Dhubs.com, 3dprintingindustry.com, 3ders.org, all3dp.com; all web sites accessed on various dates between March 2018–October 2019) shows that well above 200 models and systems from more than 40 vendors are currently offered worldwide. The corresponding prices for the great majority of systems falls mostly in the low to medium range (a few hundred to a few thousand dollars) but there are also high-end industrial scale systems available priced at several tens of thousands dollars. FDM/FFF printers can be supplied either fully assembled or as Do-It-Yourself (DIY) kits that are assembled by the user. In the case of DIY kits, the buyer will usually take into account the perceived difficulty of assembly and whether sufficient instructions are provided by the manufacturer; both factors that depend on the level of user’s experience with 3D printers and are, therefore, quite subjective. In order, therefore, to avoid this non-objective parameter, printers provided as assembly kits were excluded from the analysis.

Based on the results of the preliminary analysis, a homogeneous initial population of assembled FDM/FFF printers, i.e., printers based on the thermoplastic material extrusion technology, offered at a price below USD 5000, has been constructed (273 printers). Low cost 3D printers based on other technologies, such as SLA or DLP, were excluded from the analysis because they are quite different in terms of materials and technical features. The corresponding final list contains several models from various companies including, but not limited to, the following (in alphabetical order): Anet, Craftunique, Creality, Dremel, Flashforge, Leapfrog, Lulzbot, Makerbot, Makergear, Monoprice, Prusa3D, Qidi, Raise3D, Ultimaker, and Wanhao. The majority of these printers are developed and offered according to the “low-cost online business model”, whose main aim is to provide quality printers at low cost, with emphasis being placed at speed, ease-of-use, and durability [52].

In terms of materials, all identified printers can process standard PLA and ABS filaments. Depending on the configuration and capabilities of each printer, several other thermoplastic materials can be used. Some of the most commonly used thermoplastic materials for FDM/FFF printers and their basic properties and price range are summarized in

Table 1. Basic properties and price range for common FDM materials [53,54,55,56,57].

Material	Tensile Strength (MPa)	Young’s Modulus (GPa)	Stiffness	Impact Resistance	Price (USD/kg)
ABS	22–38	0.8–2.3	High	Medium	10–40
ASA	33–55	2.0–2.2	High	Medium	30–50
PA	39–85	1.0–2.4	Middle	High	25–65
PC	60–72	1.8–2.3	High	Medium	40–75
PET	50–53	2.1–3.6	High	Medium	20–60
PLA	39–65	1.3–3.3	High	Low	10–40
TPE	26–43	0.4–0.6	Low	High	30–70

. Printers equipped with two or more nozzles are considered capable of employing a secondary material, such as water-soluble PVA, for sacrificial support structures. Identified printers do not incorporate a heated chamber, since this a patent-protected feature of Stratasys and would, therefore, significantly limit the initial population. They can, however, incorporate a heated platform or an enclosed chamber. Other non-standard features, such as a moving/conveyer build surface or a rotating build table for continuous printing and automatic part removal, were not considered too, for the same reason.

In terms of software, the corresponding printers are usually accompanied by specially-developed slicing software but can, also, be programmed employing popular, freely available slicers such as Cura (ultimaker.com/software/ultimaker-cura, accessed on 26 April 2021), Prusaslicer (www.prusa3d.com/prusaslicer/, accessed on 26 April 2021), Craftware (craftbot.com/software, accessed on 26 April 2021) and/or Repetier (www.repetier.com/, accessed on 26 April 2021). The above software tools allow the user to appropriately define basic printer properties and parameters (workspace shape and dimensions, number of nozzles, filament diameter, etc.) to fit the specific printer. They offer, also, basic slicing features such as the definition of build parameters (build orientation, layer thickness, extrusion/bed temperatures, infill pattern and percentage, deposition speeds, etc.) and the generation of support structures. They include, also, modules for simulating/previewing the generated G-code file. Advanced features such as multi-material support, mesh modifiers, break-away support structures, variable layer height definition and special infill patterns (e.g., gyroid, Hilbert, or adaptive cubic) are, also, available.

For the identified printers, technical data concerning specific parameters were collected from relevant web sources, mostly from the manufacturers’ official web sites but also from various 3D printing forums and sites. The technical characteristics, which were employed as parameters in the proposed DEA evaluation, were: build volume (effective machine volume), number of extruder heads, layer thickness (Z resolution), nozzle diameter, the existence or not of a heated plate, resolution of movement in the X-Y (layer) plane, variety of materials that can be processed by the printer, nominal power, printing speed, whether the printer is open source or not, weight, and, finally, the printer’s market price. The 3D printer has been considered as a system whose inputs are associated with cost features, whereas its outputs are associated with its technical characteristics and capabilities. Therefore all the above parameters, but the last one, were evaluated as outputs, whilst market price was considered the sole input parameter. Taking the above statement into consideration, the primary objective of the evaluation is to evaluate whether DMUs can proportionally reduce their excess resources (cost) while keeping their characteristics constant. Thus input oriented DEA has been employed in the current analysis.

It is evident that regarding certain parameters, enhanced 3D printer performance is associated with higher corresponding values. It is preferable, e.g., for a printer to have a higher printing speed or a larger printing volume. Nevertheless, there are also parameters related to the printer’s performance that we prefer to be as small as possible, e.g., the minimum layer thickness or X-Y resolution possible. To overcome the different, ascending or descending, nature of the parameters, which cannot be handled by DEA, a simple transformation of values for descending-nature parameters was performed. In this case, the ‘transformed’ values of parameter were computed by subtracting the actual parameter value from the highest corresponding value that was observed at the selected group of printers. This way, the ‘transformed’ parameter obtains its lowest value, i.e., zero, when the actual parameter attains its worst, i.e., its maximum, value. Ascending parameters are retained unchanged. Binary parameters, such as whether a printer is open source or is equipped with a heated platform, were given a value of one, when the answer was negative, and two otherwise. Being open-source was considered advantageous because it allows easier and lower-cost equipment maintenance and modifications. Finally, the parameter associated with material variety obtained values from 1 to 4 according the number of different materials that the printer could reportedly process.

Since the final number of systems, for which the majority of data were available, was quite high a smaller sample should be selected, for further analysis with prices that span the whole price range. The size of the final sample was determined by taking into account the number of outputs and inputs and the corresponding guidelines given in the studies of Bowlin [58] and Dyson et al. [59]. A random selection was performed after dividing the existing sample into six groups. Each group had a price range of a thousand of dollars (e.g., the first group from USD 0 to 999, the second from USD 1000 to 1999, etc.). From each group the 14% was randomly selected. For this purpose Minitab 17 was used. The final set of 30 3D printers and their technical specifications/parameters and prices are presented in

Table 2. Technical specifications/parameters (outputs) and price (input) of the 3D printers sample.

Printer #	Build Volume (mm3)	Num. of Extruders	Layer Thickness (μm)	Nozzle Diameter (mm)	Heated Plate	X-Y Resolution (μm)	No of Materials	Nominal Power (W)	Printing Speed (mm/s)	Open Source	Weight (kg)	Price (USD)
1	1459	1	50	0.35	No	15	4	20	60	No	1.00	315.00
2	2627	1	150	0.40	Yes	11	4	220	30	No	5.00	499.99
3	8365	1	100	0.40	Yes	1	2	150	10	No	8.00	599.00
4	4900	2	100	0.40	Yes	11	2	200	40	No	12.00	675.00
5	5186	2	100	0.40	Yes	11	2	300	80	No	25.00	699.00
6	8000	2	100	0.40	Yes	100	2	154	40	Yes	15.00	799.00
7	2458	1	20	0.40	No	12.5	4	175	250	No	9.30	799.99
8	10,450	2	100	0.40	No	50	4	150	120	Yes	18.00	999.00
9	9987	1	100	0.40	Yes	4	2	200	200	No	14.00	1099.00
10	6084	1	250	0.40	No	20	2	150	30	Yes	8.00	1190.00
11	3658	1	50	0.50	Yes	100	4	150	275	Yes	8.55	1250.00
12	9987	1	100	0.40	Yes	4	4	200	200	Yes	14.00	1299.00
13	9052	1	100	0.40	Yes	11	2	240	300	Yes	17.00	1099.00
14	7611	1	50	0.35	No	50	1	70	120	No	10.00	1899.00
15	7800	1	100	0.40	Yes	20	4	500	100	Yes	12.00	2537.00
16	6338	2	100	0.40	Yes	11	1	150	40	No	12.60	3492.00
17	5640	1	100	0.35	Yes	50	4	380	80	Yes	20.00	3495.00
18	36,995	2	50	0.35	Yes	17	4	920	300	No	37.00	4175.26
19	7196	1	100	0.40	Yes	12	4	240	100	No	10.00	500.00
20	4894	2	100	0.40	Yes	11	2	300	100	Yes	15.00	977.00
21	5769	1	100	0.40	No	100	1	120	50	Yes	6.70	669.11
22	12,474	1	50	0.40	No	150	1	150	200	Yes	30.00	1975.00
23	4449	1	100	0.40	No	11	2	350	180	No	10.00	1267.50
24	10,467	1	50	0.35	Yes	10	4	360	200	Yes	13.00	1775.00
25	14,934	1	50	0.35	Yes	12.5	4	400	150	Yes	12.50	1931.85
26	10,680	1	250	0.50	Yes	40	4	360	150	Yes	11.00	2295.00
27	10,609	1	20	0.40	Yes	12	3	221	300	Yes	11.00	2500.00
28	39,000	2	20	0.40	Yes	12.7	4	300	120	Yes	30.00	2650.00
29	28,373	1	10	0.40	Yes	12.5	4	600	150	Yes	40.00	2829.00
30	40,469	1	50	0.40	Yes	21	4	600	250	No	38.50	4499.00

.

For DEA according to the CCR and BCC models, an in-house DEA software was used, the code of which has been developed by the authors and runs in a Microsoft Excel environment. For context-dependent DEA, DEA Excel Solver was used as proposed by Zhu [60].

4. Results and Discussion

Before proceeding to the actual evaluation of the 3D printers with DEA, a correlation analysis of the 11 output parameters was performed in order to identify the minimum number of parameters that could be efficiently employed in DEA, i.e., investigate whether some of the initial selected parameters could be ignored without affecting the evaluation results. A reduction in the number of parameters is desirable because it lowers the associated computational burden and generally reduces the time required for the analysis.

The results of the correlation analysis of output parameters, performed with Minitab 17, are presented in

Table 3. Correlation coefficient (in bold) and p-value (in italics) for 3D printer output parameters.

	Build Volume	Num. of Extruders	Layer Thickness	Nozzle Diameter	Heated Plate	X-Y Resolution	Number of Materials	Nominal Power	Printing Speed	Open Source
Number of Extruders	0.190
	0.315
Layer Thickness	0.364	0.039
	0.048	0.836
Nozzle Diameter	0.123	0.052	0.380
	0.518	0.786	0.038
Heated Plate	0.281	0.193	0.029	−0.163
	0.132	0.306	0.878	0.391
X-Y Resolution	0.110	0.036	−0.032	0.256	0.355
	0.563	0.849	0.866	0.173	0.054
Num. of Materials	0.324	−0.176	0.163	−0.039	0.305	0.298
	0.081	0.351	0.390	0.837	0.101	0.110
Nominal Power	−0.698	−0.104	−0.167	−0.154	−0.428	−0.262	−0.442
	0.000	0.583	0.377	0.416	0.018	0.161	0.015
Printing Speed	0.366	−0.245	0.433	−0.123	0.100	−0.005	0.283	−0.384
	0.047	0.193	0.017	0.517	0.601	0.981	0.130	0.036
Open Source	0.038	−0.081	−0.080	−0.211	0.081	−0.379	0.033	0.013	−0.015
	0.841	0.670	0.673	0.263	0.670	0.039	0.864	0.946	0.939
Weight	−0.833	−0.292	−0.374	−0.113	−0.256	0.063	−0.154	0.694	−0.355	−0.048
	0.000	0.117	0.042	0.551	0.172	0.742	0.418	0.000	0.055	0.800

. It can be observed that no strong correlation between any two of the output parameters exists. Nevertheless, there are pairs of parameters which seem to be somewhat correlated, as indicated by the fact that the corresponding p-values are lower than 0.05, but weakly, as indicated by the corresponding correlation coefficient values.

Among the initial set of output parameters, four seem to be more often correlated with others: build volume, nominal power, X-Y resolution, and speed. In order to check whether some of these parameters could be omitted from the analysis, a preliminary DEA evaluation of the printers’ set was conducted. This evaluation involved two sets of computations, the first taking into account all eleven outputs while the second only seven of them, excluding the four correlated parameters. The sole input parameter in both evaluations was the printers’ price and both CCR and BCC, input-oriented DEA was performed. The corresponding preliminary evaluation results are presented in

Table 4. Results of CCR and BCC DEA evaluation of printers, considering all eleven output parameters and only seven of them.

Printer #	CCR		BCC
	11 Outputs	7 Outputs	11 Outputs	7 Outputs
1	1.0000	1.0000	1.0000	1.0000
2	1.0000	1.0000	1.0000	1.0000
3	1.0000	1.0000	1.0000	1.0000
4	1.0000	0.9333	1.0000	1.0000
5	0.9818	0.9013	1.0000	0.9657
6	1.0000	0.7885	1.0000	1.0000
7	1.0000	0.4528	1.0000	1.0000
8	0.8146	0.6306	1.0000	1.0000
9	0.7818	0.4550	1.0000	0.4550
10	0.5081	0.5034	1.0000	0.5034
11	0.8208	0.4875	1.0000	1.0000
12	0.6614	0.3849	1.0000	0.3849
13	1.0000	0.5450	1.0000	0.5450
14	0.3276	0.1659	1.0000	0.1659
15	0.2832	0.2402	1.0000	0.6307
16	0.2015	0.1804	1.0000	0.1933
17	0.1842	0.1743	0.5079	0.5079
18	0.7055	0.1509	1.0000	1.0000
19	1.0000	1.0000	1.0000	1.0000
20	0.7088	0.6448	1.0000	0.8178
21	0.9121	0.8965	1.0000	1.0000
22	0.4888	0.3064	1.0000	0.8101
23	0.5199	0.2485	0.5478	0.2485
24	0.5189	0.3433	1.0000	1.0000
25	0.4412	0.3154	1.0000	1.0000
26	0.3704	0.2655	1.0000	0.5447
27	0.4629	0.2436	1.0000	1.0000
28	1.0000	0.2377	1.0000	1.0000
29	0.6848	0.2157	1.0000	1.0000
30	0.6129	0.1169	1.0000	0.3556

.

Examining the values in Table 4 is quite obvious that when taking into account all output parameters the BCC model leads to a very high number of efficient systems (28 out of 30), i.e., almost all printers are evaluated as efficient. The CCR model, on the other hand, provided a broader distribution of efficiency scores that permits a more useful assessment of printer efficiency. This was, to some extent, expected since the BCC frontier is more flexible and envelops the data in a tighter manner. Another observation is that omitting the four output parameters (build volume, nominal power, X-Y resolution, and speed) leads to a reduction of efficiency scores for most printers in both models. This reduction affects also the number of efficient units identified for both models; something that is exhibited more vividly in the case of BCC DEA where efficient units are reduced from 28 to 16. A possible explanation for this deterioration of scores for some of the efficient printers is that these systems have relatively high values for the omitted output parameters.

Since the exclusion of all four parameters led to significant changes in the initially obtained scores, it has been decided to further explore whether some of the four output parameters under investigation could be excluded from further analysis. This investigation entailed the examination of different combinations of outputs, which included the seven output parameters that were definitely uncorrelated according to previous analysis (see Table 3), plus one or two of the parameters that exhibited weak correlations. The investigated combinations of outputs are presented analytically in

Table 5. Further examined combinations of output parameters (√: evaluated, ✕: excluded).

Parameters	Combinations
	A	B	C	D	AB	AC	AD	BC	BD	CD
Extruder Head	√	√	√	√	√	√	√	√	√	√
Layer Thickness	√	√	√	√	√	√	√	√	√	√
Nozzle Diameter	√	√	√	√	√	√	√	√	√	√
Heated Plate	√	√	√	√	√	√	√	√	√	√
Number of Materials	√	√	√	√	√	√	√	√	√	√
Open Source	√	√	√	√	√	√	√	√	√	√
Weight	√	√	√	√	√	√	√	√	√	√
Building Volume (A)	√	✕	✕	✕	√	√	√	✕	✕	✕
X-Y Resolution (B)	✕	√	✕	✕	√	✕	✕	√	√	✕
Nominal Power (C)	✕	✕	√	✕	✕	√	✕	√	✕	√
Printing Speed (D)	✕	✕	✕	√	✕	✕	√	✕	√	√

. It should be noted that triple combinations of the four outputs under investigation were not considered simply because it does not make a lot of sense, computation-wise, to ignore just a single output/specification.

The DEA evaluation results, employing CCR and BCC models for the 10 different combinations of output parameters (shown in Table 5) are presented in

Table 6. Evaluated efficiency of 3D printers, according to the CCR (a) and the BCC (b) model, for the output combinations of Table 5.

(a)
Printer #	A	B	C	D	AB	AC	AD	BC	BD	CD
1	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
2	1.0000	1.0000	1.0000	0.6487	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
3	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
4	1.0000	0.9333	0.9333	0.9333	1.0000	1.0000	1.0000	0.9333	0.9333	0.9333
5	0.9657	0.9013	0.9013	0.9013	0.9657	0.9657	0.9818	0.9013	0.9013	0.9013
6	1.0000	0.7885	0.7885	0.7885	1.0000	1.0000	1.0000	0.7885	0.7885	0.7885
7	0.5075	0.4528	0.4528	1.0000	0.5075	0.5075	1.0000	0.4528	1.0000	1.0000
8	0.8001	0.6306	0.6306	0.6306	0.8001	0.8001	0.8146	0.6306	0.6306	0.6306
9	0.6272	0.4619	0.4588	0.6218	0.6279	0.6280	0.7818	0.4619	0.7052	0.7052
10	0.5034	0.5034	0.5034	0.5081	0.5034	0.5034	0.5081	0.5034	0.5081	0.5081
11	0.4875	0.4875	0.4875	0.8205	0.4875	0.4875	0.8208	0.4875	0.8205	0.8205
12	0.5306	0.3908	0.3882	0.5261	0.5312	0.5313	0.6614	0.3908	0.5966	0.5966
13	0.6062	0.5450	0.5450	0.9913	0.6062	0.6062	1.0000	0.5450	0.9913	0.9913
14	0.3276	0.1659	0.1659	0.2465	0.3276	0.3276	0.3276	0.1659	0.2546	0.2546
15	0.2603	0.2402	0.2402	0.2461	0.2603	0.2603	0.2832	0.2402	0.2461	0.2461
16	0.2005	0.1804	0.1804	0.1804	0.2014	0.2007	0.2005	0.1804	0.1804	0.1804
17	0.1743	0.1743	0.1743	0.1770	0.1743	0.1743	0.1842	0.1743	0.1770	0.1770
18	0.7012	0.1509	0.1509	0.2609	0.7012	0.7012	0.7055	0.1509	0.2609	0.2609
19	1.0000	1.0000	1.0000	0.8402	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
20	0.7059	0.6448	0.6448	0.6448	0.7059	0.7059	0.7088	0.6448	0.6448	0.6448
21	0.8965	0.8965	0.8966	0.9121	0.8965	0.8966	0.9121	0.8966	0.9121	0.9121
22	0.4452	0.3064	0.3064	0.4224	0.4452	0.4452	0.4888	0.3064	0.4224	0.4224
23	0.3251	0.2559	0.2485	0.4925	0.3323	0.3251	0.5187	0.2559	0.4902	0.4902
24	0.4220	0.3433	0.3433	0.4700	0.4220	0.4220	0.5189	0.3433	0.4700	0.4700
25	0.4048	0.3154	0.3154	0.3658	0.4048	0.4048	0.4412	0.3154	0.3658	0.3658
26	0.3342	0.2655	0.2655	0.3079	0.3342	0.3342	0.3704	0.2655	0.3079	0.3079
27	0.3125	0.2436	0.2436	0.4358	0.3125	0.3125	0.4629	0.2436	0.4358	0.4358
28	1.0000	0.2377	0.2377	0.2377	1.0000	1.0000	1.0000	0.2377	0.2377	0.2377
29	0.6848	0.2157	0.2157	0.2498	0.6848	0.6848	0.6848	0.2157	0.2498	0.2498
30	0.6091	0.1169	0.1169	0.1778	0.6091	0.6091	0.6129	0.1169	0.2028	0.2028
(b)
Printer #	A	B	C	D	AB	AC	AD	BC	BD	CD
1	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
2	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
3	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
4	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
5	0.9657	0.9657	0.9657	1.0000	0.9657	0.9657	1.0000	0.9657	1.0000	1.0000
6	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
7	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
8	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
9	0.6272	0.5205	0.4950	1.0000	0.6571	0.6474	0.9483	0.5205	1.0000	1.0000
10	0.5034	0.5034	0.5034	0.5404	0.5034	0.5034	1.0000	0.5034	1.0000	1.0000
11	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
12	0.5306	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
13	0.6096	0.5450	0.5450	1.0000	0.6096	0.6096	1.0000	0.5450	1.0000	1.0000
14	0.4945	0.1659	0.1659	0.2465	0.4945	1.0000	0.5754	0.1659	1.0000	1.0000
15	0.6307	1.0000	0.6307	1.0000	1.0000	0.6307	0.6307	1.0000	0.6307	0.6307
16	0.2009	0.1933	1.0000	1.0000	1.0000	1.0000	0.2009	1.0000	1.0000	1.0000
17	0.5079	0.5079	0.5079	0.2858	0.5079	0.5079	0.5079	0.5079	0.5079	0.5079
18	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
19	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
20	0.8178	1.0000	0.8178	1.0000	1.0000	0.8178	1.0000	1.0000	1.0000	1.0000
21	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
22	0.8436	0.8101	1.0000	0.6329	0.8436	1.0000	0.8514	1.0000	1.0000	1.0000
23	0.3284	0.3125	0.2485	0.5438	0.3415	0.3284	0.5187	0.3125	0.4902	0.4902
24	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
25	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
26	1.0000	1.0000	0.5447	1.0000	1.0000	1.0000	1.0000	1.0000	0.5447	0.5447
27	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
28	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
29	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
30	0.7013	0.3803	0.3556	0.1778	0.7013	0.7013	0.8095	0.3803	0.4890	0.4890

a and Table 6b respectively. Comparing the results for the CCR model presented in Table 6a to that of the analysis performed with all 11 outputs (Table 4) indicated that out of the 10 different output combinations the one that gives efficiency scores closest to those calculated using the complete set of output parameters, is combination (AD), i.e., employing the seven ‘uncorrelated’ outputs and the parameters of building volume and printing speed. A similar examination of efficiency scores for the BCC model (compare the corresponding column of Table 4 with the figures in Table 6b) shows that the ‘closest to the complete set of outputs’ combination is (CD), i.e., adding to the analysis the parameters of nominal power and printing speed. It is interesting to note, however, that in the BCC model almost all the 3D printers achieve maximum efficiency, same as observed in the preliminary analysis. Obviously, such a result has limited practical value as a filtering tool for assessing the performance of a 3D printer. The CCR method seems, therefore, more suitable for the problem at hand. Furthermore, as suggested in previous studies [61], the CCR model seems the most suitable method for evaluating sets of relatively homogeneous units, such as those investigated in the current study.

To further assess the suitability of the CCR method, statistical testing has been performed on the specific printer sample, as described in [45]. The test was implemented in R program language and employed the bootstrapping method for testing the following hypotheses:

Hypothesis 0 (H0).

3D Printers excibit constant returns to scale (CCR).

Hypothesis 1 (H1).

3D Printers excibit variable returns to scale (BCC).

As indicated by the test results, H0 can be accepted at a 5% confidence level; hence the CCR model has been employed for further evaluation.

Having excluded the BCC model from further analysis and reduced the number of outputs to nine (building volume, number of extruders, layer thickness, nozzle diameter, presence of heated bed, number of materials, printing speed, open-source character, and weight) the analysis proceeded to the evaluation of an extended sample of 3D printers. The extended sample included eight additional 3D printers, numbered as printers 31–38 (see

Table 7. Technical specifications/parameters and price of the additional top-rated 3D printers.

Printer #	Build Volume (mm3)	Num. of Extruders	Layer Thickness (μm)	Heated Plate	Num. of Materials	Printing Speed (mm/s)	Open Source	Weight (kg)	Price (USD)
31	1701	1	100	Yes	4	55	No	4.50	215.61
32	7342	1	100	Yes	4	100	No	10.00	317.64
33	7200	1	100	Yes	4	70	Yes	10.00	399.00
34	4905	2	100	Yes	4	100	No	11.00	899.00
35	3366	1	100	No	1	100	No	11.50	161.95
36	19,503	1	50	Yes	4	200	Yes	14.97	2500.00
37	4786	1	100	No	1	150	No	8.84	829.99
38	1875	1	100	No	1	150	No	5.00	249.00

), which were highly rated and received favorable reviews from experts at specialized web sites (all3dp.com, www.aniwaa.com, 3dforged.com; all web sites accessed on various dates between July 2018–October 2019). The actual number of highly rated printers in the sample was nine, since printer 24, which was randomly included in the initial sample, received favorable reviews, also.

The DEA evaluation results for the extended sample, according to the CCR model, are presented in the second column of

Table 8. Efficiency scores for the extended sample of 38 printers, for different combinations of output parameters.

Printer #	Efficiency Scores (All Nine Outputs Considered)	Nozzle Diameter Excluded	Open Source Excluded	Nozzle Diameter and Open Source Excluded
1	0.9131	0.8369	0.9131	0.8369
2	0.4675	0.4675	0.4675	0.4675
3	0.6257	0.6257	0.5892	0.5892
4	0.4799	0.4799	0.4799	0.4799
5	0.4634	0.4634	0.4634	0.4634
6	0.5378	0.5378	0.5378	0.5378
7	0.5749	0.5749	0.5749	0.5749
8	0.4203	0.4203	0.4203	0.4203
9	0.4151	0.4151	0.4151	0.4151
10	0.2722	0.2722	0.2306	0.2306
11	0.3930	0.3930	0.3930	0.3930
12	0.3512	0.3512	0.3512	0.3512
13	0.4421	0.4421	0.4421	0.4421
14	0.1906	0.1884	0.1906	0.1884
15	0.1576	0.1576	0.1356	0.1356
16	0.0928	0.0928	0.0928	0.0928
17	0.1029	0.1029	0.0926	0.0825
18	0.4545	0.4545	0.4545	0.4545
19	0.6353	0.6353	0.6353	0.6353
20	0.3315	0.3315	0.3315	0.3315
21	0.4841	0.4841	0.3923	0.3923
22	0.2820	0.2820	0.2820	0.2820
23	0.2358	0.2358	0.2358	0.2358
24	0.2570	0.2570	0.2570	0.2570
25	0.2527	0.2527	0.2463	0.2463
26	0.2108	0.2108	0.2055	0.2055
27	0.2022	0.2022	0.2022	0.2022
28	0.6493	0.6493	0.6493	0.6493
29	0.4425	0.4425	0.4425	0.4425
30	0.3955	0.3955	0.3955	0.3955
31	1.0000	1.0000	1.0000	1.0000
32	1.0000	1.0000	1.0000	1.0000
33	0.9738	0.9738	0.7961	0.7961
34	0.4001	0.4001	0.4001	0.4001
35	1.0000	1.0000	1.0000	1.0000
36	0.3459	0.3459	0.3459	0.3459
37	0.2927	0.2927	0.2927	0.2927
38	0.9756	0.9756	0.9756	0.9756

. The results of the analysis show a wide distribution of efficiency scores among printers and a rather clear distinction between efficient and less efficient units. In an effort to further reduce the number of output parameters to simplify the data collection and analysis process, the influence of nozzle diameter and open-source character on the evaluation results was examined. The evaluated efficiencies, excluding both and each of the above parameters are presented also in Table 8.

Comparing the results considering all nine parameters (second column) to those obtained without considering nozzle diameter (third column), hardly any difference is observed. This is definitely not the case for the efficiency ratings evaluated when the open-source character (fourth column) or both parameters are ignored. Interestingly enough, the efficiency scores in the last two columns of Table 8 are almost identical. These two observations indicate that ignoring the nozzle diameter does not notably alter the overall 3D printer evaluation results and implies that the specific parameter could be also omitted. Such a behavior could be probably attributed to the fact that there is only a very limited variation of this particular output/specification among the 38 3D printers examined (the great majority of printers is equipped with nozzles of 0.40 mm diameter).

Examining the efficiency scores for the nine top-rated printers in the sample (printers 31–38 and printer 24), we observe that only five of them were evaluated as highly efficient. These are printers 35, 32, and 31, which display maximum efficiency, as well as printers 33 and 38, which achieve efficiency scores close to one. The remaining four top-rated systems (printers 24, 34, 36, and 37), on the other hand, failed to achieve high efficiency scores in the DEA evaluation. It should also be noted that all other printers, which were not considered as top choices from experts, did not obtain high efficiency scores, with the sole exception of printer 1 who was evaluated relatively favorably.

Some interesting observations can also be extracted by examining the relationship between efficiency and the price. Further investigation of this relationship is of interest because price is the sole input parameter in efficiency evaluation. As is clearly obvious in Figure 1, where the plot of a printer’s price against its efficiency score is presented, relatively expensive 3D printers (price > USD 2000) are not evaluated as highly efficient. On the contrary, all printers that achieved high efficiency scores were relatively low priced; the three printers that achieved maximum efficiency were, in fact, among the cheapest ones. Accordingly, the low efficiency scores of the four underperforming top-rated printers (printers 24, 34, 36, and 37) seems to be due to their relatively high price.

Focusing on the correlation between price and efficiency we observe a significant negative correlation, with −0.621 correlation coefficient and 0.000 p-value. It is interesting to note that the specific correlation coefficient value is quite close to the corresponding value computed by Doyle and Green [18] for dot matrix printers (−0.45). Above results could, also, be interpreted as an indication of higher economies of scale for manufacturers of relatively cheap printers, who can presumably withstand smaller profit margins due to higher numbers of units sold. This competitive advantage could explain the corresponding higher efficiency scores [30]. Possible correlations between efficiency scores and output parameters have also been examined, but no significant correlation values were observed.

Another interesting observation from the analysis of DEA results is that among the three efficient printers, printer 35 appears to be the efficient peer, or equivalently serves as the reference, for 31 ‘inefficient’ printers, whilst printers 32 and 31 are the efficient peers for 22 and 11 ‘inefficient’ printers respectively. A possible reading of this could be that printer 31 could be targeted at a specific market segment (niche) while printer 35 could have a broader market appeal. It should be noted that this characterization refers to the number of market competitors rather to the number of potential buyers.

Context-Dependent DEA

To further investigate the applicability of the DEA approach in the problem of assessing the efficiency of 3D printers, context-dependent DEA was applied. As described earlier, context-dependent DEA classifies DMUs in successive efficient frontiers/groups and assigns at each DMU attractiveness and progress values that can help in further assessing the technical efficiency of a DMU and how this can be improved.

The context-dependent evaluation of the extended sample of 3D printers resulted in the formation of eight efficient frontiers/groups, which are presented in the first column of

Table 9. Attractiveness context levels.

Group	Printer #	Attractiveness Context Levels
		Level 2	Level 3	Level 4	Level 5	Level 6	Level 7	Level 8
E1	35	1.7984	3.0874	5.1250	7.8265	11.6538	15.9097	30.7525
	32	2.5123	4.2058	6.1516	8.8740	12.1638	15.2122	22.0060
	31	3.7011	5.8362	9.0627	13.0734	17.9199	22.4108	32.4196
E2	33		1.8363	2.8822	4.3037	5.2254	6.7551	13.1391
	1		2.1164	3.7079	5.6020	8.3684	11.1573	21.6357
	38		2.6041	3.7441	5.8371	9.0426	13.2610	26.3178
E3	5			1.2861	3.5157	5.0768	6.9127	10.0000
	4			1.3319	3.6407	5.2520	7.1585	10.3556
	13			1.3809	1.9895	2.2748	5.0928	11.9256
	28			1.4568	1.7369	2.3537	3.6606	9.1510
	6			1.5284	3.0757	4.5255	6.0476	8.7484
	21			1.6872	2.5336	3.3849	4.4185	8.6969
	7			1.7142	2.4478	3.1156	6.7114	13.6525
	3			1.9903	2.8667	3.6565	4.7639	9.3356
	19			2.1007	2.8554	4.1699	5.3906	10.4850
	2			2.1018	3.1101	4.7823	5.9680	12.2327
E4	9				1.1820	1.9301	3.7006	7.9504
	18				1.2037	1.6405	2.4886	6.4054
	29				1.2160	1.6316	2.6420	6.2362
	37				1.5829	2.4868	3.9784	7.8954
	11				1.6669	2.1061	2.1439	9.6113
	8				2.4600	3.6550	4.8368	6.9970
	20				2.5154	3.7010	4.9458	7.1546
	34				2.7336	4.0615	5.3749	7.7753
E5	24					1.2463	1.7239	4.9225
	36					1.3108	2.2711	4.8749
	30					1.4314	2.1813	5.5737
	23					1.6738	2.9673	6.2041
	12					1.7090	3.7198	6.7263
	10					1.8405	2.1281	4.6992
E6	16						1.3837	2.0017
	14						1.7813	2.7607
	27						2.4536	5.2425
	22						2.2292	4.4241
	25						1.8830	3.4711
E7	15							1.9287
	26							2.8936
E8	17

, where the corresponding printers’ attractiveness scores for the successive levels of analysis are also included. It can be observed that as the evaluation context level increases so does a printer’s attractiveness score. It should also be noted that the relative ranking of printers according to the attractiveness score in each group, does not necessarily remains the same through all eight levels of analysis. Printer 32, for example, at the first group of printers E¹, has higher attractiveness from printer 35 at the first six levels of analysis level, but this ranking is reversed at the last two levels. The relative ranking of printers in the second group E², on the other hand, remains the same at all levels. It is, therefore, obvious that the evaluation context affects the ranking, as has been observed and in previous studies [62]. An examination of the attractiveness score of top-rated printers shows that the five of them that achieved high efficiency in previous analysis are also found in the first two efficient frontiers/groups, as expected. Among them, printer 31 is the most attractive choice at all levels of analysis. The remaining four top-rated printers are accordingly placed in the fourth (printers 37 and 34) and the fifth efficient frontiers (printers 24 and 36).

In

Table 10. Progress context levels.

Group	Printer #	Progress Context Levels
		Level 7	Level 6	Level 5	Level 4	Level 3	Level 2	Level 1
E1	35
	32
	31
E2	33							1.0269
	1							1.1949
	38							1.0250
E3	5						1.3183	2.1581
	4						1.2731	2.0840
	13						1.4933	2.2620
	28						1.2261	1.5402
	6						1.2185	1.8593
	21						1.6433	2.0658
	7						1.4424	1.7395
	3						1.3511	1.5983
	19						1.1558	1.5741
	2						1.2026	2.1392
E4	9					1.2792	1.6468	2.4091
	18					1.4174	1.7517	2.2004
	29					1.4604	1.7993	2.2601
	37					1.3620	2.1778	2.3076
	11					1.1546	2.1326	3.4167
	8					1.1106	1.5546	2.3790
	20					1.1174	1.8426	3.0164
	34					1.1418	1.5936	2.4995
E5	24				1.3391	1.8663	2.5791	3.7353
	36				1.2207	1.8503	2.2798	2.8912
	30				1.1452	1.6315	2.0131	2.5288
	23				1.3555	1.9205	3.1735	4.2413
	12				1.0102	1.5119	1.9465	2.8475
	10				1.0718	1.8755	2.9459	3.6740
E6	16			1.4210	3.6467	4.9878	6.2848	10.7811
	14			1.1162	1.6215	2.8485	3.8819	5.3071
	27			1.1434	1.5890	2.1604	3.1069	4.9452
	22			1.0190	1.4227	2.0253	2.5387	3.5465
	25			1.0575	1.4547	2.2216	2.9843	3.9578
E7	15		1.2928	1.4774	2.1756	2.5370	5.5289	6.3469
	26		1.1341	1.3831	1.7573	1.1131	3.5710	4.7431
E8	17	1.3776	1.8091	2.0353	3.0391	3.4950	8.5785	9.7171

the printers’ progress values are presented. An examination of progress and attractiveness values of printers in group E² reveals that printer 38 is the most efficient unit of the group. It should be noted that the specific printer obtained the second-best score at the initial input-oriented DEA, as can also be seen in Figure 2. As noted, however, in the study of [63], this is not necessarily the case. Printer 17, for example, is the sole member of the last context level, which could very well mean that it is the worst choice, despite the fact that it did not attain the worst efficiency score, printer 16 did. An examination of progress scores for both printers also shows that at some context levels (levels 4, 5, and 7) printer 16 requires more effort to improve than printer 17. Finally, it should be noted that the efficiency scores for printers belonging in the E² group are significantly lower from the corresponding scores for the rest of the printers (see Figure 2). This assessment is validated by the fact that E² printers exhibit the lowest progress scores at the first context level.

5. Conclusions

In the present study, DEA is proposed as a method of evaluating the technical efficiency of 3D printers. In order to identify the most suitable DEA approach for the specific task, both CCR and BCC input-oriented models of DEA are employed for assessing the efficiency of a sample of 38 3D printers, which are based on FDM/FFF technology and priced at less than USD 5000. The specific sample comprised mostly of systems randomly selected from a wide list of currently available systems, plus a small number of systems that received top ratings and reviews from 3D printing experts. The initial DEA evaluation model involved the assessment of price as the sole input parameter and of eleven 3D printer characteristics/specifications as output parameters. Employing correlation analysis and successive DEA evaluations for different combinations of outputs, their number was reduced to eight, namely, building volume, number of extruders, layer thickness, the presence of a heated bed, number of materials, printing speed, open-source character, and weight. According to the results of the analysis, CCR DEA seems to be the most appropriate model for the specific problem.

Employing CCR DEA, only three out of nine top-rated 3D printers achieved maximum efficiency, while another two exhibited efficiency scores greater than 0.9. The remaining four top-rated 3D printers were evaluated with efficiency scores lower than 0.5. To further investigate the efficiency, context-dependent DEA has been applied. According to the results of context-dependent DEA, five of the top-rated printers were assigned in the top-two efficient frontiers—a result that confirmed previous evaluations and rankings—while four were evaluated as relatively attractive in lower context levels. A possible interpretation of the result is that printers in the top-two levels are good all-around alternatives that combine good technical features with low price, while the remaining four are more attractive at specific contexts, where several other factors such as brand name, user base, and customer support are also considered. An examination of the price-efficiency relationship for the specific sample showed that there is a significant reciprocal relationship between the two values.

Regarding the applicability of the DEA method for the problem at hand, it could be argued that the results of the study show that DEA can be a useful approach for a preliminary assessment of the technical efficiency of 3D printers, based on data which are readily and easily available from the world-wide-web and without the need for subjective judgments regarding their values or relative importance. Relying on online sources of data may be considered a weakness of the proposed approach, but we believe that data reliability can be assured with a careful selection of the employed online sources. Considering the relatively large number of 3D printers offered world-wide and the rapidly changing landscape of the corresponding market, DEA may be employed for identifying a relatively small set of efficient systems out of an initial high number of possible alternatives, in relatively short time. This way 3D printer manufacturers may evaluate in short time the strengths and weaknesses of their products, compared to those offered by the competition, and adjust their product development and marketing strategies. Potential buyers could, also, consider the identified set of efficient units as a list of candidates for further evaluation where other quite important factors such as production quality and repeatability, system robustness, ease-of-use, and quality of after-sales service are assessed. The evaluation of all these factors requires extensive study and testing, and their importance to the buyer is of a more-or-less subjective nature. In this context, the proposed approach could be enhanced by incorporating a second evaluation step, where other multi-criteria decision-making methods are employed to further assess the performance of efficient printers. Future research could, also, focus on the investigation of other DEA approaches, which address some of the limitations of the classic models. In this context, a comparative study of convex and non-convex DEA approaches, would be particularly of interest.