1. Introduction
In data centers, information-technology equipment (ITE) expends an enormous amount of electricity on information processing. Data centers are thus known as facilities that consume huge amounts of electricity. In fact, it is estimated they account for 1% of global electricity consumption [
1]. Although, almost all the power consumption of the ITE in a data center is converted into sensible heat, the failure rate of the ITE increases as its inlet temperature increases [
2]. From the standpoint of preventing “information-communication stagnation” due to failure of ITE, the ITE is therefore cooled 24 h a day, 365 days a year. As an operational target for cooling, the thermal guidelines [
2] presented by the American Society of Heating, Refrigerating, and Air-Conditioning Engineers (ASHRAE) TC 9.9 are often cited. As for a data center, heating, ventilation, and air-conditioning (HVAC) consumes 30–40% of the energy of the whole facility [
3]. Accordingly, better energy conservation is strongly demanded. Especially in the case of a data center, fans consume a large amount of power because the sensible-heat load of the housed ITE is processed with a large volume of air, which accounts for about 26% of the power consumption of the HVAC [
4]. For that reason, improving the efficiency of air-conditioning airflow has been actively researched.
Through full-scale experiments, four airflow configurations (namely, raised-floor supply/room return; raised-floor supply/ceiling return; ceiling-supply/room return; and ceiling-supply/raised-floor return) were compared and evaluated according to the evaluation index (based on dimensionless room temperature and standard deviation) proposed by Nakao et al. [
5]. Supply-air volume for each perforated panel in an experimental facility equipped with a raised-floor computer-room air-conditioning (CRAC) unit was measured, and phenomena such as decreased supply-air volume and backflow near the CRAC were investigated by Rambo et al. [
6]. However, methods such as full-scale experiments and reduced-model experiments face problems such as the cost of constructing the test environment and the difficulty of changing conditions on a large scale. Accordingly, as recommended in ASHRAE TC 9.9, as a method for studying optimization and analysis of air flow [
4], thermal and air-flow simulations based on computational fluid dynamics (CFD) are being actively used.
CFD is an effective study tool for comparing airflow schemes (which usually require large-scale experiments) and for optimizing parameters that require multiple levels. In a study by Shrivastava et al., CFD was used to compare thermal performances of seven types of airflow schemes that combine raised-floor supply, raised-floor return, ceiling supply, and ceiling return [
7]. In another study, as for the ventilation design of these airflow schemes, height of the ceiling, airflow rate through perforated panels, and position of return vents were studied as parameters [
8]. And a parameter study of ceiling height in multiple airflow schemes was conducted by Sorell et al. [
9]. Airflow distribution from perforated panels was analyzed by CFD by using raised-floor height and aperture ratio of the panels as parameters, and a method to make the airflow distribution uniform was proposed by Patankar et al. [
10,
11]. As a parameter study by CFD aimed at improving the thermal environment, the effect of airflow separation (by, for example, aisle enclosure and blank panels, was verified by Wibron et al. [
12] and Wang et al. [
13]. Moreover, optimum conditions for the outlet angle of the aperture panel were studied by Ni et al. [
14]. In addition, as a method for improving airflow in the maintenance phase of an existing data center, a baffle plate was proposed by Yuan et al., who optimized its size and mounting angle [
15].
In this way, CFD makes it possible to compare airflow schemes and conduct parameter studies on each element of a data center without conducting large-scale experiments, thereby improving efficiency of research. However, the results of CFD analysis change according to whatever element is modeled. Now, we consider combining actual measurements taken at a data center with the results of CFD simulation. If we try to build a CFD model with the minimum number of elements and fit it to measurement data by only setting those elements, the setting may deviate greatly from the theoretical values or actual values.
As for the raised-floor air-supply method, the raised floor is used not only for distributing air flow, but also for providing space for laying power supplies, network cables, air-conditioning pipes, and other infrastructure. VanGilder et al. argued that this housed infrastructure acts as obstacles that may interfere with the airflow, so it is desirable to include these obstacles in airflow analysis of the raised floor [
16]. And Joshi et al. reported that “distributed air leakage”, which occurs in an area equivalent to 0.1–0.5% of the exposed floor area of the fixed panels (due to abrasion of tile seals), corresponds to 10–15% of the air volume passing through the perforated panels [
17]. Karki et al. treated air leakage from fixed panels of raised floors as a research theme [
18]. Joshi et al. also showed that pressure loss in the raised-floor-supply/ceiling-return system is about 250 Pa [
17]. Accordingly, since the pressure difference inside the rooms in a data center is large, it is considered that the gap between structural components (such as the above-mentioned raised-floor fixed panels) affects the CFD simulation result. In addition, accompanying the high-density housing of ITE, airflow separation to suppress the rise of ITE inlet temperature is becoming more important. Typical methods of airflow separation are: (i) cold-aisle/hot-aisle rack arrangement, (ii) aisle containment (which physically separates the cold aisle and the hot aisle), and (iii) panels that prevent backflow of ITE outlet air from places where ITE is not yet mounted on the mounting rails in racks (hereinafter, “blank panels”). On the contrary, by separating the space in this manner, large pressure or temperature differences are generated between the spaces. And such pressure difference promotes air leakage (as mentioned above). Furthermore, in a data center—composed of material with low thermal resistance (such as steel plate or aluminum), it is considered that heat transmission due to temperature differences has a considerable influence.
In light of the above-described background, in the present study, in a simulation room of a data center adopting the raised-floor-supply/transverse return method (using cold-aisle containment (CAC) and blank panels), the elements that are thought to affect the results of CFD analysis were modeled, and the factors that affect prediction accuracy of the analysis were revealed. Presence of obstructions, thermal resistance of structural components and aperture ratio of air gaps were selected as the elements. Moreover, a method for improving accuracy of predicting the thermal environment of a data center by CFD simulation using parameter tuning is proposed. In addition, in the CFD model after tuning, the heat-insulation performance is examined in terms of improving the thermal environment and reducing energy consumption. High heat generation in data centers is a major problem, and air segregation, which is one of the solutions, has reached a fairly high level. However, no papers have yet referred to the heat-insulation performance as a means of further improving the thermal environment. This article suggests that advanced insulation performance contributes to the improving ventilation efficiency, and quantifies the effect of improving heat-insulation performance.
2. Parameter-Tuning Method
The design of experiments (DOE)—developed by R. A. Fisher of the United Kingdom for streamlining agricultural experiments—is a method for assignment and analysis of experiments [
19]. Compared with the conventional one-factor-at-a-time approach, this DOE can obtain sufficient information in a small number of experiments, so it makes it possible to study the effects of multiple factors on a certain process [
20].
The aim of this study is to improve prediction accuracy of CFD simulations by using this DOE. The parameter-tuning method for multiple characteristics proposed by Nishina et al. is used so that the same CFD model can be accurately reproduced even if the air volume of the air-conditioner varies [
21]. As for this multi-character parameter-tuning method, before the two steps that respectively reduce deviation and bring values closer, a step for reducing the difference in deviation patterns with respect to the signal factor for each characteristic is added.
The DOE for parameter tuning when K types of characteristics are given as P
k (k = 1, 2, …, K) is shown in
Table 1. This is a direct-product experiment in which the parameter-tuning factor is the inner factor, and the signal factor (G) is the outer factor. Mixed-level orthogonal array L
18 is used for the inner orthogonal array. Here, x
jk is taken as the experimental measurement of signal factor G
j at the j-th level in regard to characteristic P
k, and c
ijk is taken as the simulation result of characteristic P
k and signal factor G
j in regard to the i-th analysis of the inner orthogonal array. The difference between the simulation results and the experimental measurements is taken as y
ijk, given as
The procedure of parameter tuning is described as follows. When the optimum level differs in each step, the level of the first step is basically given the highest priority, and the level of the second step is given next priority.
1. First Step
The condition regarding the reproducibility of the deviation pattern of the signal factor for each characteristic is set. The factorial effect of the tuning factor is calculated with S
i, calculated by using the following formula, as the analysis characteristic. For each factor, a level with a small average S
i value is selected.
2. Second Step
The analysis characteristic for the variation of signal factors is defined as the sum of squared deviation S
i’, and the condition to minimize the value of S
i’ is given as:
3. Third Step
Mean value for each analysis is obtained, and the factorial effect of the tuning factor on the mean value is calculated. A level with an average value close to zero is selected.
The handling of the interaction between factors when using the orthogonal array is described hereafter. Generally, in an orthogonal array, the main effect of a factor appears as the effect of the column to which the factor is assigned, and the interaction between factors appears in a different column from the column to which the factor is assigned. When another factor is assigned to the column in which the interaction appears, the effect of that factor and this interaction appear to be confounding [
22]. In this study, the interaction is not considered so as to select the optimum level with high reproducibility, even if the interaction has a certain influence. Therefore, in regard to the orthogonal array, mixed-system orthogonal array L
18 (in which interaction does not appear in a specific column; i.e., interactions appear almost uniformly in all columns) is used, and the superiority or inferiority between the levels is evaluated by using the average value of the analysis characteristic for each level. Moreover, prediction accuracy is confirmed at the change of level in each step, and unexpected deterioration of accuracy due to interaction is eliminated.
3. Parameter Tuning
3.1. Overview of Verification Room
The verification room simulating a data center is shown in
Figure 1, and the specifications of the experimental equipment are detailed in
Table 2. The rack layout in the verification room is a cold-aisle/hot-aisle type, and the cold aisle is fitted with cold-aisle containment (CAC) that surrounds with horizontal components that pass over the racks. At the center of the verification room (hot-aisle part), the room, raised floor, and ceiling are divided by temporary partitions (antistatic flameproof films), and the target range of verification room is 11.0 m long and 5.0 m deep. Of the 22 19-inch racks, 20 racks are equipped with one heat generator with a 2-kW heater, and the other two (unused) racks are blocking the door side of the racks. The heat generators (with height of 6U) are installed at heights of 23U to 28U from the rack bottom.
As for the CFD model, the following conditions are set: airflow of the CRAC, air-supply temperature of the CRAC, airflow at the heat generator, and calorific value of the heat generator. They are values measured at the points listed in
Table 3 or calculated values based on the measured values. The air volume of the CRAC was calculated from average air velocity and effective area of the inlet of the CRAC, and the air volume of the heat generator was calculated from the power consumption and the inlet/outlet temperature difference. Inlet-air temperature of the heat generator is a prediction target of the CFD simulation.
3.2. Overview of CFD Model
Commercial software was used for CFD analysis. The simulation set-up is shown in
Table 4. The discretization algorithm used is SIMPLEC (Semi-Implicit Method for Pressure-Linked Equations Consistent) based on the orthogonal finite-volume method [
23]. As the turbulence model, the standard k-ε model, which is commonly chosen in CFD analysis of data centers [
7,
8,
13,
14,
15,
16,
18,
24], was selected. Fundamental governing equations are the continuity equation, the Navier–Stokes equation and the energy equation. The buoyancy is considered in the Navier–Stokes equation using the Boussinesq approximation. This CFD software does not consider the surface roughness of the materials. The walls enclosing the verification room are modeled as adiabatic. Convergence is considered when calculation error in the entire analysis area is 10
−3.5. VanGilder et al. reported that grid size used for CFD analysis significantly affects prediction accuracy in a rough range [
24]. In this study, to eliminate the effect of different grid sizes as much as possible, the same number of partitions (23,400,000 grids) was used for all models.
A model of the verification room is shown in
Figure 2. Aperture ratios are assigned to the raised-floor perforated panels of the cold aisle and rack door, respectively. The raised-floor perforated panel has alternately arranged aperture ratios of 25% and 50%. In the CFD model, the aperture ratio is uniform (37.5%). A ratio of 76% (catalog value) is assigned to aperture of the rack door.
3.3. Signal Factors and Characteristics
To improve the accuracy of the CFD model, the goal of this study is to be able to reproduce the simulation results with the same model even if the airflow supplied by the air conditioner varies. Therefore, for the experiments conducted in the verification room, fan frequency of the CRAC was set to 45 Hz (G
1), 54 Hz (G
2), and 63 Hz (G
3) with air-supply volume of the CRAC as a signal factor. The conditions assigned in the experiments are listed in
Table 5. In addition, the inlet temperature of the heat generator, which is the prediction target, is measured 10 times (namely, 10 racks), as shown in
Table 3. Since 10 temperature measurements are targeted, a multi-characteristic DOE is applied as a combination of 10 characteristics.
3.4. Prameter-Tuning Factors
Factors that are considered to affect the results of CFD analysis are extracted as listed in
Table 6. As for the level indicating the presence or absence of a model, the first level has a model, and the second level has no model. As for the levels related to the numerical values of the model, the first level is a theoretical value or a measured value, the second level is half the value of the first level, and the third level is not considered in the model. However, the second level of thermal resistance (factor C), calculated by thermal conductivity and thickness of the materials, is twice the thickness of the first level. Column H, to which no factor is assigned, is the error column (e).
Each parameter-tuning factor is outlined as follows.
A. Presence or Absence of Obstructions
The raised floor frame, airflow guide, piping and cables, and air-conditioner inlet structure (shown in
Figure 3) are modeled as obstacles. Here, the airflow guide is a steel plate—obliquely fixed immediately below the air-conditioner outlet—for smoothly introducing air-conditioned air into the raised-floor space.
B. Aperture Ratio of CRAH
The unused computer-room air handler (CRAH) adjacent to the CRAC is tuned off. However, a pressure difference between the inlet and outlet ports occurs, so it is possible that the inside of the CRAH becomes a leakage path and affect the CFD results. Static pressure inside the CRAH during rated operation were obtained from the P-Q characteristic curve of the internal fans as well as rated air capacity and rated external static pressure of the CRAH, and resistance coefficient was calculated from internal static pressure of the CRAH and airflow velocity on the inlet surface. An aperture ratio corresponding to the resistance coefficient is assigned to the inlet port of the CRAH.
C. Thermal Resistance
Heat transmission of the raised-floor fixed panel, rack case, heat generating case, CAC, and blank panel are considered. Thermal-resistance value was obtained from thermal conductivity and thickness of the materials, and the heat-transfer coefficient was calculated by using the automatic calculation function of the CFD software.
D. Aperture Ratio of Fixed Panels
Measured values were used for the aperture ratio of raised-floor fixed panels. The air-gap size between panels was about 10 mm per 26 fixed panels (with width of 300 mm).
E. Aperture Ratio of Blank Panels
The aperture ratio is separately assigned to the following blank panels: (i) a blank panel to be installed on the mounting rail in the rack in which ITE is not installed; (ii) a side blank panel that blocks the gap between the rack side and mount rail; and (iii) a lower blank panel that blocks gap at the bottom of the rack and the bottom edge of the blank panels and side blank panels. Measured values are used for each aperture ratio.
F. Presence or Absence of Gap between Racks
The gap between the racks, namely, 6 mm, is considered in the model. In the case that no gap exists, the space between the racks is closed with a rectangular parallelepiped model with a width of 6 mm. For the third level, the dummy-level technique is used, and a standard first level is substituted as the dummy level.
G. Aperture Ratio of CAC
Although no large gaps exist in the CAC itself, a gap of about 2 mm exists in the 80-mm fixing member between the upper part of the rack and the CAC horizontal members. This measured value is used as the aperture ratio of the CAC.
3.5. Implementation of Parameter Tuning
The signal factor and parameter-tuning factor are assigned as shown in
Table 1 and analyzed on the basis of the DOE. From the analysis results shown in
Appendix A, analytical characteristics S
i, S
i’, and
were calculated and averaged at each level of the fitting factor to create the factor-effect diagram shown in
Figure 4. As for analysis characteristic S
i, the level reflecting any of the parameter-tuning factors, including the dummy level of factor F, in the CFD model is smaller, and it can be said that the reproducibility of the results when the air-supply capacity of the CRAC is varied is good. On the other hand, if theoretical values and measured values (first level) of aperture ratio of the CRAH (B), thermal resistance (C), and aperture ratio of the blank panels (E) are taken as conditions for the CFD model, the reproducibility becomes worse than that of the second level.
In the first step of parameter tuning, a level with a small analysis characteristic Si is selected. In the second step, the level at which analysis characteristic Si’ is small is selected, and in the third step, the level at which analysis characteristic is close to zero is selected. Basically, the level selected in the first step is given the highest priority, and the level selected the second step is given second priority. When the level is changed in a lower-order step, the optimum level selected in a higher-order step will be changed. Consequently, the factors that change the level in each step are limited to a single factor. Prediction accuracy is then confirmed after the change. The indexes taken to confirm prediction accuracy are: (i) average deviation () between the simulated heat-generator inlet temperatures of 10 racks and the corresponding measurement results and (ii) residual sum of squares, Se.
The levels to be selected by parameter tuning in each step are listed in
Table 7, and the confirmation results concerning prediction accuracy are shown in
Figure 5. As described above, in the first step, the level at which analysis characteristic S
i was small for each factor was selected. In the second step, parameter-tuning factor D with the largest decrease in analysis characteristic S
i’ was changed from the first level to the second level. Prediction accuracy is confirmed after the second step. Residual sum of squares S
e and average deviation
remained almost unchanged. In the third step, the third level of factor D, at which the analysis characteristic is close to zero, was selected. As shown in the factor-effect diagram, change in prediction accuracy after the third step increased residual sum of squares S
e; however, average deviation
was close to zero, and prediction accuracy was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) and parameter-tuning factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 simulations. With the proposed parameter-tuning method, by performing the analysis assigned to mixed-level orthogonal array L18 in terms of only the number of signal factors and performing all three steps for confirming the tuning result in terms of only the number of signal factors, an optimum combination was derived after 63 trials.
Reproducibility of the actual inlet temperature of a heat generator is confirmed as follows. The measurements and simulation results of heat-generator inlet temperature for each rack when CRAC fan frequency was varied [45 Hz (G
1), 54 Hz (G
2), and 63 Hz (G
3)] are plotted in
Figure 6. Each step of the simulation is considered first. The simulation results of the first step and those of the second step show no significant difference under any CRAC airflow conditions. Next, compared to the first and second steps, the third step reveals that the heat-generator inlet temperature decreased, especially in the racks far from the CRAC, namely, racks 1–5 to 11 and 2–5 to 11. It is considered that the reasons for this result are twofold: (i) there is no bypass air flowing from inside of the raised floor to the hot aisle via the gap in the fixed panels and (ii) sufficient air-conditioning air supply passed through the perforated panel in the raised floor.
Next, the simulation results of the third step after parameter tuning are compared with the measurements. As for the first row of racks, although deviations in temperature of 1 to 1.5 °C and 3 °C exist for rack numbers 1-1 and 1-11, as for the other racks, the experimental measurements could be roughly reproduced. Also, as for the second row, the experimental measurements could be predicted accurately. Rack number 1-11 shows large deviations, which suggest that a factor that not reflected in the CFD model is increasing the heat-generator inlet temperature. Under the assumption that there is a gap in the part where the vertical member of the CAC and the rack number 1-11 contact, the results that reflect this gap in the CFD model for CRAC fan frequency of 63 Hz (G
3) is shown in
Figure 7. As for the result of changing the gap in intervals of 10 mm, when the gap was 30 mm, the simulated heat-generator inlet temperature of rack number 1-11 was close to the measured value. Also, the assumed gap did not adversely affect the prediction results of other inlet temperatures.
As mentioned above, in the existing literature [
17,
18], the influence of the air gap between the raised-floor fixed panels is mentioned. However, prediction accuracy of this model was higher if the gap was not reflected in the model. In this CFD model, the bypass route that passes through the stopped CRAH is modeled, and as a bypass route, it exists in parallel with the gap between the raised-floor fixed panels. Since this opening is larger than the gap between the raised-floor fixed panels, the influence of the presence or absence of the model of the fixed-panel gap may be reduced.
5. Concluding Remarks
In this study, the aim was to improve the prediction accuracy of ITE inlet temperature in CFD simulation of a verification room simulating a raised-floor supply/transverse-return data center utilizing cold-aisle containment and blank panels. In the verification room, obstructions, gaps between components, and heat transmission of components are considered as factors that may affect the results of CFD analysis. In addition, the factors were tuned so that prediction accuracy could be maintained with the same model even if the CRAC supply-air volume was varied. To ensure the parameter tuning is efficient, the “design of experiments” (DOE) method was used. To tune the parameter-tuning factor and its level under the three conditions of CRAC supply-air volume, it is necessary to perform 2916 simulations by brute force; however, the proposed method can derive the optimum level of the tuning factor in 63 simulations. Comparing the average ITE inlet temperature of 10 racks with the experimental measurements and CFD simulation results after parameter tuning showed that the former was within ±0.2 °C under all CRAC airflow conditions. Although the factors and levels adopted depend on the target room, the proposed parameter-tuning method is considered to be versatile in terms of improving the accuracy of CFD simulations.
Moreover, the fact that the CFD model after tuning incorporates heat flow through the structural components was focused on, and improving the thermal environment was investigated. As a result of improving the insulation performance of the raised-floor fixed panels, rack cases, aisle containment, and blank panels by insulating with 10-mm-thick glass wool, average ITE inlet temperature was decreased by about 1 °C. This result was rearranged by air-management metrics, and a method of evaluating the influence of CRAC supply-air volume and thermal-insulation performance on the thermal environment was demonstrated. It was shown that improving the heat-insulation performance of the structural members makes it possible to reduce power consumption of the CRAC fan by 26%, while keeping the recirculation ratio constant. For the target thermal environment in question, it is desirable to calculate reduction of running cost by narrowing the CRAC supply-air volume and initial cost increase due to installing advanced heat insulation on a trial basis and compare them in regard to life-cycle cost.
High-density mounting of ITE is progressing, and air-segregation methods such as aisle containment and blank panels have been implemented; however, in the future, higher ventilation efficiency (cold/hot air segregation) will be required. Reduction of heat transmission by improving thermal-insulation performance is an approach for cold/hot air segregation that differs from airflow segregation because it suppresses heat transfer in the partitioned space. Improving thermal-insulation performance of structural components is likely to be one solution to the problem of high heat generation in data centers.