Improvement in Runtime Speed for Frequency Domain Soil–Structure Interaction Analysis Using a Coarray Parallel Processing Technique

Abstract

In this paper, we propose a new algorithm for introducing Coarray Fortran (CAF) into a program that dynamically analyzes the frequency domain in soil–structure interactions. This was implemented in KIESSI-3D, which is frequency domain soil-structure interaction analysis computer code based on finite and infinite element techniques, and the performance of analysis speed based on the number of images (or coarrays) and cores per image was evaluated. For the performance evaluation, we used examples of full-scale nuclear power plant buildings with a shallow foundation model as well as a deep foundation model. In terms of analysis speed, the new KIESSI-3D improved the analysis speed by an average of 2.78 times for the shallow foundation problem and 2.69 times for the deep foundation problem, compared with the existing KIESSI-3D, because of the new algorithm and CAF. Further, as the number of cores and the internal memory size in the computer systems increased, the efficiency of also parallel processing increased.

1. Introduction

Since large interaction effects between structures and soil appear in heavy structures such as nuclear power plants and LNG storage tanks, the dynamic soil–structure interaction (SSI) is considered without fail in seismic design [1]. According to the ASCE 4, SSI effects can be neglected for a hard rock condition with shear wave velocity is greater than or equal to 2550 m/s (=8500 ft/s) (a rare case in practice). Therefore, the continuously increasing need for seismic designs that consider SSI can be accomplished using the frequency and time domain analysis methods.

The frequency domain analysis approach approximates both structure and soil as linear viscoelastic bodies, performs independent analyses for discrete frequencies, and then converts the frequency response into the time domain to calculate the structure and soil responses [2]. This method has been combined with various other approaches, including the finite element, boundary element, boundary integral [3], and flexible volume methods [4], and has been used effectively in practice since the 1970s. SASSI (System for Analysis of Soil–Structure Interaction) is used worldwide as representative commercial software for the frequency domain analysis [4,5,6,7].

Although analyzing the SSI time domain is generally more expensive than frequency domain analysis, it can be effective for analyzing the nonlinear response history, which cannot be handled accurately using frequency domain analysis. Currently, time domain analysis is generally carried out using finite elements for the nonlinear structures as well as near-field and far-field soil models in which boundary condition elements such as viscous boundaries, perfectly matched discrete layers, and perfectly matched layers are used to simulate the effects of infinite soil domain [8,9,10]. General-purpose finite element analysis software is effective at describing the nonlinear behavior of structures and soil. However, a numerical model for the time domain SSI analysis was verified by comparing time-domain solutions for a small seismic input with those obtained using the corresponding frequency domain technique [1,11]. This is one of the reasons why more efficient frequency domain technologies are being developed.

Moreover, the development of multi-core CPUs and parallel computational programming technologies, such as open multi-processing (OpenMP), message passing interface (MPI), coarray, and the Fortran compiler’s DO CONCURRENT construct, has enabled cost-effective analyses of more complex models with shorter computation times [12,13,14].

Table 1. Comparison of features of the parallel processing methods [15].

Parallel Method/Language	Coarray Fortran (CAF)	DO CONCURRENT	OpenMP	MPI
Fortran standard	Yes	Yes	No	No
Shared memory	Yes	Possibly	Yes	Yes
Distributed memory	Yes	Possibly	No	Yes
Ease of use	Easy	Easy	Easy	Hard
Flexibility	High	Poor	Limited	High
Incremental improvement	Yes	Yes	Yes	No
Performance	High	Poor	Limited	High

shows a comparison of the advantages and disadvantages of these methods [15]. In OpenMP, parallelization is performed using a shared memory method that uses computer resources (core and internal memory) to perform operations simultaneously. MPI is considered as the best parallelization technique in terms of analysis rate improvement and memory management because it has been continuously upgraded since its inception to configure multiple computers simultaneously as a single computer; the computers perform tasks independently of each other and share data through internal communication [13]. However, the method is rather inefficient because a construct different from the standard Fortran is required for communication between the computers, so that the entire source code needs to be modified to enable parallelization. The DO CONCURRENT construct has been used since Fortran 2008 as a simple replacement for the DO loop construct to express loop-level parallelism [16,17], with the intent of enabling more effective automatic parallel execution. However, it was found that the performance is poor [15,18].

The coarray method avoids the complexity of MPI and enables gradual parallelization of the existing source codes. This method has been adopted as a standard for Fortran compilers since Fortran 2008 [16,17]. Coarray Fortran (CAF) has been widely applied in recent years as it facilitates the parallelization of the existing source codes [19,20]. Figure 1 shows an example of the OpenMP and CAF parallel processing methods using 12 cores. Evidently, in the OpenMP construct all the cores and memory are used for iterative tasks such as do loops, whereas in the CAF method the user exploits as many computer resources as desired, dividing them into images that are independently run units, and the program is run independently for each image. Data transmission and reception between the images can be performed using a simple rule [21].

In this study, CAF was applied to the KIESSI-3D program to optimize the source code such thus reduce the analysis runtime. KIESSI-3D is a frequency domain SSI analysis program based on the finite element and dynamic infinite element methods, as illustrated in Figure 2 [22,23,24,25,26]. To verify the effectiveness of the developed parallel processing technique developed, analysis rate performance was evaluated using computer systems of various specifications. As an analysis example for performance evaluation, a full-scale nuclear power plant containment building was used with both shallow and deep foundation models.

2. Coarray Parallel Processing Technique

KIESSI-3D currently uses the OpenMP method and the Intel/MKL PARDISO direct sparse solver (2019 version). It calculates mass and stiffness matrices of the finite elements, and mass and stiffness matrices of the infinite elements and effective seismic load for each frequency to perform the frequency domain analyses, as illustrated in Figure 3. Furthermore, it stores the results for all frequency points shared after carrying out all the analyses on the hard drive. Because the code was parallelized using the OpenMP, the analysis rate varies depending on the number of cores. However, since the analysis rate achieved by the OpenMP is not linearly proportional to the number of cores, it was necessary to introduce a new parallel processing method, and thus CAF was adopted in this study. The source code was optimized to carry out an efficient analysis using this method. In addition, the analysis procedure was established and the program was modified so that the CAF functions could be maximized. A conceptual analysis diagram of our proposed new version of KIESSI-3D incorporating CAF is shown in Figure 4.

The modified KIESSI-3D consists of OpenMP parts running on a single image and two OpenMP+CAF parts running on multiple images, as shown in Figure 4. The OpenMP parts oversee the information required for all single-image analyses and distribute the information to the other images. At this time, each individual image acquires data in parallel for the system equation corresponding to the pre-allocated frequency points, analyzing the data using OpenMP. The proposed technique can be summarized, as follows:

• Step 1. Assign frequency points to be analyzed in each image.
• Step 2. Run all images simultaneously to construct element matrices of infinite elements and effective seismic loads (OpenMP+CAF part).
• Step 3. Construct system matrices by running a single image and save them to the hard drive frequency by frequency.
• Step 4. Run all images simultaneously for the pre-assigned frequency points to solve the matrix equation and save the solution to the hard drive (OpenMP+CAF part). At this time, OpenMP is used to run on each image.
• Step 5. Construct frequency domain responses by running a single image, which is done by collecting and interpolating solutions at the frequency points for further post-processing.

3. Numerical Evaluation

In order to evaluate the applicability and performance of the KIESSI-3D with CAF incorporated, two SSI analysis problems were selected as representative of real nuclear power plant structures supported by a shallow and a deep foundation, as shown in Figure 5. In addition, eight computer systems as shown in

Table 2. Specifications of the computer systems used in this study.

PC Identification	Number of CPUs	Cores/CPU	CPU Model (Clock Speed)	RAM (GB)	Runtime for 45 Frequencies with Ca1nc1 Option (Minutes)
					Shallow Foundation	Deep Foundation
PC-1	1	8	Intel Xeon W-2245 (3.9 GHz)	256	11.7	31.9
PC-2	1	16	Intel i9-9960X (3.1 GHz)	128	12.7	34.6
PC-3	2	6	Intel Xeon X5690 (3.46 GHz)	64	47.6	141.4
PC-4	2	8	Intel Xeon E5-2687w (3.1 GHz)	192	19.2	58.5
PC-5	2	12	Intel Xeon E5-2687w v4 (3.0 GHz)	192	20.1	54.0
PC-6	2	16	Intel Xeon Gold 6142 (2.6 GHz)	384	13.6	37.0
PC-7	2	24	Intel Xeon Gold 6248R (3.0 GHz)	768	12.4	34.4
PC-8	2	32	AMD EPYC 7601 (2.2 GHz)	384	62.2	121.3

were used for the evaluation of various computer environments. Two are single-CPU systems and six are dual-CPU systems. Analysis rates were compared by number of images (or number of coarrays) and cores used in the individual images. In Figure 6 and Figure 7, the number of images and cores per image can be labeled as ca{$n_1$}nc{$n_2$}, where $n_1$ is the number of images, $n_2$ is the number of cores per image, and the total number of cores used for the analysis is $n_1$×$n_2$. For instance, for ca2nc4 the total number of cores used for analysis is 8. It should be noted here that the number of single-frequency solution processes in each coarray is the number of frequency points used in these analyses (NF in Figure 3 and Figure 4) divided by the number of images. Forty-five frequency points were used in the soil–structure interaction analyses (i.e., NF = 45). Thus, the number of single-frequency solution processes in each image is 22 or 23 for ca2nc{$n_2$}, 11 or 12 for ca4nc{$n_2$}, and so on.

The speedup ratios corresponding to the cores used in the soil–structure interaction analysis for a shallow foundation are summarized in Figure 6. As shown in Figure 6a, the result of using only one image (ca1nc{$n_2$}) is the case where only the OpenMP functions were used, without using the CAF functions. In Figure 6b, it can be seen that in the cases where the six dual-CPU systems are used, the performance of the OpenMP operation is not improved; rather, it degrades even when a large number of cores (larger than the number of cores for a single CPU) are used. This performance degradation can be attributed to factors such as the overhead due to communication between the CPUs. In addition, scalability, which can be defined as the ability to retain performance levels when adding additional processors, decreases when the number of cores per image exceeds four, even in sections where the performance increases with the increase in the number of cores. Based on these results, optimal performance can be expected when the number of cores per image does not exceed four.

The numbers of images and cores per image depend on the resources of the computer system (number of cores, RAM capacity) as well as the size of the problem (RAM capacity necessary for one image). As an example, in the case of a 12-core PC, various image–core number combinations (ca2nc6, ca4nc3, ca6nc2, ca12nc1, etc.) are possible. Notably, since the size of the internal memory (RAM) used by each image is the same, if the number of images increases then the amount of RAM used will also increase. That is, the number of images is determined by the size of the problem (the size of RAM used by one image) and total size of the RAM available. As an example, if one image uses 40 GB of RAM, and the total RAM of the computer system is 192 GB, then the maximum number of usable images is four.

When CAF is applied, the extent of performance improvement for the shallow foundation example depends on the number of images and cores per image, as shown in Figure 6. The extent of performance improvement for the deep foundation example was similar to that for the shallow foundation example, as shown in Figure 7.

In general, CAF+OpenMP parallelization performance improves as the number of images increased. However, OpenMP+CAF parallelization performance of converges for Intel CPU systems when the number of cores per image did not exceed four. Moreover, when the optimal image–core number combination for the six dual-CPU computer systems listed in Table 2 was used, the analysis rate for the shallow foundation example improved as the number of cores of the computer system increased, as shown in Figure 8a. In addition, with the implementation of the OpenMP+CAF algorithm as shown in Figure 4, the new KIESSI-3D code analysis rate was between 1.74 and 4.65 times faster than for the existing code, which used only OpenMP. In the case of the deep foundation example, the new KIESSI-3D code analysis rate was between 1.85 and 4.89 times faster than that of the existing code, as shown in Figure 8b.

The performance improvement measured by the speedup ratio (S) in parallel processing programming, as characterized by the parallelization ratio (P) of codes, can be expressed simply by Amdahl’s law as follows [27]:

$$S=\frac{1}{(1-P)+P/N} \mathrm{or} P=\frac{1-1/S}{1-1/N}$$

(1)

where N is the number of CPU cores.

The extent of performance improvement according to the number of cores and parallelization ratio shown in Equation (1) can be illustrated as shown in Figure 9. Even if the same number of cores is used, the efficiency of the operation will decrease if the parallelization ratio is low, indicating that the parallelization ratio is an important indicator in parallel processing programming. In addition, the parallelization ratio P achieved by the OpenMP+CAF algorithm shown in Figure 4 was calculated based on the speedup ratio S shown in Figure 8, and summarized in Figure 10. From this figure, it can be seen that the parallelization ratio increases as the number of cores increases. Specifically, when the number of cores is 64, the maximum parallelization ratio is 0.988, which is close to the maximum value of 1. These results confirm the effectiveness of the new KIESSI-3D source code for multi-core computer systems.

4. Conclusions

In this study, a new algorithm for introducing a coarray Fortran (CAF) parallel processing scheme into an existing computer program parallelized only by OpenMP for dynamic frequency-domain soil–structure interaction analysis was proposed and implemented in KIESSI-3D. The speed-enhancement performance of the algorithm according to the number of images (or coarrays) and the number of cores per image was evaluated. As analysis examples for performance evaluation, a full-scale nuclear power plant containment building with both shallow and deep foundation models was used. The results of the numerical-example analyses showed that the new KIESSI-3D improved the analysis rate by an average of 2.78 times for the shallow foundation problem and 2.69 times for the deep foundation problem compared with the existing KIESSI-3D, thanks to the new algorithm and the CAF incorporation. In addition, as the number of cores and the size of the internal memory of the computer system increased, the parallel processing efficiency increased. Furthermore, when the number of cores was 64, the parallelization ratio of the source code was shown to be 0.977 to 0.988, which is close to the theoretical maximum value of 1. Therefore, it can be confirmed that the new source code is an effective modification for a multi-core computer system.