**1. Introduction**

During the service life, engineering structures are subjected to various inherent deterioration processes of structure such as corrosion, fatigue, material creep, and so on. As a result, the deformation of the degraded structure will deviate from the original one. On this end, the structural deformation in eventually used as an important index in the structural health monitoring [1,2]. For instance, the external load and deformation of the structure system generally follows the below relation:

$$\left[d\right] = \left[K\right]^{-1}\left[f\right].$$

where {*d*} stands for the deformation state; [*K*] is the stiffness matrix of the structure; "*f*# represent the effect induced by the external load. When any damage or deterioration occurs in the structure, the stiffness matrix [*K*] will change correspondingly, which in turns lead to the inevitable change in the deformation state {*d*}. Therefore, the change in the deformation state can be utilized to evaluate the health state of the structure. The present study focuses on the direct extraction of the overall deformation, rather than approximating the deformation through the data measured at several discrete points. The major advantage of the overall deformation data is to eliminate the error in structural health evaluation caused by insufficient measurement.

Traditionally, the structural deformation can be measured by leveling, total station, GPS, vibration sensors, and other equipment. At present, these methods can accurately and rapidly measure the deformation information of structures. However, only the limited key points of the targeting structure can be measured using the above methods, which often lead to insufficient data and, moreover, the insensitivity to structural deterioration [3]. Obviously, the direct solution is to largely increase the number of sensors installed on the structure. However, it is both time and budget consuming, which is not applicable in engineering practices. Alternatively, the digital image full-field structural morphology measurement can be a very ideal solution, which can take the advantages of both the structural damage identification method and digital image processing technology. Therefore, it is very crucial to effectively utilize the structural image features to extract the full-field deformation information of the structure.

In recent years, digital image processing technology is eventually employed to measure the overall deformation of the structure. As a kind of remote sensing technique, photogrammetry does not need any contact with the objects, and this can be a grea<sup>t</sup> advantage in the deformation monitoring of structures. "Photogrammetry" is to set up a base station in a stable area on the front of the target, and then shoot the target, so as to ge<sup>t</sup> the shape and motion state of the target according to the image [4]. According to different imaging distances, photogrammetry can be divided into "space photogrammetry", "close-range photogrammetry", and "microscopic photogrammetry" [5]. In structural deformation monitoring, close-range photogrammetry has broad prospects for development [6], and "Close-range photogrammetry" means that the distance between the base station and the measured structure is within 300 m [7]. Feng et al. [8] presents a comprehensive review on the recent development of computer vision-based sensors for structural displacement response measurement and their applications for SHM. Importation issues critical to successful measurement are discussed in detail, including how to convert pixel displacements to physical displacements, how to achieve sub-pixel resolutions, and what to cause measurement errors and how to mitigate the errors. However, the article also clearly points out that in many respects, the vision-based sensor technology is still in its infancy. The majority studies have still been focused on measurements of small-scale laboratory structures or field measurements of large structures at a limited number of points for a short period of time. Rolands Kromanis et al. [9] introduces a low-cost robotic camera system (RCS) for accurate measurement collection of structural response. The low-cost RCS provides very accurate vertical displacements. The measurement error of the RCS is 1.4%. Serena Artese et al. [10] proposed a bridge monitoring system, which combines camera and laser indicator; the elastic line inclination is measured by analyzing the single frames of an HD video of

the laser beam imprint projected on a flat target. The inclination of the elastic line at the support was obtained with a precision of 0.01 mrad. Ghorban et al. [11] measured the overall deformation of the masonry wall subjected to cyclic loads, using the 3D image correlation technology. The displacement, rotation, and interface slip between the reinforced concrete column and masonry were measured. Wang et al. [12] used the close-range photogrammetry technology to monitor the displacement of tunnel caverns. The measured results were compared with the values measured by mechanical convergence meter, and the difference between the two methods is no more than ±2 mm at the measuring distance of 8 m. This accuracy meets the requirement of general tunnel deformation monitoring. Reference [13] studied the application of sub-pixel displacement measurement method in soil strain monitoring. Based on the spatial correlation function iteration, the sub-pixel displacement of soils was measured. Zang et al. [14] applied the close-range photogrammetry technology in measuring bridge deflection and proved that a desirable accuracy can be achieved, i.e., ±1 mm. However, the accuracy is greatly affected by the positioning of the artificial marking points required by the method [15]. Although the above studies validated the feasibility of the application of close-range photogrammetry technology in structural deformation monitoring, the above methods are still unable to measure the structural overall deformation.

In order to explore the feasibility of photogrammetry in structural overall deformation monitoring, Ivan Detchev et al. [16] explored the use of consumer-grade cameras and projectors for the deformation monitoring of structural elements. A low-cost digital camera deformation monitoring system is proposed. Static load tests of concrete beams are carried out in the laboratory. The experiments proved that it was possible to detect sub-millimeter-level overall deformations given the used equipment and the geometry of the setup. However, this technology requires high texture characteristics of the structure surface and needs to project random pattern on the structure surface, which is difficult to achieve in the actual bridge structure deformation monitoring. On another hand, the close-range photogrammetry requires the measuring equipment to be located in the orthogonal projection position of the measured surface, which is usually difficult in engineering practice. Taking the bridge structure as an example, the surroundings near the bridge are usually complex, such as mountains, rivers, and trees, which makes it difficult for the camera to maintain the orthogonal projection position with respect to the measuring bridge.

In the deformation monitoring of structures, environmental factors must be considered. The complex geographical conditions of the structures means the photogrammetric camera is unable to work in the ideal measuring position. Therefore, it is necessary to study a new photogrammetric method for the overall deformation of structures under the condition that the camera is in the inclined position. In view of the actual needs of the structure deformation monitoring, this paper studies the overall deformation monitoring method under the condition of tilt photography. The steel truss concrete composite beam specimens were made in the laboratory. The static deformation images of the specimens were obtained by oblique photography. The overall deformation of the specimens was obtained by perspective transformation and edge detection technology. The error sources of this method were analyzed. This research is a comprehensive application of photogrammetry and digital image processing technology in the field of structure deformation monitoring. Its research foundation has been carefully verified and published in many publications [17,18]. The research results can alleviate the problem of insufficient deformation data in damage identification. In addition, compared with traditional photogrammetric methods, this study also highlights the advantages of flexible placement of camera positions in actual measurement work.

#### **2. Orthogonal Projection and Global Deformation Acquisition Method of Structures**

#### *2.1. Perspective Transformation of Digital Image*

On the basis of unchanged image content, the image pixel position is transformed, which is called image geometric transformation [19]. It mainly includes translation, rotation, zooming, reflection, and slicing. Usually, compound transformations, such as the perspective transformation, can be divided into a series of basic transformations. According to the perspective principle [20], when photographed under the condition of non-orthogonal projection, the image of the measured structure will deform. As a result, the true shape of the structure can be obtained only when the camera is in the orthogonal projection position of the measured surface. The process of mathematical transformation from oblique projection center to orthogonal projection center is called the perspective transformation. Figure 1 illustrates the basic model of perspective transformation.

**Figure 1.** Perspective transformation.

A 3D Cartesian coordinate system can be established, in which the projection center of the camera is selected as the origin, called the camera coordinate. Meanwhile, the image plane is set as the x-y plane, and the focus of the plane is located at [0, 0, *f* ](*f* > <sup>0</sup>). A 2D Cartesian coordinate system can be established on the plane where the object is measured, called the measuring coordinate. The origin of the measuring coordinate system is [*<sup>x</sup>*0, *y*0, *<sup>z</sup>*0]*<sup>T</sup>* in the camera coordinate. The unit vectors in the *x*-axis direction are [*<sup>u</sup>*1, *u*2, *<sup>u</sup>*3]*<sup>T</sup>*, the unit vectors in the *y*-axis direction are [*<sup>v</sup>*1, *v*2, *<sup>v</sup>*3]*<sup>T</sup>*, and the vector relation can be written as follows: 

$$\begin{cases} \begin{aligned} \boldsymbol{u}\_{1}\boldsymbol{v}\_{1} + \boldsymbol{u}\_{2}\boldsymbol{v}\_{2} + \boldsymbol{u}\_{3}\boldsymbol{v}\_{3} = 0\\ \boldsymbol{u}\_{1}^{2} + \boldsymbol{u}\_{2}^{2} + \boldsymbol{u}\_{3}^{2} = \boldsymbol{v}\_{1}^{2} + \boldsymbol{v}\_{2}^{2} + \boldsymbol{v}\_{3}^{2} = 1 \end{aligned} \end{cases} \tag{1}$$

The points of coordinates [*u*, *v*]*<sup>T</sup>* in the measuring plane can be expressed in the camera coordinate system as the vector below,

$$
\mu \begin{bmatrix} u\_1 \\ u\_2 \\ u\_3 \end{bmatrix} + \upsilon \begin{bmatrix} v\_1 \\ v\_2 \\ v\_3 \end{bmatrix} + \begin{bmatrix} x\_0 \\ y\_0 \\ z\_0 \end{bmatrix} \tag{2}
$$

Assuming that the coordinate of the point in the original imaging plane is [*x*, *y*, <sup>0</sup>]*<sup>T</sup>*, ∃*k* ∈ *R* the following expression can be derived,

$$
\mu \begin{bmatrix} u\_1 \\ u\_2 \\ u\_3 \end{bmatrix} + v \begin{bmatrix} v\_1 \\ v\_2 \\ v\_3 \end{bmatrix} + \begin{bmatrix} x\_0 \\ y\_0 \\ z\_0 \end{bmatrix} - \begin{bmatrix} 0 \\ 0 \\ f \end{bmatrix} = k \begin{bmatrix} 0 \\ 0 \\ f \end{bmatrix} - \begin{bmatrix} x \\ y \\ 0 \end{bmatrix} \tag{3}
$$

Comparing the preceding formula, it yields

$$-\mathbf{k}\begin{bmatrix}\mathbf{x} \\ \mathbf{y}\end{bmatrix} = \mathbf{u}\begin{bmatrix}\boldsymbol{u}\_{1} \\ \boldsymbol{u}\_{2}\end{bmatrix} + \mathbf{v}\begin{bmatrix}\boldsymbol{v}\_{1} \\ \boldsymbol{v}\_{2}\end{bmatrix} + \begin{bmatrix}\boldsymbol{x}\_{0} \\ \boldsymbol{y}\_{0}\end{bmatrix} = \begin{bmatrix}\boldsymbol{u}\_{1} & \boldsymbol{v}\_{1} & \boldsymbol{x}\_{0} \\ \boldsymbol{u}\_{2} & \boldsymbol{v}\_{2} & \boldsymbol{y}\_{0}\end{bmatrix}\begin{bmatrix}\boldsymbol{u} \\ \boldsymbol{v} \\ 1\end{bmatrix}\tag{4}$$

$$kf = uu\_3 + vv\_3 + z\_0 - f = \begin{bmatrix} u\_3 & v\_3 & z\_0 - f \\ & & \\ & 1 \end{bmatrix} \begin{bmatrix} u \\ v \\ 1 \end{bmatrix}.\tag{5}$$

Equation (5) can be rewritten as the following,

$$k - k = \begin{bmatrix} \ -\frac{\nu\_3}{f} & -\frac{\nu\_3}{f} & -\frac{z\_0 - f}{f} \end{bmatrix} \begin{bmatrix} u \\ v \\ 1 \end{bmatrix}. \tag{6}$$

Combining Equations (6) and (4), it leads to

$$-k \begin{bmatrix} \mathbf{x} \\ \mathbf{y} \\ 1 \end{bmatrix} = \begin{bmatrix} \boldsymbol{\mu}\_1 & \boldsymbol{\upsilon}\_1 & \boldsymbol{\varkappa}\_0 \\ \boldsymbol{\mu}\_2 & \boldsymbol{\upsilon}\_2 & \boldsymbol{\nu}\_0 \\ -\frac{\boldsymbol{\mu}\_3}{f} & -\frac{\boldsymbol{\upsilon}\_3}{f} & -\frac{\boldsymbol{\varepsilon}\_0 - f}{f} \end{bmatrix} \begin{bmatrix} \boldsymbol{\mu} \\ \boldsymbol{\upsilon} \\ 1 \end{bmatrix}. \tag{7}$$

For convenience, a parameter matrix M is introduced, as shown below,

$$
\mathcal{M} = \begin{bmatrix}
 u\_1 & v\_1 & x\_0 \\ 
 u\_2 & v\_2 & y\_0 \\ 
 -\frac{u\_3}{f} & -\frac{v\_3}{f} & -\frac{z\_0 - f}{f}
\end{bmatrix}.
\tag{8}
$$

If the focus [0, 0, *f* ]*T* is not on the measuring plane, the matrix M is a nonsingular matrix. Under normal working conditions, the focus will not be on the measuring plane, so the matrix M can usually be treated as a nonsingular matrix. The focal length and spatial position of the camera will change when the camera moves to a new position to capture the target structure. It can also be considered that the camera imaging plane is fixed, the focal length and the actual spatial position of the structure are changed. Make the coordinates of the camera focus change to [0, 0, *f*]*<sup>T</sup>*, and the original coordinates of the measuring plane change to *x*0, *<sup>y</sup>*0, *z*0 *T*. The unit vectors of the *x*, *y* axes in the measuring plane become *u*1, *<sup>u</sup>*2, *u*3 *T*, *v*1, *v*2, *v*3 *T*. Similarly, there is ∃*k* ∈ *R*, making the coordinate point [*u*, *v*]*<sup>T</sup>* on the measuring plane, and its corresponding imaging point [*<sup>x</sup>*, *y*, 0]*T* should satisfy:

$$-k'\begin{bmatrix} x' \\ y' \\ 1 \end{bmatrix} = \begin{bmatrix} u\_1' & v\_1' & x\_0' \\ u\_2' & v\_2' & y\_0' \\ -\frac{u\_3'}{f'} & -\frac{v\_3'}{f'} & -\frac{z\_0' - f'}{f'} \end{bmatrix} \begin{bmatrix} u \\ v \\ 1 \end{bmatrix}.\tag{9}$$

The parameter matrix M' is denoted as:

$$M' = \begin{bmatrix} u\_1' & v\_1' & x\_0' \\ u\_2' & v\_2' & y\_0' \\ -\frac{u\_3'}{f'} & -\frac{v\_3'}{f'} & -\frac{z\_0' - f'}{f'} \end{bmatrix}. \tag{10}$$

Comparison of Equations (7) and (9) leads to

$$\mathbf{x} - \mathbf{x}' \begin{bmatrix} \mathbf{x}' \\ \mathbf{y}' \\ 1 \end{bmatrix} = \mathbf{M}' \begin{bmatrix} \mathbf{u} \\ \mathbf{v} \\ 1 \end{bmatrix} = -k\mathbf{M}'\mathbf{M}^{-1} \begin{bmatrix} \mathbf{x} \\ \mathbf{y} \\ 1 \end{bmatrix} \tag{11}$$

assuming:

$$\mathbf{M}^{'} \cdot \mathbf{M}^{-1} = \begin{bmatrix} m\_{11} & m\_{12} & m\_{13} \\ m\_{21} & m\_{22} & m\_{23} \\ m\_{31} & m\_{32} & m\_{33} \end{bmatrix} . \tag{12}$$

Accordingly, the following expansions are introduced:

⎧⎪⎪⎪⎨⎪⎪⎪⎩*k x* = *k*(*<sup>m</sup>*11*<sup>x</sup>* + *m*12*x* + *<sup>m</sup>*13) *k x* = *k*(*<sup>m</sup>*21*<sup>x</sup>* + *m*22*x* + *<sup>m</sup>*23) *k* = *k*(*<sup>m</sup>*31*<sup>x</sup>* + *m*32 *y* + *<sup>m</sup>*33) . (13)

Therefore, there is:

$$\begin{cases} \text{x}' = \frac{m\_{11}x + m\_{12}x + m\_{13}}{m\_{31}x + m\_{32}y + m\_{33}}\\ \text{y}' = \frac{m\_{21}x + m\_{22}x + m\_{23}}{m\_{31}x + m\_{32}y + m\_{33}} \end{cases} \tag{14}$$

The coordinates (*x*, *y*) are the imaging point of the original image, which is transformed into a new imaging point (*<sup>x</sup>*, *y*) after the perspective transformation. Based on the above analysis, the proposed process can convert the original oblique structural image into orthophoto-projection image, which provides technological foundation for monitoring the overall deformation of the structure.
