Initial Structure Design and Optimization of an Automotive Remote Head-Up Display

Abstract

Aiming at the problems of difficult construction and occlusion of the initial structure of an automotive augmented reality head-up display (AR-HUD), a method to quickly build the initial structure of an automotive AR-HUD is proposed. Firstly, the position of the mirrors in the initial structure is calculated based on the Rodrigues rotation formula. Secondly, the position of the mirrors is restricted by constraints during optimization to prevent the problem of structural occlusion. Finally, a virtual image display system with a visual distance of 7.5 m and a field of view angle of 10° × 5° is designed. The image quality analysis of the optimized system shows that the light spots in each field of view are all within Airy spots. At the spatial cutoff frequency of the virtual image plane of the optical system, the modulation transfer function (MTF) value of the full field of view is basically greater than 0.5, and the distortion is less than 1%. Finally, using the image for simulation, the results of the simulation image are satisfying, which proves the validity and feasibility of the structural design. It provides a useful reference for the structural design of the remote head-up display system.

1. Introduction

With the development of the automobile industry, head-up display (HUD) technology has gradually been applied to car driving. It can project driving information as images to the front of the windshield of the car and provide drivers with real-time driving information, thus improving driving safety and comfort [1,2].

Augmented reality head-up displays (AR-HUDs) are a research hotspot that has arisen in recent years. Compared with traditional HUDs, AR-HUDs can achieve a longer projection distance and a larger field of view angle, superimpose virtual prompts in the real world, and describe and perceive information intuitively and clearly to achieve a friendly and natural interactive interface [3,4,5]. Aiming at the AR-HUD structure design, Huang Xingzhou [6] et al. designed an optical path structure with a virtual image visual distance of 7.5 m and a virtual image field of view angle of 9.8° × 5.5° by using a free-form surface and off-axis reflected optical path structure and analyzed the tolerance of the system. Chen Xiaowei et al. [7] designed a virtual image display light path with a virtual visual distance of 10 m and a field of view angle of 10° × 5° by using two free-form surfaces and an off-axis double reflection system and conducted simulation analysis on light damage. Finally, the display effect was simulated by UI images. However, the above design only analyzes the design index and results and lacks a detailed demonstration and derivation for the construction of the initial structure. In terms of initial HUD structure design, Ma Donglin et al. [8] proposed a HUD initial structure design method based on off-axis triple inversion and analyzed HUD structural constraints and optimization strategies. Zhang Yangliu et al. [9] conducted an initial design of a HUD optical system based on a seed curve expansion algorithm, verified the results, and analyzed the tolerance of the system. However, the above initial structure design method is only used for short-distance projection head-up display systems, and there is no mature design method suitable for long-distance display systems currently.

Existing methods lack a detailed demonstration and derivation for the construction of the initial AR-HUD structure. Inspired by the theory of the Rodrigues rotation formula, we propose a method to design the initial AR-HUD structure. To overcome the difficulty of structural occlusion, we take the example of an off-axis triple inverse structure.

Our contributions are as follows:

(1)

The design index is proposed according to the current industry AR-HUD standard.

(2)

A method of initial AR-HUD structure design based on the Rodrigues rotation formula is proposed, and combined with the characteristics of the off-axis triple inverse structure, the problem of structural occlusion in the optimization process is solved.

(3)

We design an example according to the above method and analyze the design results.

This article is organized as follows: Section 2 presents the basic theory and design objective, and Section 3 describes the proposed initial structure design based on the Rodrigues rotation formula and anti-occlusion optimization method. In Section 4, we design an example and discuss the experimental results. In Section 5, we summarize the content of this paper.

2. Basic Theory and Design Objective

2.1. Design Principle

The light path diagram of the HUD structure includes five main parts: source image, reflectors M₁ and M₂, windshield, and eyebox. As shown in Figure 1, light is emitted from the source image at a certain divergence angle and enters the human eye after being reflected by reflectors M₁ and M₂ and the windshield. Finally, it converges at a certain distance to form a virtual image, which enables drivers to see driving information and the road at the same time.

2.2. Design Parameter Analysis

In the process of AR-HUD design, the main design parameters include virtual image visual distance, field of view angle, eyebox size, source image size, and image quality requirements.

In the AR-HUD projection process, the projected image needs to be combined with the real scene. Considering driving comfort, the virtual visual distance of the AR-HUD required in the industry is between 7.5 m and 20 m [10]. This paper adopts the 7.5 m virtual visual distance commonly used in the industry as the design objective.

The field of view angle refers to the angle between the edge of the observed object and the center of the observation point, and in the head-up display system, it is just the angle between the edge of the formed virtual image and the center of the pupil of the human eye. The current industry standard for AR-HUD generally sets the field of view angle to about 10° × 5°.

As shown in Figure 2 below, eyebox refers to the range through which the driver’s eyes can adjust during driving without affecting the sharpness of the virtual image. This data is an empirical statistical value, which is usually set according to relevant research and the demands of automobile manufacturers. In this paper, the empirical value is set at 130 mm × 50 mm.

As shown in Figure 3 below, the visual distance of the virtual image is 7.5 m, and the field of view angle is set to 10° × 5°. The horizontal imaging size $X$ can be calculated as

$$X=2d\cdot \tan \frac{{10}^{\circ }}{2}=1312.33\mathrm{mm}.$$

(1)

Similarly, the imaging size in the vertical direction can be calculated as 654.91 mm.

The typical resolution of the human eye is 1′. When the field of view angle is 10° × 5°, in order for the driver to clearly observe the virtual image generated by the HUD system, the image resolution is required to be greater than 600 × 300 pixels. In this paper, the DLP5530S-Q1 automotive digital micromirror device (DMD), produced by Texas Instruments, is selected as the source image module with a resolution of 1152 × 576 pixels, which can achieve a high-performance augmented reality HUD, gaining a wider field of view and greater driver eye movement range.

Based on the virtual image resolution of 1152 × 576 pixels, the size of each pixel of the horizontal virtual image μ can be calculated as

$$\mathsf{\mu }=\frac{X}{R_x},$$

(2)

where $X$ is the horizontal imaging size and $R_x$ is the virtual image resolution at x direction. It can be calculated at about 1.14 mm.

To obtain the analytical modulation transfer function (MTF) value [6] of the optical transfer function, according to Nyquist’s sampling theorem, the resolution of the optical system in the virtual image space $MT{F}_{vis}$ is at least

$$MT{F}_{vis}=\frac{1}{2\mu }\approx 0.44lp/\mathrm{mm}.$$

(3)

According to the analysis of the above technical indicators, considering the compact HUD structure, the machinability of a free-form surface in the later stage, and optimizing the layout, the specific parameters and indicators of the HUD system in this paper are shown in

Table 1. AR-HUD system parameter settings.

HUD System Parameter	Value
Wavelength (mm)	486–656 (visible light)
Virtual image distance (m)	7.5
Angle of field (°)	10 × 5
Eyebox (mm)	130 × 50
Source image size (mm)	92.08 × 42.04
Virtual image space MTF	>[email protected] lp/mm
Distortion (%)	<2

below:

3. Proposed Methods

3.1. Initial Structure Design Based on the Rodrigues Rotation Formula

At present, the initial structural design of HUD is basically aimed at the short-distance display system, that is, the optical system with a virtual visual range of between 2 and 2.5 m. And there is no detailed design scheme for the long-distance projection AR-HUD. In this paper, based on the off-axis triple inverse structure, the following AR-HUD initial structure design method is proposed, combined with the HUD installation position and prototype volume constraints.

According to the internal structure size of the instrument panel, the maximum volume of the automotive HUD that can be installed is about 15 L. In the initial design, in order to prevent the occlusion of the lens in the later stage as much as possible, the reflector space should be extended as far as possible with a certain installation gap. The initial placement position is roughly shown in Figure 4. The spacing between the components can be measured according to the installation position.

When designing the initial structure of the HUD, it usually starts from the reverse, that is, the virtual image plane is regarded as the object plane, which is conducive to the construction of the initial light path. In order to facilitate the initial structural design, the source image, reflectors M₁ and M₂, and windshield are set in the same meridian plane as the main ray intersection. The optical path structure shown in Figure 5 can be obtained by pursuing the main ray. In the figure, the vertical axis always corresponds to the y axis.

In Figure 5, $S_0,S_1,S_2,S_3,S_4$ are intersection points of the main ray with the eyebox, windshield, free-form surfaces M₁ and M_2, and source image surface, respectively. The tilt angle of the windshield surface can be measured in advance, and the direction vector $\overrightarrow{\mathit{N}}$ can be calculated from the surface profile data. The reflection angle $\alpha$ of the surface can be calculated as follows:

$$\alpha =\arccos (\frac{\overrightarrow{S_1{S}_0}}{\left|\overrightarrow{S_1{S}_0}\right|}\cdot \frac{\overrightarrow{N}}{\left|\overrightarrow{N}\right|}).$$

(4)

Given that the angle of reflection $\alpha$ is equal to the angle of incidence, the reflected light $\overrightarrow{S_1{S}_2}$ can be calculated as

$$\frac{\overrightarrow{S_1{S}_2}}{\left|{\overrightarrow{S_{1}S}}_2\right|}=2\cdot \frac{\overrightarrow{N}}{\left|\overrightarrow{N}\right|}\cdot \cos \alpha -\frac{\overrightarrow{S_1{S}_0}}{\left|{\overrightarrow{S_{1}S}}_0\right|}.$$

(5)

According to the HUD installation position measurement before the design, $\left|\overrightarrow{S_1{S}_2}\right|$ is about 230 mm, and then the vector $\overrightarrow{S_1{S}_2}$ can be calculated. According to the installation spacing of each surface in Figure 4, the reflection angle and reflected light vector parameter coordinates $\overrightarrow{S_2{S}_3}$, $\overrightarrow{S_3{S}_4}$ of the surface $S_3,S_4$ can be calculated.

After calculating the reflection vector of each surface, it is necessary to calculate the rotation angle and tilt angle of each surface, including the three x, y, and z axis directions, which can help to calculate the rotation angles. Assuming that the direction unit vector of the surface $S_2$ before rotation is $\mathit{S}=(s_x,s_y,s_z)$, and the direction vector after rotation is $\mathit{T}=(t_x,t_y,t_z)$, then the rotation matrix satisfies

$${\mathit{T}}^T=\mathit{R}{\mathit{S}}^T,$$

(6)

$$\mathit{R}=\left(\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right),$$

(7)

where $r_{ij}$ represents each term of the rotation matrix $\mathit{R}$. Assuming that the angle between vectors $\mathit{S}$ and $\mathit{T}$ is $\beta$, the rotation matrix can be obtained according to the Rodrigues rotation formula [11]:

$$\mathit{R}=\mathit{I}+(1-\cos (\beta ))\cdot {\tilde{\mathit{K}}}^2+\tilde{\mathit{K}}\cdot \sin \beta ,$$

(8)

where $\mathit{I}$ is the unit matrix, and $\tilde{\mathit{K}}$ is the cross product matrix, which can be represented with $k=[{\begin{array}{ccc}{k}_x& k_y& k_z]\end{array}}^T$:

$$\tilde{\mathit{K}}=\left(\begin{array}{ccc}0& -k_x& k_y\\ k_z& 0& k_x\\ -k_y& k_x& 0\end{array}\right)$$

(9)

Assuming that the angles between the rotation vector $\mathit{R}$ and the $x,y,\mathrm{z}$ axis directions are $\lambda ,\mu ,\delta$, respectively, the rotation angle can be further derived as

$$\begin{array}{c}\lambda =\arctan (r_{32},r_{33})\hfill \\ \mu =\arctan (-r_{31},\sqrt{r_{32}{}^2+r_{33}^2})\hfill \\ \delta =\arctan (r_{21},r_{11})\hfill \end{array}.$$

(10)

Similarly, the rotation angle of the surplus surface in the initial structure can be calculated step by step according to the light path, such as mirror 1 and the image source. At this time, the position and deflection angle of each plane of the initial structure have been determined, and the initial system can be built.

3.2. Anti-Occlusion Optimization Method Based on the Off-Axis Triple Inverse Structure

Due to the advantages of compact structure, no shading, a large field of view, and so on [12], the off-axis triple inversion is usually used as the initial structure for optimal design in the HUD design process. However, because its structure is very flexible, the eccentricity of a small distance or the adjustment of a small angle of rotation may bring about problems in the optimization process, such as structural occlusion. Inspired by the research on the off-axis three-reverse control occlusion method [13], as shown in Figure 6 below, by controlling the height of the light at the edge of the adjacent surface in the optimization process, the light occlusion problem in the optimization process can be prevented. The constraint conditions can be expressed as follows:

$$\begin{array}{c}{y}_{1_{down}}-y_{3_{up}}>a\hfill \\ y_{2_{down}}-y_{4_{up}}>b\hfill \\ y_{3_{down}}-y_{5_{up}}>c\hfill \end{array},$$

(11)

where $y_{i_{down}}$ denotes the $i$th ordinate below the first surface, $y_{i_{up}}$ denotes the $i$th ordinate above the first surface, and $a,b,c$ are constants.

In the structure of this paper, the source image plane in the HUD can be regarded as surface 5 in Figure 6, surface M₁ as surface 3, surface M₂ as surface 4, and the windshield can be regarded as surface 2. According to the relationship between the relevant structure and surface position, as shown in Figure 7, the following expression can be used to control the surface position:

$$\begin{array}{l}\min {\mathrm{MF}}^2=\frac{{\displaystyle \sum _{i=1}^n{W}_i{(V_i-T_i)}^2}}{{\displaystyle \sum _{i=1}^n{W}_i}}\\ s.t.\begin{array}{c}{x}_{Q_4}-x_{P_2}\ge X\\ y_{P_3}-y_{Q_2}\ge Y\\ z_{Q_3}-z_{P_1}\ge Z\end{array}\end{array},$$

(12)

where MF represents the value of the system optimization function, $W_i$ represents the weight of the $i$th optimization, $V_i$ represents the actual value of the $i$th optimization variable, $T_i$ represents the standard value of the $i$th optimization variable, and $x_{Q_i}$ represents the $x$ direction coordinate value of the $Q_i$. $X,Y,Z$ are constants and need to be determined according to the relative position of the HUD structure and the size of the reflection mirrors. In this experiment, they are set to 10 mm, 20 mm, and 200 mm, respectively.

In Zemax, an optimization function is used to express the constraint condition (12), and the weight of the constraint term is set to limit the structure so as to prevent the mirror position from serious deviation, occlusion, and structure chaos during the optimization.

4. Simulation Design and Results

4.1. Design Example

In the design process, due to the irregularity of the windshield surface, the aberration introduced cannot be ignored, so it is necessary to fit the windshield surface in advance. The free-form surface expression XY extended polynomial in Zemax is used to fit the front windshield of the automobile. The XY extended polynomial is as follows:

$$z=\frac{\rho r^2}{1+\sqrt{1-(1+k){\rho }^2{r}^2}}+\sum _i^{}{A}_i{x}^m{y}^n$$

(13)

where $r^2=x^2+y^2$, $i=\frac{{(m+n)}^2+m+3n}{2+1}+1$, and $z$ represents the vector height difference in the direction of the z axis, $\rho$ represents the curvature, $k$ represents the conical surface constant, $A_i$ represents the polynomial coefficient, and $m$ and $n$ represent the corresponding $x$ and $y$ order, respectively.

The windshield point cloud data of a certain vehicle is extracted by LiDAR, and then the obtained data are fitted in Matlab using Formula (13). The windshield data after fitting is shown in

Table 2. Windshield fitting results.

Radius (mm)	Conic	(X1Y0)	(X0Y1)	(X2Y0)	(X1Y1)	(X0Y2)
−7526.66	−2.789	2.308	−3.569	−7.601	2.430	−3.564

.

After calculating the expression of the windshield, the initial structure is reverse-built according to the method proposed in Section 3.1. Considering the difficulty and processing requirements of the later processing of the free-form surface, the XY extended polynomial with stronger processability is used to design the two free-form mirrors. In the design process, in order to control aberration, only the even terms of the folded mirror M₁ are set as variables to participate in optimization to reduce the complexity of the surface. The maximum degree of the main mirror M₂ and the folded mirror M₁ is set to 14 terms, that is, the maximum degree of the polynomial $x+y$ is 4. After the system is built, the M₁ and M₂ mirror curvature radius, cone coefficient, free-form surface type, and so on are optimized as variables.

If the position and structure of the surface shape are not controlled during optimization, the structural occlusion problem, as shown in Figure 8a, may occur because the surface is very flexible. In Zemax, the occlusion problem of the structure is not easily discoverable because the occlusion problem of light is not considered. By using the optimization function to limit the edge position of the structure, the occlusion control effect, as shown in Figure 8b, can be achieved.

In the process of optimization, it is necessary to always pay attention to the optical trace diagram and optical path structure of each surface. When it is found that the light deviation is too large, it is necessary to fine-tune the deflection angle and eccentricity of the free-form surface and correct the deviation caused by calculation errors and other factors in the initial structural design. However, the adjustment amount should not be too large; otherwise, the structure may be invalid. In the optimization process, the number of variables is gradually increased, or a part of the variables is fixed, and the result is gradually optimized until the structure and image quality meet the design requirements. After simple optimization, the partial structure diagram of the AR-HUD is shown in Figure 9 below.

According to the system data, the visual distance of the virtual image of the optical path map is 7.5 m, and the imaging size of the virtual image is 1312.67 mm × 654.87 mm. The imaging position of the virtual image plane in the display of the optical path light map is shown in Figure 10 below.

In the process of optical path structure design, the windshield is involved as a reflecting surface. Therefore, the surfaces to be designed and determined are folding mirror M₁, main mirror M₂, and the windshield. Among them, the spacing between each surface, the x and y axis rotation, and the y axis offset are the main parameters; the specific data are shown in

Table 3. Optical path structure parameters.

Surface	Radius (mm)	Interval (mm)	Tilt x (°)	Tilt y (°)
M1	−12,494.98	−140.64	−39.229	0.223
M2	−106.487	−154.320	−12.525	3.225
Windshield	−7526.66	286.972	61.836	−6.956

below. The optimized M₁ and M₂ profile data are shown in

Table 4. Folding mirror M₁ surface type and data.

Surface Type	Radius (mm)	Conic	X2Y0	X1Y1	X0Y2	X3Y0	X2Y1	X1Y2
Extend polynomial	−12,494.98	1.000	3.479	-	−0.661	−0.006	-	-
			X0Y3	X4Y0	X3Y1	X2Y2	X1Y3	X0Y4
			−0.078	0.412	-	−0.084	-	−0.107

and

Table 5. Folding mirror M₂ surface type and data.

Surface Type	Radius (mm)	Conic	X2Y0	X1Y1	X0Y2	X3Y0	X2Y1	X1Y2
Extend polynomial	−106.487	−1.026	51.925	−0.107	48.937	−0.036	0.055	−0.005
			X0Y3	X4Y0	X3Y1	X2Y2	X1Y3	X0Y4
			−0.057	0.218	-	−0.500	-	−0.268

. Finally, the vision distance of the virtual image display optical path of the AR-HUD system is 7.5 m, and the virtual vision field angle is 10.145° × 5.072°, which accords with the design goal.

4.2. Image Quality Analysis

Imaging optical design must correct the aberration of the optical system, and the determination of aberration tolerance can effectively guide the optical design. In the process of HUD design, spot diagrams, optical modulation transfer functions, and distortion are usually used to evaluate the image quality.

Spot diagrams are tools that describe the image quality of an optical system, which refers to the distribution of the image obtained by imaging a point light source in the image plane. The size and shape distribution of the spot are directly related to the aberration of the optical system. The smaller the distribution area of the spots is, the smaller the aberration is. In addition, the more concentrated the spots are, the better the image quality of the system is. The pupil of the human eye is regarded as the aperture stop of the system, and the diameter of the pupil of the human eye is usually between 1 and 4 mm. In this paper, the diameter of the entrance pupil is set at 4 mm. The spot diagram distribution of each field of view on the image plane is shown in Figure 11 below. The RMS radius of the center field of view is 16.407 μm, and the maximum RMS radius of the edge field of view is only 89.011 μm, which is less than the experience value of AR-HUD design (100 μm). The distribution of visible light spots in the nine fields of view shows that they are all within Airy spots, the shape of the spots is relatively regular, and they are concentrated near the center of the circle, indicating that the whole system has small aberrations and good imaging quality.

In optical systems, the modulation transfer function (MTF) can be used to describe factors such as image quality, contrast, resolution, and sharpness. The specific calculation process can be referred to in [14]. In general, the MTF curve is expressed as the response to the spatial frequency of the input signal, with a value between 0 and 1, indicating the ability of the system to transmit a certain spatial frequency component. High-quality optical systems typically have high MTF values, which means that they can efficiently transmit signals with a high spatial frequency component, thus providing higher resolution and better image quality. The MTF curve of the AR-HUD system designed in this paper in the image square space is shown in Figure 12 below. The MTF curve of the full field of view is basically greater than 0.5 at the maximum Nyquist frequency of 0.44 lp/mm, and the performance of the entire curve is close to the diffraction limit, indicating that the overall imaging quality is good.

Figure 13a–d below are the distortion maps obtained by the system in the scanning mode of +y, +x, −y, and −x respectively. The distortion values of the system in the four scanning directions are 0.3110%, 0.3111%, 0.3617%, and 0.3342%, which are far less than the set standard value of 2%, indicating that the distortion of the system is small. The results meet the design requirements.

The image simulation tool provided by Zemax software (OpticStudio 19.4 Premium) is used to simulate and analyze the imaging performance of the system, and the original optical position image is compared with the simulation image to reflect the effect of the system in actual imaging. The specific effect is shown in Figure 14 below. It can be seen from the grid simulation diagram that the system has a slight tilt to the left during imaging, and some image details are missing. However, in the actual navigation image simulation, the information can be displayed normally, without affecting the effective transmission of information, and the system can meet the actual application requirements.

5. Conclusions

By analyzing AR-HUD structure, this paper proposes an initial structure design method based on the Rodrigues rotation formula, which effectively solves the complicated problem of initial HUD structure construction. At the same time, using the off-axis three-mirror system anti-occlusion solution, the position of the mirror in the HUD system is restricted by constraints, which solves the problem that the system structure is easy to run off and the light is blocked in the process of software optimization. Finally, the simulation analysis of the designed system shows that system distortion is controlled within 2%, the virtual image plane of the system is at a Nyquist spatial frequency of 0.44 lp/mm, the MTF value of the full field of view is close to 0.5, and the simulation image has a good imaging effect that can meet the requirements of human eye observation, proving the correctness and effectiveness of the structural design. It provides some references for the subsequent initial structure design and control of AR-HUDs.