A Novel Conservative Chaotic System Involved in Hyperbolic Functions and Its Application to Design an Efficient Colour Image Encryption Scheme

Abstract

It is well known that, compared to low-dimension chaotic systems, three-dimensional chaotic systems have a wider parameter range, more complicated behaviour, and better unpredictability. This fact motivated us to introduce a novel image encryption method that employs a three-dimensional chaotic system. We proposed a novel three-dimensional conservative system that can exhibit chaotic behaviour involving hyperbolic functions. The dynamical behaviours of the proposed system are discovered by calculating Lyapunov exponents and bifurcation diagrams. Thereafter, we designed an image encryption method based on the proposed system and a $4\times 4$ self-invertible matrix. A modified Diffie–Hellman key exchange protocol was utilised to generate the self-invertible key matrix $K_m$ employed in the diffusion stage. Our approach has three main stages. In the first stage, the proposed three-dimensional system utilises the original image to create three sequences, two of which are chosen for confusion and diffusion processes. The next stage involves confusing the image’s pixels by changing the positions of pixels using these sequences. In the third stage, the confused image is split into sub-blocks of size $4\times 4$, and each block is encrypted by multiplying it with $K_m$. Simulation findings demonstrated that the proposed image scheme has a high level of security and is resistant to statistical analysis, noise, and other attacks.

1. Introduction

Nowadays, individuals with various profiles often participate in the sharing of multimedia material such as images on the internet and other communication networks. To ensure confidential transmission and the accuracy of the received data, techniques designed to secure this type of information play a crucial role. Image encryption differs from text encryption owing to inherent properties such as the high correlation between neighbouring pixels and redundancy. Thus, the design of robust secure, resilient, and effective image encryption algorithms will always remain at the forefront of commercial and scientific endeavours.

Many image encryption techniques have been suggested and developed by researchers, such as Arnold transformation-based [1,2,3], quantum system-based [4,5,6], compression sensing-based [7,8,9], cellular automata-based [10,11,12], DNA computing-based [13,14,15], elliptic curves-based [16,17,18], and chaotic system-based [19,20,21,22,23] approaches. The approach of digital image encryption based on chaotic systems is considered to be more effective than other approaches due to their ability to improve unpredictability, which makes encryption more resistant to attacks. These systems are generally categorised into one-dimensional systems such as Logistic map [24], higher-dimensional chaotic systems such as Lorenz system [25] and Rössler system [26], and fractional-order systems such as Chen system [27] (for more details, see [28]). The higher-dimensional systems outperform low-dimensional ones in terms of randomness, nonlinearity, and generated key sequences, which contain a large keyspace.

In the literature, the use of multidimensional chaotic systems for image encryption is well documented. For instance, Naseer et al. [29] introduced a method for image encryption that uses a three-dimensional chaotic system and a permutation box for the substitution phase and an S-box for the permutation phase. However, the provided image encryption does not achieve plain image sensitivity. Maazouz et al. [30] introduced a novel three-dimensional chaotic system and explained its function in creating S-boxes, using the resulting S-boxes to develop a new image encryption method. However, the final cyphered image is produced after encryption several rounds. Zheng et al. [31] introduced a new image encryption that combines the improved two-dimensional logistic sine map and dynamic DNA sequence encryption. They showed the DNA scheme’s security problem, which does not change during the encryption process, can be solved using their approach. Kumar and Girdhar [32] suggested an image encryption technique with a confusion stage depending on two-dimensional Logistic maps and a diffusion stage depending on a Lorenz–Rossler chaotic system. Wang et al. [33] employed a six-dimensional hyperchaotic system in conjunction with DNA coding to propose an image encryption method, which executes diffusion encryption in multiple rounds. Hu et al. in [34] suggested a Fibonacci dynamical system-based cloud model-based colour image encryption technique combined with a matrix convolution algorithm. This combination improves the security of encrypted images in terms of the randomness of bits.

An assessment of the literature reveals that several colour image encryption methods have security vulnerabilities, such as susceptibility to noise attacks, low keyspace, and failure to withstand statistical attacks. In addition, encrypting colour images efficiently is a significant challenge, since they have more data than greyscale images. These limitations motivated us to develop a three-dimensional chaotic system with complex dynamic behaviour to ensure desired properties, such as dissipative and great sensitivity to small changes in control parameters and initial conditions. Such properties are analogous to image encryption requirements. The main aim was to design a novel encryption method utilising the proposed system and the self-invertible matrix generated by a modified Diffie–Hellman key exchange protocol.

This paper’s contributions are outlined below:

• The proposed conservative system is employed, where its initial conditions depend on the input image to create the first set of secret keys required to scramble the plain image;
• The first use of the Diffie–Hellman key exchange protocol with a self-invertible matrix;
• The modified Diffie–Hellman key exchange protocol is proposed and combined with the self-invertible matrix $K_m$ of size $4\times 4$ to obtain the second set of keys;
• Based on the shared secret key matrix $K_m$, the diffusion stage is performed;
• One of the main drawbacks of the matrix generated by the modified Diffie–Hellman key exchange protocol is that the inverse of the matrix does not always exist, which affects the decryption process. To avoid this problem, we utilised the self-invertible matrix, which reduces the computational process requirements during decryption and increases the unpredictability of pixels distribution during encryption;
• The keyspace size is increased to $2^{521}$ by using two sets of keys, making brute-force assaults impossible;
• The three-dimensional conservative chaotic system, together with the key matrix $K_m$, provides a high level of security;
• The suggested method was evaluated against several assaults to measure how well it performs in terms of robustness, effectiveness, and cryptanalysis resistance.

This paper is structured as follows. The stability of the proposed conservative system is investigated and possible types of equilibria are determined in Section 2. The conservativeness of the proposed system is also discussed in this section. In Section 3, a modified Diffie–Hellman Key Exchange Protocol, which was combined with the proposed conservative system, is provided to design a new image encryption technique. In Section 4, the proposed encryption method is described. The security evaluation and results are presented in Section 5. A summary of the entire study is presented in Section 6 as its conclusion.

2. Conservative Chaotic System

Chaotic systems can be found in different areas such as weather forecasting [35], telecommunications [36], and biological modelling [37], and can be classified into different categories by physical (number of dimensions), dynamical (number of wings), and algebraic features (number of equilibrium points).

It is well known that almost all classical chaotic systems are dissipative, while a few are conservative systems. The conservative ones are an old category of dynamical systems studied by Euler, Lagrange, and Hamilton, whose phase space volume is conserved. The characteristics of conservative systems are as follows:

• The sum of finite-time local Lyapunov exponents is zero;
• Divergence of the vector field is zero;
• Local Lyapunov dimension is equal to the order of the system;
• The Hamiltonian energy of the system is invariable.

Sprott in [38] raised a fascinating question of whether new chaotic dynamical systems are still needed. His answer to the question motivated us to propose a novel chaotic system with some properties and applications as follows:

$$\left\{\begin{array}{cc}\dot{x_1}=x_2,\hfill & \\ \dot{x_2}=-x_1-x_2{x}_3,\hfill & \\ \dot{x_3}=(cosh{x}_2-1)-\gamma cos{x}^2-\beta cos{x}_2,\hfill \end{array}\right.$$

(1)

where $\gamma ,\beta$ are the system parameters and $x_1,x_2,x_3$ are system variables.

There are several chaotic systems based on the system in [38] with different features, but our proposed chaotic system has the following unique characteristics:

• The system is conservative for $\gamma =\beta =1$ but dissipates for $\gamma =0.3$ and $\beta =1$. Note that the semianalytical and seminumerical technique was used to demonstrate the conservativeness of the new system;
• The proposed system (1) has a stable or unstable line, or no equilibria;
• Coexisting attractors appear.

The properties of the proposed conservative system to analyse the global dynamics are explored in the following two sections.

2.1. Dynamical Analysis

Let us consider the following algebraic equations:

$$\left\{\begin{array}{cc}{x}_2=0,\hfill & \\ \\ -x_1-x_2{x}_3=0,\hfill & \\ \\ (cosh{x}_2-1)-\gamma cos{x_1}^2-\beta cos{x}_2=0.\hfill \end{array}\right.$$

(2)

Observe that system (1) has a line of equilibria at $(0,0,x_3)$ if $\gamma =\beta =0$, or else it has no equilibrium for either $\gamma =0$ or $\beta =0$. In the case of no equilibrium, the proposed system can be classified as a chaotic system with a hidden strange attractor, a basin of attraction that does not contain neighbourhoods of equilibria.

Additionally, to set the stability of system (1), we consider the Jacobian matrices at the possible equilibrium point, i.e., at the origin $(0,0,0)$ when $\gamma =\beta =0$. The Jacobian matrix of the proposed system at the origin point is given by

$$J_0=\left(\begin{array}{ccc}0& 1& 0\\ \\ -1& -x_2& -x_2\\ \\ 2\gamma x_{1}sin\left(x_1^2\right)& \beta sin{x}_1+sinh\left(x_2\right)& 0\end{array}\right),$$

(3)

and the corresponding characteristic polynomial of $J_S$ is

$$P(J_S,\lambda )=a_3{\lambda }^3+a_1\lambda ,$$

(4)

where $a_3=a_1=-1$, while the eigenvalues of $J_0$ are ${\lambda }_1=0$ and $0\pm i.$ According to the Routh–Hurwitz stability criterion, it is easy to see that system (1) has stable equilibrium. Obviously, in the case of no equilibrium, the proposed system (1) shows a symmetry around the $x_3$-axis, such that $(x_1,x_2,x_3)\to (-x_1,-x_2,x_3)$. The system’s attractors are depicted in Figure 1 with initial data $(1,1,1)$ and $\gamma =0.3$ and $\beta =1.$

When $\gamma =0.3$ and $\beta =1,$ the conservativeness of system (1) can be testified by the divergence as follows:

$$V=\frac{\partial \dot{x_1}}{\partial x_1}+\frac{\partial \dot{x_2}}{\partial x_2}+\frac{\partial \dot{x_3}}{\partial x_3}=-x_3\left(t\right).$$

(5)

By (5), the conservativeness is not clear, since the dissipation is shown by the time-averaged value $-x_3\left(t\right)$ through the trajectory. It was established in [39] that the average value ($\overline{r}=\Delta x_i/\Delta t$) of any function can be computed by

$$\overline{r}\left(t\right)=\underset{t\to \infty }{lim}\left(\frac{{\int }_{t_0}^{t}r\left(t\right)dt}{t-t_0}\right).$$

(6)

One can verify that the average of $x_3\left(t\right)$ of system (1) is zero. Furthermore, the Lyapunov exponents (LEs) of system (1) are $L_1=0.00289,L_2=0,L_3=-0.00289$, with ${\sum }_{j=1}^3{L}_j=0$ (see Figure 2). The Kaplan–Yorke fractional dimension presents the complexity of the attractor and is defined by

$$D_{KY}=j+\frac{{\sum }_{i=1}^j{\lambda }_i}{|{\lambda }_j|},$$

(7)

where j is the largest integer satisfying ${\sum }_{i=1}^j{\lambda }_i\ge 0$ and ${\sum }_{i=1}^{j+1}{\lambda }_i<0$. The corresponding fractional dimension of system (1) is 3. Therefore, this indicates a strange attractor. Thus, the dimension is equivalent to the number of system variables, showing the proposed system’s conservativeness. System (1) is conservative, with dynamics shown in Figure 1.

For $\gamma =0.3$ and $\beta =1,$ system (1) is invariant to the transformation $t\to -t$; hence, it is time-reversible with LEs that are symmetric around zero. Additionally, a positive definite energy function in quadratic form can be provided as the Hamiltonian function:

$$H=\frac{1}{2}\left({x_1}^2+{x_2}^2+{x_3}^2\right).$$

(8)

Considering the time derivative of $H,$ we have

$$\dot{H}=x\dot{x_1}+x_2\dot{x_2}+x_3\dot{x_3}=-x_3{x}_2{x_2}^2+x_3(cosh{x}_2-1)-x_{3}cos\left(x_1^2\right)-x_{3}cos{x}_2,$$

(9)

which demonstrates that, if $\overline{x_3}\left(t\right)$ is zero, the energy of system (1) is invariable. The system is, thus, conservative by nature. It should be noted that any attempt to simplify system (1) will result in a non-conservative system, as illustrated in Table 1.

The bifurcation diagram was obtained by plotting the local maxima of the state variable $x_3\left(t\right)$ when $\beta =1$ and changing the value of $\gamma$ in the interval $[-1,12]$, and, also, when $\gamma =1$ and changing the value of $\beta$ in the interval $[-10,10]$, as shown in Figure 2.

Table 1. Two modified chaotic systems of system (1).

Modified System	LEs	${\mathit{D}}_{\mathit{KY}}$	Phase Portrait
$\left\{\begin{array}{cc}\dot{x_1}=x_2\hfill & \\ \dot{x_2}=-x_1-x_2{x}_3\hfill & \\ \dot{x_3}=(cosh{x}_2-1)-0.3cos{x}_1^2\hfill \end{array}\right.$	$L_1=0.020785$	2.486	Figure 3a
	$L_2=0$
	$L_3=-0.0265$
$\left\{\begin{array}{cc}\dot{x_1}=x_2;\hfill & \\ \dot{x_2}=-x_1-x_2{x}_3\hfill & \\ \dot{x_3}=(cosh{x}_1-0.1)-cos{x}_2\hfill \end{array}\right.$	$L_1=0.020785$	2.498	Figure 3b
	$L_2=0$
	$L_3=-12.405390$

Table 1. Two modified chaotic systems of system (1).

Modified System	LEs	${\mathit{D}}_{\mathit{KY}}$	Phase Portrait
$\left\{\begin{array}{cc}\dot{x_1}=x_2\hfill & \\ \dot{x_2}=-x_1-x_2{x}_3\hfill & \\ \dot{x_3}=(cosh{x}_2-1)-0.3cos{x}_1^2\hfill \end{array}\right.$	$L_1=0.020785$	2.486	Figure 3a
	$L_2=0$
	$L_3=-0.0265$
$\left\{\begin{array}{cc}\dot{x_1}=x_2;\hfill & \\ \dot{x_2}=-x_1-x_2{x}_3\hfill & \\ \dot{x_3}=(cosh{x}_1-0.1)-cos{x}_2\hfill \end{array}\right.$	$L_1=0.020785$	2.498	Figure 3b
	$L_2=0$
	$L_3=-12.405390$

2.2. Coexisting Attractors

To investigate the phenomena of coexisting attractors, we need to check the influence of the parameters and initial conditions on the behaviours of system (1). We have observed that, if $\gamma =0$ or $\beta =0$, the proposed system reveals chaotic dynamics with no equilibrium. Figure 4 displays a few coexisting attractors of system (1). Figure 4a shows a chaotic attractor coexists, whereas Figure 4b shows quasi-periodic coexistence.

Note that, according to the eigenvalues obtained at the origin, we have Hopf-zero equilibrium, but there is no Hopf-zero bifurcation at the origin coordinates (i.e., there is no periodic solution (attractor) that may coexist except the trivial solution at the origin); this fact can be easily proven by the average method.

Figure 1. Phase portrait of the proposed system (1) when $\gamma =1$ and $\beta =0.3$.

Figure 1. Phase portrait of the proposed system (1) when $\gamma =1$ and $\beta =0.3$.

Figure 2. (a) Bifurcation diagram of the novel system (1) when varying parameter $\gamma \in [-2,12]$. (b) Bifurcation diagram of the novel system (1) when varying parameter $\beta \in [-2,20]$. (c) Lyapunov exponents of the system (1) when $\gamma =0.3$ and $\beta =1$.

Figure 2. (a) Bifurcation diagram of the novel system (1) when varying parameter $\gamma \in [-2,12]$. (b) Bifurcation diagram of the novel system (1) when varying parameter $\beta \in [-2,20]$. (c) Lyapunov exponents of the system (1) when $\gamma =0.3$ and $\beta =1$.

Figure 3. Two modified systems of system (1).

Figure 3. Two modified systems of system (1).

Figure 4. (a) A chaotic attractor coexists when $\gamma =1$ and $\beta =0,$ (b) Quasi-periodic coexist when $\gamma =1.9$ and $\beta =4$.

Figure 4. (a) A chaotic attractor coexists when $\gamma =1$ and $\beta =0,$ (b) Quasi-periodic coexist when $\gamma =1.9$ and $\beta =4$.

3. Modified Diffie–Hellman Key Exchange Protocol (MDHKEP)

Diffie–Hellman key exchange agreement was established by Diffie and Hellman in the 1970s [40]. A modification of the protocol based on the original one was introduced as follows. Assume two users A and B decide to communicate over an insecure channel. They should first choose the two public parameters shown below:

• p—a large prime number;
• G = $\left(\begin{array}{cc}{G}_{11}& G_{12}\\ G_{21}& G_{22}\end{array}\right)$$\in M_{2\times 2}\left(F_p\right)$—a random matrix, such that $1\le G_{ij}\le p-1$ and $i,j=1,2$.

Thereafter, User A randomly picks his private key $d_A$ from the interval $[1,p-1]$ and calculates his public key as follows:

$$P_A=G^{d_A}={\left(\begin{array}{cc}{G}_{11}& G_{12}\\ G_{21}& G_{22}\end{array}\right)}^{d_A}modp.$$

(10)

User B also randomly picks his private key $d_B\in [1,p-1]$ and calculates his public key as follows:

$$P_B=G^{d_B}={\left(\begin{array}{cc}{G}_{11}& G_{12}\\ G_{21}& G_{22}\end{array}\right)}^{d_B}modp.$$

(11)

By doing so, users A and B will be able to successfully exchange the parameters required to compute the initial shared key, as shown below:

$$K_1={\left(P_B\right)}^{d_A}={\left({\left(\begin{array}{cc}{G}_{11}& G_{12}\\ G_{21}& G_{22}\end{array}\right)}^{d_B}\right)}^{d_A}={\left(\begin{array}{cc}{G}_{11}& G_{12}\\ G_{21}& G_{22}\end{array}\right)}^{d_B{d}_A}modp=\left(\begin{array}{cc}{k}_{11}& k_{12}\\ k_{21}& k_{22}\end{array}\right)$$

(12)

$$K_2={\left(P_A\right)}^{d_B}={\left({\left(\begin{array}{cc}{G}_{11}& G_{12}\\ G_{21}& G_{22}\end{array}\right)}^{d_A}\right)}^{d_B}={\left(\begin{array}{cc}{G}_{11}& G_{12}\\ G_{21}& G_{22}\end{array}\right)}^{d_A{d}_B}modp=\left(\begin{array}{cc}{k}_{11}& k_{12}\\ k_{21}& k_{22}\end{array}\right)$$

(13)

Next, users A and B create the final shared secret key matrix $K_m$ by employing the suggested technique in Acharya et al. [41], where each user can build his $4\times 4$ self-invertible key matrix $K_m$. Let

$$K_m=\left(\begin{array}{cccc}{k}_{11}& k_{12}& k_{13}& k_{14}\\ k_{21}& k_{22}& k_{23}& k_{24}\\ k_{31}& k_{32}& k_{33}& k_{34}\\ k_{41}& k_{42}& k_{43}& k_{44}\end{array}\right)$$

(14)

be a self-invertible matrix that can be partitioned as

$$K_m=\left(\begin{array}{cc}{K}_{11}& K_{12}\\ K_{21}& K_{22}\end{array}\right)$$

(15)

such that

$$K_{11}=\left(\begin{array}{cc}{k}_{11}& k_{12}\\ k_{21}& k_{22}\end{array}\right)mod256,$$

(16)

$$K_{12}=\left(\begin{array}{cc}{k}_{13}& k_{14}\\ k_{23}& k_{24}\end{array}\right),$$

(17)

$$K_{21}=\left(\begin{array}{cc}{k}_{31}& k_{32}\\ k_{41}& k_{42}\end{array}\right),$$

(18)

$$K_{22}=\left(\begin{array}{cc}{k}_{33}& k_{34}\\ k_{43}& k_{44}\end{array}\right).$$

(19)

The value of each $K_m$ entry (i.e., $K_{12}$, $K_{21}$, and $K_{22}$) can be found by solving the following equations: $K_{12}=(I_2-K_{11})mod256$, $K_{21}=(I_2+K_{11})mod256$, and $K_{22}=-K_{11}mod256$, where $I_2$ is the identity matrix.

4. Methodology of the Proposed Encryption Algorithm

We divide the input colour image of size $M\times N\times 3$ into three channels of size $M\times N$. We then separately encrypt each channel. The proposed encryption technique is built on three major phases. The first stage is to construct the chaotic sequences. The original colour image is confused in the second stage, followed by diffusion of the confused image. Figure 5 demonstrates how the colour image “Peppers” from the input was divided into three greyscale images. The chaotic sequences are then generated using these images. These sequences are then used to scramble these images and diffuse them using the key matrix $K_m$. Finally, the cyphered colour image is formed by merging the three cyphered images. This section also depicts the decryption procedure to recover the input colour image. The suggested image encryption/decryption algorithm is shown in a diagram in Figure 6.

4.1. Creating a Chaotic Sequence

The chaotic system’s initial conditions, as stated in Equation (1), depend on the image that will be transformed to vector $\omega$. The initial conditions can be determined by

$$x_1=\frac{{\sum }_{i=1}^{MN}\omega \left(i\right)+MN}{(2^{23}+MN)}$$

(20)

$$x_i=mod({10}^7{x}_{i-1},1),i=2,3,$$

(21)

where the length of the image vector $\omega$ is indicated by $MN$ and the modulus after division is denoted by the $mod(·)$. Next, iterate the chaotic system (1) to obtain the sequence $\delta$ and select the sequences $(x_1$ and $x_3$). After that, sort $\delta$ ascendantly and return the position of the sorted pixels as a vector $\theta$ of size $MN$.

4.2. Image Confusion

Image confusion is the process of shifting the location of pixels without affecting their value. The sequence $\delta$ in this method scrambles the image vector $\omega$, which is defined as

$$\rho =\omega \left(\theta \right),$$

(22)

where $\rho$ is the scrambled vector created by replacing the pixel location in $\omega$ with the values in the vector $\theta$.

Figure 5. The colour image encryption flowchart of the proposed algorithm.

Figure 5. The colour image encryption flowchart of the proposed algorithm.

Figure 6. The flowchart of the encryption and decryption in the suggested algorithm.

Figure 6. The flowchart of the encryption and decryption in the suggested algorithm.

4.3. Image Diffusion

The invertible key matrix $K_m$ is utilised to modify the image pixel values as follows:

• The vector $\theta$ is reshaped into a matrix $\mu$ with size $MN$;
• The matrix $\mu$ is split into sub-matrices of size $4\times 4$;
• Each sub-matrix then is multiplied by the invertible key matrix $K_m$ to obtain the cypher image C as follows:

$$\left(\begin{array}{cccc}{C}_{i,j}& C_{i,j+1}& C_{i,j+2}& C_{i,j+3}\\ C_{i+1,j}& C_{i+1,j+1}& C_{i+1,j+2}& C_{i+1,j+3}\\ C_{i+2,j}& C_{i+2,j+1}& C_{i+2,j+2}& C_{i+2,j+3}\\ C_{i+3,j}& C_{i+3,j+1}& C_{i+3,j+2}& C_{i+3,j+3}\end{array}\right)=$$

$$\left(\begin{array}{cccc}{\mu }_{i,j}& {\mu }_{i,j+1}& {\mu }_{i,j+2}& {\mu }_{i,j+3}\\ {\mu }_{i+1,j}& {\mu }_{i+1,j+1}& {\mu }_{i+1,j+2}& {\mu }_{i+1,j+3}\\ {\mu }_{i+2,j}& {\mu }_{i+2,j+1}& {\mu }_{i+2,j+2}& {\mu }_{i+2,j+3}\\ {\mu }_{i+3,j}& {\mu }_{i+3,j+1}& {\mu }_{i+3,j+2}& {\mu }_{i+3,j+3}\end{array}\right)\times \left(\begin{array}{cccc}{k}_{11}& k_{12}& k_{13}& k_{14}\\ k_{21}& k_{22}& k_{23}& k_{24}\\ k_{31}& k_{32}& k_{33}& k_{34}\\ k_{41}& k_{42}& k_{43}& k_{44}\end{array}\right)mod\left(256\right)$$

(23)

where $i=1:4:M,j=1:4:N.$

4.4. The Decryption Algorithm

The decryption process uses the same set of keys and proceeds in the opposite order of the encryption algorithm’s steps, as shown in Figure 6.

5. Experiments and Performance Analysis

The trial simulation setup consisted of a computer with a Windows 11 operating system, 8 GB of RAM, a 1.30 GHz central processing unit, and Matlab R2018A. In these experiments, we used Airplane, Baboon, Lena, Peppers, and Tree images with sizes of $512\times 512$ and $256\times 256$ as the plaintext images.

5.1. Encryption and Decryption Results

The encryption and decryption results for the proposed method are displayed in Figure 7. As shown, the decrypted images were similar to the original image, whilst the cypher images were noise-like and unidentifiable, which shows that the proposed approach has effective encryption and decryption outcomes. We next checked the image encryption scheme against some possible assaults.

5.2. Statistical Analysis

In this section, we show results from checking the following statistical indices:

I.

Histogram and Chi-Square Test

To visually describe the distribution of pixel intensities in the image, we need to consider the image histogram that gives a graphic view. The fact that pixel intensities are constrained to a given range causes histograms for original images to often be inconsistent. Thus, cryptanalysts can leverage this characteristic to intercept the encryption by utilising histogram-based attacks. Figure 8 depicts the colour image, Peppers, and histograms of red, green, and blue channels that indicate the encrypted analogue of the original image. The original image has non-uniform histograms, but the encrypted image has highly uniform distributions of pixel intensities in all channels, which shows the suggested method can withstand statistical attack. The Chi-square test was employed to determine the homogeneity of a histogram and is calculated via the following equation:

$${\chi }^2=\sum _{i=1}^{256}\frac{{(V_i-g)}^2}{g},$$

(24)

where $V_i$ represents the grey recurrence value of i and $g=\frac{V}{256}.$ The value of ${\chi }^2(\alpha ,d)$ was 293.2478, with $\alpha =0.05$ as the significance level and $d=255$ as the degree of freedom [22].

Table 2. Chi-square test results.

Images	Size	R	G	B
Airplane	$256\times 256$	246.7344	211.7890	323.0156
Baboon	$256\times 256$	236.4766	292.7109	245.2656
Lena	$256\times 256$	285.3906	251.5547	236.9609
Peppers	$256\times 256$	237.2813	292.7109	242.5313
Tree	$256\times 256$	248	214.8516	230.0938

shows the ${\chi }^2$ values of the cyphered image. Since all the values were less than 293, the histogram of the images encrypted with our suggested approach was uniform.

II.

Entropy

One of the most significant measures in dynamical systems theory and security is entropy, which is a statistical test for calculating randomness in an image, defined as

$$H\left(m\right)=-\sum _{i=1}^{T}P\left(m_i\right){log}_{2}P\left(m_i\right),$$

(25)

where $P\left(m_i\right)$ is the probability of the element $m_i$, and T represents the total number of elements $m_i$. For an encrypted image, eight is the ideal value of entropy [16]. In this paper, we used the suggested approach to encrypt colour images and determine the entropy values of the cyphered images, as shown in

Table 3. The values of entropy for various colour images in three channels.

Images	Size	R	G	B
Airplane	$512\times 512$	7.9993	7.9992	7.9993
Baboon	$512\times 512$	7.9991	7.9993	7.9992
Lena	$512\times 512$	7.9992	7.9993	7.9993
Peppers	$512\times 512$	7.9993	7.9994	7.9992
Tree	$512\times 512$	7.9993	7.9993	7.9994
Airplane	$256\times 256$	7.9973	7.9977	7.9964
Baboon	$256\times 256$	7.9974	7.9968	7.9973
Lena	$256\times 256$	7.9969	7.9972	7.9974
Peppers	$256\times 256$	7.9974	7.9968	7.9973
Tree	$256\times 256$	7.9973	7.9976	7.9975

. In a second experiment, we used our technique and some other existing methods [2,9,12,18,23] to encrypt Lena’s colour image.

Table 4. Comparison of the entropy of the Lena image in three channels.

Scheme	Size	R	G	B	Average
Proposed	$256\times 256$	7.9969	7.9972	7.9974	7.9972
[2]	$256\times 256$	7.9899	7.9890	7.9894	7.9894
[9]	$256\times 256$	7.9948	7.9958	7.9950	7.9952
[12]	$256\times 256$	7.9974	7.9970	7.9969	7.9971
[18]	$256\times 256$	7.9972	7.9972	7.9971	7.9972
[23]	$256\times 256$	7.9972	7.9965	7.9963	7.9967

displays the computed entropy values. In general, the average entropy for the cyphered image produced by our approach beat the outcomes of the other techniques [2,9,12,23] or was similar to the outcomes of the other techniques [18]. The results indicate that the proposed method has higher unpredictability of image information compared to other existing methods, as Table 4 shows.

5.3. Keyspace Analysis

For a good encryption scheme, the keyspace must be large enough to withstand brute-force assaults. The suggested algorithm’s key consists of the starting values $x_1, x_2$, and $x_3$, as well as the parameters $\gamma$ and $\beta$. The keyspace size can reach ${10}^{70}\simeq 2^{233}$ if the accuracy is ${10}^{-14}$. The algorithm also uses the keys $d_A,d_B,P_A,P_B,K_1$, and $K_m$. The size of these keys depends on the value of p in $F_p$, where the keys $d_A,d_B$ are of size $\tau$, whereas those for $P_A$ and $P_B$ are of size $4\tau$. The shared secret key $K_m$ has a size of 128 bits, since it is a $4\times 4$ matrix whose elements of 8 bits belong to the field $F_p$. The keyspace analysis for the suggested algorithm may, thus, be determined as the following: $2^{\tau +\tau +4\tau +4\tau +128+233}=2^{10\tau +361}\ge 2^{128}$. Attackers who select a 16-bit value for p will need to carry out $2^{10\left(16\right)+361}=2^{521}$ operations to defeat the method, which is robust enough to withstand brute-force attacks.

Table 5. Comparison of keyspace values.

Scheme	Proposed	[2]	[9]	[18]	[23]	[34]
Keyspace	$2^{521}$	$2^{470}$	$2^{448}$	$2^{128}$	$2^{478}$	$2^{219}$

compares the obtained keyspace of the suggested image encryption technique with the literature’s techniques. The findings in Table 5 demonstrate that the suggested technique has a larger keyspace than existing methods [2,9,12,18,23].

5.4. Key Sensitively Test

A robust image encryption scheme must be sensitive to even minor changes in the initial conditions of the secret key employed. The recovered image becomes noisy and scrambled when this key is slightly changed. The private key parameters were ($p=5003$, $G=\left(\begin{array}{cc}6& 592\\ 245& 2079\end{array}\right)$, $d_A=3$, $d_B=59$, $x_1=1.169208378754845$, $x_2=0.903830949650254$, $x_3=0.949650254333392$). The plain image of Peppers was encrypted in this experiment using the private key, and the result is displayed in Figure 9(a1). The original image was not recovered as in Figure 9(a2) when the cyphered image was decrypted in Figure 9(a1) by altering $x_2$ (one of the private keys) to $x_2+{10}^{-12}$ while keeping the other parameters the same as above. The original image was only recovered with the correct private key, as shown in Figure 9(a3).

5.5. Anti-Differential Attack

The ability to withstand differential attack is a crucial measure of plaintext sensitivity. Attackers typically alter a plain image’s pixels slightly and attempt to determine differences between the original encrypted image and the modified encrypted image to find a weakness in the encryption algorithms, such as attaining information about the secret key. Two measures, denoted by NPCR and UACI, can be used to test the ability to resist differential attacks. The equations of NPCR and UACI are as follows:

$$\mathrm{NPCR}=\frac{1}{MN}\sum _{i=1}^M\sum _{j=1}^{N}dist(i,j)\times 100\%,$$

(26)

and

$$\mathrm{UACI}=\frac{1}{255\times MN}\sum _{i=1}^M\sum _{j=1}^N|C_1(i,j)-C_2(i,j)|\times 100\%,$$

(27)

where $C_1(i,j)$ and $C_2(i,j)$ are the cypher images created by two original images with a one-pixel difference, and $dist(i,j)$ is mathematically expressed by

$$dist(i,j)=\left\{\begin{array}{cc}0,if{C}_1(i,j)=C_2(i,j);\hfill & \\ \\ 1,if{C}_1(i,j)\ne C_2(i,j).\hfill \end{array}\right.$$

(28)

To guarantee the security of the algorithm, the UACI value must be near $0.3340$, and NPCR must be near $0.9960$.

Table 6. UACI and NPCR test results for various colour images in three channels.

Images	Size	UACI			NPCR
		R	G	B	R	G	B
Airplane	$512\times 512$	0.3351	0.3333	0.3353	0.9962	0.9959	0.9961
Baboon	$512\times 512$	0.3348	0.3346	0.3348	0.9960	0.9961	0.9960
Lena	$512\times 512$	0.3346	0.3343	0.3354	0.9960	0.9960	0.9964
Peppers	$512\times 512$	0.3340	0.3352	0.3341	0.9959	0.9961	0.9962
Tree	$512\times 512$	0.3340	0.3342	0.3335	0.9960	0.9960	0.9962
Airplane	$256\times 256$	0.3351	0.3358	0.3349	0.9963	0.9964	0.9960
Baboon	$256\times 256$	0.3329	0.3354	0.3348	0.9964	0.9964	0.9959
Lena	$256\times 256$	0.3343	0.3353	0.3342	0.9961	0.9963	0.9967
Peppers	$256\times 256$	0.3352	0.3354	0.3371	0.9961	0.9964	0.9962
Tree	$256\times 256$	0.3341	0.3347	0.3334	0.9961	0.9961	0.9961

and

Table 7. UACI and NPCR of encrypted Lena image with size $256\times 256$ in three channels.

Scheme	UACI				NPCR
	R	G	B	Average	R	G	B	Average
Proposed	0.3343	0.3353	0.3342	0.3346	0.9961	0.9963	0.9967	0.9964
[2]	0.3340	0.3367	0.3361	0.3356	0.9965	0.9962	0.9961	0.9963
[9]	0.3335	0.3343	0.3341	0.3340	0.9960	0.9961	0.9961	0.9961
[12]	-	-	-		-	-	-	-
[18]	0.3333	0.3346	0.3341	0.3340	0.9965	0.9975	0.9965	0.9968
[23]	0.3307	0.3076	0.2787	0.3057	0.9962	0.9962	0.9962	0.9962

display the comparative results of UACI and NPCR with respect to other methods [2,9,12,18,23]. The average values of UACI and NPCR for the cyphered image Lena outperformed the techniques [9,12,18,23] in terms of UACI and outperformed the techniques [2,9,12,23] in terms of NPCR. The results reveal that UACI and NPCR average values were perfect, so that the proposed approach can effectively survive differential attacks.

5.6. Noise Attack

When images are transmitted over media channels, they are often subjected to noise. If the cyphered image is exposed to noise, it may influence the image quality after decryption. We contaminated the cypher image “Peppers” of size $512\times 512$ with salt and pepper noise with densities $10\%,20\%,$ and $30\%$; the analogous decrypted images are shown in Figure 10a–a2. From the below figures, one can see that these images were noisy but readable.

5.7. Correlation Analysis

Let $N_p$ represent the neighbouring pixels. In the usual images, $N_p$ values are very close to each other, which means neighbouring pixels are highly correlated in the original images. This characteristic may be exploited by attackers to break the cypher scheme. Thus, $N_p$ have to be highly uncorrelated in the encrypted images. The horizontal, vertical, and diagonal correlation coefficient between any two adjacent pixels, x and y, is calculated by

$$C_{x,y}=\frac{{\sum }_{i=1}^{K_t}(x_i-E\left(x\right))(y_i-E\left(y\right))}{\sqrt{{\sum }_{i=1}^{K_t}{(x_i-E\left(x\right))}^2}\sqrt{{\sum }_{i=1}^{K_t}{(y_i-E\left(y\right))}^2}}$$

(29)

where $E(.)$ indicates the expected values of the random variables, $x_i$ and $y_i$ represent greyscale values of $N_p$, and $K_t$ denotes the total number of pixels involved in the computations.

In Figure 11, one can observe that the coefficients of correlation in the cypher images in all three (horizontal, vertical, and diagonal) directions were almost zero. Therefore, the connected pixels in the cyphered images were highly uncorrelated. The findings displayed in Table 8 show that the correlation values of the various colour images were all close to 0, indicating that the proposed method cracked the strong correlation between neighbouring pixels in all tested images. Table 9 shows a comparison of correlation values of the suggested encryption method and existing encryption methods. One can see that our technique’s correlation values were closer to zero, indicating that the proposed method has superior protection against statistical attacks.

Table 8. Correlation coefficient test results of different encrypted images in three channels.

Image	Size	Directions	R	G	B
Airplane	$512\times 512$	H	0.0032	−0.0011	0.0010
		V	0.0035	0.0006	−0.0008
		D	−0.0002	0.0033	0.0018
Baboon	$512\times 512$	H	−0.0018	−0.00007	0.0020
		V	−0.0004	−0.00100	−0.0008
		D	0.0016	−0.00030	0.0031
Lena	$512\times 512$	H	−0.00006	−0.0003	−0.0018
		V	0.00150	−0.0020	−0.0011
		D	0.00010	−0.0011	0.0021
Peppers	$512\times 512$	H	−0.0004	0.0016	0.0025
		V	0.0006	0.0032	0.0016
		D	−0.0022	0.0006	0.0043
Tree	$512\times 512$	H	0.0002	−0.0010	−0.0018
		V	0.0014	0.0025	0.0008
		D	−0.0006	−0.0002	−0.0029
Airplane	$256\times 256$	H	−0.0067	−0.0024	0.0062
		V	0.0057	−0.0047	−0.0007
		D	−0.0004	−0.0079	−0.0002
Baboon	$256\times 256$	H	0.0020	−0.0072	0.0043
		V	−0.0037	−0.0044	0.0039
		D	−0.0043	0.0062	0.0012
Lena	$256\times 256$	H	−0.00004	0.0004	−0.0027
		V	−0.00370	−0.0044	0.0039
		D	0.00400	0.0060	−0.0002
Peppers	$256\times 256$	H	−0.0014	−0.0072	0.0048
		V	0.0022	0.0070	−0.0010
		D	0.0020	0.0062	−0.0039
Tree	$256\times 256$	H	0.0051	−0.0017	−0.0002
		V	−0.0055	0.0085	−0.0014
		D	0.0053	0.0021	−0.0067

Table 8. Correlation coefficient test results of different encrypted images in three channels.

Image	Size	Directions	R	G	B
Airplane	$512\times 512$	H	0.0032	−0.0011	0.0010
		V	0.0035	0.0006	−0.0008
		D	−0.0002	0.0033	0.0018
Baboon	$512\times 512$	H	−0.0018	−0.00007	0.0020
		V	−0.0004	−0.00100	−0.0008
		D	0.0016	−0.00030	0.0031
Lena	$512\times 512$	H	−0.00006	−0.0003	−0.0018
		V	0.00150	−0.0020	−0.0011
		D	0.00010	−0.0011	0.0021
Peppers	$512\times 512$	H	−0.0004	0.0016	0.0025
		V	0.0006	0.0032	0.0016
		D	−0.0022	0.0006	0.0043
Tree	$512\times 512$	H	0.0002	−0.0010	−0.0018
		V	0.0014	0.0025	0.0008
		D	−0.0006	−0.0002	−0.0029
Airplane	$256\times 256$	H	−0.0067	−0.0024	0.0062
		V	0.0057	−0.0047	−0.0007
		D	−0.0004	−0.0079	−0.0002
Baboon	$256\times 256$	H	0.0020	−0.0072	0.0043
		V	−0.0037	−0.0044	0.0039
		D	−0.0043	0.0062	0.0012
Lena	$256\times 256$	H	−0.00004	0.0004	−0.0027
		V	−0.00370	−0.0044	0.0039
		D	0.00400	0.0060	−0.0002
Peppers	$256\times 256$	H	−0.0014	−0.0072	0.0048
		V	0.0022	0.0070	−0.0010
		D	0.0020	0.0062	−0.0039
Tree	$256\times 256$	H	0.0051	−0.0017	−0.0002
		V	−0.0055	0.0085	−0.0014
		D	0.0053	0.0021	−0.0067

Table 9. Comparison of correlation coefficients of encrypted Lena image in three channels.

Scheme	Size	Directions	R	G	B
Proposed	$256\times 256$	H	−0.00004	0.0004	−0.0027
		V	−0.00370	−0.0044	0.0039
		D	0.00400	0.0060	−0.0002
[2]	$256\times 256$	H	0.00040	−0.0050	−0.0103
		V	−0.00140	−0.0144	0.0010
		D	0.00200	0.0073	−0.0040
[9]	$256\times 256$	H	−0.00200	−0.0013	−0.0059
		V	−0.00220	−0.0041	0.0014
		D	0.00690	0.0059	0.0035
[12]	$256\times 256$	H	0.00080	0.0002	0.0274
		V	0.00130	−0.0001	−0.0002
		D	0.00060	−0.0048	−0.0055
[18]	$256\times 256$	H	0.00090	0.0007	0.0007
		V	−0.00090	−0.0009	−0.0007
		V	0.00170	0.0009	0.0009
[23]	$256\times 256$	H	0.00073	0.0031	−0.0051
		V	−0.00054	0.0008	0.0033
		D	0.00147	−0.0015	−0.0062

Table 9. Comparison of correlation coefficients of encrypted Lena image in three channels.

Scheme	Size	Directions	R	G	B
Proposed	$256\times 256$	H	−0.00004	0.0004	−0.0027
		V	−0.00370	−0.0044	0.0039
		D	0.00400	0.0060	−0.0002
[2]	$256\times 256$	H	0.00040	−0.0050	−0.0103
		V	−0.00140	−0.0144	0.0010
		D	0.00200	0.0073	−0.0040
[9]	$256\times 256$	H	−0.00200	−0.0013	−0.0059
		V	−0.00220	−0.0041	0.0014
		D	0.00690	0.0059	0.0035
[12]	$256\times 256$	H	0.00080	0.0002	0.0274
		V	0.00130	−0.0001	−0.0002
		D	0.00060	−0.0048	−0.0055
[18]	$256\times 256$	H	0.00090	0.0007	0.0007
		V	−0.00090	−0.0009	−0.0007
		V	0.00170	0.0009	0.0009
[23]	$256\times 256$	H	0.00073	0.0031	−0.0051
		V	−0.00054	0.0008	0.0033
		D	0.00147	−0.0015	−0.0062

6. Conclusions

This paper presents a novel three-dimensional conservative chaotic system that involves a hyperbolic function and can behave chaotically. Lyapunov exponents and bifurcation diagrams were computed to determine the system’s dynamical characteristics, which are crucial to employ such a system in image encryption. The suggested technique was utilised in this three-stage encryption process to modify the location of the image pixels. The colour image is diffused and separated into a set of $4\times 4$ blocks, and then each block’s pixel values are changed using a self-invertible matrix of size $4\times 4$ created using MDHKEP. The novel encryption technique demonstrated excellent sensitivity to even small changes in the distribution of pixels, so that a completely different encrypted image was formed. The results of this study guarantee the suggested algorithm’s ability to withstand all common assaults.