Quantifying Wildlife Abundance: Negative Rayleigh Modeling of Line Transect Data

Abstract

This study introduces a negative Rayleigh detection model for estimating population abundance in line transect surveys. The model satisfies key detection conditions and provides a detailed analysis of its probability density function, moments, and other essential characteristics. Parameters are estimated using three methods: moment estimator, maximum likelihood estimator, and Bayesian estimator. The model’s performance is evaluated through simulations, comparing its estimators to those from established models. An empirical application using perpendicular distance data further assesses the model, with goodness-of-fit statistics demonstrating its advantages over traditional methods.

1. Introduction

Quantifying wildlife abundance is a critical aspect of ecological and conservation research, providing essential data for managing and protecting species. Accurate estimates of wildlife populations are crucial for understanding species distribution, assessing the impact of environmental changes, and informing conservation strategies. Among the various methods used to estimate wildlife abundance, line transect sampling is widely recognized for its efficiency and reliability. Line transect sampling is a widely used method in ecology for estimating the abundance or density of organisms within a particular habitat. The method involves laying a transect line (usually a long, straight line) through the habitat and recording the distance and location of all individuals or groups of individuals observed along the line. The data collected during line transect sampling can be used to estimate the density or abundance of the population being studied, using a variety of statistical methods. These methods typically involve estimating the probability of detection for each individual or group observed along the transect line and using this probability to correct the observed counts for incomplete detection. Line transect sampling can be conducted using a variety of techniques, including walking the transect line, driving along the transect line, or flying over the habitat in an aircraft or drone. The method can be used to study a wide range of organisms, including plants, insects, birds, mammals, and fish.

One of the distinct advantages of line transect sampling is its capability to efficiently collect data on a large number of individuals or groups within a relatively short period of time. This method provides valuable insights into population densities and distributions across different habitats. However, like any sampling technique, line transect sampling comes with its inherent limitations that must be considered. A primary limitation is the requirement for a straight transect line that accurately represents the habitat being studied. Deviations from this ideal can introduce biases in population estimates, particularly if habitats are irregular or fragmented. Moreover, the method’s reliance on linearity may overlook spatial complexities and variations in habitat structure that influence species distributions. Observer bias is another critical consideration. If the same observer collects all the data, their individual perception and experience can inadvertently influence data collection, potentially leading to biased estimates. Standardizing observer protocols and employing multiple observers can help mitigate this issue. Furthermore, line transect sampling may underestimate the abundance or density of certain organisms. Some species may be cryptic or have behaviors that make them difficult to detect along the transect line, especially if they occur in low densities or blend into their surroundings. This detection limitation can skew population estimates, particularly for rare or elusive species. Despite these challenges, line transect sampling remains a valuable tool in ecological research, providing quantitative data that contribute to conservation efforts, habitat management, and species monitoring. Addressing these limitations through rigorous study design, standardized protocols, and complementary sampling methods can enhance the reliability and applicability of line transect data in ecological studies.

The detection function, also known as the conditional function $g\left(x\right)$, is used by Buckland et al. [1] to express the model mathematically. The definition of the detection function is:

$$g\left(x\right)=\mathrm{P}\left(\mathrm{observing}\mathrm{an}\mathrm{object}\mathrm{at}\mathrm{a}\mathrm{random}\mathrm{location}\mathrm{given}\mathrm{its}\mathrm{perpendicular}\mathrm{distance}\mathrm{is}x\right)$$

This truncation could happen either before or after the survey. The distance beyond which no additional observations are recorded is known as the greatest perpendicular distance, or w. To evaluate the effectiveness of a detection model, several assumptions are made regarding the function $g\left(x\right)$ (see Buckland et al. [1] and Miller and Thomas [2]). It is expected that as the distance x increases, the probability $g\left(x\right)$ of detecting an object randomly decreases monotonically, reflecting the higher likelihood of missing animals further away from the transect line. Mathematically, if $x_1>x_2$, then $g\left(x_2\right)>g\left(x_1\right)$. Additionally, items that are exactly on the transect path will always be visible (i.e., $g\left(0\right)=1$), which denotes perfect detection along the transect path. The detection probability should also approach one as the distance approaches zero, indicating high detection probability in close proximity (see Eberhardt [3]). Since the slope at $x=0$ is zero (i.e., $g^{\prime }\left(0\right)=0$), indicating a flat slope at zero distance, the tangent to the function is a horizontal line at zero distance. Consequently, these requirements are satisfied by the shape of $g\left(x\right)$. The shoulder conditions are the set of presumptions that any detection model needs to take into account. The following traits are used by Buckland [4] and Buckland et al. [5] to characterize line transect sampling:

• N items are dispersed at random throughout area A of the study zone, with an abundance of $D=N/A$, the average population density.
• The likelihood that an object is spotted on the line is 1, hence $g\left(0\right)=1$.
• In the first place of sighting, objects are found.
• There is only one count per item.
• Perpendicular distances are measured carefully and accurately.
• The distribution of objects occurs independently of the line.
• There will be a few, possibly a lot, of missed objects.

Researchers Burnham and Anderson [6] and Seber [7] found that the following expression was correlated with the population density D of objects in a given area:

$$D={\displaystyle \frac{E\left(n\right)f\left(0\right)}{2L}}.$$

(1)

In this case, $f\left(0\right)$ is a parameter that guarantees that $f\left(x\right)$, for $0\le x\le w$, is a probability density function (PDF) for the perpendicular distance data. L is the length of the transect line. $E\left(n\right)$ is the anticipated value of detections. $f\left(x\right)$, which is dependent on an unknown parameter and belongs to a known family of functional forms, is estimated using the perpendicular distances (see Burnham and Anderson [6]). The following describes the relationship between this PDF and $g\left(x\right)$:

$$f\left(x\right)={\displaystyle \frac{g\left(x\right)}{\mu }},$$

(2)

where the normalization factor $\mu ={\int }_0^{w}g\left(x\right)dx$ ensures that $f\left(x\right)$ is scaled appropriately. If and only if $f^{\prime }\left(0\right)=0$, then $f\left(x\right)$ satisfies the shoulder criteria, according to Eberhardt [8]. It is easy to confirm this equivalency. Thus, $f\left(x\right)$ has the same form as $g\left(x\right)$ but is scaled to have an area under the curve equal to one. It follows a monotonically declining trend. In relation to the suggested model for $f\left(x\right)$, this scaling is essential for a precise estimation of $f\left(0\right)$ (see Crain et al. [9]).

Other techniques that can be used to determine the population density D include:

$$\widehat{D}={\displaystyle \frac{n\widehat{f}\left(0\right)}{2L}}.$$

(3)

In this research, we assume $w=\infty$ for simplicity’s sake. Nevertheless, w can be substituted with a big enough value for an accurate approximation in real-world scenarios. The total number of objects detected is indicated by the parameter n, and the appropriate estimate for $f\left(0\right)$ is $\widehat{f}\left(0\right)$. In the literature, numerous parametric and nonparametric techniques for estimating $f\left(0\right)$ have been suggested. See the works of Burnham et al. [10], Quinn and Gallucci [11], and Pollock [12] for additional developments in this methodology. The parametric approach is the main emphasis of this study. Specifically, the maximum likelihood technique is used to estimate parameters within the parametric detection model, which allows the estimator for $f\left(0\right)$ and the population abundance parameter D to be estimated.

The article’s remaining sections are arranged as follows: In Section 2, we suggest a negative Rayleigh detection model (NRDM) that meets the shoulder criteria and construct the probability density function (PDF) that goes along with it. This section also introduces some notable features of the suggested model. Section 3 explores the moments of the NRDM and discusses related measures associated with it. Section 4 focuses on the maximum likelihood estimations of the detection model parameters and presents the corresponding confidence intervals. In Section 5, we employ Mathematica 10 for simulation purposes. We provide plots and graphs to visually examine the performance of the estimated parameters. The simulation allows us to calculate the population abundance along with its confidence interval. A practical dataset of perpendicular distance is also utilized to illustrate the utility of the NRDM. Under the suggested model, we obtain pertinent measurements for this collection of data. Finally, in Section 6, based on the study’s findings, we give our concluding thoughts and insights.

2. The Proposed Detection Model

In fact, a less accurate estimator may result from the one-parameter model $g\left(x\right)$’s potential inability to accurately represent the true form of the detection curve. We provide an enhanced detection function for the negative Rayleigh detection model (NRDM) in order to overcome this constraint.

The following detection function defines the NRDM:

$$g\left(x\right)=g(x;u)=e^{-{\displaystyle \frac{x^2}{2u^2}}},x>0,u>0.$$

(4)

The detection function $g\left(x\right)$ can be utilized to characterize the detection probability along the transect path if it satisfies the shoulder criteria. An object is detected with certainty and probability one, or $g\left(0\right)=1$, when it is immediately on the transect route.

The detection function’s first derivative with regard to x looks like this:

$$g^{\prime }\left(x\right)=g^{\prime }(x;u)=-{\displaystyle \frac{x}{u^2}}{e}^{-{\displaystyle \frac{x^2}{2u^2}}}.$$

(5)

A more versatile model for determining the detection probability is the one-parameter NRDM, u. The upcoming sections of the paper delve into the specifics of this model, encompassing its attributes and techniques for estimation. This implies that for $u>1$, $g^{\prime }\left(0\right)=0$. Given that $g\left(x\right)$ monotonically decreases for each $x\in (0,\infty )$, these shoulder requirements are further supported by a variety of forms (see Figure 1).

The matching PDF of the NRDM is derived as a normalized $g\left(x\right)$.

$$f\left(x\right)=f(x;u)={\displaystyle \frac{\sqrt{\frac{2}{\pi }}}{u}}{e}^{-{\displaystyle \frac{x^2}{2u^2}}}.$$

(6)

The PDF plots for various parametric values are displayed in Figure 2. Equation (6)’s first derivative with respect to x is

$$f^{\prime }\left(x\right)=-{\displaystyle \frac{x\sqrt{\frac{2}{\pi }}{e}^{-\frac{x^2}{2u^2}}}{u^3}}.$$

(7)

Indeed, expression (7) shows a direct proportionality between $g\left(x\right)$ and $f\left(x\right)$. As a result, $f\left(x\right)$ and $g\left(x\right)$ exhibit similar characteristics, including $f^{\prime }\left(0\right)=0$ for $u>1$ and a monotone decreasing shape. This implies that the proposed model can serve as a reliable estimator for $f\left(0\right)$, satisfying the “Shape Criterion” as described by Burnham et al. [10]. The reliability of the model is demonstrated in Figure 1 and Figure 2, which highlight the wide range of shapes that the detection function $g\left(x\right)$ can take, as it adjusts $f\left(x\right)$ based on the parameter u. Furthermore, it is worth noting that all plots exhibit a progressive decay to zero as $x\to \infty$, a desirable characteristic for the considered detection model.

Now, let us talk about some NRDM statistical data. The parameter $f\left(0\right)$ of Equation (6) is first obtained by

$$f\left(0\right)={\displaystyle \frac{\sqrt{\frac{2}{\pi }}}{u}}$$

(8)

and the population parameter D is

$$D={\displaystyle \frac{nf\left(0\right)}{2L}}={\displaystyle \frac{n\sqrt{\frac{2}{\pi }}}{2Lu}}.$$

The estimation of the population abundance D relies on estimating the parameter $f\left(0\right)$. This parameter is crucial in the estimation process. In order to estimate $f\left(0\right)$ and subsequently estimate D, a parametric form (shape) for the detection function (or PDF) is required. Therefore, the approach used to estimate D is referred to as a parametric method. This method involves specifying a specific parametric form for the detection function, which allows for the estimation of the parameter $f\left(0\right)$ and ultimately the estimation of D. The parametric method provides a structured framework for estimating population abundance based on the chosen parametric model for the detection function.

3. Some Statistical Properties

The proposed detection function $g\left(x\right)$’s rth moments of the PDF $f\left(x\right)$ are derived in this part, and they aid in the estimation of the parameter $f\left(0\right)$, allowing us to calculate the population abundance.

The following formula can be used to determine the rth moment of a random variable X given the probability density function (PDF) given in (6):

$${\mu }_r^{\prime }={\int }_0^{\infty }{x}^r{\displaystyle \frac{\sqrt{\frac{2}{\pi }}}{u}}{e}^{-{\displaystyle \frac{x^2}{2u^2}}},dx.$$

After some algebra, we have

$${\mu }_r^{\prime }={\displaystyle \frac{2^{r/2}{u}^r\Gamma \left(\frac{1+r}{2}\right)}{\sqrt{\pi }}}.$$

(9)

The mean and variance can be obtained by using $r=1,2$ as

$${\mu }_1^{\prime }=E\left(X\right)=\sqrt{{\displaystyle \frac{2}{\pi }}}u,$$

(10)

and

$${\sigma }^2=Var\left(X\right)=u^2-{\displaystyle \frac{2u^2}{\pi }}.$$

(11)

For some parameter values, the central tendency, variability, asymmetry, and peakedness can be described, where ${\mu }_r^{\prime }$ denotes the rth moment of the random variable X, $\mu$ represents the mean of X, and ${\sigma }^2$ stands for the variance of X. According to Ghitany et al. [13] and Wackerly et al. [14], skewness and kurtosis provide insights into the asymmetry and the shape of the distribution, respectively.

Table 1. Mean (M), variance (V), skewness (S), and kurtosis (K) for some values of the parameters.

Parameter $\left(\mathit{u}\right)$	M	V	S	K
0.5	0.39894	0.09084	0.99527	22.28491
0.6	0.47873	0.13081	0.99527	22.28491
0.7	0.55851	0.17805	0.99527	22.28459
0.8	0.63830	0.23256	0.99527	22.28491
0.9	0.71809	0.29433	0.99527	22.28491
1.0	0.79788	0.36338	0.99527	22.28491
1.1	0.87766	0.43970	0.99520	22.28311
1.2	0.957461	0.52326	0.99527	22.28491
1.3	1.03725	0.61411	0.99527	22.28491
1.4	1.11703	0.71222	0.99527	22.28491
1.5	1.19682	0.81760	0.99527	22.28491
1.6	1.27661	0.93025	0.99527	22.28491

presents the numerical values for the mean, variance, skewness, and kurtosis of X for different parameter values of u whose R code is given in Appendix A. The graphical representation of these statistics is shown in Figure 3. It is observed that as the parameter values increase, these statistical measures typically decrease. However, for any fixed value of u, the skewness and kurtosis remain constant.

4. Inference

This section focuses on making statistical inferences about the parameter $f\left(0\right)$ when the parameter u is unknown. We employed three methods for this purpose: the moment estimator (ME), the maximum likelihood (ML) approach, and the Bayesian estimation (BE) method. Each method involves estimating the value of u that optimizes the likelihood function. Consider the probability density function (PDF) $f\left(x\right)$ in conjunction with the recommended detection function.

4.1. Moment Estimator of $f\left(0\right)$

The initial moment of the model is given by Equation (10).

$$E\left(X\right)={\mu }_1^{\prime }=\sqrt{{\displaystyle \frac{2}{\pi }}}u=f\left(0\right)u^2,$$

which leads to estimate $f\left(0\right)$ as

$$\widehat{f}\left(0\right)={\displaystyle \frac{\overline{x}}{u^2}},$$

where $\widehat{f}\left(0\right)$ is the moment estimator (ME) for $f\left(0\right)$. The current estimations of u must be entered here. Equation (9) can be used for this, by setting $r=1$. This gives us

$$\sqrt{{\displaystyle \frac{2}{\pi }}}u={\displaystyle \frac{{\displaystyle \sum _{i=1}^n}{x}_i}{n}},$$

(12)

Solving the equations described above allows us to derive moment estimates of u, denoted as $\widehat{u}$. Consequently, the moment estimator of $f\left(0\right)$ is given by ${\widehat{f}}_M\left(0\right)=f\left(0\right)$.

4.2. Maximum Likelihood Estimator of $f\left(0\right)$

To obtain the maximum likelihood estimator of $f\left(0\right)$, we need to find the ML estimator of u. For the probability function based on the PDF $f\left(x\right)$, to determine the maximum likelihood estimate of the parameter u for the given function $f\left(x\right)$, we need to discover the value of u that maximizes the likelihood function $L\left(u\right)$. The likelihood function is provided by:

$$L\left(u\right)=\sum _{i=1}^{n}f(x_i;u)$$

where $x_i$ are the observed data points. Taking the natural logarithm of the likelihood function, we obtain:

$$\sum _{i=1}^{n}logf(x_i;u)=-nlogu-(n/2)log(2/\pi )-(1/\left(2u^2\right))\sum _{i=1}^n{x}_i^2$$

(13)

To determine the greatest probability estimate of u, we need to differentiate the above formula with regard to u, make it equal to zero, and solve for u.

$${\displaystyle \frac{d}{du}}lnL\left(u\right)=-n/u+(1/u^3)\sum _{i=1}^n{x}_i^2=0$$

Therefore, the maximum likelihood estimate of u for the given function $f\left(x\right)$ is:

$$u=\sqrt{(1/n)\sum _{i=1}^n{x}_i^2}$$

Note that this is the sample standard deviation of the data points $x_i$. One can compute the highest likelihood estimate using a numerical technique like the Newton–Raphson method. Let $\widehat{u}$ be the maximum likelihood estimator of u. Then, the maximum likelihood estimator of the parameter $f\left(0\right)$ is given by ${\widehat{f}}_{ML}\left(0\right)=f\left(0\right)$.

4.3. Bayesian Estimator of $f\left(0\right)$

This section covers the Bayesian estimate (BE) for the unknown parameters u and $f\left(0\right)$, respectively. For the Bayesian parameter estimation, Tolba et al. [15], Alsadat et al. [16], Bhat et al. [17], and Chinedu et al. [18] all considered loss functions, including squared error loss (SEL) functions. For the variables u and $f\left(0\right)$, we can consider applying independent gamma priors by using PDF $f\left(x\right)$ in the parameter prior detection function.

$${\pi }_1\left(u\right)\propto u^{s_1-1}{e}^{-q_{1}u},u>0,s_1>0,q_1>0,$$

(14)

$${\pi }_2\left(f\left(0\right)\right)\propto f{\left(0\right)}^{s_2-1}{e}^{-q_{2}f\left(0\right)},f\left(0\right)>0,s_2>0,q_2>0,$$

where the historical data on the unknown parameters are reflected in the selection of the hyper-parameters $s_j,q_j,j=1,2$. The joint prior for $\Omega =(u,f(0\left)\right)$ is given by

$$\pi (\Omega )\propto {\pi }_1\left(u\right){\pi }_2\left(f\left(0\right)\right)$$

(15)

$$\pi (\Omega )\propto u^{s_1-1}f{\left(0\right)}^{s_2-1}{e}^{-q_{1}u-q_{2}f\left(0\right)}.$$

With respect to the observed data $x=(x_1,x_2,\dots ,x_n)$, the relevant posterior density is given by

$$\pi (\Omega |x)={\displaystyle \frac{\pi (\Omega )\ell (\Omega )}{{\int }_u{\int }_{f\left(0\right)}\pi (\Omega )\ell (\Omega )dudf\left(0\right)}}.$$

(16)

Thus, the posterior density function can be represented by

$$\begin{array}{ccc}& & \pi (\Omega |x)\propto u^{s_1-1}f{\left(0\right)}^{s_2-1}{e}^{-q_{1}u-q_{2}f\left(0\right)}\hfill \\ & & \times \sum _{i=1}^n{\displaystyle \frac{\sqrt{\frac{2}{\pi }}}{u}}{e}^{-{\displaystyle \frac{x^2}{2u^2}}}.\hfill \end{array}$$

(17)

The Bayes estimator for any function, like $l(\Omega )$, under the squared error loss (SEL) function, is given by

$${\widehat{\Omega }}_{BEsel}=E[l(\Omega )|x]={\int }_{\Omega }l(\Omega )\pi (\Omega /x)d\Omega .$$

(18)

They can be quantitatively computed using any statistical software, including Mathematica version 13.2, Python version 3.10.6, R version 4.3.0, and so on.

5. Simulation Study and Results

A simulation study evaluating the effectiveness of several estimators of the detection function $f\left(0\right)$ is presented in this section. The suggested model estimators were the maximum likelihood (ML) estimator ${\widehat{f}}_{\mathrm{ML}}\left(0\right)$ and the negative Rayleigh detection model (NRDM) estimator ${\widehat{f}}_{\mathrm{NRDM}}\left(0\right)$, which was computed using the method of moments’ estimator ${\widehat{f}}_{\mathrm{MOM}}\left(0\right)$. We also took into account already-existing estimators, namely, the Weighted Exponential Model (WEM) estimator ${\widehat{f}}_{\mathrm{WEM}}\left(0\right)$, the Half-Normal Model (HNM) estimator ${\widehat{f}}_{\mathrm{HNM}}\left(0\right)$, and the Negative Exponential Model (NEM) estimator ${\widehat{f}}_{\mathrm{NEM}}\left(0\right)$.

The simulation relied on two alternative target detection functions: the Exponential Power (EP) detection function and the Beta Exponential (BE) detection function. The simulation was run with various sample sizes and truncated points for each detection function. To replicate the perpendicular distances, two distinct target detection functions and observations $x_1,x_2,\dots ,x_n$ of $X_1,X_2,\dots ,X_n$ with sample sizes of $n=50$, 100, 200, 300, and 500 were taken into consideration. These detection functions were chosen based on the requirement that they reflect the range of shapes that could occur in the specified field (see Eidous [19]). Perpendicular distances were simulated using the target models listed below:

• The Exponential Power (EP) detection function, introduced by Pollock [12], provides a flexible model for detection probability and is defined as:

  $$f\left(x\right)={\displaystyle \frac{e^{-x^{\xi }}}{\Gamma \left(1+\frac{1}{\xi }\right)}},x>0,;\xi >1.$$

  (19)
  In this function, $\Gamma \left(x\right)$ denotes the gamma function, which is integral in various statistical distributions.
• Eberhardt’s Beta Exponential (BE) detection function [8] models detection probability with the formula:

  $$g\left(x\right)=(1+\xi ){\left(1-x\right)}^{\xi },0<x<1,;\xi >0.$$

  (20)
  This function is particularly useful in scenarios where the detection probability decreases linearly.

To mimic the data, every model was clipped at certain distances w. The truncated points of the detection functions for BE and EP were $w=0.4,1.3,1.1,2.0$ and $w=5.1,3.2,2.4,2.2$, respectively. For both models, a number of (arbitrary) values of $\xi$ were chosen. We evaluate sample sizes of $n=50$, 100, 200, 300, and 500 for each model, while the samples for the randomly generated perpendicular distances were 1500 times larger. The results of the simulation research, along with the relative bias (RB) and relative mean square error (RME), are shown in

Table 2. When the Exponential Power (EP) model was used to simulate the data, RB-1 and RME-1 were used for the different estimators.

			Estimator
$\mathit{n}$	$\mathit{\xi }$	$\mathit{w}$	${\widehat{\mathit{f}}}_{\mathbf{MOM}}\left(\mathbf{0}\right)$		${\widehat{\mathit{f}}}_{\mathbf{ML}}\left(\mathbf{0}\right)$		${\widehat{\mathit{f}}}_{\mathbf{NEM}}\left(\mathbf{0}\right)$		${\widehat{\mathit{f}}}_{\mathbf{HNM}}\left(\mathbf{0}\right)$		${\widehat{\mathit{f}}}_{\mathbf{WEM}}\left(\mathbf{0}\right)$
			RB-1	RME-1	RB-1	RME-1	RB-1	RME-1	RB-1	RME-1	RB-1	RME-1
50	1.2	5.1	0.1182	0.3641	−0.1656	0.1501	0.0521	0.1510	0.3851	0.3942	0.1820	0.2131
100	1.2	5.1	0.0801	0.3421	−0.1672	0.1120	0.0531	0.1131	0.3882	0.3921	0.1812	0.1973
200	1.2	5.1	0.0781	0.3300	−0.1691	0.0623	0.0433	0.0804	0.3940	0.3964	0.1892	0.1964
300	1.2	5.2	0.0628	0.3270	−0.1712	0.0512	0.0333	0.0511	0.3961	0.3921	0.1791	0.1922
500	1.2	5.1	0.0421	0.3210	−0.1752	0.0321	0.0234	0.0452	0.3943	0.3930	0.1672	0.1861
50	1.3	3.2	−0.0081	0.2412	−0.1333	0.1391	0.4021	0.4351	0.1332	0.1644	0.0909	0.1571
100	1.3	3.2	−0.0291	0.2341	−0.1371	0.1382	0.3974	0.4147	0.1354	0.1512	0.0879	0.1250
200	1.3	3.2	−0.0390	0.2322	−0.1394	0.1360	0.3891	0.3984	0.1421	0.1501	0.0812	0.1046
300	1.3	3.2	−0.0592	0.2310	−0.1412	0.1341	0.3864	0.3912	0.1322	0.1422	0.0801	0.1000
500	1.3	3.2	−0.0631	0.2301	−0.1481	0.1312	0.3811	0.3901	0.1210	0.1401	0.0752	0.0241
50	2.1	2.4	−0.0190	0.1511	−0.1000	0.1040	0.6011	0.6254	0.0233	0.1091	0.2453	0.2791
100	2.1	2.4	−0.0431	0.1122	−0.1020	0.06312	0.5936	0.6088	0.0178	0.0781	0.2399	0.2601
200	2.1	2.4	−0.0400	0.0940	−0.1100	0.0400	0.5751	0.5810	0.0050	0.0501	0.2256	0.2341
300	2.1	2.4	−0.0471	0.0901	−0.1154	0.0212	0.5423	0.5561	0.0021	0.0401	0.2200	0.2120
500	2.1	2.4	−0.0498	0.0641	−0.1192	0.0144	0.5351	0.5206	0.0013	0.0201	0.2123	0.2101
50	2.6	2.2	−0.0021	0.0980	−0.0646	0.0747	0.7045	0.7244	0.1160	0.1520	0.3251	0.3510
100	2.6	2.2	−0.0091	0.0820	−0.0686	0.0734	0.6970	0.7081	0.1086	0.1290	0.3201	0.3342
200	2.6	2.2	−0.0160	0.0453	−0.0696	0.0710	0.6965	0.7026	0.1066	0.1186	0.3190	0.3274
300	2.6	2.2	−0.0213	0.0312	−0.0745	0.0684	0.6921	0.7001	0.1024	0.1125	0.3154	0.3210
500	2.6	2.2	−0.0625	0.0154	−0.0780	0.0631	0.6901	0.6922	0.1001	0.1097	0.3122	0.3171

and

Table 3. RB-2 and RME-2 for the different estimators when the data were simulated from the Beta (BE) model.

			Estimator
$\mathit{n}$	$\mathit{\xi }$	$\mathit{w}$	${\widehat{\mathit{f}}}_{\mathbf{MOM}}\left(\mathbf{0}\right)$		${\widehat{\mathit{f}}}_{\mathbf{ML}}\left(\mathbf{0}\right)$		${\widehat{\mathit{f}}}_{\mathbf{NEM}}\left(\mathbf{0}\right)$		${\widehat{\mathit{f}}}_{\mathbf{HNM}}\left(\mathbf{0}\right)$		${\widehat{\mathit{f}}}_{\mathbf{WEM}}\left(\mathbf{0}\right)$
			RB-2	RME-2	RB-2	RME-2	RB-2	RME-2	RB-2	RME-2	RB-2	RME-2
50	1.1	0.4	0.0850	0.0994	0.0700	0.0911	0.5092	0.5551	0.1914	0.2234	0.2841	0.2564
100	1.1	0.4	0.0764	0.0813	0.0626	0.0731	0.4885	0.5185	0.1892	0.2042	0.2431	0.2221
200	1.1	0.4	0.0442	0.0621	0.0443	0.0473	0.4013	0.5113	0.1726	0.1901	0.2013	0.2011
300	1.1	0.4	0.0321	0.0381	0.0286	0.0351	0.3967	0.5004	0.1531	0.1842	0.1922	0.1950
500	1.1	0.4	0.0174	0.0292	0.0093	0.0271	0.3513	0.4821	0.1482	0.1673	0.1917	0.1940
50	1.6	1.3	0.1010	0.1286	0.1723	0.1896	0.4114	0.4855	0.1794	0.2021	0.1681	0.2008
100	1.6	1.3	0.0921	0.1132	0.1433	0.1574	0.4321	0.4174	0.1572	0.1891	0.1462	0.1864
200	1.6	1.3	0.0801	0.1060	0.1116	0.1204	0.3233	0.3513	0.1386	0.1557	0.1174	0.1703
300	1.6	1.3	0.0635	0.1017	0.1064	0.1183	0.3212	0.3341	0.1298	0.1425	0.1063	0.1628
500	1.6	1.3	0.0413	0.0933	0.1006	0.1126	0.3173	0.3112	0.1135	0.1395	0.1012	0.1575
50	2.1	1.1	0.1129	0.1605	0.1253	0.1417	0.3993	0.4320	0.1985	0.2255	0.2101	0.1834
100	2.1	1.1	0.0935	0.1554	0.1075	0.1176	0.2544	0.3312	0.1627	0.2112	0.2065	0.1713
200	2.1	1.1	0.0774	0.1432	0.0663	0.0855	0.2216	0.2800	0.1456	0.1895	0.1944	0.1546
300	2.1	1.1	0.0473	0.1376	0.0512	0.0733	0.2197	0.2678	0.1324	0.1695	0.1912	0.1341
500	2.1	1.1	0.0353	0.1192	0.0475	0.0612	0.2178	0.2509	0.1116	0.1525	0.1821	0.1299
50	2.8	2.0	0.1520	0.1995	0.1417	0.1675	0.2512	0.3196	0.2495	0.2751	0.19922	0.2183
100	2.8	2.0	0.1341	0.1743	0.1342	0.1545	0.2403	0.2931	0.2001	0.2494	0.1704	0.1983
200	2.8	2.0	0.1194	0.1531	0.0922	0.1024	0.2313	0.2221	0.1972	0.2212	0.1581	0.1664
300	2.8	2.0	0.1107	0.1283	0.0901	0.1004	0.2283	0.2026	0.1837	0.2182	0.1423	0.1526
500	2.8	2.0	0.1046	0.1192	0.0834	0.0926	0.2204	0.2004	0.1722	0.2121	0.1382	0.1454

.

The main conclusions drawn from the simulation study were as follows:

• All estimators exhibited a decreasing relative bias (RB) and relative mean error (RME) as the sample size increased, indicating that the estimators become more reliable with larger sample sizes.
• The ML estimator ${\widehat{f}}_{\mathrm{ML}}\left(0\right)$ consistently outperformed the NRDM estimator ${\widehat{f}}_{\mathrm{NRDM}}\left(0\right)$ in terms of RB and RME. This suggests that ${\widehat{f}}_{\mathrm{ML}}\left(0\right)$ is a superior estimator compared to ${\widehat{f}}_{\mathrm{MOM}}\left(0\right)$.
• For all considered target detection functions, the suggested estimators ${\widehat{f}}_{\mathrm{ML}}\left(0\right)$ demonstrated a reduced RB and RME in comparison to the NEM, HNM, and WEM estimators. This shows that the suggested estimators are better than the alternatives.
• We plotted the RMEs for ${\widehat{f}}_{\mathrm{ML}}\left(0\right)$, ${\widehat{f}}_{\mathrm{NEM}}\left(0\right)$, ${\widehat{f}}_{\mathrm{HNM}}\left(0\right)$, and ${\widehat{f}}_{\mathrm{WEM}}\left(0\right)$, respectively, in Figure 4, Figure 5, Figure 6 and Figure 7. These numbers allowed us to conclude that our suggested estimations were reasonable and performed exceptionally well for all target detection functions that were taken into consideration.

6. Model Accordance and Practical Data Utilization

Since model selection establishes the framework for data analysis and prediction, it is in fact an essential step in the modeling process. A thorough analysis of the data’s properties, including its distribution, variability, and possible sources of error, is necessary to select the best model. It also entails evaluating the trade-off between the accuracy of the model’s representation of the data and its complexity. The Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC) are two examples of information criteria that can be used for model selection, in addition to likelihood-based and Bayesian approaches. By offering a numerical representation of the model’s goodness of fit and a penalty for model complexity, these techniques enable us to compare models with various numbers of parameters.

But there is no one-size-fits-all approach to model selection; rather, the best approach relies on the specifics of the research topic and the properties of the data. Expert judgment and topic expertise may occasionally be required to reach a final decision. After a model has been chosen, it is crucial to validate it to make sure it can accurately fit new data and is not overfitting the original data. Model validation entails evaluating the model’s goodness of fit using a variety of statistical metrics, including mean squared error, R-squared, and cross-validation, and testing the model on a different dataset that was not used for model fitting.

6.1. Comparative Models

The suggested detection model’s performance was compared to a number of other detection models using various goodness-of-fit evaluations. The models that were used for comparison are displayed in

Table 4. Comparative models.

Model Name	Detection Function $\mathit{g}(\mathit{x};·)$	Probability Density Function $\mathit{f}(\mathit{x};·)$
New Two-Parameter Detection Model (NDM-1) [20]	$g(x;v,u)=(1+u{x}^v)e^{-u{x}^v}$	$f(x;v,u)={\displaystyle \frac{v^2{u}^{1/v}}{(v+1)\Gamma (1/v)}}(1+u{x}^v)e^{-u{x}^v},v,u>0$
Model (2013)-2 [21]	$g(x;v,u)={(2-e^{-ux})}^v{e}^{-ux}$	$f(x;v,u)={\displaystyle \frac{u(1+v)}{2^{v+1}-1}}{(2-e^{-ux})}^v{e}^{-ux},v,u>0$
Generalized Exponential Model (GEM-3) [11]	$g(x;v,u)=e^{(1/v){(x/u)}^v}$	$f(x;v,u)={\displaystyle \frac{e^{(1/v){(x/u)}^v}}{u{v}^{1/v}\Gamma (1+1/v)}},v,u>0$
Exponential Power Series Model (EPSM-4) [12]	$g(x;v,u)=e^{{(x/u)}^v}$	$f(x;v,u)={\displaystyle \frac{e^{{(x/u)}^v}}{u\Gamma (1+1/v)}},v,u\ge 0$
Reverse Logistic Model (RLM-5) [8]	$g(x;v,u)={\displaystyle \frac{(1+u)e^{-vx}}{1+u{e}^{-vx}}}$	$f(x;v,u)={\displaystyle \frac{vu(1+u)e^{-vx}}{(1+u)(1+u{e}^{-vx})log(1+u)}},v,u\ge 0$
Exponential Quadratic Model (EQM-6) [10]	$g(x;v,u)=e^{(-vx-u{x}^2)}$	$f(x;v,u)={\displaystyle \frac{2u{e}^{(-vx-u{x}^2)}}{e^{-v^2/4u}\sqrt{\pi }\mathrm{erfc}(v/2\sqrt{u})}},v,u\ge 0$
Weighted Half-Normal Model (WHNM-7) [22]	$g(x;u)=(2-e^{-x^2/2u})e^{-x^2/2u}$	$f(x;\lambda )={\displaystyle \frac{2}{2\sqrt{2}-1}}(2-e^{-x^2/2\lambda })e^{-x^2/2\lambda },\lambda >0$
Model (2015)-8 [19]	$g(x;u)={(1-0.5e^{-ux/2})}^{2}4e^{-ux}$	$f(x;u)={\displaystyle \frac{24u}{11}}{(1-0.5e^{-ux/2})}^2{e}^{-ux},u>0$
Weighted Exponential Model (WEM-9) [23]	$g(x;u)=(2-e^{-ux})e^{-ux}$	$f(x;u)={\displaystyle \frac{2u}{3}}(2-e^{-ux})e^{-ux},u\ge 0$
Negative Exponential Model (NEM-10) [24]	$g(x;u)=e^{-ux}$	$f(x;u)=u{e}^{-ux},u\ge 0$

.

6.2. Data on the Perpendicular Length in Meters

Dataset: Measurements in meters for parallel wooden stake distances.

The dataset contains 67 stakes collected along a single path of length 1000 m, from a population of 150 stakes. The true density is given as 0.00375 stake/meter. Based on earlier research by Karunamuni and Quinn [25] and Zhang [26], the value of $f\left(0\right)$ was given as 0.11029. The sagebrush meadow’s zone size was calculated to be 40,000 ${\mathrm{m}}^2$ using the equation $D=N/A$, where $N=150$ stakes.

Table 5. Measurements of wooden stake distances (in meters).

2.020	0.450	10.400	3.610	0.920	1.000	3.400	2.900	8.160	6.470	5.660
2.950	3.960	0.090	11.820	14.230	2.440	1.610	6.500	8.270	4.850	1.470
18.600	0.410	0.400	0.200	11.590	3.170	7.100	10.710	3.860	6.050	6.420
3.790	15.240	3.470	3.050	7.930	18.150	10.050	4.410	1.270	13.720	6.250
3.590	9.040	7.680	4.890	9.100	3.250	8.490	6.080	0.400	9.330	0.530
1.230	1.670	4.530	3.120	3.050	6.600	4.400	4.970	3.170	7.670	18.160
4.080

presents the data, which have been reviewed and reported by Burnham et al. [10], Barabesi [27] and Strindberg et al. [28]. In contrast to the other values, the perpendicular distance of 31.31 in Table 5 considerably departs from the transect path. Therefore, it is suggested to remove this extreme value or truncation point from the dataset to improve the accuracy of the estimation, as mentioned in Zhang [26]. Based on this information, the decision was made to eliminate the mentioned extreme value or truncation point from the dataset to ensure the accuracy and reliability of the estimation process.

Table 6. Descriptive statistics and theoretical NRDM metrics of the wooden stakes’ dataset.

n	Mean (M1)	Median (M2)	Standard Deviation (SD)	Skewness (S)	Kurtosis (K)	$\frac{\mathit{Skewness}}{\mathit{Kurtosis}}$
67	5.73202	4.41244	4.55641	1.10832	3.80334	0.29143
67	5.82552	4.47211	4.40122	1.31834	3.61219	0.30581

provides the basic statistics of the wooden stakes’ dataset and presents the ratio of skewness to kurtosis for the wooden stakes’ dataset. This ratio provides insights into the distribution’s shape and helps assess how well the data align with theoretical expectations of the negative Rayleigh detection model (NRDM). The box plot and TTT plot in Figure 8 offer more detail.

The study used maximum likelihood estimators, or MLEs, to ascertain the probability density function (PDF) of each detection model and assess how well it matched the collected data. Commonly used criteria used to compare the fit performance of models considered for the study of real datasets include the p value of the Kolmogorov–Smirnov test, the Bayesian information criterion (BIC-1), the Hannan-Quinn information criterion (HQIC-1), the Akaike information criterion (AIC-1), and the corrected AIC (CAIC-1). They are shown in

Table 7. The dataset for wooden stakes using the MLEs, $A^{\ast }$, $W^{\ast }$, K-S statistic, and p-value.

Detection Models	$\widehat{\mathit{v}}$	$\widehat{\mathit{u}}$	${\mathit{A}}^{\ast }$	${\mathit{W}}^{\ast }$	K-S	p-Value
NRDM	-	7.3013	0.2323	0.0198	0.0698	0.8981
NDM-1	1.2846	0.1165	0.2642	0.2671	0.0841	0.7315
Model (2013)-2	2.1209	0.23507	0.2732	0.0384	0.0860	0.7018
GEM-3	1.7378	6.9116	0.2722	0.0419	0.0884	0.6706
EPSM-4	1.7378	9.4988	0.2720	0.0419	0.0886	0.6701
RLM-5	0.2839	6.7169	0.2597	0.0409	0.08908	0.6624
EQM-6	0.0073	0.0397	0.2793	0.0442	0.0910	0.6362
WHNM-7	-	42.697	0.8712	0.0413	0.1040	0.4611
Model (2015)-8	-	0.2368	0.3773	0.0703	0.1137	0.3521
WEM-9	-	0.2044	0.5023	0.0978	0.1260	0.2365
NEM-10	-	0.1746	1.0225	0.2064	0.1587	0.0681

and

Table 8. Information criteria and log likelihood ℓ for the wooden stakes’ dataset.

Detection Models	-ℓ	AIC-1	AICC-1	BIC-1	HQIC-1	CAIC-1
NRDM	181.828	365.655	365.716	367.859	366.527	365.717
NDM-1	181.663	367.326	367.513	371.735	369.070	368.002
Model (2013)-2	182.031	368.061	368.249	372.471	369.806	368.260
GEM-3	181.653	367.304	367.492	371.714	369.049	367.494
EPSM-4	181.653	367.304	367.492	371.714	369.049	367.495
RLM-5	181.769	367.537	367.724	371.946	369.282	367.724
EQM-6	181.657	367.312	367.500	371.721	369.057	367.501
WHNM-7	181.753	367.504	367.592	371.814	369.059	367.513
Model (2015)-8	182.166	368.331	368.393	368.536	367.204	366.399
WEM-9	182.527	367.653	367.814	369.257	367.925	367.118
NEM-10	183.863	369.724	369.786	371.929	370.597	369.787

, and their Mathematica code is given in Appendix B. Our model had the lowest information criteria values and the highest p value for the Kolmogorov–Smirnov test, as can be seen. Therefore, it can be concluded that the proposed model is effective in this scenario.

Given the information provided, an analysis was conducted to estimate the population abundance ($\widehat{D}$) and the maximum likelihood estimate of the true density, ${\widehat{f}}_{ML}\left(0\right)$, using the maximum likelihood estimators (MLEs) from many studied models. The sample standard deviation is shown as $\widehat{SD}$, while the theoretical standard deviation of the models is indicated by $SD$. The ${\widehat{f}}_{ML}\left(0\right)$ and $|\widehat{SD}-SD|$ results are shown in

Table 9. For the wooden stakes’ dataset, population abundance D and estimated $f\left(0\right)$ were calculated using $|\widehat{SD}-SD|$.

Detection Models	${\widehat{\mathit{f}}}_{\mathit{ML}}\left(0\right)$	$\left(\widehat{\mathit{D}}\right)$	$|\widehat{\mathit{SD}}-\mathit{SD}|$
NRDM	0.10921	0.00360	0.02601
NDM-1	0.11401	0.00383	0.02902
Model (2013)-2	0.09502	0.00322	0.26616
GEM-3	0.11821	0.00400	0.03212
EPSM-4	0.11801	0.00397	0.03202
RLM-5	0.12002	0.00401	0.03643
EQM-6	0.12207	0.00404	0.03411
WHNM-7	0.09404	0.00327	0.33963
Model (2015)-8	0.12913	0.00431	0.41672
WEM-9	0.13604	0.00453	0.66911
NEM-10	0.17402	0.00581	1.17652

. The table shows that the NRDM (nonlinear Rayleigh detection model) model had the smallest absolute difference between the sample standard deviation ($\widehat{SD}$) and the theoretical standard deviation ($SD$) when using the p-value, K-S statistic, W*, A*, and MLEs. Furthermore, among the detection models that were tested, the estimates of ${\widehat{f}}_{ML}\left(0\right)$ and $\widehat{D}$ produced by the NRDM model were fairly close to the true values, which were reported to be 0.1127 and 0.0037, respectively, for the data under consideration. These findings were consistent with the true values of $f\left(0\right)$ and D discovered in previous studies. Finally, with only minor deviations between the sample standard deviation and theoretical standard deviation, the NRDM model yielded the best fit to the true values of $f\left(0\right)$ and D for the examined data. Table 9 provides a summary of these results.

Further proof of the NRDM’s competitiveness can be found in Figure 9, which plots the estimated survival functions and PDFs of the compared models for the considered dataset based on each model’s survival function and PDF after their parameters were replaced with the corresponding MLEs from Table 7.

7. Concluding Remarks

7.1. Conclusions

This study introduced and evaluated the nonlinear Rayleigh detection model (NRDM), a new one-parameter detection model for line transect data. The NRDM demonstrates a monotonically decreasing perpendicular distance and satisfies the shoulder criterion, indicating its promising potential. Characterized by its flexibility and reliance on two parameters, the NRDM requires numerical techniques, such as the Newton–Raphson method, for parameter estimation. Simulation results highlighted the model’s strong performance in estimating population abundance, showing improved accuracy compared to existing models. The NRDM also showed better fit and parameter estimation when applied to empirical datasets of perpendicular distances.

7.2. Limitations

The study has several limitations. Primarily, the use of simulated data to assess the NRDM’s performance may not fully represent the complexities and variability of real-world field conditions. Additionally, factors such as species behavior, habitat features, and variations in survey design were not comprehensively addressed, which may impact the model’s applicability. These limitations suggest that while the NRDM shows promising results, its effectiveness in real-world scenarios requires further scrutiny.

7.3. Future Work

Future research should focus on validating the NRDM across a range of ecological contexts and field conditions to confirm its robustness and applicability. Investigations should explore the model’s performance under different survey designs and incorporate factors like species behavior and habitat characteristics. Additionally, refining the NRDM and assessing potential modifications will be crucial for enhancing its utility in various ecological studies. Continued exploration will help ensure the NRDM’s effectiveness in practical applications and improve its adaptability to diverse research needs.