https://doi.org/10.53656/math2023-5-2-ont

2023/5, стр. 452 - 464

ON THE DYNAMICS OF A ClASS OF THIRD-ORDER POLYNOMIAL DIFFERENCE EQUATIONS WITH INFINITE NUMBER OF PERIOD-THREE SOLUTIONS

Резюме: We study the local dynamics and global character of third-order polynomial difference in the first octant of initial conditions with infinite number of prime period-three solutions (three cycles). It is also presented the case when the observed difference equation may be extended to the whole \(\mathbb{R}^{3}\).

Ключови думи: basins of attraction; difference equation; equilibrium; stability; periodthree solution

1. Introduction

Let \(f\left(x_{n}, x_{n-1}, x_{n-2}\right)=a x_{n}^{k}+a x_{n-1}^{k}+a x_{n-2}^{k}+b\), where \(a, b \gt 0\) and \(k \in\) \(\mathbb{N}\). In this paper we study the local stability of equilibriums, theglobal stability character, the periodic behavior and the boundedness of solutions of polynomial third-order difference equation of type

(1)\[ x_{n+1}=x_{n-2} f\left(x_{n}, x_{n-1}, x_{n-2}\right) \]

with initial conditions \(x_{-2}, x_{-1}, x_{0}\) in the first octant. So our results will be more special, as well as more precise and hence easy to be applied. In our research of dynamics of (1) we develop a new method of proofs and omit the we ll-known theory of monotone maps (specially developed for planar maps applied to po lynomial maps), and in particular competitive and cooperative maps, which guarantee the existence and uniqueness of the stable and unstable manifolds for the fixed and periodic points (Kulenović & Merino 2010). Furthermore, the difference equation (1) has infinitely many period-three solutions and we expose the explicit form of the surface that separates the first octant into two basins of attraction of a locally stable zero equilibrium and the point at infinity. The most investigated types of difference equations are polynomial difference equations and polynomial maps in the plane \(\mathbb{R}^{2}\). Also, it is very important to mentionthat the polynomial difference equations with simple form but rich dynamic were observed in the complex domain (Milnor 2000; Morosawa et al. 2000).

The basins of attraction may have very complicated structures even for very simple-looking maps (the chaotic maps may have Cantor sets as a basin of attraction, (Kulenović & Merino 2002)). The first results on polynomial difference equations, based on elementary proofs, we can find in (Amleh et al. Part I 2008) and (Amleh et al. Part II 2008). More precisely, all obtained results described only a part (box) of the basins of attraction of equilibrium(s) and period-two solutions. The theory of monotone maps (cooperative and competitive maps) was developed (Brett & Kulenović 2009), (Kulenović & Merino 2010), which provided the existence and uniqueness of the stable, unstable, and central manifolds (onedimensional curve) for the equilibrium(s) and periodic points. All their results can be extended and generalized.

From theorems proved in (Brett & Kulenović 2009), (Kulenović & Merino 2010), applied on polynomial maps, follow the results of general second-order polynomial difference equations (Bektešević et al. 2014). All the solutions from the described in (Bektešević et al. 2014) regions of initial conditions (basins of attractions) in the first quadrant tend to equilibrium points, period-two solutions or at infinity, except for the case of infinitely many period-two solutions. In (Bektešević et al. 2018), the case of infinitely many period-two solutions is completely investigated and the corresponding difference equation is a special case of the equation

\[ x_{n+1}=a x_{n} x_{n-1}+a x_{n-1}^{2}+b x_{n-1} . \]

Further (Bektešević et al. 2021) we have extended and improved our research to the difference equation

\[ x_{n+1}=a x_{n}^{m} x_{n-1}+a x_{n-1}^{m+1}+b x_{n-1} \] for \(m \gt 1\). Since the difference equation \(x_{n+1}=x_{n-1}^{m+1} x_{n}^{m}\) can be solved explicitly (Elaydi 2005), in (Bektešević et al. 2022) we have considered the difference equation of type

\[ x_{n+1}=x_{n-1} P\left(x_{n-1}\right) P\left(x_{n}\right), \] where \(P(0) \gt 0\) and \(P(x)\) is polynomial with nonnegative coefficients and initial conditions \(x_{-1}\) and \(x_{0}\) are arbitrary nonnegative numbers. This difference equation represents an example of a difference equation for which the boundary of the region of initial conditions when all solutions tend to infinity can be found explicitly and represent a planar curve.

In the planar case, all definitions of stability of equilibrium points and the main result for studying local stability of equilibrium(s) can be found in (Kulenović & Ladas 2001). The special case of rational difference equation, with linear terms in numerator and denominator, investigated by (Kulenović & Ladas 2001) is a polynomial difference equation that can be solved exactly. The monograph (Kulenović & Ladas 2001) includes well-known difference equations such as Riccati difference equations (forbidden set problem), Pielou’s discrete delay logistic model, Lyness’s equation, and also contains a large number of open problems and conjectures. (Camouzis & Ladas 2008) is represented by the third-order rational difference equation with linear terms in numerator and denominator with nonnegative parameters and initial conditions. The book (Kulenović & Ladas 2001) contains the basic results for the development of theory difference equations of order greater than two. All definitions of stability and known results for linearized stability analysis and global dynamics are obtained from (Agarwal 1992), (Alligood et al. 1997), (Devaney 1992), (Elaydi 2000), (Elaydi 2005), (Guckenheimer & Holmes 1983), (Hale & Kocak 1991).

We first list some results needed for the proofs of our theorems.

Suppose that the function \(F=F\left(u_{0}, u_{1}, \ldots, u_{k}\right)\) is continuously differentiable in some open neighborhood of an equilibrium point \(\bar{x}\). Denote by \(q_{i}=\tfrac{\partial F}{\partial u_{i}}(\bar{x}, \bar{x}, \bar{x})\), \(i=0,1, \ldots, k\), the partial derivative of \(F\) with respect to \(u_{i}\), evaluated at the point \(\bar{x}\) of the difference equation of order \(k+1\)

(2)\[ x_{n+1}=F\left(x_{n}, x_{n-1}, \ldots, x_{n-k}\right), \quad n=0,1, \ldots \]

Then the equation

(3)\[ y_{n+1}=q_{0} y_{n}+q_{1} y_{n-1}+\cdots+q_{k} y_{n-k}, \quad n=0,1, \ldots \]

is called the linearized equation of (2) around the equilibrium point \(\bar{x}\), and the equation

(4)\[ \lambda^{k+1}-q_{0} \lambda^{k}-\cdots-q_{k-1} \lambda-q_{k}=0 \]

is called the characteristic equation of (3).

The following, known as Linearized Stability Theorem, is very useful in determining the local stability character of the equilibrium point \(\bar{x}\) of (2).

Theorem 1.2.1 (see Camouzis & Ladas 2008): Assume that the function \(F\) is a continuously differentiable function defined on some open neighborhood of an equilibrium point \(\bar{x}\). Then the following statements are true:

a. If all the roots of (4) have an absolute value less than one, then the equilibrium point \(\bar{x}\) of (2) is locally asymptotically stable.

b. If at least one root of (4) has an absolute value greater than one, then the equilibrium point \(\bar{x}\) of (2) is unstable.

If there exists a root of (4) with absolute value equal to one, then the equilibrium point \(\bar{x}\) is called non-hyperbolic. Otherwise, the equilibrium point \(\bar{x}\) of (2) is called hyperbolic.

The next theorem gives a sufficient condition for all roots of an equation to lie inside a unit disk.

Theorem 1.2.5 (see Camouzis & Ladas 2008): If \(q_{0}, q_{1}, \ldots, q_{k}\) are the real numbers such that \(\left|q_{0}\right|+\left|q_{1}\right|+\cdots+\left|q_{k}\right| \lt 1\) then all roots of (4) lie inside the unit disk.

The next theorem is known as the comparison result. It is very useful toestablish bounds for the solutions of nonlinear equations in terms of the solutions of equations with known behavior.

Theorem 1.4.1 (see Camouzis & Ladas 2008): Let \(I\) be a real interval, let \(k\) be a positive integer, and let \(F: I^{k+1} \rightarrow I\) be an increasing in all its arguments function. Assume that the real sequences \(\left\{x_{n}\right\}_{n=-k}^{\infty},\left\{y_{n}\right\}_{n=-k}^{\infty}\) and \(\{z\}_{n=-k}^{\infty}\) are such that \(x_{n} \leq y_{n} \leq z_{n}\) for all \(-k \leq n \leq 0\), and satisfiy the inequalities or

\[ x_{n+1} \leq F\left(x_{n}, \ldots, x_{n-k}\right), y_{n+1}=F\left(y_{n}, \ldots, y_{n-k}\right), z_{n+1} \geq F\left(z_{n}, \ldots, z_{n-k}\right) \] for each \(n=0,1, \ldots\) Then \(x_{n} \leq y_{n} \leq z_{n}\) for all \(n \gt 0\).

2. Equilibrium points

Proposition 1: Equation (1) always has the zero equilibrium and an unique positive equilibrium iff \(b \in(0,1)\).

Proof: Fixed points of (1) are the nonnegative solutions of the equation

\[ \bar{x}\left(3 a \bar{x}^{k}+b-1\right)=0 \] so, (1) always has zero equilibrium. Clearly, (1) has an unique positive equilibrium \(\bar{x}_{+}\)iff

\[ 3 a \bar{x}^{k}=1-b \gt 0 \]

3. Analysis of local stability

Set \(F(x, y, z)=z f(x, y, z)\), where \(f\) is given by (1). Then \(F\) is continuously differentiable,

\[ \tfrac{\partial F}{\partial x}=z \tfrac{\partial f}{\partial x}=a k z x^{k-1}, \tfrac{\partial F}{\partial y}=z \tfrac{\partial f}{\partial y}=a k z y^{k-1}, \]

and

\[ \tfrac{\partial F}{\partial z}=f(x, y, z)+z \tfrac{\partial f}{\partial z}=f(x, y, z)+a k z^{k} \] Let \(\bar{x}\) be an equilibrium of (1) and

\[ \begin{gathered} q_{1}(\bar{x})=\tfrac{\partial F}{\partial x}(\bar{x}, \bar{x}, \bar{x})=a k \bar{x}^{k}, q_{2}(\bar{x})=\tfrac{\partial F}{\partial y}(\bar{x}, \bar{x}, \bar{x})=a k \bar{x}^{k} \\ q_{3}(\bar{x})=\tfrac{\partial F}{\partial z}(\bar{x}, \bar{x}, \bar{x})=a(k+3) \bar{x}^{k}+b \end{gathered} \] denote partial derivatives of the function \(F(x, y, z)\) evaluated in \((\bar{x}, \bar{x}, \bar{x})\). Then the equation

\[ y_{n+1}=q_{1} y_{n}+q_{2} y_{n-1}+q_{3} y_{n-2}, \quad n \in \mathbb{N}_{0} \]

is called the linearized equation of (1) and the equation

(5)\[ \lambda^{3}-q_{1} \lambda^{2}-q_{2} \lambda-q_{3}=0 \]

is the corresponding characteristic equation.

Proposition 2: The zero equilibrium of (1) is one of the following: a) locally asymptotically stable if \(b \in(0,1)\);

b) nonhyperbolic and locally stable if \(b=1\);

c) unstable if \(b \gt 1\).

Proof: For zero equilibrium \(\bar{x}=0\) of (1) we have:

\[ q_{1}(0)=0, q_{2}(0)=0 \text { and } q_{3}(0)=b \]

a) Since \(\left|q_{1}\right|+\left|q_{2}\right|+\left|q_{3}\right|=b \lt 1\), by applying Theorem 1.2.5, we deduce that all roots of the characteristic equation lie inside the unit disk, so in a view of Theorem 1.2.1 the zero equilibrium is locally asymptotically stable.

b) If \(b=1\), then characteristic equation (5) associated with zero equilibrium is \(\lambda^{3}=1\). The statement of proposition follows from Theorem 1.2.1.

c) If \(b \gt 1\), then characteristic equation (5) associated with zero equilibrium is \(\lambda^{3}=b\), where \(\lambda=\sqrt[3]{b} \gt 1\). The statement of proposition follows from Theorem 1.2.1.

Proposition 3: If \(b \in(0,1)\), then the positive equilibrium of (1) is unstable and nonhyperbolic.

Proof: For positive equilibrium of (1) we have \(3 a \bar{x}^{k}=1-b\) and

\[ \begin{gathered} q_{1}(\bar{x})=\tfrac{k(1-b)}{3} \gt 0, \quad q_{2}(\bar{x})=\tfrac{k(1-b)}{3} \gt 0 \\ q_{3}(\bar{x})=\tfrac{(k+3)(1-b)}{3}+b \gt 1 \end{gathered} \]

In this case, characteristic equation (5) can be written

\[ \left(\text { as } \lambda^{2}+\lambda+1\right)\left(\lambda-\tfrac{k(1-b)+3}{3}\right)=0 \] and it has exactly one positive root \(\lambda_{1}=1+\tfrac{k(1-b)}{3} \gt 1\) and two complex roots \(\lambda_{2,3}\) such that \(\left|\lambda_{2,3}\right|=1\). Now, the statement of proposition is a consequence of Theorem 1.2.1.

Proposition 4: If \(b \gt 1\), then every solution of (1) tends to infinity.

Proof: Every solution \(\left\{x_{n}\right\}\) of (1) satisfies the inequality

\[ x_{n+1}=x_{n-2} f\left(x_{n}, x_{n-1}, x_{n-2}\right) \geq b x_{n-2} \] which is because the comparison result from Theorem 1.4.1 implies that \(x_{n} \geq y_{n}\), where \(y_{n}\) is the solution of \(y_{n+1}=b y_{n-2}\). We will find the solution of the last difference equation in the form \(\lambda^{n}\), where in general \(\lambda\) is a complex number. Substituting this value into \(y_{n+1}=b y_{n-2}\) we have the following polynomial cubic equation \(\lambda^{3}=b\), or equivalently

\[ \lambda^{3}-(\sqrt[3]{b})^{3}=(\lambda-\sqrt[3]{b})\left(\lambda^{2}+\lambda \sqrt[3]{b}+(\sqrt[3]{b})^{2}\right)=0 \]

After straightforward calculation we get

\[ \lambda_{1}=\sqrt[3]{b} \text { and } \lambda_{2,3}=\sqrt[3]{b} \tfrac{-1 \pm \sqrt{3} i}{2} \] Since, \[ \tfrac{-1+\sqrt{3} i}{2}=\cos \tfrac{2 \pi}{3}+i \sin \tfrac{2 \pi}{3} \] then general solution of the diffrence equation is given by

\[ \begin{aligned} y_{n}=\alpha_{1}(\sqrt[3]{b})^{n}+\alpha_{2} & (\sqrt[3]{b})^{n}\left(\tfrac{-1+\sqrt{3} i}{2}\right)^{n}+\alpha_{3}(\sqrt[3]{b})^{n}\left(\tfrac{-1-\sqrt{3} i}{2}\right)^{n} \\ = & \alpha_{1}(\sqrt[3]{b})^{n}+\alpha_{2}(\sqrt[3]{b})^{n}\left(\cos \tfrac{2 \pi}{3}+i \sin \tfrac{2 \pi}{3}\right)^{n} \\ + & \alpha_{3}(\sqrt[3]{b})^{n}\left(\cos \tfrac{2 \pi}{3}-i \sin \tfrac{2 \pi}{3}\right)^{n} \end{aligned} \] According to the De Moivre's Theorem, \[ \begin{aligned} y_{n}=\alpha_{1}(\sqrt[3]{b})^{n} & +\alpha_{2}(\sqrt[3]{b})^{n}\left(\cos \tfrac{2 n \pi}{3}+i \sin \tfrac{2 n \pi}{3}\right) \\ & +\alpha_{3}(\sqrt[3]{b})^{n}\left(\cos \tfrac{2 n \pi}{3}-i \sin \tfrac{2 n \pi}{3}\right) \end{aligned} \] which become

\[ y_{n}=\alpha_{1}(\sqrt[3]{b})^{n}+\left(\alpha_{2}+\alpha_{3}\right)(\sqrt[3]{b})^{n} \cos \tfrac{2 n \pi}{3}+i\left(\alpha_{2}-\alpha_{3}\right)(\sqrt[3]{b})^{n} \sin \tfrac{2 n \pi}{3} \] Finally \[ y_{n}=c_{1}(\sqrt[3]{b})^{n}+(\sqrt[3]{b})^{n}\left(c_{2} \cos \tfrac{2 n \pi}{3}+c_{3} \sin \tfrac{2 n \pi}{3}\right) \]

Where \(c_{1}=\alpha_{1}, c_{2}=\alpha_{2}+\alpha_{3}, c_{3}=i\left(\alpha_{2}-\alpha_{3}\right) \in \mathbb{R}\) are such that \(y_{n} \geq 0\) for all \(n \in \mathbb{N}\). Hence, \(y_{n} \rightarrow \infty\) which implies \(x_{n} \rightarrow \infty\).

4. Periodic solutions of prime period three

Proposition 5: Equation (1) has infinitely many prime period-three solutions. All period-three solutions belong the surface \(f(x, y, z)=1\) except the point ( \(\bar{x}_{+}, \bar{x}_{+}, \bar{x}_{+}\)) where \(\bar{x}_{+}\)is the positive equilibrium of (1).

Proof: The period-three solutions \(\phi, \psi, \omega\) of (1) satisfy the system

\[ \phi=\phi f(\omega, \psi, \phi), \psi=\psi f(\phi, \omega, \psi), \omega=\omega f(\psi, \phi, \omega) \]

Since \(f(\omega, \psi, \phi)=f(\phi, \omega, \psi)=f(\psi, \phi, \omega)\) and \(0 \leq \phi \lt \psi \lt \omega\), we deduce that \(f(\phi, \omega, \psi)=1\). Therefore, every point of the set \(\mathcal{S}=\{(x, y, z): f(x, y, z)=\) \(1\} \backslash\left\{\left(\bar{x}_{+}, \bar{x}_{+}, \bar{x}_{+}\right)\right\}\)is a prime period-three solution of (1). Thus the surface \(f(x, y, z)=1\) separates the first octant of the initial condition into two regions and \(\mathcal{J}=\left\{(x, y, z) \in \mathbb{R}^{3}: x \geq 0, y \geq 0, z \geq 0\right\}\).

5. Main result, Analysis of global stability

The next our main result describes the global behavior of all solutions of (1).

Theorem 1: Consider the difference equation (1) with initial conditions \(x_{-2}, x_{-1}, x_{0} \in \mathcal{J}\) and \(b \in(0,1)\). Then (1) has a zero equilibrium and a unique positive equilibrium \(\bar{x}_{+}\). The surface \(\mathcal{S}\) separates the set \(\mathcal{J}\) into two regions: the region below the surface \(\mathcal{S}\) is the basin of attraction of the point \(E_{0}(0,0,0)\) and the region above the surface \(\mathcal{S}\) is the basin of attraction of the point at infinity and every point on \(\mathcal{S}\) except \(E_{+}\left(\bar{x}_{+}, \bar{x}_{+}, \bar{x}_{+}\right)\)is a period-three solution of (1).

Proof: According to Proposition 1, the equation (1) has zero equilibrium and unique positive equilibrium \(\bar{x}_{+}=\sqrt[k]{\tfrac{1-b}{3 a}}\). By applying Proposition 2 we conclude that the zero equilibrium is locally asymptotically stable (sink). According to Proposition 3, the positive equilibrium \(\bar{x}_{+}\)is an unstable nonhyperbolic point. From Proposition 5 it follows that the equation (1) has infinitely many prime period-three solutions and all of them belong to the set \(\mathcal{S}\).

We also clame that the surface \(f(x, y, z)=1\) is a graph of the decreasing function \(z=z(x, y)\) in both variables on set \(\mathcal{J}\). Indeed, from \(f(x, y, z)=1\) weget \(z=\sqrt[k]{c-x^{k}-y^{k}}\), where \(c=\tfrac{1-b}{3 a}\) and \(\tfrac{\partial z}{\partial x}=\tfrac{-k x^{k-1}}{\sqrt[k]{\left(c-x^{k}-y^{k}\right)^{k-1}}} \leq 0, \tfrac{\partial z}{\partial y}=\tfrac{-k y^{k-1}}{\sqrt[k]{\left(c-x^{k}-y^{k}\right)^{k-1}}} \leq 0\).

It is easy to see that the function \(f(x, y, z)\) is an increasing function in each of its arguments \(x, y, z\). Now, let \(\left\{x_{n}\right\}\) be a solution of (1) for initial values \(x_{-2}, x_{-1}, x_{0} \in \mathcal{J}\) such that the point \(\left(x_{-2}, x_{-1}, x_{0}\right)\) lies below the surface \(\mathcal{S}\). That yield s \(f\left(x_{-2}, x_{-1}, x_{0}\right) \lt 1\). One can easily see that \(f(x, y, z)\) is a symmetric function, and so

\[ f(x, y, z)=f(y, x, z)=f(z, y, x)=f(x, z, y) . \]

Thus, because of the monotonicity of \(f\) in all of its arguments, we get

\[ \begin{gathered} x_{1}=x_{-2} f\left(x_{0}, x_{-1}, x_{-2}\right)=x_{-2} f\left(x_{-2}, x_{-1}, x_{0}\right) \lt x_{-2} \\ x_{2}=x_{-1} f\left(x_{1}, x_{0}, x_{-1}\right) \lt x_{-1} f\left(x_{-2}, x_{0}, x_{-1}\right) \\ =x_{-1} f\left(x_{-2}, x_{-1}, x_{0}\right) \lt x_{-1} \\ x_{3}=x_{0} f\left(x_{2}, x_{1}, x_{0}\right) \lt x_{0} f\left(x_{-1}, x_{-2}, x_{0}\right)=x_{0} f\left(x_{-2}, x_{-1}, x_{0}\right) \\ \lt x_{0} \end{gathered} \] Therefore, \(\left(x_{-2}, x_{-1}, x_{0}\right)\) and \(\left(x_{1}, x_{2}, x_{3}\right)\) are two “North-East ordered” points:

\[ \left(x_{-2}, x_{-1}, x_{0}\right) \preccurlyeq_{N E}\left(x_{1}, x_{2}, x_{3}\right) \]

This means that the point \(\left(x_{1}, x_{2}, x_{3}\right)\) also belows to the surface \(\mathcal{S}\) and hence \(f\left(x_{1}, x_{2}, x_{3}\right) \lt 1\). Similarly, one can find that

\[ \begin{gathered} x_{4}=x_{1} f\left(x_{3}, x_{2}, x_{1}\right)=x_{1} f\left(x_{1}, x_{2}, x_{3}\right) \lt x_{1} \\ x_{5}=x_{2} f\left(x_{4}, x_{3}, x_{2}\right) \lt x_{2} f\left(x_{1}, x_{3}, x_{2}\right)=x_{2} f\left(x_{1}, x_{2}, x_{3}\right) \lt x_{2} \\ x_{6}=x_{3} f\left(x_{5}, x_{4}, x_{3}\right) \lt x_{3} f\left(x_{2}, x_{1}, x_{3}\right)=x_{3} f\left(x_{1}, x_{2}, x_{3}\right) \lt x_{3} \end{gathered} \]

Continuing in this way we obtain that

\[ (0,0,0) \preccurlyeq_{N E} \ldots \preccurlyeq_{N E} E_{m} \preccurlyeq_{N E} \ldots \preccurlyeq_{N E} E_{2} \preccurlyeq_{N E} E_{1} \preccurlyeq_{N E} E_{0} \] where \(E_{m}\left(x_{-2+3 m}, x_{-1+3 m}, x_{3 m}\right)\) for \(m \in \mathbb{N}_{0}\). All this leads that those subsequences \(\left\{x_{3 n}\right\},\left\{x_{3 n+1}\right\}\) and \(\left\{x_{3 n+2}\right\}\) are monotonically decreasing and bounded below by zero. Since below the surface \(\mathcal{S}\) there is no period-three solution, we deduce that \(x_{3 n} \rightarrow 0, x_{3 n+1} \rightarrow 0\), , and \(x_{3 n+2} \rightarrow 0\).

On the other hand, suppose that \(\left\{x_{n}\right\}\) is a solution of (1) with initial values \(x_{-2}, x_{-1}, x_{0} \in \mathcal{J}\), such that the point \(\left(x_{-2}, x_{-1}, x_{0}\right)\) lies above the surface \(\mathcal{S}\). Then \(f\left(x_{-2}, x_{-1}, x_{0}\right) \gt 1\). In view of the method used above,

\[ E_{0} \preccurlyeq_{N E} E_{1} \preccurlyeq_{N E} E_{2} \preccurlyeq_{N E} \ldots \preccurlyeq_{N E} E_{m} \preccurlyeq_{N E} \ldots, \] where \(E_{m}\left(x_{-2+3 m}, x_{-1+3 m}, x_{3 m}\right), m \in \mathbb{N}_{0}\). Hence, the subsequences \(\left\{x_{3 n}\right\}\), \(\left\{x_{3 n+1}\right\}\), and \(\left\{x_{3 n+2}\right\}\) of solution \(\left\{x_{n}\right\}\) of (1) are monotonically increasing and tend to the point at infinity.

Let \(\mathcal{B}(0,0,0), \mathcal{B}\left(\bar{x}_{+}, \bar{x}_{+}, \bar{x}_{+}\right), \mathcal{B}(\infty)\) denote the basins of attraction of zero equilibrium, positive equilibrium of (1), and the point at infinity, respectively. Finally \[ \begin{gathered} \mathcal{B}(0,0,0)=\{(x, y, z) \in \mathcal{J}: f(x, y, z) \lt 1\}, \mathcal{B}(\infty)=\{(x, y, z) \in \mathcal{J}: f(x, y, z) \gt 1\} \\ \mathcal{B}\left(\bar{x}_{+}, \bar{x}_{+}, \bar{x}_{+}\right)=\left\{\left(\bar{x}_{+}, \bar{x}_{+}, \bar{x}_{+}\right)\right\} \end{gathered} \]

In some special cases, we can extend our research to the whole space \(\mathbb{R}^{3}\).

Theorem 2: Consider the difference equation (1) for \(k=2\), where initial conditions \(\left(x_{-2}, x_{-1}, x_{0}\right) \in \mathbb{R}^{3}\) and \(b \in(0,1)\). Then (1) has a zero equilibrium, a positive equilibrium \(\bar{x}_{+},\) and a negative equilibrium \(\bar{x}_{+}\), and a negative equilibrium \(\bar{x}_{-}=\bar{x}_{+}\). In this case, the surface \(\mathcal{S}\) is a sphere:

\[ x^{2}+y^{2}+z^{2}=\tfrac{1-b}{a} \]

which separates the set \(\mathbb{R}^{3}\) into two regions: the region inside the sphere \(\mathcal{S}\) is the basin of attraction of the point \(E_{0}(0,0,0)\) and the region outside the sphere \(\mathcal{S}\) is the basin of attraction of the point at infinity. Every point on \(\mathcal{S}\) except \(E_{+}\left(\bar{x}_{+}, \bar{x}_{+}, \bar{x}_{+}\right)\) and \(E_{-}\left(\bar{x}_{-}, \bar{x}_{-}, \bar{x}_{-}\right)\)is a period-three solution of (1).

Proof: Because of Theorem 1, the equation (1) has a locally asymptotically stable zero equilibrium, unstable nonhyperbolic equilibrium \(\bar{x}_{+}=\sqrt{\tfrac{1-b}{3 a}}\), and infinitely many the prime period-three solutions that belong to sphere \(\mathcal{S}\). One can show that the negative equilibrium \(\bar{x}_{-}\)has the same characteristic equation as the positive equilibrium \(\bar{x}_{+}\). By applying Proposition 3, we prove that the negative equilibrium \(\bar{x}_{-}\)is also an unstable nonhyperbolic point.

Let \(A_{0}\left(x_{-2}, x_{-1}, x_{0}\right)\) be the interior point of sphere \(\mathcal{S}\). Denote by \(d(A, B)\) the distance between two points \(A\) and \(B\). According to Theorem 1, every solution \(\left\{x_{n}\right\}\) tends to the zero equilibrium provided \(A_{0} \in \mathcal{J}\). Now consider the two sequences of the points \(\left\{A_{n}\left(x_{n-2}, x_{n-1}, x_{n}\right)\right\}\) and real numbers \(\left\{\rho_{n}\right\}\), where \(\rho_{n}=d\left(A_{n}, E_{0}\right)=\) \(\sqrt{x_{n}^{2}+x_{n-1}^{2}+x_{n-2}^{2}}, n \in \mathbb{N}_{0}\). If we assume that point \(A_{n}\) is the interior point of sphere \(\mathcal{S}\), then

\[ x_{n}^{2}+x_{n-1}^{2}+x_{n-2}^{2} \lt \tfrac{1-b}{a} \text { or } \rho_{n}^{2} \lt \tfrac{1-b}{a} \]

So

\[ \rho_{n+1}^{2}=x_{n+1}^{2}+x_{n}^{2}+x_{n-1}^{2} \] and from (1) we find \(x_{n+1}^{2}\), which implies consequently

\[ \begin{gathered} \rho_{n+1}^{2}=x_{n-2}^{2}\left(a\left(x_{n}^{2}+x_{n-1}^{2}+x_{n-2}^{2}\right)+b\right)^{2}+x_{n}^{2}+x_{n-1}^{2}, \\ \rho_{n+1}^{2}=x_{n-2}^{2}\left(a \rho_{n}^{2}+b\right)^{2}+x_{n}^{2}+x_{n-1}^{2}, \\ \rho_{n+1}^{2} \lt x_{n-2}^{2}\left(a \tfrac{1-b}{a}+b\right)^{2}+x_{n}^{2}+x_{n-1}^{2}, \\ \rho_{n+1}^{2} \lt x_{n-2}^{2}+x_{n}^{2}+x_{n-1}^{2}=\rho_{n}^{2} . \end{gathered} \]

Hence \(\rho_{n+1} \leq \rho_{n}\), and the point \(A_{n+1}\) is also an interior point of sphere \(\mathcal{S}\) closer to point \(E_{0}\) than \(A_{n}\). By applying mathematical induction with \(A_{0}\) being an interior point of the sphere \(\mathcal{S}\), we obtain that all \(A_{n}\) are interior points of \(\mathcal{S}\) satisfying

\[ \rho_{0} \gt \rho_{1} \gt \cdots \gt \rho_{n} \gt \rho_{n+1} \gt \cdots, \quad\left(\rho_{k}=d\left(A_{k}, E_{0}\right)\right) . \]

So the sequence of distances \(\left\{\rho_{n}\right\}\) is decreasing, bounded below by zero and hence convergent. Since there is no interior period-three solution of (1) in sphere \(\mathcal{S}\), the subsequences \(\left\{\left|x_{3 n}\right|\right\},\left\{\left|x_{3 n+1}\right|\right\}\) and \(\left\{\left|x_{3 n+2}\right|\right\}\) must approach the zero equilibrium.

The case when \(A_{0}\left(x_{-2}, x_{-1}, x_{0}\right)\) is an outer point of sphere \(\mathcal{S}\) is similar and will be omitted. One can show that \(\rho_{n+1} \gt \rho_{n} \gt \tfrac{1-b}{a}\) for all \(n \in \mathbb{N}_{0}\), which means that the subsequences \(\left\{\left|x_{3 n}\right|\right\},\left\{\left|x_{3 n+1}\right|\right\}\), and \(\left\{\left|x_{3 n+2}\right|\right\}\) are monotonically increasing. Since on the sphere \(\mathcal{S}\) is no period-three solution or equilibrium point of (1), we deduce that \(\left\{\left|x_{3 n}\right|\right\},\left\{\left|x_{3 n+1}\right|\right\}\), and \(\left\{\left|x_{3 n+2}\right|\right\}\) must tend to infinity. The figures 1 and 2 are visual illustrations of Theorem 1 for \(k=1\) and Theorem 2 for \(k=2\).

6. Conclusions

In general, the polynomial difference equations and their applications are a great source of ideas for finding an approximate solution of difference equations at all in this dynamic area of research. In the planar (two-dimensional) case, the theory of monotonic maps guarantees the existence of unique stable manifold (onedimensional increasing/decreasing smooth curve). This manifold passes through the positive equilibrium point (saddle point or a nonhyperbolic point) which separates the first quadrant of initial conditions into two disjoint regions. By now we are not able to find the equation of a stable manifold, but able to find asymptotic approximations of this equation. Moreover, bringing the above considered map to the normal form around the equilibrium solutions, cosidering the period-two solutions and using the method of undetermined coefficients makes possible to obtain some local approximations of the considered manifold. In the threedimensional case, no theory provides to us with stable and unstable manifolds through equilibrium points, so we are forced to develop another method and techniques of research to understand the dynamics of third-order polynomial difference equations.

Results in this paper may be used for observation and investigation of difference equations of type

(6)\[ x_{n+1}=x_{n-2} f\left(x_{n}, x_{n-1}, x_{n-2}\right), \]

where \(f\left(x_{n}, x_{n-1}, x_{n-2}\right)=a_{1} x_{n}^{k}+a_{2} x_{n-1}^{k}+a_{3} x_{n-2}^{k}+b\) with \(a_{1}, a_{2}, a_{3}, b \gt 0\) and \(k \in \mathbb{N}\).

If \(m=\min \left\{a_{1}, a_{2}, a_{3}\right\}\) and \(M=\max \left\{a_{1}, a_{2}, a_{3}\right\}\), then

\[ f_{1}\left(x_{n}, x_{n-1}, x_{n-2}\right) \leq f\left(x_{n}, x_{n-1}, x_{n-2}\right) \leq f_{2}\left(x_{n}, x_{n-1}, x_{n-2}\right) \] where

\[ \begin{gathered} f_{1}\left(x_{n}, x_{n-1}, x_{n-2}\right)=m x_{n}^{k}+m x_{n-1}^{k}+m x_{n-2}^{k}+b, \\ f_{2}\left(x_{n}, x_{n-1}, x_{n-2}\right)=M x_{n}^{k}+M x_{n-1}^{k}+M x_{n-2}^{k}+b . \end{gathered} \]

By applying Theorem 1.4.1 (Comparison result) on difference equations

\[ \begin{aligned} & x_{n+1}=x_{n-2} f_{1}\left(x_{n}, x_{n-1}, x_{n-2}\right) \\ & x_{n+1}=x_{n-2} f\left(x_{n}, x_{n-1}, x_{n-2}\right) \\ & x_{n+1}=x_{n-2} f_{2}\left(x_{n}, x_{n-1}, x_{n-2}\right) \end{aligned} \] one can find a part of basins of attraction of zero equilibrium and point at infinity of (6). If \(k=1\), surface \(\mathcal{S}\) from Theorem 1 becomes a real plane in \(\mathbb{R}^{3}\).

REFERENCES

AGARWAL R., 1992. Difference Equations and Inequalities, Theory, Methods and Applications, Marcel Dekker Inc., New York.

ALLIGOOD K. T., SAUER T & YORKE J. A., 1997. CHAOS An Introduction to Dynamical Systems, Springer-Verlag, New York, Berlin, Heidelberg, Tokyo.

AMLEH A. M., CAMOUZIS E. & LADAS G., 2008. On the dynamics of rational difference equation, Part I, International Journal of Difference Equation, vol.3, no. 1, pp. 1 – 35.

AMLEH A. M., CAMOUZIS E. & LADAS G., 2008. On the dynamics of rational difference equation, Part II, International Journal of Difference Equation, vol. 3, no. 2, pp. 195 – 225.

BEKTEŠEVIĆ J., HADŽIABDIĆ V., MEHULJIĆ M., METOVIĆ S. & PERVAN N., 2022. The Global Behavior of a Certain Difference Polynomial Equation. International Journal of Difference Equation, vol. 2, no. 17, pp. 305 – 317.

BEKTEŠEVIĆ J., HADŽIABDIĆ V., MEHULJIĆ M. & MUJIĆ N., 2018. The Global Behavior of a Quadratic Difference Equation. Filomat, vol. 32, no. 18, pp. 6203 – 6210.

BEKTEŠEVIĆ J., HADŽIABDIĆ V. & MEHULJIĆ M., 2021. The Global Behavior of a Certain Difference Polynomial Equation. Filomat,vol. 35, no. 11, pp. 3901 – 3908.

BEKTEŠEVIĆ J., KULENOVIĆ M. R. S. & PILAV E., 2014. Global Dynamics of Quadratic Second Order Difference Equation in the First Quadrant. Applied Mathematics and Computation, vol. 227, pp. 50 – 65.

BRETT A. & KULENOVIĆ M. R. S., 2009. Basins of attraction of equilibrium points of monotone difference equations. Sarajevo Journal of Mathematics vol. 5, no. 18, pp. 211 – 233.

CAMOUZIS E. & LADAS G., 2008. Dynamics of Third Order Rational Difference Equations with Open Problems and Conjectures. London: Chapman and Hall/CRC Boca Raton.

DEVANEY R. L., 1992. A First Course in Chaotic Dynamical Systems: Theory and Experiment, Addison-Wesley, Reading, MA

ELAYDI S., 2000. Discrete Chaos, Chapman, and Hall/CRC Boca Raton, FL

ELAYDI S., 2005. An Introduction to Difference Equations (Third Edition). New York: Springer Science + Business Media.

GUCKENHEIMER J. & HOLMES P., 1983. Nonlinear Oscillations, Dynamical Systems and Bifurcations of Vector Fields, Springer-Verlag, New York, Berlin, Heidelberg, Tokyo

HALE J. & KOCAK H., 1991. Dynamics and Bifurcations, Springer-Verlag, New York, Berlin, Heidelberg, Tokyo

KULENOVIĆ M. R. S. & LADAS G., 2001. Dynamics of Second Order Rational Difference Equations with Open Problems and Conjectures. London: Chapman and Hall/CRC Boca Raton.

KULENOVIĆ M. R. S. & MERINO O., 2002. Discrete Dynamical Systems and Difference Equations with Mathematica. London: Chapman & Hall/CRC, Boca Raton.

KULENOVIĆ M. R. S. & MERINO O., 2010. Invariant manifolds for competitive discrete systems in the plane. International Journal of Bifurcation and Chaos, vol. 20, no. 8, pp. 2471 – 2486.

MILNOR J., 2000. Dynamics in One Complex Variable. New York: Stony Brook.

MOROSAWA S., NISHIMURA Y., TANIGUCHI M. & UEDA T., 2000. Holomorphic dynamics. Cambridge University Press UK.