Open Problems on Information and Feedback Controlled Systems

Abstract

Feedback or closed-loop control allows dynamical systems to increase their performance up to a limit imposed by the second law of thermodynamics. It is expected that within this limit, the system performance increases as the controller uses more information about the system. However, despite the relevant progresses made recently, a general and complete formal development to justify this statement using information theory is still lacking. We present here the state-of-the-art and the main open problems that include aspects of the redundancy of correlated operations of feedback control and the continuous operation of feedback control. Complete answers to these questions are required to firmly establish the thermodynamics of feedback controlled systems. Other relevant open questions concern the implications of the theoretical results for the limitations in the performance of feedback controlled flashing ratchets, and for the operation and performance of nanotechnology devices and biological systems.

1. Introduction

Control over physical systems exerted by an external agent is ubiquitous in physics and engineering, where it is used for purposes such as stabilizing unstable dynamics or increasing system performance. Studying the properties of controlled systems is a fundamental task for engineering purposes and from a fundamental physical point of view [1].

The operation of the controller modifies the system dynamics, and the system in turn can also influence the controller decisions. Controllers are called closed-loop or feedback controllers if their actions are dependent on the state of the system and called open-loop if otherwise. In other words, closed-loop or feedback controllers actuate the system using the information about the system state, while open-loop controllers actuate the system independently of the system state.

It is intuitively clear that using information about the system state is potentially useful to improve the controller action on the system and increase the system performance (e.g., the power output or the efficiency). However, the ceiling of such performance enhancement from making use of system state information is not clearly quantified. In particular, a complete and general development is lacking for calculating the performance in feedback controlled systems using thermodynamic principles. The aim of this paper is to present and put in context several open questions in this field.

In the following section we briefly review some selected results about the relation between information and feedback. Next, in Section 3, we present several of the still unsolved questions and discuss their implications. Finally, our general conclusions are presented in Section 4.

2. State-of-the-Art

In this section, we introduce some helpful results in the context of the below-mentioned open problems. The results presented here will be useful for one to get insight into the open questions and address their answers.

In SubSection 2.1 we comment some of the main results found in the framework of Maxwell’s demon and computation theory [2,3,4,5,6,7,8]. In SubSection 2.2 we summarize recent results about the relation between the achievable decrease of entropy and the information about the system gathered by the controller [9,10,11]. In SubSection 2.3 we show recent results concerning performance and information in a relevant stochastic system, namely, a feedback flashing ratchet. In SubSection 2.4 we summarize recent results on the thermodynamics of feedback controlled systems. Finally, in SubSection 2.5 we comment several very recent results that generalize statistical physics identities and relations to feedback controlled systems.

2.1. Maxwell’s Demon

The trade-off between information and feedback control has been a difficult task since the birth of Maxwell’s demon [2], a hypothetical being that gathers information about a system to decrease the entropy of it without performing work. Szilard’s seminal work [3] contains the basic ingredients of the interrelation between information theory and thermodynamics, which was later precisely stated in the Landauer’s principle: The erasure of one bit of information at temperature T implies an energetic cost of at least $k_{B}Tln2$ [4]. Bennett [5] pointed out that the Landauer’s principle is essential to preserve the second law of thermodynamics in feedback systems, as the controller must erase its memory after each cycle to allow the whole system to truly operate cyclically. How to achieve the shorter description for the memory record of the demon in order to minimize the energetic erasure cost was established by Zurek [6,7] by using an algorithmic complexity approach. He proposed to compress the memory record of the demon before the erasure. This compression of the memory can be done, in principle, without energy expenditure—i.e., reversibly. The reversibility of the compression process is related to the fact that a memory can be erased at zero energy cost provided its state is known, i.e., stored in another memory [8]. This implies that the minimum energy required to erase the memory of the demon is not given by the number of bits stored, but by the minimum number of bits in which the information contained in the memory can be stored (after compression). This important result raises the attention on the relevance of considering whether the information is redundant or not.

2.2. Information-Theoretic Limits of Control

A fundamental magnitude in information theory [12] is the Shannon entropy of a random variable X. If X takes values x with probability $p_X\left(x\right)$, then the Shannon entropy is given, in nats ($1\text{ bit}=ln2\text{ nats}$), by

$$H\left(X\right)=-\sum _x{p}_X\left(x\right)ln{p}_X\left(x\right)$$

(1)

and it is a measure of the average uncertainty of the random variable [12]. (When X is the macrostate of a system and x is its microstate, $H\left(X\right)$ is related with the statistical physics entropy S of the system by $S=k_{B}H$). On the other hand, the conditional entropy of X given another random variable C is

$$H\left(X\right|C)=\sum _c{p}_C\left(c\right)H\left(X\right|C=c)$$

(2)

with $H\left(X\right|C=c)=-{\sum }_x{p}_{X|C}\left(x\right|c)ln{p}_{X|C}\left(x\right|c).$ From these magnitudes, the mutual information between two random variables, which is the information that one variable contains about the other one, is computed as

$$I(X;C)=H\left(X\right)-H\left(X\right|C)$$

(3)

Hence, the mutual information can be viewed as the reduction of uncertainty of one of the random variables due to the knowledge of the other [12].

Fundamental limits on the controllability of dynamical systems can be studied in the light of information theory [9,10,11]. Recently, it has been established that the maximum amount of entropy $\Delta H_{\text{closed}}$ that can be extracted from any dynamical system by a single closed-loop actuation is upper bounded by the maximum entropy decrease achievable by open-loop control $\Delta H_{\text{open}}$ plus the mutual information between the controller C and the state of the system X [10,11]:

$$\Delta H_{\text{closed}}\le \Delta H_{\text{open}}+I(X;C)$$

(4)

This result has provided insight into the relations between information and feedback, explicitly quantifying and showing one of the limitations of attainable performance improvement with using certain amount of system information. The analogous results for the physical entropy of a physical system has been recently obtained for repeated closed-loop actuations, see Section 2.4 and [13]. Also recently the application of information theory to biochemical reaction networks has allowed to show the limits of the ability to control fluctuations in molecular abundances inside the cells as a function of the number of signaling events [14].

2.3. Feedback Controlled Ratchets and Information

Let us now briefly review some related results applied to Brownian ratchets, viz., stochastic systems that are capable of rectifying thermal fluctuations by working out of equilibrium. Recently, feedback controlled versions of Brownian ratchets have been studied [15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30] and experimentally realized [31]. (See Figure 1 for a Brownian ratchet that is based on the flashing of a potential.)

Figure 1. In a feedback flashing ratchets a controller with information of the state of the system rectifies the thermal fluctuations of Brownian particles by switching on and off a periodic potential (which can be asymmetric, as the one depicted in the plot, or symmetric). This rectification of thermal fluctuations generates particle flux even against an external force, thus performing work [15,16,17,18,19,20,21,22,23,31,32].

Figure 1. In a feedback flashing ratchets a controller with information of the state of the system rectifies the thermal fluctuations of Brownian particles by switching on and off a periodic potential (which can be asymmetric, as the one depicted in the plot, or symmetric). This rectification of thermal fluctuations generates particle flux even against an external force, thus performing work [15,16,17,18,19,20,21,22,23,31,32].

These feedback controlled ratchets can improve the performance over its open-loop counterparts thanks to the information the controller gets about the system state. These information-dependent ratchets are relevant in chemical and biological systems [33,34,35], and also in nanotechnology devices [15,16,17,18,19,20,21,22,23,31]. Other ratchets and Brownian motors inspired by Maxwell’s demon present a directed motion when there is a temperature difference between two different parts of the motor (one part could be thought as the controller and the other as the controlled system) [36,37]. However, for these later systems no systematic study on the role of information has been performed. On the other hand, for feedback flashing ratchets, relations between the information I (about the system) gathered by the controller per step and the performance have been recently obtained [38,39]. These relations have been established for a collective feedback flashing ratchet compounded of one or few particles, considering the optimal protocol for one particle and its generalization to few particles. In this protocol the controller gathers the sign of the net force over the particles if the potential would be on. If the sign is measured with a perfect measurement device and it is transmitted to the controller through a noiseless channel, the controller gets one bit of information about the system (the maximum value of the information I per step for this system). However, if the measurement or the transmission is imperfect, the controller gets less than one bit of information about the system, and this limits the increase of performance improvement that can be obtained thanks to the information.

Let us call $v_{\text{open}}$ the maximum flux that can be achieved with an open-loop control protocol, and $v_{\text{closed}}$ the maximum flux that can be obtained with a closed-loop control protocol that uses an amount of information I per measurement step. The computations in [38] have shown that the maximum increase in the flux that can be obtained has an upper bound proportional to the square root of the information I received in each step, i.e.,

$$v_{\text{closed}}-v_{\text{open}}\le C_1\sqrt{I}$$

(5)

(with $C_1$ being a constant depending on the system’s characteristics).

An analogous bound has been found for the maximum power output in [39]. Let us call $P_{\text{open}}$ the maximum power output that can be obtained with an open-loop control, and $P_{\text{closed}}$ the maximum power output that can be obtained with a closed-loop control that uses an amount of information I per measurement step. It has been shown in [39] that the maximum increase in the power output has an upper bound proportional to the information I gathered by the controller per step, i.e.,

$$P_{\text{closed}}-P_{\text{open}}\le C_{2}I$$

(6)

(with $C_2$ being a constant depending on the system’s characteristics).

The previous bounds [Equation (5) and Equation (6)] provide insight into the limitations of performance enhancement that can be obtained with a given amount of information. In particular, they have already been used to assess the feasibility of an experimental realization of feedback ratchets before its implementation [22,23,31]. In addition, these bounds are an example of the type of predictions that a general theory would provide about the relations between information and feedback control.

2.4. Thermodynamics of Feedback Controlled Systems

Correlated repeated actuation of the feedback controller is present when the controller repeatedly actuates the system without waiting for the relaxation of the system to equilibrium between actuations. (Thus, adiabatic approximations [40] cannot be applied.) In the correlated repeated actuation, the information received by the controller from the different measurements has redundancies. These redundancies have to be consistently taken into account as they reduce the possibilities to increase the performance up to the bounds that would be predicted from the known results for uncorrelated repeated actuations [2,3,4,5].

Recently, the entropy reduction of a system due to the correlated repeated actuation of an external controller has been computed [13]. This was the main lacking ingredient to establish the thermodynamics of feedback controlled systems (i.e., to be able to compute the thermodynamic potentials and its relations for these systems). In the following paragraphs we include an excerpt of [13] summarizing the main results.

A control step of a controller involves: measuring the system, deciding the control action to take given the measurement output, and acting on the system following the control action chosen. From the point of view adopted in [13], i.e., the point of view of the system, we are just concerned about which action the controller has taken on the system, whose consequence is to modify the evolution of the system. We are not concerned about the state of the controller.

Let us call $X_k:=X\left(t_k\right)$ the macrostate of a general physical system at the kth control step of the controller (at time $t_k$), and $C_k$ the action taken by the controller (it is not a specification of the state of the controller). The entropy just before the first measurement is given by the statistical entropy

$$S_1^b=-k_B\sum _{x\in \mathcal{X}}{p}_{X_1}\left(x\right)ln{p}_{X_1}\left(x\right)=:k_{B}H\left(X_1\right)$$

(7)

with $\mathcal{X}$ the set of possible microstates of the system. If the measurement implies the control action $C_1=c$, the entropy of the system decreases to

$$k_{B}H\left(X_1\right|C_1=c):=-k_B\sum _{x\in \mathcal{X}}{p}_{X_1|C_1}\left(x\right|c)ln{p}_{X_1|C_1}\left(x\right|c)$$

(8)

because knowing that the control action is $C_1=c$ gives additional information about the actual microstate of the system, due to the correlation between measurements by the controller and control actions. Therefore, the average entropy after the first control step can be obtained by the following weighted average over the set $\mathcal{C}$ of all possible control actions:

$$S_1^a=\sum _{c\in \mathcal{C}}{p}_{C_1}\left(c\right)k_{B}H\left(X_1\right|C_1=c)=:k_{B}H\left(X_1\right|C_1)$$

(9)

Hence the average variation of the entropy at the first step is

$$\Delta S_1=S_1^a-S_1^b=k_{B}H\left(X_1\right|C_1)-k_{B}H\left(X_1\right)=:-k_{B}I(X_1;C_1)$$

(10)

i.e., it is $k_B$ times the (minus) mutual information [12] between $X_1$ and $C_1$ given in nats (ln 2 nats = 1 bit).

After the first measurement we have a set of macrostates (one for each possible control action). Each of the macrostates evolves to give an entropy $k_{B}H\left(X_2\right|C_1=c)$ just before the second control step. The average entropy of the system after the second step is

$$S_2^a=\sum _{c,c^{\prime }\in \mathcal{C}}{p}_{C_2{C}_1}(c^{\prime },c)k_{B}H\left(X_2\right|C_2=c^{\prime },C_1=c)=k_{B}H\left(X_2\right|C_2,C_1)$$

(11)

and the average variation of the entropy at this second control step is

$$\Delta S_2=S_2^a-S_2^b=k_{B}H\left(X_2\right|C_2,C_1)-k_{B}H\left(X_2\right|C_1)=-k_{B}I(X_2;C_2|C_1)$$

(12)

This conditioning of the mutual information shows that the entropy of the system is only reduced by new non-redundant information. Analogously we get for the average entropy reduction of the k-th step

$$\Delta S_k=-k_{B}I(X_k;C_k|C_{k-1},\dots ,C_1)=-k_{B}H\left(C_k\right|C_{k-1},\dots ,C_1)+k_{B}H\left(C_k\right|C_{k-1},\dots ,C_1,X_k)$$

(13)

Therefore, the total average entropy reduction due to the information used in M control steps is $\Delta S_{\mathrm{info}}={\sum }_{k=1}^M\Delta S_k$, i.e.,

$$\Delta S_{\mathrm{info}}=-k_B\sum _{k=1}^{M}I(C_k;X_k|C_{k-1},\dots ,C_1)$$

(14)

or equivalently

$$\Delta S_{\mathrm{info}}=-k_{B}H(C_M,\dots ,C_1)+k_B\sum _{k=1}^{M}H\left(C_k\right|C_{k-1},\dots ,C_1,X_k)$$

(15)

The first term in this last equation gives the maximum entropy decrease attainable, and the second term gives the reduction on this maximum due to decorrelations between the control actions and the state of the system (decorrelations that can be due to errors and noises affecting the controller operation). These results have a broad range of applications including deterministic and non-deterministic feedback controls, and they indicate that the average entropy reduction due to the information used can be computed in terms of the joint probabilities for the state of the system and the control actions history.

These results have also allowed us to establish the thermodynamics of feedback controlled systems. As an application, we have computed the efficiency of isothermal feedback controlled system, and as an example, we have applied the results to compute the efficiency of a Markovian particle pump. This isothermal feedback controlled system can extract work from a single heat bath, thanks to the use of information, see Figure 2 and [13]. For an experimental demonstration of a Markovian particle pump, see [41]. Very recently other motors that extract work from a single heat bath have been studied [42,43,44,45].

Figure 2. Markovian particle pump with two lattice sites between barriers. The particle is in contact with a thermal bath at temperature T, and an external controller measures the particle location and raises the closest barrier to the left. The result is that the particle is further and further constrained to the right, increasing its potential energy. This is a simple feedback controlled system that extracts useful work from the entropy reduction due to the information about the system used by the external feedback controller [13].

Figure 2. Markovian particle pump with two lattice sites between barriers. The particle is in contact with a thermal bath at temperature T, and an external controller measures the particle location and raises the closest barrier to the left. The result is that the particle is further and further constrained to the right, increasing its potential energy. This is a simple feedback controlled system that extracts useful work from the entropy reduction due to the information about the system used by the external feedback controller [13].

2.5. Generalization of Identities and Relations to Feedback Controlled Systems

Very recently, several important identities and nonequilibrium equalities of Statistical Physics have been generalized to enlarge their range of applicability to feedback systems. Sagawa and Ueda [46,47,48,49] have extended the fluctuation theorem and the Jarzynski relation [50] to feedback controlled systems even for multiple measurements. Once again, they show that the mutual information plays a crucial role to reformulate the nonequilibrium statistical mechanics to feedback controlled processes. Abreu and Seifert have extended the Hatano and Sasa equality to feedback controlled processes [45], considering systems that, even for fixed control parameters, have nonequilibrium steady states with dissipation. These results also provide bounds for the power output that can be extracted under isothermal conditions. The detailed fluctuation theorem has also been generalized to processes with feedback in [51,52,53]. All these results broadly extend the available resources to study feedback controlled systems and to find the bounds to their performance.

3. Open Problems

In this section we present several open problems in the context of the connections between the study of feedback controlled systems and information theory. We have selected those questions that we think would imply a greater progress in the understanding of these connections, and would further advance the understanding of feedback controlled systems. The questions we have selected mainly concern the establishment of bounds to the performance and the establishment of the thermodynamics of feedback controlled systems. Other open questions on the links between feedback control and information theory are raised in the Bechhoefer review on control [1]. Regarding other open questions that concern the implications of the theoretical results for the performance limitations of particular systems, due to their particular relevance we discuss here feedback controlled flashing ratchets, and also, nanotechnology devices and biological systems.

3.1. Feedback Controlled Systems: Limitations to the Performance and Thermodynamics

Feedback controlled systems are usually designed with the aim that the controller profits from the information about the system state it receives in order to increase the performance of the system. Therefore, the first natural main question is how much the performance can be increased with a certain amount of information, or, stated otherwise, which are the bounds of the performance increase. The second main question is the establishment of the thermodynamics of feedback controlled systems accounting for thermodynamic effects of the information transfer. This second question can be seen as included in the first one, as, for instance, the computation of the maximum efficiency involves a bound in the work that can be extracted from the feedback controlled system.

The case of a single operation of a feedback controller has been extensively studied in the context of Maxwell’s demon and in computation theory. In this case the main thermodynamical results are known; see Section 2.1 and Section 2.2.

On the other hand, the study of the repeated operations of a feedback controller presents new ingredients. We have to distinguish whether the operations of the feedback controller are correlated or uncorrelated. In the latter case, the system has lost memory of the previous actuation of the controller when its status is measured again to decide the next actuation by the controller. Therefore, the uncorrelated operations of the feedback can be decomposed as a succession of independent single operations, and thus the known previous results can be applied. However, the case of correlated repeated operation of the feedback controller has fundamental new ingredients that must be taken into account.

3.1.1. Correlated Repeated Actuation of the Feedback Controller

In [13] (see Section 2.4) the entropy reduction due to the repeated operation of the controller has been derived from the point of view of the feedback controlled system. This point of view is also used to derive the result of [9,10,11] (see Section 2.2). From this point of view, the controller is an external agent and we are not concerned about its internal state. We are only concerned about the effects on the system due to its interaction with the controller. One of the effects is the reduction of the system’s entropy due to the additional determination of the system’s macrostate through the information obtained from the system by the controller.

Therefore, there are still open questions to be answered. One particular open question involves the precise relations between these recent results and the algorithmic complexity approach adopted by Zurek [6,7] to study the correlated repeated actuation of a feedback controller from the perspective of the system plus controller. This study has allowed Zurek to establish the minimum energetic cost for erasing a memory that has stored the measurements the controller has processed (see Section 2.1). This minimum energetic cost also sets thermodynamic constraints that lead to a maximum entropy reduction for the system.

Another open question is the application or the extension of the results found in [13] to find bounds for other performance measures of feedback controlled systems, for example, the maximum attainable flux or power.

3.1.2. Continuous Actuation of the Feedback Controller

Continuous actuation of the feedback controller is effectively present when the controller repeatedly actuates on the system with a period shorter than any of the characteristic time scales of the system dynamics. It is therefore a limiting case of the correlated repeated actuation discussed above. However, the continuous time limit of the results found in [13] is not direct because of the nontrivial relationship between differential entropy and discrete entropy (see [12]). Therefore, even the extension of the results in [13] to the continuous actuation limit remains still an open problem.

Some techniques of filtering theory might be useful for the study of the information and entropy flows associated with continuous-time processes in non-equilibrium statistical mechanical systems [54]. In addition, methods of optimal control theory have been proven useful to compute limits on how much work can be extracted from linear systems [55].

3.2. Feedback Controlled Flashing Ratchets: Limitations to the Performance and Thermodynamics

Feedback flashing ratchets are relevant feedback controlled systems that would provide relevant applications of the general results obtained for feedback controlled systems. In addition, they can provide enlightening examples to guide the progress in the solution of the previous open problems.

The application of the results of [13] to feedback controlled flashing ratchets has not been done yet. It will allow to establish bounds on the entropy reduction attainable for feedback flashing ratchet and to establish the thermodynamics of these systems, which are theoretically and technologically relevant. Therefore, this is another relevant open problem.

Previously, in Section 2.3, we have summarized the bounds that have been found in [38,39] for the performance increase thanks to the information used by the feedback controller, when the performance is measured by the flux [38] or by the power output [39]. These relation were stated in terms of the information per actuation. However, how to restate these relations in terms of the total amount of information per unit time (i.e., only computing non-redundant information per unit time) is still an open question. This restatement will help to put these results [38,39] in relation with those found in [13] for the entropy reduction and the thermodynamics of a general feedback controlled system.

3.3. Limitations to the Operation of Nanotechnology Devices and Biological Systems

In many nanotechnology devices and biological systems, one is interested in increasing the performance of one part of the system through the bidirectional interaction of other part of the system [15,16,17,18,19,20,21,22,23,31,33,34,35]. Therefore, we can think of the subsystem whose performance we want to increase as a feedback controlled system, and consider the other interacting part as the controller. Thus, the bounds of the performance of the feedback controlled systems will imply limitations to the operation of nanotechnology devices and biological systems. Finding these implications is still a very relevant open question.

An illustrative example of this type of questions is the previously mentioned application of information theory to biochemical reaction networks to show the limits on the ability to control fluctuations in molecular abundances inside the cells as a function of the number of signaling events [14].

This shows that a deeper knowledge of the thermodynamics and the statistical physics of feedback controlled systems will bring a deeper understanding of nanotechnology devices and biological systems, and consequently it will increase our capacity of designing and controlling them.

4. Conclusions

Feedback or closed-loop control is present whenever there is a controller or external agent that gathers information on the state of the system and uses this information to decide the action it will perform on the system. Feedback control allows to increase the performance, and it is present in many systems with interest for physicists, engineers and biologists [1].

We have briefly reviewed the main ideas of the historical results by Landauer [4], Bennett [5], and Zurek [6,7], as well as the more recent results on the limits to the control of a general system [10,11], the limits to the performance of feedback controlled ratchets [38,39], the thermodynamics of feedback controlled systems [13], and the generalization of statistical physics identities and relations to feedback controlled systems [45,46,47,48,49,51,52,53]. This brief review has helped us to give the context for the open questions that we have presented in Section 3. These questions concern the quantification of the limits to the performance increase that can be obtained thanks to a feedback that uses a certain amount of information and the establishment of the thermodynamics of feedback controlled systems. Answering these questions is an important task for the theoretical understanding of feedback controlled systems, which has important technological applications. Feedback flashing ratchets [15,16,17,18,19,20,21,22,23,31] and Markovian particle pumps [13,41] are examples of paradigmatic motors whose deeper understanding will benefit the comprehension of the biological and technological devices they model, and boost new applications. Control engineers have also addressed similar topics from the point of view of dynamical systems [54,55] using models and techniques of optimal control theory [1,56] as the Linear-Quadratic-Gaussian control or the Kalman filter. The development of the links between physics, information theory and control theory will allow to quantify and to establish constraints to the increase of performance reachable with a given amount of information in general feedback controlled systems.