% Upper-case A B C D E F G H I J K L M N O P Q R S T U V W X Y Z % Lower-case a b c d e f g h i j k l m n o p q r s t u v w x y z % Digits 0 1 2 3 4 5 6 7 8 9 % Exclamation ! Double quote " Hash (number) # % Dollar $ Percent % Ampersand & % Acute accent ' Left paren ( Right paren ) % Asterisk * Plus + Comma , % Minus - Point . Solidus / % Colon : Semicolon ; Less than < % Equals = Greater than > Question mark ? % At @ Left bracket [ Backslash \ % Right bracket ] Circumflex ^ Underscore _ % Grave accent ` Left brace { Vertical bar | % Right brace } Tilde ~ % ---------------------------------------------------------------------| % --------------------------- 72 characters ---------------------------| % ---------------------------------------------------------------------| % % Optimal Foraging Theory Revisited: Chapter 3. Optimization Objectives % % (c) Copyright 2007 by Theodore P. Pavlic % \chapter{Statistical Optimization Objectives for Solitary Behavior} \label{ch:optimization_objectives} The efficacy of any particular behavior may be measured quantitatively in various ways. In this chapter, we approach the problem of combining appropriate statistics so that the utility of solitary behaviors can be measured for a given application. Choosing a static behavior to maximize some unit of expected value is analogous to choosing investments to maximize future returns. Reflecting this analogy, behavioral ecology has borrowed methods from investment theory and capital budgeting for behavioral analysis. We also use these methods, collectively known as \index{modern portfolio theory (MPT)|see{finance, modern portfolio theory}}\index{MPT|see{finance, modern portfolio theory}}\acro[\index{finance!modern portfolio theory (MPT)|indexdef}modern portfolio theory~(MPT)]{MPT}{\index{finance"!modern portfolio theory (MPT)"|indexglo}modern portfolio theory}, to analyze our model; however, we generalize the classical \ac{OFT} approach. This approach not only allows it to be applied to engineering problems, but it also provides answers to some of the criticisms of the theory. Additionally, we suggest new ways of describing optimal agent behavior and relationships among existing methods. The major purpose of this chapter is to introduce functions that combine statistics of the agent model to measure the utility of solitary behaviors. Behaviors that maximize these functions may be called \emph{optimal}. In \longref{sec:obj_fn_structure}, we define the structure of the optimization functions that are interesting to us. In \longref{sec:oft_approach_optimization}, we describe the optimization approach used frequently in classical \ac{OFT}. In \longref{sec:alternate_optimization_objectives}, we propose an alternate approach and give new or refined optimization objectives for analyzing agent behavior. Finally, in \longref{sec:future_econ_directions}, we briefly discuss how insights from \index{post-modern portfolio theory (PMPT)|see{finance, post-modern portfolio theory}}\index{PMPT|see{finance, post-modern portfolio theory}}\acro[\index{finance!post-modern portfolio theory (PMPT)|indexdef}post-modern portfolio theory~(PMPT)]{PMPT}{\index{finance"!post-modern portfolio theory (PMPT)"|indexglo}post-modern portfolio theory} may inspire new optimization approaches in both agent design in engineering and agent analysis in biology. All results discussed in this chapter will be qualitative and justified graphically. Specific analytical optimization results for some of the objectives discussed here are given in \longref{ch:optimization_results}. \section{Objective Function Structure} \label{sec:obj_fn_structure} Optimization functions usually combine multiple optimization \emph{objectives} in a way that captures the relative value of each of those objectives. In our case, each of our objectives is a statistic taken from the model in \longref{ch:model}. Therefore, in \longref{sec:stats_of_interest}, we present statistics that could serve as objectives for optimization and methods for combining them. In \longref{sec:optimization_constraints}, we discuss motivations for constraining the set of feasible behaviors and show how these constrained sets can be incorporated into optimization. Finally, in \longref{sec:optimization_choice_effects}, we discuss the importance of exploring a variety of optimization criteria. \subsection{Statistics of Interest} \label{sec:stats_of_interest} \longref{tab:common_stats} shows some obvious choices for statistics to be used as optimization objectives. % \begin{table}\centering \begin{tabular}{@{}r*{4}{c}@{}} \toprule & \multicolumn{2}{c}{Means} & \multicolumn{2}{c}{Variances}\\ \cmidrule(r){2-3}\cmidrule(l){4-5} {Net Gain Statistics:} & $\E(G_1)$ & $\E(\oft{G}_1)$ & $\var(G_1)$ & $\var(\oft{G}_1)$\\ {Cost Statistics:} & $\E(C_1)$ & $\E(\oft{C}_1)$ & $\var(C_1)$ & $\var(\oft{C}_1)$\\ {Time Statistics:} & $\E(T_1)$ & $\E(\oft{T}_1)$ & $\var(T_1)$ & $\var(\oft{T}_1)$\\ \bottomrule \end{tabular} \caption[Common Statistics for Solitary Optimization]{Common statistics used in optimization of solitary agent behavior.} \label{tab:common_stats} \end{table} % However, other statistics like $\E(G^N/T^N)$ (\ie, average gain per unit time) or $\E( (G^N + C^N)/C^N )$ (\ie, average efficiency) for all $N \in \N$ could also be relevant. \index{stochasticity!statistics!skewness|(}Economists \citep[\eg,][]{CDHP97,Kane82,KSY93,Lai91,Tsiang72} might argue that the skewness\footnote{\index{skewness|see{stochasticity, statistics, skewness}}\index{stochasticity!random variable|(}For a random variable $X$, its \emph{skewness} is a measure of the \index{symmetry|see{stochasticity, distributions, symmetry}}\index{stochasticity!distributions!symmetry}symmetry of its (\aimention{Henri L. Lebesgue}Lebesgue) probability density $f_X$. The standard definition of skewness is $\E((X - \E(X))^3)/\std(X)^3$. Note that this is a scaled version of the third \index{stochasticity!statistics!central moments}central moment.\index{stochasticity!random variable|)}} of each of these random variables would be a reasonable statistic to study because it may be desirable to have random variables that are distributed asymmetrically (\eg, net gains that are more often high than low)\footnote{This might be called \index{skewness preference}skewness preference. It is also desirable to optimize skewness simply to prevent deleterious asymmetry.}.\index{stochasticity!statistics!skewness|)} Of course, any one of these statistics may not capture all relevant objectives of a problem. For example, it may be desirable to maximize both $\E(G_1)$ and $-\E(T_1)$ (\ie, minimize $\E(T_1)$); however, it may not be possible to accomplish both of these simultaneously. Therefore, here we discuss the construction of compound objectives that allow for optimization with respect to multiple criteria. \index{multiobjective optimization|see{optimality, multiobjective optimization}}% \index{compound objectives|see{optimality, multiobjective optimization}}% \index{optimality!compound objectives|see{optimality, multiobjective optimization}}% \index{optimality!multiobjective optimization|(}Take a problem with $m \in \N$ relevant optimization objectives. For all objective functions to be minimized, replace the function with its additive or multiplicative inverse (\ie, replace a function $f$ with the function $-f$ or, for functions with strictly positive or strictly negative ranges, $1/f$); therefore, the ideal objective is to maximize all $m$ functions. Collect these $m$ objective functions into $m$-vector $\v{x}$ where $\v{x} = \{x_1,x_2,\dots,x_m\}$. Use the weighting vector $\v{w} \in \R_{\geq0}^m$ with $\v{w} = \{w_1,w_2,\dots,w_m\}$ to represent the relative value of each of these objectives. Therefore, the compound objective functions % \begin{equation} w_1 x_1 + w_2 x_2 + \cdots + w_m x_m \quad \text{ or } \quad \min\{ w_1 x_1, w_2 x_2, \dots, w_m x_m \} \label{eq:compound_optimization_objectives} \end{equation} % represent different ways to combine all $m$ objectives. \index{optimality!multiobjective optimization!linear combination}The former of these two compound objectives is a linear combination of statistics (\ie, $\v{w}^\T \v{x}$), and an optimal behavior for this function will be \index{Pareto optimal|see{optimality, Pareto efficient}}\index{Pareto efficient|see{optimality, Pareto efficient}}\index{optimality!Pareto optimal|see{optimality, Pareto efficient}}\index{optimality!Pareto efficient}\aimention{Vilfredo Pareto}Pareto efficient\footnote{\index{optimality!Pareto efficient|(indexdef}To be \emph{\aimention{Vilfredo Pareto}Pareto efficient} or \emph{\aimention{Vilfredo Pareto}Pareto optimal} means that any deviation that yields an increase in one objective function will also result in a decrease in another objective function. \aimention{Vilfredo Pareto}Pareto optimal solutions characterize tradeoffs in optimization objectives. If deviation from some behavior will increase all objective functions, then that behavior cannot be \aimention{Vilfredo Pareto}Pareto efficient. The set of all \aimention{Vilfredo Pareto}Pareto efficient solutions is called the \index{frontier|see{optimality, Pareto efficient}}\index{Pareto frontier|see{optimality, Pareto efficient}}\index{optimality!Pareto frontier|see{optimality, Pareto efficient}}\emph{\aimention{Vilfredo Pareto}Pareto frontier}.\index{optimality!Pareto efficient|)indexdef}} with respect to the $m$ objective functions. Maximization of the latter of these two compound objectives represents a \index{optimality!multiobjective optimization!maximin}\emph{maximin} optimization problem. \index{Lagrange multiplier method|see{optimality, Lagrange multiplier method}}\index{optimality!Lagrange multiplier method}\aimention{Joseph-Louis Lagrange}Lagrange multiplier methods (\ie, \index{Kuhn-Tucker conditions|see{optimality, KKT conditions}}\index{KKT conditions|see{optimality, KKT conditions}}\index{Karush-Khun-Tucker (KKT) conditions|see{optimality, KKT conditions}}\aimention{William Karush and Harold W. Kuhn and Albert W. Tucker}\acro[\index{optimality!KKT conditions}Karush-Khun-Tucker~(KKT)]{KKT}{\index{optimality"!KKT conditions"|indexglo}Karush-Kuhn-Tucker} conditions) \citep{Bertsekas95} can be used to study the optimal solutions to both forms in \longref{eq:compound_optimization_objectives}.% \index{optimality!multiobjective optimization|)} \subsection{Optimization Constraints}% \label{sec:optimization_constraints}% \index{optimization constraints|(} In a given foraging problem, it is not necessarily the case that all modeled behaviors are applicable or even possible. That is, optimization analysis must be considered with respect to a set of feasible behaviors. The following are some examples of constraints that have been found in the literature; suggestions for how those constraints could be implemented in this model are also given. % \begin{description} \item\emph{Time Constraints:} \index{optimization constraints!time}The economics-inspired graphical foraging model of \citet{Rapport71} considers level \emph{indifference curves} of an energy function. Each of these curves represents a set of combinations of prey where each combination returns the same energetic gain to the forager. \Citeauthor{Rapport71} then assumes that the forager has a finite lifetime and surrounds all prey combinations that can be completed in this time with a boundary called the \emph{consumption frontier}\footnote{The consumption frontier is a Pareto frontier. Diets on this frontier return the greatest gain for their foraging time.}. The optimal diet combination is the point of tangency between the consumption frontier and some indifference curve. In other words, this is the combination of prey items that returns the highest energetic gain for the given finite lifetime. We can quantify this idea by maximizing $\E(G(t))$ subject to the constraint $t \leq \overline{T}$ where $\overline{T} \in \R_{>0}$. Because \citeauthor{Rapport71} gives a qualitative explanation for the observations in \citet{Murdoch69}, the analytical application of our model with this time constraint could give a quantitative explanation. \item\emph{Nutrient Constraints:} \index{optimization constraints!nutrients}\citet{Pulliam75} optimizes a point gain per unit time function similar in form to $\E(\oft{G}_1)/\E(\oft{T}_1)$, but the notion of nutrient constraints is added. That is, there are $m \in \N$ nutrients and all tasks of type $i \in \{1,2,\dots,n\}$ return quantity $\rho_{ij}$ of nutrient $j \in \{1,2,\dots,m\}$. \Citeauthor{Pulliam75} then calls $M_j \in \R_{\geq0}$ a minimum amount of nutrient $j$ that must be returned from processing. The goal is to maximize the rate of point gain while maintaining this minimum nutrient level. These nutrient constraints could be added to our model as well. As \citeauthor{Pulliam75} notes, under these constraints, optimal behaviors often include partial preferences. In the unconstrained classical \ac{OFT} problem, it is sufficient for optimality to either process all or none of tasks of a particular type; however, with nutrient constraints it may be necessary for optimality that only a fraction of the encountered tasks of a certain type be processed\footnote{In \longref{ch:optimization_results}, we generalize the classical \ac{OFT} result to show that over a closed interval of preference probabilities, sufficiency is associated with the endpoints. The results of \citet{Pulliam75} effectively make that interval a function of nutrition requirements; under these constraints, partial preferences may be necessary for optimality.}. \item\emph{Encounter-Rate Constraints:} \index{optimization constraints!encounter-rate}\citet{GS83} and \citet{PP06} explore the optimization of a point gain per unit time function as well; however, the impact of speed choice on imperfect detection\index{speed-accuracy tradeoff|(} is also introduced. That is, with perfect detection, an increase in speed will most likely come with an increase in encounter rate with tasks of every type. However, when detection errors can occur, the relationship between encounter rate and speed may be arbitrarily nonlinear. If this exact relationship is not known, it may be sufficient to restrict search speed to a range where detection is reliable. If the impact of search speed were added to our model (\eg, if encounter-rate was parameterized by speed), this restriction could be modeled as constraints on search speed. The resulting optimal behavior would include a search speed that provides the optimal encounter rates subject to imperfect detection.\index{speed-accuracy tradeoff|)} \end{description} % Any optimization function of a form in \longref{eq:compound_optimization_objectives} subject to a finite number of equality or non-strict inequality constraints\footnote{A \emph{strict} inequality constraint uses $<$ or $>$; therefore, a \emph{non-strict} or \emph{weak} inequality constraint uses $\leq$ and $\geq$.} may be analyzed with \index{optimality!Lagrange multiplier method}\aimention{Joseph-Louis Lagrange}Lagrange multiplier methods. Therefore, in principle, a wide range of constrained optimization problems can be studied. \index{optimization constraints|)} \subsection{Impact of Function Choice on Optimal Behaviors} \label{sec:optimization_choice_effects} As discussed in \longref{sec:oft_long_term_rate}, classical \ac{OFT} results come from maximizing the long-term rate of gain (\eg, $\E(\oft{G}_1)/\E(\oft{T}_1)$). This choice follows from the argument of \citet{PPC77} that optimizing this long-term rate synthesizes the two extremes, energetic maximization and time minimization, of a general model of foraging given by \citet{Schoener71}. This rate approach is taken by \citet{Pulliam75} whose quantitative results show that the optimal diet predicted by a rate maximizer depend only on the encounter rates with prey types in the diet. However, \citet{Rapport71} focusses only on gain maximization (in finite time) and shows that the optimal diet depends on encounter rates with all prey types. These two results are very different, and the only justification for using the first result follows from a purely intuitive argument from \citet{PPC77}. However, the result from \citeauthor{Rapport71} is entirely valid from a perspective of the foundational work of \citeauthor{Schoener71}. Therefore, it is clear that one optimization criterion will not fit all problems. Clearly, is important to investigate other functions that may be more appropriate for specific problems. \section{Classical OFT Approach to Optimization}% \label{sec:oft_approach_optimization}% \index{solitary agent model!classical analysis!optimization|(} As discussed by \citet{SC82}, classical \ac{OFT} approaches optimization from two perspectives which are both based on evolutionary arguments. The first analyzes behaviors that optimize of the asymptotic limit of rate of net gain. The second assumes the agent must meet some energetic requirement and maximizes its probability of success. The former, which we describe in \longref{sec:oft_long_term_rate}, is called \emph{rate maximization}, and the latter, which we describe in \longref{sec:oft_risk_sensitivity}, is described as being \emph{risk sensitive}. Both approaches develop optimal static behaviors for the solitary agent. \subsection{Maximization of Long-Term Rate of Net Gain}% \label{sec:oft_long_term_rate}% \index{long-term average rate of net gain|(}% \index{rate maximization|(} \index{long-term average rate of net gain!justification|(}% In biological contexts, it is expected that natural selection will favor foraging behaviors that provide greater future reproductive success, a common surrogate for \aimention{Charles R. Darwin}Darwinian fitness. So, functions mapping specific behaviors to quantitative measures of reproductive success can be optimized to predict behaviors that should be maintained by natural selection. \Citet{Schoener71} defines such a model, and while quantities in the model are too difficult to define for most cases, behaviors predicted by the model fall on a continuum from foraging time minimizers (when energy is held constant) to energy maximizers (when foraging time is held constant). In other words, behaviors should be excluded if there exists another behavior that has both a higher energy return and a lower time. \Citet{PPC77} argue that the \emph{rate} of net energy intake is the most general function to be maximized as it captures both extremes on the \citeauthor{Schoener71} continuum by asserting an upward pressure on energy intake and a downward pressure on foraging time. This will allow a forager to achieve its energy consumption needs while also leaving it enough time for other activities such as reproduction and predator avoidance. This interpretation is only valid over the space of behaviors with positive net energetic intake. For example, rate maximization puts an upward pressure on foraging time for behaviors that return negative net energetic intake. This is not recognized by \citeauthor{PPC77}, and the continuum of behaviors described by \citeauthor{Schoener71} explicitly exclude these time maximizers. However, from a survival viewpoint, it makes sense that foragers facing a negative energy budget should maximize time foraging. Therefore, rate maximization encapsulates two conditional optimization problems; it trades off net gain and total time in a way that is dependent upon energy reserves.% \index{long-term average rate of net gain!justification|)}% \index{long-term average rate of net gain!limit|(}% The rate of net energy intake can be defined in different ways. Using the terms from \longref{ch:model}, it could be defined as $\oft{G}(t)/t$ or $\E( \oft{G}(t) )/t$ for any $t \in \R_{\geq0}$ or $\oft{G}^N/\oft{T}^N$ or $\E( \oft{G}^N/\oft{T}^N )$ for any $N \in \N$. However, \citeauthor{PPC77} also argue that rates should be calculated over the entire lifetime of the forager. Thus, rather than taking a particular $t \in \R_{\geq0}$ or $N \in \N$, the asymptotic limits of these ratios should be taken. Conveniently, \longref{eq:oft_payoff_long_rate} shows that all of these limits are equivalent. By \longref{eq:oft_payoff_long_rate_equiv}, % \begin{equation} \begin{split} \frac{\E\left(\oft{G}_1\right)}{\E\left(\oft{T}_1\right)} &= \frac {\E\left(\oft{G}^{N_*}\right)}{\E\left(\oft{T}^{N_*}\right)} = \aslim\limits_{N \to \infty} \frac{\oft{G}^N}{\oft{T}^N} = \lim\limits_{N \to \infty} \E\left(\frac{\oft{G}^N}{\oft{T}^N}\right)\\ &= \frac{\E\left(\oft{G}(t_*)\right)}{\E\left(\oft{T}(t_*)\right)} = \aslim\limits_{t \to \infty} \frac{\oft{G}(t)}{t} = \lim\limits_{t \to \infty} \frac{\E\left( \oft{G}(t) \right)}{t}\\ \end{split} \label{eq:oft_payoff_long_rate_summary} \end{equation} % for any $t_* \in \R_{>0}$ and $N_* \in \N$. For this reason, the ratio of expectations $\E(\oft{G}_1)/\E(\oft{T}_1)$ has received significant interest in classical \ac{OFT} \citep[\eg,][]{HouMc99,SC82,SK86}. We call this ratio the \emph{long-term (average) rate of net gain}. Note that by \longref{eq:RoE_equivalence} this ratio plays an identical role in our analysis approach when we consider the asymptotic case.% \index{long-term average rate of net gain!limit|)}% \subsubsection{Opportunity Cost and Pareto Optimality}\aimention{Vilfredo Pareto} %\citet[ch.~4]{HouMc99} \index{opportunity cost|see{long-term average rate of net gain, opportunity cost}}% \index{long-term average rate of net gain!opportunity cost|(}% \Citet{HouMc99} provide an interesting interpretation of $\E(\oft{G}_1)/\E(\oft{T}_1)$. They define constant $\oft{\gamma}^* \in \R$ to be the maximum value of $\E(\oft{G}_1)/\E(\oft{T}_1)$ (\ie, the long-term rate of net gain) over the set of feasible agent behaviors. They then treat rate $\oft{\gamma}^*$ as a factor converting time spent between encounters to maximum points possible from that time. Therefore, $\oft{\gamma}^*$ converts time into its equivalent \emph{opportunity cost} (\ie, gain paid per unit time). They show that the behavior that maximizes % \begin{equation} \E\left(\oft{G}_1 - \oft{\gamma}^* \oft{T}_1\right) \label{eq:HouMc99_approach} \end{equation} % will also be the behavior that achieves the maximum long-term rate of gain $\oft{\gamma}^*$. So, maximizing the long-term rate of gain is equivalent to maximizing the per-cycle \emph{gain} after being discounted by the opportunity cost of the cycle time\footnote{There is a related result by \citet{ES84} that predicts the optimal behavior on simultaneous encounters. This is described by both \citet{HouMc99} and \citet{SK86}, and \citeauthor{HouMc99} show this simultaneous encounter result to follow from the opportunity cost result.}. Solving for this behavior can be done analytically only if $\oft{\gamma}^*$ is known, and so the method of \citeauthor{HouMc99} numerically solves for the optimal behavior using iteration, which could be a weakness of this approach. However, it demonstrates an important interpretation of $\E(\oft{G}_1)/\E(\oft{T}_1)$ as the opportunity cost of time. Not surprisingly, this also shows that the behavior that maximizes the long-term rate of gain is \aimention{Vilfredo Pareto}Pareto optimal with respect to maximization of $\E(\oft{G}_1)$ and (maximization) minimization of $\E(\oft{T}_1)$ when $\oft{\gamma}^* > 0$ ($\oft{\gamma}^* < 0$); that is, this optimal behavior represents a particular tradeoff between net gain and total time. This \aimention{Vilfredo Pareto}Pareto interpretation casts $\oft{\gamma}^*$ as the relative importance of minimizing time, which is consistent with notion of opportunity cost\footnote{When $\oft{\gamma}^* < 0$, the relative importance of minimizing time is \emph{negative}, which indicates that $|\oft{\gamma}^*|$ is the relative importance of \emph{maximizing} time (\ie, an opportunity \emph{gain}).}. The numerical approach to finding $\oft{\gamma}^*$ and the corresponding optimal behavior is equivalent to sliding along a continuum of \aimention{Vilfredo Pareto}Pareto efficient solutions (\ie, tradeoffs of net gain and total time).% \index{long-term average rate of net gain!opportunity cost|)}% \subsubsection{Equilibrium Renewal Process as an Attractive Alternative} \index{equilibrium process rate of net gain|see{equilibrium renewal process rate of net gain}}% \index{equilibrium renewal process rate of net gain|(}% \Citet{Cha73} note that it is desirable to derive the \emph{equilibrium renewal process} rate of net gain. That is, introduce a $T_1 \in \R_{>0}$ and redefine the process to start after $T_1$ foraging time has past. Hence, runtime $t$ represents the length of the interval immediately after time $T_1$, and so quantity of interest to \citeauthor{Cha73} is $\E(G(t))/t$, which represents the average rate of net gain returned to an agent when the agent is in equilibrium with its environment (\ie, after the decay of any initial transients). However, they point out that this rate is only known for such a process if it is additionally assumed that the net gain on each \ac{OFT} cycle is independent of the total time of each \ac{OFT} cycle (in particular, the processing time of each cycle). In that case, $\E(G(t))/t$ can also be expressed as the ratio $\E(\oft{G}_1)/\E(\oft{T}_1)$. Unfortunately, it is rare that net gain and processing time will be independent in a practical system. Analytical results are not available otherwise. For this reason, when $\E(\oft{G}_1)/\E(\oft{T}_1)$ is used it is usually assumed to be a limiting case (\ie, a rate over a long time rather than a short-term rate after a long time).% \index{equilibrium renewal process rate of net gain|)}% \subsubsection{Graphical Interpretation of Rate Maximization}% \index{long-term average rate of net gain!graphical optimization|(}% When an agent is only free to choose its (average) processing times, the tasks are said to occur in \emph{patches} or to be \emph{patchily distributed} \citep{SK86}. Take such a case with a single task type and no search or processing costs (\ie, $n=1$, $c^s=c_1=0$, $p_1=1$, and $\tau_1 \in \R_{\geq0}$). \Citet{SK86} show that this problem has an insightful graphical solution. Consider \longref{fig:oft_rate_maximization}. % %\begin{figure}[!ht]\centering \begin{figure}\centering \begin{picture}(140,109)(0,0) \put(70, 54.5){% \makebox(0,0){\scalebox{0.9}{\framebox(87.5,95){ \shortstack[c]% {$\E(\oft{G}_1) = g_1(\tau_1)$\\% $\E(\oft{T}_1) = \tau_1 + \frac{1}{\lambda}$\\% $\oft{\gamma} \triangleq \frac{g_1(\tau_1)}% {\tau_1 + \frac{1}{\lambda}}$\\% $\lambda = \lambda_1$\\% $\lozenge^* \triangleq \max\left\{\lozenge\right\}$}}}}} \end{picture} \begin{picture}(160,109)(-33,-21) % Horizontal Axis \thicklines \put(-30, 0){\vector(1, 0){144}} \put(115, 0){\makebox(0, 0)[l]{$\tau_1$}} % Vertical Axis \thicklines \put(0, -16){\vector(0, 1){90}} \put(0, 75){\makebox(0, 0)[b]{$g_1(\tau_1)$}} % Rate Line \thinlines \put(-24, 0){\line(2,1){138}} \put(20, 22){\circle*{5}} % Maximal Rate (Slope) \thinlines \put(74,67){\line(1,0){36}} \put(74,49){\line(0,1){18}} \put(76,65.5){\makebox(0, 0)[lt]{$\oft{\gamma}^*$}} % Curve, starting origin and moving CW \thinlines \qbezier(0,0)(0,12)(20,22) \qbezier(20,22)(56,40)(78,40) \qbezier(78,40)(97,40)(114,30) %\cbezier(20,22)(56,40)(90,40)(114,30) % Max and mins \thicklines \put(-4,40){\line(1,0){8}} \put(20,-4){\line(0,1){8}} \thinlines \put(-5,40){\makebox(0,0)[r]{$g_1^*$}} \put(0,40){\dashbox{3}(78,0){}} \put(20,-5){\makebox(0,0)[t]{$t^*$}} %\put(20,0){\dashbox{3}(0,20){}} % Origin strategy \thicklines \put(-24,-4){\line(0,1){8}} \thinlines \put(-24,-5){\makebox(0, 0)[t]{$\frac{-1}{\lambda}$}} \end{picture} \caption[Visualization of Classical OFT Rate Maximization]{Rate maximization in classical \ac{OFT}. It is assumed that $n=1$, $c^s = 0$, and $c_1 = 0$. The constraint that $p_1 = 1$ is also applied. The optimal processing time is denoted $t^*$, and the corresponding maximal rate is denoted $\oft{\gamma}^*$ and shown as a slope of a tangent line.} \label{fig:oft_rate_maximization} \end{figure} % The $g_1(\tau_1)$ function is plotted with respect to feasible choices of $\tau_1$ and a mark is made at the point $(-1/\lambda,0)$. For any $\tau_1$, the corresponding long-term rate of gain is the slope of a line that connects points $(-1/\lambda,0)$ and $(\tau_1,g(\tau_1))$. Therefore, the optimal $\tau_1$ (shown as $t^*$) is the one that corresponds with the line with the maximal slope, and that slope will be the maximal long-term rate of gain (shown as $\oft{\gamma}^*$). In \longref{sec:alternate_rate_tradeoffs}, we show how this graphical interpretation can be extended to the general case\footnote{We show this interpretation using our approach to defining the relevant statistics of the model; however, our method can also be applied to the classical \ac{OFT} statistics in an obvious way (\ie, with little more than a change of notation).} (\ie, with multiple types, costly searching and processing, and tasks that may or may not be patchily distributed). \index{MVT|see{marginal value theorem}}% Several conclusions can be drawn from \longref{fig:oft_rate_maximization}. For differentiable functions with $g_1(0)=0$ and $g_1'(0)>0$, the optimal processing time $t^*$ must be such that $g_1'(t^*)$ is equal to the long-term rate of gain. In particular, if $g_1$ is a concave function, then this line will be the \emph{unique} tangent line that crosses $(0,1/\lambda)$. Rate-maximization for the classical \ac{OFT} model is said to follow the \emph{\acro[\index{marginal value theorem (MVT)|indexdef}marginal value theorem~(MVT)]{MVT}{\index{marginal value theorem (MVT)"|indexglo}marginal value theorem}} \citep{Cha76,Cha73}. This means that the average time an agent processes patchily distributed tasks of a certain type is the time when the average rate of point gain for the task type drops to the average rate of point gain for the environment. That is, processing should continue until the marginal return from the next instant of processing is less than the environmental average rate of gain\footnote{This interpretation is really only accurate for a deterministic agent model. In the general stochastic agent model, the \ac{MVT} need only be observed in the first-order statistics of the gains and processing times.}.% \index{long-term average rate of net gain!graphical optimization|)}% \index{long-term average rate of net gain|)} \index{rate maximization|)} \subsection{Minimization of Net Gain Shortfall}% \label{sec:oft_risk_sensitivity}% \index{risk minimization|see{risk sensitivity}}% \index{risk sensitivity|(}\index{mean-variance analysis (MVA)|(} Because rate maximization depends only on first-order statistics, it disregards the \index{stochasticity!statistics!standard deviation}standard deviation\footnote{\index{stochasticity!statistics!standard deviation|(indexdef}For random variable $X$, the standard deviation $\std(X)$ is $\sqrt{\var(X)}$ (\ie, the square root of the variance).\index{stochasticity!statistics!standard deviation|)indexdef}} of random variables in the model. For example, an agent with a behavior that maximizes its long-term rate of net gain may bypass frequently encountered tasks with small gains regardless of any survival needs. \index{gain threshold|see{gain success threshold}}\index{survival threshold|see{gain success threshold}}\index{threshold|see{gain success threshold}}\index{success threshold|see{gain success threshold}}\index{gain success threshold}However, if the agent must meet a net gain requirement in finite time, it may be beneficial to decrease mean net gain if that decrease also comes with an decrease in the uncertainty of returns. \subsubsection{Maximization of Reward-to-Variability Ratio} \Citet{SC82} introduce a risk-sensitive agent model and an optimization approach that maximizes the \index{probability of success}probability of success. Consider a solitary agent that must acquire some minimal net gain \index{gain success threshold}$\oft{G}^T$ by a time $\oft{T} \in \R_{\geq0}$. Call $\oft{\mu}$ the expectation and $\oft{\sigma}$ the standard deviation of net gain acquired by $\oft{T}$ for some given behavior. The method states that the desired risk-sensitive behavior should maximize the objective % \begin{equation} \frac{\oft{\mu} - \oft{G}^T}{\oft{\sigma}} \label{eq:oft_zscore} \end{equation} % \index{location|see{stochasticity, distributions, location-scale family}}\index{scale|see{stochasticity, distributions, location-scale family}}\index{stochasticity!distributions!location-scale family|(} If the net gain random variable is \index{location-scale|see{stochasticity, distributions, location-scale family}} location-scale\footnote{\index{stochasticity!distributions!location-scale family|indexdef}A family of distribution functions $\Omega$ is called \emph{location-scale} if there exists some $F \in \Omega$ such that for all $F_1 \in \Omega$, there exists a \index{stochasticity!distributions!location|see{stochasticity, distributions, location-scale}}\emph{location} $m \in \R$ and \index{stochasticity!distributions!scale|see{stochasticity, distributions, location-scale}}\emph{scale} $s \in \R_{>0}$ with $F_1(x) = F( (x-m)/s )$. A random variable is location-scale if its distribution comes from such a family. This idea of a two parameter family of distribution functions comes from \citet{RS70}, and this definition of such a class of functions is due to \citet{Bawa75}; however, \citet{Meyer87} gives an equivalent definition. Examples of location-scale distributions are the \index{stochasticity!distributions!normal}normal, \index{stochasticity!distributions!exponential}exponential, \index{stochasticity!distributions!uniform}, and \index{stochasticity!distributions!double exponential}double exponential distributions.} with identical \index{stochasticity!statistics!skewness}skewness for all choices of location and scale\footnote{Location-scale distributions with mean locations and standard deviation scales will naturally have this property.}, the behavior that maximizes \longref{eq:oft_zscore} will also minimize the probability that the net gain is less than the $\oft{G}^T$ threshold\footnote{This is a sufficient condition; however, it is not necessary. Investment theoretic consequences of location-scale distributions are given by \citet{Bawa75} and \citet{Meyer87}. The multivariate case is handled by \citet{Chamberlain83} and \citet{OR83}.}. \index{gain success threshold|(}In other words, if the agent is said to be \emph{successful} when its net gain meets or exceeds $\oft{G}^T$, then the optimal behavior will maximize the probability of success\footnote{This result can be generalized slightly by considering the class of distributions where a monotonic transformation (\ie, continuously differentiable with non-negative derivative everywhere) of random variables is location-scale. The \index{stochasticity!distributions!log-normal}log-normal distribution belongs to this more general class \citep{Bawa75}.}.\index{gain success threshold|)} \paragraph{Location-Scale Justification:} By the \index{CLT|see{stochasticity, central limit theorem}}\index{central limit theorem (CLT)|see{stochasticity, central limit theorem}}\emph{\acro[\index{stochasticity!central limit theorem (CLT)}central limit theorem~(CLT)]{CLT}{\index{stochasticity"!central limit theorem (CLT)"|indexglo}central limit theorem}}, if the net gain is a sum of \iid{}\ random variables (\eg, individual cycle gains), the probability distribution of the net gain will approach a \index{stochasticity!distributions!normal}normal distribution\footnote{\index{stochasticity!distributions!normal|(indexdef}A \emph{normal} or \index{normal distribution|see{stochasticity, distributions, normal}}\index{Gaussian distribution|see{stochasticity, distributions, normal}}\emph{\aimention{Carl Friedrich Gau\ss}Gaussian} random variable $X$ with mean $\mu$ and standard deviation $\sigma$ has (\aimention{Henri L. Lebesgue}Lebesgue) probability density $f_X(x) = 1/( \sigma \sqrt{2 \pi}) \exp( -(x-\mu)^2/(2 \sigma^2) )$. Normal random variables are location-scale with location $\mu$ and scale $\sigma$ and are \index{stochasticity!distributions!symmetry}symmetric about their mean (\ie, they have zero skewness).\index{stochasticity!distributions!normal|)indexdef}} as the number of elements in the sum increases. Therefore, it may be reasonable (\eg, consider $\oft{G}^N$ as $N \to \infty$) to assume that net gains are \index{stochasticity!distributions!normal}normally distributed or at least location-scale with location-scale invariant skewness. In this case, the behavior that maximizes \longref{eq:oft_zscore} will certainly maximize the probability of success.\index{stochasticity!distributions!location-scale family|)} \paragraph{Analogous Results from Economics:} \citet{SK86} call this the \index{z-score@$z$-score|see{reward-to-variability ratio}}\emph{$z$-score model}; however, it is well-known in economics that this method was initially developed by \citet{Sharpe66} for application to optimal portfolio selection. \Citeauthor{Sharpe66} calls \longref{eq:oft_zscore} a \index{reward-to-variability ratio|indexdef}\emph{reward-to-variability ratio}\footnote{This is also known as the \index{Sharpe ratio|see{reward-to-variability ratio}}\emph{\citeauthor{Sharpe66} ratio}, which is named after the \aimention{Alfred B. Nobel}Nobel laureate who developed it.}. While economists realize that return distributions need not be \index{stochasticity!statistics!normal}normally distributed (\eg, symmetric about the mean) for the reward-to-variability ratio to minimize risk, \citet{SK86} depend on \index{stochasticity!distributions!normal}normality to justify their claims \citep[p.~134]{SK86}. Assuming \index{stochasticity!distributions!normal}normality of returns may be far too restrictive. In fact, it is desirable that returns are skewed so that the mass is concentrated on higher gains (\ie, not symmetric and therefore not \index{stochasticity!distributions!normal}normal). Therefore, by depending on consistent skewness rather than symmetry, the economic argument of reward-to-variability maximization is not only more general but also more convincing than the argument of \citeauthor{SK86}. \paragraph{Links to Risk-Sensitive Dynamic Optimization:} \index{dynamic optimization}An \index{ex post performance measure@\expost{} performance measure}\expost{} version of the \index{reward-to-variability ratio}reward-to-variability ratio is described by \citet{Sharpe94}\footnote{\Citet{Sharpe94} also discusses other related ratios which may have applicability to stochastic optimization of static behavior.}, which is typically used for measuring past performance. However, there may some opportunity to use this \expost{} ratio for dynamic optimization to derive an optimal control similar to the bang-bang\footnote{\index{bang-bang control|(indexdef}A \emph{bang-bang} control switches an actuator from one extreme to the other. In this context, the variance at each small time period could be chosen by the agent to be either its lowest value or its highest value.\index{bang-bang control|)indexdef}} control described by \citet{SK86} and \citet{McNamara83} and based on the $z$-score model. \subsubsection{Graphical Interpretation of Risk Minimization}% \index{reward-to-variability!graphical optimization|(} Again, consider an agent described by the model in \longref{ch:model} with tasks that are patchily distributed. Assume that there is a single task type and no search or processing costs (\ie, $n=1$, $c^s=c_1=0$, $p_1=1$, and $\tau_1 \in \R_{\geq0}$), but assume that the agent must acquire a net gain of $\oft{G}^T$ in $\oft{T}$ time. \Citet{SC82} use this scenario to illustrate reward-to-variability maximization, as shown in \longref{fig:oft_risk_sensitivity}. % %\begin{figure}[!ht]\centering \begin{figure}\centering \begin{picture}(150,181)(0,0) \put(75, 90.5){\makebox(0,0){ \framebox(110,55){ \shortstack[c]% {$t_a < t_b < t^* < \oft{T}$\\ $\oft{\rho}% \triangleq\frac{\oft{\mu}-\oft{G}^T}{\oft{\sigma}}$\\% $\lozenge^* \triangleq \max\left\{\lozenge\right\}$}}}} \end{picture} \begin{picture}(117,181)(2,-17) \thicklines % Horizontal Axis \put(5, 0){\vector(1, 0){105}} \put(111, 0){\makebox(0, 0)[l]{$\oft{\sigma}$}} % Vertical Axis \put(24, -15){\vector(0, 1){165}} \put(24, 151){\makebox(0, 0)[b]{$\oft{\mu}$}} % Z-score line \thinlines \put(24, 136.5){\line(2,-1){86}} \put(80, 108.5){\circle*{5}} % Maximal Z-score (slope) \thinlines \put(29,134){\line(1,0){39}} \put(68,134){\line(0,-1){19.5}} \put(67.5,132.5){\makebox(0, 0)[rt]{$\oft{\rho}^*$}} % Threshold Value \thicklines \put(20,136.5){\line(1,0){8}} \thinlines \put(19,136.5){\makebox(0,0)[r]{$\oft{G}^T$}} % Curve, starting from origin and moving CCW \thinlines \qbezier(40,121)(30,121)(30,100) \qbezier(80,108.5)(55,121)(40,121) \qbezier(92,80)(92,102.5)(80,108.5) \qbezier(85,38)(92,62)(92,80) \qbezier(54,7)(78,14)(85,38) \qbezier(24,0)(30,0)(54,7) % Example times \put(53.16,9.88){\line(7,-24){1.68}} \put(53.16,9.88){\rotatebox{286.26}% {\makebox(0,0)[r]{$t_a$}}} \put(82.12,38.84){\line(24,-7){5.76}} \put(82.12,38.84){\rotatebox{343.74}% {\makebox(0,0)[r]{$t_b$}}} \put(80,108.5){\rotatebox{63.4349}% {\makebox(0,0)[r]{$t^*\,$}}} %\put(27,100){\line(1,0){6}} %\put(34,100){\makebox(0,0)[l]{$\oft{T}$}} % Max and mins \thicklines \put(92,-4){\line(0,1){8}} \put(20,121){\line(1,0){8}} \thinlines \put(92,-5.5){\makebox(0, 0)[t]{$\oft{\sigma}^*$}} \put(92,0){\dashbox{3}(0,80){}} \put(19,121){\makebox(0, 0)[r]{$\oft{\mu}^*$}} \put(24,121){\dashbox{3}(16,0){}} \end{picture} \caption[Visualization of Classical OFT Risk-Sensitive Solutions]{Risk sensitivity in classical \ac{OFT}. Over a time period of length $\oft{T}$, $\oft{\mu}$ is the expected net gain, $\oft{\sigma}$ is the standard deviation of the net gain, and $\oft{G}^T$ is an average net gain success threshold. The curve shows $(\oft{\mu},\oft{\sigma})$ combinations for each average processing time, and $t_a$ and $t_b$ are examples of average processing times. The optimal average processing time is denoted $t^*$ and the corresponding maximal reward-to-variability ratio is denoted by $\oft{\rho}^*$ and shown as the slope of a tangent line.} \label{fig:oft_risk_sensitivity} \end{figure} % The curve shows the $(\oft{\mu},\oft{\sigma})$ combinations that result from each choice of processing time $\tau_1$, where processing times increase in a counter-clockwise direction. Because the gain threshold is so high, the optimal processing time, denoted $t^*$, is biased toward a higher-variance distribution of outcomes. In other words, the negative energy budget forces the agent to take more risks to maximize the probability of success. In \longref{sec:alternate_rate_tradeoffs}, we demonstrate how this graphical solution can be used to gain insight into the most general case of the model from \longref{ch:model}\footnote{Again, while our solution will not be in terms of the statistics used in classical \ac{OFT}, it can easily be extended with little more than a change of notation. Also, to maintain tractability, we assume \longrefs{eq:cycle_proc_gross_gain_2moment}--% \shortref{eq:cycle_net_gain_2moment}.}.% \index{reward-to-variability!graphical optimization|)}% \index{mean-variance analysis (MVA)|)}\index{risk sensitivity|)} \subsection{Criticisms of the OFT Approach} \label{sec:oft_criticisms} Despite its successes in making qualitative predictions that agree with empirical data, rate maximization of the classical \ac{OFT} approach is often criticized. \index{patch overstaying}For example, \citet{Nonacs01} tabulates 19 cases of foragers being observed to process tasks longer than predicted by rate maximization, which shows that observations tend to deviate only in one direction. \Citeauthor{Nonacs01} concludes that ``something fundamental is missing'' \citep[p.~71]{Nonacs01} from the classical \ac{OFT} approach. However, as we discussed, rate maximization exerts an upward pressure on time when energy budgets are negative. Thus, the deficiency may not be in classical \ac{OFT} but in gain models that do not properly include negative cases\footnote{A generalization of this suggestion is given in \longref{sec:alternate_rate_tradeoffs}.}. Additionally, the risk-sensitive analysis of \citet{SC82} also explains this upward pressure on time for similar reasons. However, this analysis receives significantly less attention than the rate maximization approach even though analogous risk-sensitive methods are used with great success in other fields, like \ac{MPT}. Alternate approaches choose to abandon the classical \ac{OFT} agent model \citep[\eg,][]{BK95,BW96,CB05,HR87,HG05,IHY81,Real80,TL81}; these approaches study risk aversion, satisficing\footnote{\index{satisficing|(indexdef}Satisficing describes a behavior that is suboptimal but achieves some minimum gain (\ie, ``survival of the more fit over the less fit, not necessarily of the most fit'' \citep[p.~640]{CB05}).\index{satisficing|)indexdef}}, heuristic rules, and even propose fitting empirical data directly without any functional justification for the resulting behavioral rules. We value the generality and intuitive appeal of the classical \ac{OFT} approach. Additionally, even though it may be debatable whether it can be used for evolutionary analysis in biology, optimization has natural applications in engineering. Therefore, we use our generalized agent model and build optimization objectives inspired by both classical \ac{OFT} and its critics; these objectives should have applicability to engineering and may answer some of the questions about the shortfalls of the predictive power of classical \ac{OFT}. \index{solitary agent model!classical analysis!optimization|)} \section{Generalized Optimization of Solitary Agent Behavior}% \label{sec:alternate_optimization_objectives}% \index{solitary agent model!processing-only analysis!optimization|(} Here, we discuss optimization criteria that may have utility in both engineering and biology. As discussed in \longref{sec:oft_approach_optimization}, the analysis of asymptotic behaviors may be justifiable in evolutionary analysis. However, in engineering applications, it may be more appropriate to focus on the case where an agent's finite lifetime is determined entirely by a set number of tasks the agent must \emph{complete}. Also, because design takes the place of natural selection, we need to explore different optimization objectives. Therefore, in this section, we use our generalized solitary agent model and insights from both \ac{OFT} and \ac{MPT} to generate new optimization objectives that may be applicable to both biology and engineering. We discuss our approach to handling finite lifetimes in \longref{sec:finite_task_processing}. In \longref{sec:alternate_rate_tradeoffs}, we present functions that, like classical \ac{OFT}, use rates to trade off between multiple optimization objectives. In \longref{sec:alternate_pareto_tradeoffs}, we present generalized \aimention{Vilfredo Pareto}Pareto optimal combinations of optimization objectives that allow a design to tune the relative importance of the objectives. Note that when we discuss standard deviation in variance, we assume \longrefs{eq:cycle_proc_gross_gain_2moment}--% \shortref{eq:cycle_net_gain_2moment} for simplicity. Some of the objectives that we define still operate on the sliding scale of behaviors described by \citet{Schoener71}. The primary justification in classical \ac{OFT} for using the long-term rate of gain discussed in \longref{sec:oft_long_term_rate} is that it agrees with \citeauthor{Schoener71} and makes predictions that agree \emph{qualitatively} with empirical evidence \citep{Cha73,PPC77,SK86}. However, some of our alternative objectives also agree with \citeauthor{Schoener71} and have qualitiative predictions similar to those from classical \ac{OFT}. Therefore, these criteria seem as well-suited for biological application as the long-term rate of gain. In fact, as discussed in \longref{sec:alternate_rate_tradeoffs}, the cases where our approaches yield different predictions than the classical \ac{OFT} approach may be interesting and help explain some of the empirical inconsistencies with the long-term rate of gain approach \citep[\eg,][]{BK95,HR87,Nonacs01}. \subsection{Finite Task Processing} \label{sec:finite_task_processing} One disadvantage of studying behavior that is optimal with respect to the long-term rate of gain is the necessity of justifying the use of infinite-time (\ie, infinite-cycle) approximation. A true finite-time optimization approach is desirable. There are several dynamic programming approaches to time-limited agents \citep[\eg,][]{ClkMngl00, HouMc99, MnglClk88, WaBeHaBo06} where behaviors are based on state. While this state-based approach provides can be better for the analysis animal behavior, it sometimes provides less intuition and is more difficult to implement on-line in an engineered agent that may have very limited computational abilities. Additionally, assuming that the time horizon is known or even fixed may be unrealistic. Thus, an approach that combines elements of both the infinite-time approach and the time-limited approach could be useful. Rather than fixing the time horizon, we fix the number of tasks processed. This is useful for modeling, for example, a situation where an agent expends a limited resource on each processed task. Therefore, while the total searching and processing time is finite and unknown (\ie, random), the total number of tasks to process is fixed. So, take $N^p \in \N$ to be a fixed number of tasks completed by an agent. Our objective functions are based on statistics from our modeling approach. Of course, these statistics take the total number of tasks $N^p$ as a parameter. Therefore, it makes sense that behaviors for a small number of total tasks may vary greatly from behaviors for a large number of tasks. However, in all cases behaviors will not be based on state. In particular, the behavior for the first task processed will be consistent with the behavior for the last task processed. This allows for the construction of behaviors that can be implemented on very simple engineered agents. \subsection{Tradeoffs as Ratios} \label{sec:alternate_rate_tradeoffs} Ratios are unique in their ability to apply pressure on one objective conditioned on the sign of another objective. However, the derivation of the expectation of ratios of random variables is not trivial and often requires high-order statistics of the random variables. Despite the drawbacks, we explore some ratios of statistics here that may be useful when there are no other easy ways to apply the appropriate pressures for optimization. A more general method of multiobjective optimization is explored in \longref{sec:alternate_pareto_tradeoffs}. For example, classical \ac{OFT} studies the optimization objective $\E(\oft{G}_1)/\E(\oft{T}_1)$ because it is the limiting expression of $\E(\oft{G}(t))/t$ as $t \to \infty$ and maximization of the long-term rate of gain is desirable. The analogous expression with our assumption of a finite number of tasks processed is $\E(G^{N^p}/T^{N^p})$, but this expectation has poor analytical tractability. Although not ideal, maximization of $\E(G_1)/\E(T_1)$ provides similar pressures on behaviors and has a relatively simple analytical structure. Therefore, it is reasonable to study behaviors that are optimal with respect to this latter objective. \subsubsection{Rate Maximization with Gain Threshold} \index{rate of excess net gain|(indexdef}% Consider an agent that must achieve an expected net gain of at least \index{gain success threshold}$G^T$ after $N^p$ tasks processed but must also achieve its goal in as little time as possible. This problem has aspects in common with both \index{rate maximization}rate maximization and \index{risk sensitivity}risk sensitivity. Therefore, we consider the objective function % \begin{equation} \frac{ \E(G^{N^p}) - G^T }{ \E(T^{N^p}) } \end{equation} % or, equivalently, % \begin{equation} \frac{ \E(G_1) - \frac{G^T}{N^p} }{ \E(T_1) } \label{eq:payoff_rate} \end{equation} % So, the goal of the agent is to maximize its expected \emph{excess} net gain in as little expected time possible. This may be considered a generalization of the long-term rate of gain optimization used in classical \ac{OFT}.% \index{rate of excess net gain|)indexdef} \index{rate of excess net gain|(}% \index{rate of excess net gain!graphical optimization|(}% A \index{examples!graphical solutions!rate of excess net gain|(indexdef}graphical solution to this problem is shown in \longref{fig:rate_maximization}. This graphical approach is similar to the time-constrained approach of \citet{Rapport71} that is described above. % %\begin{figure}[!ht]\centering \begin{figure}\centering \begin{picture}(168,68)(0,0) \put(0, 0){\scalebox{0.8}{\framebox(210,85){ \shortstack[c]% {$\mu \triangleq \E(G_1) = % \overline{g^p} - \overline{c^p} - \frac{c^s}{\lambda^p}% $\\% $\E(T_1) = \overline{\tau^p} + \frac{1}{\lambda^p}$\\% $\gamma \triangleq \frac{\mu - % \frac{G^T}{N^p}}{\overline{\tau^p}% + \frac{1}{\lambda^p}}$\\% $\lozenge^* \triangleq \max\left\{\lozenge\right\}$}}}} \end{picture}\\ \subfloat[Per-Cycle Perspective]{ \begin{picture}(191,155)(-39,-20) % Horizontal Axis \thicklines \put(-30, 12){\vector(1, 0){144}} %\put(-30, 0){\vector(1, 0){144}} \put(115, 12){\makebox(0, 0)[l]% {$\overline{\tau^p} + \frac{1}{\lambda^p}$}} % Vertical Axis \thicklines \put(-12, -16){\vector(0, 1){136}} %\put(0, -16){\vector(0, 1){136}} \put(-12, 121){\makebox(0, 0)[b]{$\mu$}} % Rate Line \thinlines \put(-12, 92){\line(2,-1){126}} \put(84, 44){\circle*{5}} % Maximal Rate (Slope) \thinlines \put(20,76){\line(1,0){34}} \put(54,76){\line(0,-1){17}} \put(53,75){\makebox(0, 0)[rt]{$\gamma^*$}} % Threshold Value \thicklines \put(-16, 92){\line(1,0){8}} \put(-17, 92){\makebox(0, 0)[r]{$\frac{G^T}{N^p}$}} % Curve, starting origin and moving CW \thinlines \qbezier(0,0)(10,55)(40,55) \qbezier(40,55)(58.5,55)(84,44) %\qbezier(84,44)(92,41)(114,21) \qbezier(84,44)(92,41)(100,21) % Max and mins \thicklines \put(-16,55){\line(1,0){8}} \put(100,8){\line(0,1){8}} \thinlines \put(-17,55){\makebox(0, 0)[r]{$\mu^*$}} \put(-12,55){\dashbox{3}(52,0){}} \put(100,7){\makebox(0, 0)[t]% {$\left(\overline{\tau^p}% +\frac{1}{\lambda^p}\right)^*$}} \put(100,12){\dashbox{3}(0,9){}} % Origin strategy \thicklines \put(0,8){\line(0,1){8}} \put(-16,0){\line(1,0){8}} \thinlines \put(0,17){\makebox(0, 0)[b]% {$\frac{1}{\lambda^p}$}} \put(-17,0){\makebox(0, 0)[r]% {$-\frac{c^s}{\lambda^p}$}} \end{picture} \label{fig:rate_maximization_cycle} } \subfloat[Graphical Interpretation]{ \begin{picture}(182,155)(-54,-20) % Horizontal Axis \thicklines \put(-30, 0){\vector(1, 0){144}} \put(115, 0){\makebox(0, 0)[l]{$\overline{\tau^p}$}} % Vertical Axis \thicklines \put(0, -16){\vector(0, 1){136}} \put(0, 121){\makebox(0, 0)[b]% {$\overline{g^p}-\overline{c^p}$}} % Rate Line \thinlines \put(-12, 92){\line(2,-1){132}} \put(84, 44){\circle*{5}} % Maximal Rate (Slope) \thinlines \put(20,76){\line(1,0){34}} \put(54,76){\line(0,-1){17}} \put(53,75){\makebox(0, 0)[rt]{$\gamma^*$}} % Threshold Value \thicklines \put(-4, 80){\line(1,0){8}} \put(-5, 80){\makebox(0, 0)[rt]% {\scalebox{0.8}{$\frac{G^T}{N^p}$}}} \put(-4, 92){\line(1,0){8}} \put(5, 90){\makebox(0, 0)[lb]% {$\frac{G^T}{N^p}+\frac{c^s}{\lambda^p}$}} \put(0, 80){\circle*{2}} \put(-2, 82){\circle*{1}} \put(-4, 84){\circle*{1}} \put(-6, 86){\circle*{1}} \put(-8, 88){\circle*{1}} \put(-10, 90){\circle*{1}} \put(-12, 92){\circle*{3}} \put(-14, 94){\circle*{1}} \put(-16, 96){\circle*{1}} \put(-18, 98){\circle*{1}} \put(-20, 100){\circle*{1}} \put(-22, 102){\circle*{1}} \put(-24, 104){\circle*{1}} \put(-26, 106){\circle*{1}} \put(-28, 108){\circle*{1}} \put(-30, 110){\circle*{1}} \put(-32, 112){\circle*{1}} \put(-34, 114){\circle*{1}} \put(-36, 116){\circle*{1}} \put(-38, 118){\vector(-1,1){1}} \put(-28,108){\line(0,-1){6}} \put(-28,102){\line(1,0){6}} \put(-29,105){\makebox(0, 0)[r] {\scalebox{0.8}{$-c^s$}}} % Curve, starting origin and moving CW \thinlines \qbezier(0,0)(10,55)(40,55) \qbezier(40,55)(58.5,55)(84,44) %\qbezier(84,44)(92,41)(114,21) \qbezier(84,44)(92,41)(100,21) % Max and mins \thicklines \put(-4,55){\line(1,0){8}} \put(100,-4){\line(0,1){8}} \thinlines \put(-5,54){\makebox(0, 0)[r]% {$\left(\overline{g^p}-\overline{c^p}\right)^*$}} \put(0,55){\dashbox{3}(40,0){}} \put(100,-5){\makebox(0, 0)[t]% {$\left(\overline{\tau^p}\right)^*$}} \put(100,0){\dashbox{3}(0,21){}} % Origin strategy \thicklines \put(-12,-4){\line(0,1){8}} %\put(-4,12){\line(1,0){8}} \thinlines \put(-12,-5){\makebox(0, 0)[t]% {$\frac{-1}{\lambda^p}$}} %\put(-5,12){\makebox(0, 0)[r]% %{$-\frac{c^s}{\lambda^p}$}} \end{picture} \label{fig:rate_maximization_cycle_visualization} } % \\ % \subfloat[Limiting Cases]{ % \begin{picture}(401.5,115.5)(0,0) % \put(0,0){\scalebox{.75}{ % \begin{picture}(176,154)(-36,-20) % %\put(-36,-20){\framebox(176,154){}} % % Horizontal Axis % \thicklines % \put(-30, 0){\vector(1, 0){144}} % \put(115, 0){\makebox(0, 0)[l]% % {$\overline{\tau^p}$}} % % Vertical Axis % \thicklines % \put(0, -16){\vector(0, 1){136}} % \put(0, 121){\makebox(0, 0)[b]% % {$\overline{g^p}-\overline{c^p}$}} % % Curve, starting origin and moving CW % \thinlines % \qbezier(0,0)(10,55)(40,55) % \qbezier(40,55)(58.5,55)(84,44) % %\qbezier(84,44)(92,41)(114,21) % \qbezier(84,44)(92,41)(100,21) % % Max and mins % \thicklines % \put(100,-4){\line(0,1){8}} % \thinlines % \put(100,-5){\makebox(0, 0)[t]% % {$\left(\overline{\tau^p}\right)^*$}} % \put(100,0){\line(0,1){120}} % % Limits % \put(100, 21){\circle*{10}} % \put(108,60){\rotatebox{90}{\makebox(0,0)[t]% % {\shortstack% % {Time maximizer\\for\\$c^s \to \infty$}}}} % \put(0, 0){\circle*{10}} % \put(-2,60){\rotatebox{90}{\makebox(0,0)[b]% % {\shortstack% % {Time minimizer\\for\\$c^s \to -\infty$}}}} % \put(50,80){\makebox(0,0)% % {$\lim\limits_{c^s\to\pm\infty}\gamma^*% % =\mp \infty$}} % \end{picture} % }} % End of \put\scalebox for left figure % % % \put(134,0){\scalebox{.75}{ % \begin{picture}(176,154)(-54,-20) % %\put(-54,-20){\framebox(176,154){}} % % % \end{picture} % }} % End of \put\scalebox for middle figure % % % \put(268,0){\scalebox{.75}{ % \begin{picture}(178,154)(-37,-20) % %\put(-37,-20){\framebox(178,154){}} % % Horizontal Axis % \thicklines % \put(-30, 0){\vector(1, 0){144}} % \put(115, 0){\makebox(0, 0)[l]% % {$\overline{\tau^p}$}} % % Vertical Axis % \thicklines % \put(0, -16){\vector(0, 1){136}} % \put(0, 121){\makebox(0, 0)[b]% % {$\overline{g^p}-\overline{c^p}$}} % % Curve, starting origin and moving CW % \thinlines % \qbezier(0,0)(10,55)(40,55) % \qbezier(40,55)(58.5,55)(84,44) % %\qbezier(84,44)(92,41)(114,21) % \qbezier(84,44)(92,41)(100,21) % % Max and mins % \thicklines % \put(100,-4){\line(0,1){8}} % \thinlines % \put(100,-5){\makebox(0, 0)[t]% % {$\left(\overline{\tau}\right)^*$}} % \put(100,0){\line(0,1){120}} % % Limits % \put(100, 21){\circle*{10}} % \put(108,60){\rotatebox{90}{\makebox(0,0)[t]% % {\shortstack% % {Time maximizer\\for\\$G^T \to \infty$}}}} % \put(0, 0){\circle*{10}} % \put(-2,60){\rotatebox{90}{\makebox(0,0)[b]% % {\shortstack% % {Time minimizer\\for\\$G^T \to -\infty$}}}} % \put(50,80){\makebox(0,0)% % {$\lim\limits_{G^T\to\pm\infty}\gamma^*% % =\mp \infty$}} % \end{picture} % }} % End of \put\scalebox for right figure % \end{picture} % \label{fig:rate_maximization_cycle_limit_case} % } \caption[Visualization of Rate Maximization]{Rate maximization. Curves show the \aimention{Vilfredo Pareto}Pareto frontiers for maximizing net gain mean and maximizing or minimizing total time. $G^T$ denotes the net gain survival threshold. $\gamma^*$ denotes the maximal rate, and the solutions that achieve this rate are shown as large dots at points of tangency with a line of slope $\gamma^*$. The dotted arrow in \subref{fig:rate_maximization_cycle_visualization} shows how the figure changes as with $c^s$ and $\lambda^p$. It is not necessary that $|\mu^*| < \infty$ and $(\overline{\tau^p} + 1/\lambda^p)^* < \infty$.} \label{fig:rate_maximization} \end{figure} % The curve shows the set of all \aimention{Vilfredo Pareto}Pareto efficient solutions for maximizing gain and both maximizing or minimizing time\footnote{That is, points on the curve represent the highest and lowest expected total times possible for a given expected net gain.}. In \longref{fig:rate_maximization_cycle_visualization}, the axes are shifted to highlight the impact of parameter changes on the solution\footnote{The encounter rate $\lambda^p$ may only be considered to be an environmental parameter if, for all $i \in \{1,2,\dots,n\}$, $p_i$ is not free for the agent to choose. The analogous graphical interpretation of results on classical \ac{OFT} statistics does not have this constraint. However, in both cases the shape of the \aimention{Vilfredo Pareto}Pareto frontier curve will change with changes in the overall encounter rate if the individual task type encounter rates are not also scaled by the same factor.}. \longref{fig:oft_rate_maximization} was generated in an identical way; however, the single gain curve $g_1(\tau_1)$ is replaced with a curve representing the \aimention{Vilfredo Pareto}Pareto-efficient-average-processing-gain curve. This graphical interpretation of rate maximization applies to any case rather than just the single type patch case\footnote{Recall that $\overline{g^p}=g_1$ and $\overline{\tau^p}=\tau_1$ for the $n=1$ (and $p_1=1$) case.}. Additionally, search and processing costs are shown. In fact, when $G^T = 0$, it is clear that the cost of searching plays the same role as a gain success threshold. The line whose slope is the rate may be viewed as \emph{pivoting} at the point $(-1/\lambda^p,G^T/N^p+c^s/\lambda^p)$, and so this may be called the \emph{threshold pivot point}. For this particular $\overline{g^p}$ and $\overline{\tau^p}$, several observations can be made. % \begin{description} \item\emph{Cycle Lifetime and Gain Threshold:} The per-cycle net gain threshold $G^T/N^p$ depends upon the number of lifetime cycles $N^p$. Therefore, the length of lifetime has an impact on the gain budget and therefore the optimal strategy. \item\emph{Negative Gain Budgets:} \index{patch overstaying!rational explanation}Whenever the expected gain is less than the gain success threshold, the average time spent processing tasks (which the agent can control through both preference probability and processing time) will increase. Graphically, whenever $G^T/N^p + c^s/\lambda^p > (\overline{g^p}-\overline{c^p})^*$, an increase in $G^T$ will bring an increase in the $\overline{\tau^p}$ solution. Therefore, rate maximization in classical \ac{OFT} may predict low processing times compared to observations because costs or required energy thresholds are being neglected. \item\emph{Sunk Cost Effect and Concorde Fallacy:} \index{sunk-cost effect}\index{Concorde fallacy|see{sunk-cost effect}}\citet{HA99} review both the \emph{sunk cost effect} and the \emph{Concorde fallacy}, which are well-known in economics and biology. These describe the same behavior, except the former is typically applied to humans and the latter is typically applied to animals. Agents are said to manifest this behavior when showing a ``greater tendency to continue an endeavor once an investment in'' gain has been made \citep[p.~591]{HA99}. In other words, an agent will continue to process tasks evidently \emph{because} of the \emph{cost} of prior processing. \citeauthor{HA99} suggest that this is due to the use of an ordinarily adaptive heuristic rule; however, we suggest that rate maximization may also explain this effect. Consider the situation shown in \longref{fig:rate_maximization_cycle_visualization}, but replace the strictly positive $(\overline{g^p} - \overline{c^p})$-$\overline{\tau^p}$ curve with a version of itself that has been mirrored around the $\tau^p$-axis. In this case, the optimal average processing time is $(\tau^p)^*$. That is, it is better to face a negative energy budget after a long period of time rather than facing a negative energy budget after a short period of time, even though the later negative energy budget is worse (\ie, it may be better to be short in gain after much effort than being short in gain after no effort). \end{description} % Therefore, rate maximization explains some effects that may otherwise be considered to be \index{rationality|see{ecological rationality}}\index{ecological rationality}irrational.% \index{examples!graphical solutions!rate of excess net gain|)indexdef}% \index{rate of excess net gain|)}% \index{rate of excess net gain!graphical optimization|)}% \subsubsection{Efficiency}% \index{efficiency|see{excess efficiency}}% \index{excess efficiency|(indexdef} Now, take an agent that must achieve an expected \emph{gross} gain of at least \index{gain success threshold}\index{gross gain success threshold}$G_g^T$ after $N^p$ tasks processed but must achieve its goal while accumulating as little cost as possible. We consider the objective function % \begin{equation} \frac{ \E(G^{N^p}) + \E(C^{N^p}) - G_g^T }{ \E(C^{N^p}) } \end{equation} % or, equivalently, % \begin{equation} \frac{ \E(G_1) + \E(C_1) - \frac{G_g^T}{N^p} }{ \E(C_1) } \label{eq:payoff_efficiency} \end{equation} % So, the goal of the agent is to maximize its expected \emph{excess} gross gain while also accumulating as little cost as possible. This is a notion of the agent's \emph{efficiency} (\ie, maximal benefit-to-cost ratio). While there is no direct pressure on time minimization in this objective, all costs in the model depend linearly on time. Therefore, minimization of cost indirectly has a time minimization effect as well.% \index{excess efficiency|)indexdef} \index{excess efficiency|(}% \index{excess efficiency!graphical optimization|(}% A \index{examples!graphical solutions!excess efficiency|(indexdef}graphical solution to this problem is shown in \longref{fig:efficiency_maximization}. % %\begin{figure}[!ht]\centering \begin{figure}\centering \begin{picture}(168,68)(0,0) \put(0, 0){\scalebox{0.8}{\framebox(210,85){ \shortstack[c]% {$\E(G_1) + \E(C_1) = \overline{g^p}$\\% $\E(C_1) = \overline{c^p} + \frac{c^s}{\lambda^p}$\\% $\epsilon \triangleq \frac{\overline{g^p} - % \frac{G^T}{N^p}}{\overline{c^p}% + \frac{c^s}{\lambda^p}}$\\% $\lozenge^* \triangleq \max\left\{\lozenge\right\}$}}}} \end{picture}\\ \subfloat[Per-Cycle Perspective]{ \begin{picture}(194,155)(-42,-21) % Horizontal Axis \thicklines %\put(-30, 12){\vector(1, 0){144}} \put(-30, 0){\vector(1, 0){144}} \put(115, 0){\makebox(0, 0)[l]% {$\overline{c^p} + \frac{c^s}{\lambda^p}$}} % Vertical Axis \thicklines \put(-12, -16){\vector(0, 1){136}} %\put(0, -16){\vector(0, 1){136}} \put(-12, 121){\makebox(0, 0)[b]{$\overline{g^p}$}} % Rate Line \thinlines \put(-12, 92){\line(2,-1){126}} \put(84, 44){\circle*{5}} % Maximal Rate (Slope) \thinlines \put(20,76){\line(1,0){34}} \put(54,76){\line(0,-1){17}} \put(53,75){\makebox(0, 0)[rt]{$\epsilon^*$}} % Threshold Value \thicklines \put(-16, 92){\line(1,0){8}} \put(-17, 92){\makebox(0, 0)[r]{$\frac{G_g^T}{N^p}$}} % Curve, starting origin and moving CW \thinlines \qbezier(0,0)(10,55)(40,55) \qbezier(40,55)(58.5,55)(84,44) %\qbezier(84,44)(92,41)(114,21) \qbezier(84,44)(92,41)(100,21) % Max and mins \thicklines \put(-16,55){\line(1,0){8}} \put(100,-4){\line(0,1){8}} \thinlines \put(-17,55){\makebox(0, 0)[r]% {$\left(\overline{g^p}\right)^*$}} \put(-12,55){\dashbox{3}(52,0){}} \put(100,-4){\makebox(0, 0)[t]% {$\left(\overline{c^p}% +\frac{c^s}{\lambda^p}\right)^*$}} \put(100,0){\dashbox{3}(0,21){}} % Origin strategy \thicklines \put(0,-4){\line(0,1){8}} \put(-16,0){\line(1,0){8}} \thinlines \put(0,-5){\makebox(0, 0)[t]% {$\frac{c^s}{\lambda^p}$}} \end{picture} \label{fig:efficiency_maximization_cycle} } \subfloat[Graphical Interpretation]{ \begin{picture}(160,155)(-33,-21) % Horizontal Axis \thicklines \put(-30, 0){\vector(1, 0){144}} \put(115, 0){\makebox(0, 0)[l]{$\overline{c^p}$}} % Vertical Axis \thicklines \put(0, -16){\vector(0, 1){136}} \put(0, 121){\makebox(0, 0)[b]{$\overline{g^p}$}} % Rate Line \thinlines \put(-12, 92){\line(2,-1){126}} \put(84, 44){\circle*{5}} % Maximal Rate (Slope) \thinlines \put(20,76){\line(1,0){34}} \put(54,76){\line(0,-1){17}} \put(53,75){\makebox(0, 0)[rt]{$\epsilon^*$}} % Threshold Value \thicklines \put(-4, 92){\line(1,0){8}} \put(5, 90){\makebox(0, 0)[lb]% {$\frac{G_g^T}{N^p}$}} \put(0, 92){\circle*{2}} \put(-2, 92){\circle*{1}} \put(-4, 92){\circle*{1}} \put(-6, 92){\circle*{1}} \put(-8, 92){\circle*{1}} \put(-10, 92){\circle*{1}} \put(-12, 92){\circle*{3}} \put(-14, 92){\circle*{1}} \put(-16, 92){\circle*{1}} \put(-18, 92){\circle*{1}} \put(-20, 92){\circle*{1}} \put(-22, 92){\circle*{1}} \put(-24, 92){\circle*{1}} \put(-26, 92){\circle*{1}} \put(-28, 92){\circle*{1}} \put(-30, 92){\vector(-1,0){1}} % Curve, starting origin and moving CW \thinlines \qbezier(0,0)(10,55)(40,55) \qbezier(40,55)(58.5,55)(84,44) %\qbezier(84,44)(92,41)(114,21) \qbezier(84,44)(92,41)(100,21) % Max and mins \thicklines \put(-4,55){\line(1,0){8}} \put(100,-4){\line(0,1){8}} \thinlines \put(-5,54){\makebox(0, 0)[r]% {$\left(\overline{g^p}\right)^*$}} \put(0,55){\dashbox{3}(40,0){}} \put(100,-5){\makebox(0, 0)[t]% {$\left(\overline{c^p}\right)^*$}} \put(100,0){\dashbox{3}(0,21){}} % Origin strategy \thicklines \put(-12,-4){\line(0,1){8}} %\put(-4,12){\line(1,0){8}} \thinlines \put(-12,-5){\makebox(0, 0)[t]% {$\frac{-c^s}{\lambda^p}$}} %\put(-5,12){\makebox(0, 0)[r]% %{$-\frac{c^s}{\lambda^p}$}} \end{picture} \label{fig:efficiency_maximization_cycle_visualization} } \caption[Visualization of Efficiency Maximization]{Efficiency maximization. Curves show \aimention{Vilfredo Pareto}Pareto frontiers for maximizing gross gain mean and minimizing total cost. $G_g^T$ denotes the gross gain survival threshold. $\epsilon^*$ denotes the maximal efficiency, and the solutions that achieve this efficiency are shown as large dots at points of tangency with a line of slope $\epsilon^*$. The dotted arrow in \subref{fig:efficiency_maximization_cycle_visualization} shows how the figure changes as with $c^s$ and $\lambda^p$. It is not necessary that $|(\overline{g^p})^*| < \infty$ and $|(\overline{c^p} + c^s/\lambda^p)^*| < \infty$.} \label{fig:efficiency_maximization} \end{figure} % Here, the curve shows the set of all \aimention{Vilfredo Pareto}Pareto efficient solutions for maximizing gross gain and both maximizing or minimizing cost\footnote{That is, points on the curve represent the highest and lowest expected cost possible for a given expected gross gain.}. In \longref{fig:efficiency_maximization_cycle_visualization}, the axes are shifted to highlight the impact of parameter changes on the solution\footnote{As with the graphical interpretation of rate maximization, the effects of parameter changes are simpler to explore using classical \ac{OFT} statistics because the overall encounter rate is not influenced by the preference probabilities, which are often considered to be decision variables that describe agent behavior.}. The line whose slope is the efficiency may be viewed as \emph{pivoting} at the point $(-c^s/\lambda^p,G_g^T/N^p)$, and so this may be called the \emph{threshold pivot point}. For this particular $\overline{g^p}$ and $\overline{c^p}$, several observations can be made. % \begin{description} \item\emph{Cycle Lifetime and Gain Threshold:} As with rate maximization, the per-cycle gross gain threshold $G_g^T/N^p$ depends upon the number of lifetime cycles $N^p$. Therefore, the length of lifetime has an impact on the gain budget and the optimal strategy. \item\emph{Negative Gain Budgets:} Assuming the gain threshold is high, the optimal behavior moves toward has a higher cost and a lesser gross gain as the gross gain threshold is increased. In other words, the number of points \emph{lost} per unit cost (\ie, the loss efficiency) is actually lower at a higher cost. \item\emph{Negative Search Costs:} Assume that $c^s < 0$ and $G_g^T \leq 0$. In this case, the optimal solution is the one such that $-c^s/\lambda = \overline{c^p}$. In other words, when facing a positive energy budget and a search \emph{gain}, the most efficient solution is the one where search gains are equal to processing costs. A similar result holds for negative average search costs. Essentially, whenever the gain threshold pivot point is below (or encircled by) the \aimention{Vilfredo Pareto}Pareto frontiers, the optimal solution will be the one that makes $\E(C_1)=0$. \item\emph{Sunk Cost Effect and Concorde Fallacy:} \index{sunk-cost effect}Efficiency maximization provides an explanation for the sunk cost effect and the Concorde fallacy similar to the explanation for rate maximization. If the $\overline{g^p}$-$\overline{c^p}$ curve in \longref{fig:efficiency_maximization_cycle_visualization} is mirrored about the $\overline{c^p}$-axis, the resulting optimal cost will be $(\overline{c^p})^*$. That is, processing cost is maximized in this case. Roughly, it is ``better'' to have a negative gain budget at high cost than low cost. \end{description} % As with rate maximization, an analysis of this optimization objective suggests explanations for behaviors that might normally be considered \index{ecological rationality}irrational.% \index{examples!graphical solutions!excess efficiency|)indexdef}% \index{excess efficiency!graphical optimization|)}% \index{excess efficiency|)}% \subsubsection{Risk-Sensitivity: Reward-to-Variability Ratios}% \index{reward-to-variability ratio|(} As discussed, \citet{Sharpe66} introduces an index called the \emph{reward-to-variability ratio}\footnote{Again, this is often called the \emph{\citeauthor{Sharpe66} ratio} by those other than \citeauthor{Sharpe66}.}, a measure of future performance that balances expected return with standard deviation of return. When returns are location-scale distributed with identical skewness for each location-scale choice (\eg, normally distributed), maximization of this index is identical to minimization of risk of return shortfall.% \index{reward-to-variability ratio|)} \index{reward-to-variability ratio|(indexdef}% Let \index{net gain success threshold|see{gain success threshold}}\index{gain success threshold}$G^T$ represent a (deterministic) net gain threshold of success for the $N^p$ cycles in an agent's lifetime. In our context\footnote{For simplicity, we assume \longrefs{eq:cycle_proc_gross_gain_2moment}--% \shortref{eq:cycle_net_gain_2moment}.}, the reward-to-variability ratio is expressed by % \begin{equation*} \frac{ \E(G^{N^p}) - G^T }{ \std( G^{N^p} ) } \quad \text{ or, equivalently, } \quad \sqrt{N^p} \frac{ \E(G_1) - \frac{G^T}{N^p} }{ \std( G_1 ) } \end{equation*} % That is, this is a ratio of \emph{excess} returns to the standard deviation of those returns.% \index{reward-to-variability ratio|)indexdef}% \index{reward-to-variability ratio|(}% \index{reward-to-variability ratio!graphical optimization|(}% A \index{examples!graphical solutions!reward-to-variability ratio|(indexdef} graphical solution to this problem is shown in \longref{fig:risk_sensitivity_sharpe}. % %\begin{figure}[!ht]\centering \begin{figure}\centering \begin{picture}(168,68)(0,0) \put(0, 0){\scalebox{0.8}{\framebox(210,85){ \shortstack[c]% {$\mu \triangleq \E(G_1) = % \overline{g^p} - \overline{c^p} - \frac{c^s}{\lambda^p}% $\\% $\sigma \triangleq \std(G_1) = \sqrt{% \var\left(g^p - c^p\right) % + \left(\frac{c^s}{\lambda^p}\right)^2% }$\\% $\rho \triangleq \frac{\mu - % \frac{G^T}{N^p}}{\sigma}$\\% $\lozenge^* \triangleq \max\left\{\lozenge\right\}$}}}} \end{picture}\\ \subfloat[Lifetime Perspective]{ \begin{picture}(214,149)(-30,-22) % Horizontal Axis \thicklines \put(-10, 12){\vector(1, 0){145}} \put(137, 12){\makebox(0, 0)[l]{$\std(G^{N^p})$}} % Vertical Axis \thicklines \put(12, -15){\vector(0, 1){125}} \put(12, 111){\makebox(0, 0)[b]{$\E(G^{N^p})$}} % Reward-to-Variability Line \thinlines \put(12, 94){\line(2,-1){123}} \put(100, 50){\circle*{5}} % Maximal Reward-to-Variability (Slope) \thinlines \put(16,92){\line(0,-1){29}} \put(16,63){\line(1,0){58}} \put(16,64){\makebox(0, 0)[lb]{$\sqrt{N^p} \rho^*$}} % Threshold Value \thicklines \put(8, 94){\line(1,0){8}} \put(7, 94){\makebox(0, 0)[r]{$G^T$}} % Curve, starting from upper left and moving CW \thinlines \qbezier(35,35)(35,60)(51,60) \qbezier(51,60)(80,60)(100,50) \qbezier(100,50)(120,40)(120,33) \qbezier(120,33)(120,10)(24,-5) % Max and mins \thicklines \put(120,8){\line(0,1){8}} \put(8,60){\line(1,0){8}} \thinlines \put(120,7){\makebox(0, 0)[t]{$\sqrt{N^p} \sigma^*$}} \put(120,12){\dashbox{3}(0,21){}} \put(7,60){\makebox(0, 0)[r]{$N^p \mu^*$}} \put(12,60){\dashbox{3}(39,0){}} % Origin strategy \thicklines \put(24,8){\line(0,1){8}} \put(8,-5){\line(1,0){8}} \thinlines \put(16,17){\makebox(0, 0)[lb]% {$\sqrt{N^p} \frac{|c^s|}{\lambda^p}$}} %\put(24,-5){\dashbox{3}(0,17){}} \put(7,-3){\makebox(0, 0)[r]% {$-N^p \frac{c^s}{\lambda^p}$}} %\put(12,-5){\dashbox{3}(12,0){}} \end{picture} \label{fig:risk_sensitivity_sharpe_lifetime} } \subfloat[Per-Cycle Perspective]{ \begin{picture}(160,149)(-14,-22) \thicklines % Horizontal Axis \put(-10, 12){\vector(1, 0){145}} %\put(137, 12){\makebox(0, 0)[l]{$\std(G_1)$}} \put(137, 12){\makebox(0, 0)[l]{$\sigma$}} % Vertical Axis \put(12, -15){\vector(0, 1){125}} %\put(12, 111){\makebox(0, 0)[b]{$\E(G_1)$}} \put(12, 111){\makebox(0, 0)[b]{$\mu$}} % Reward-to-Variability Line \thinlines \put(12, 85.2){\line(5,-2){123}} \put(110, 46){\circle*{5}} % Maximal Reward-to-Variability (Slope) \thinlines \put(20,82){\line(0,-1){20}} \put(20,62){\line(1,0){50}} \put(22,64){\makebox(0, 0)[lb]{$\rho^*$}} % Threshold Value \thicklines \put(8,85.2){\line(1,0){8}} \put(7,85.2){\makebox(0,0)[r]{$\frac{G^T}{N^p}$}} % Curve, starting from upper left and moving CW \thinlines \qbezier(35,30)(35,55)(51,55) \qbezier(51,55)(80,55)(110,46) \qbezier(110,46)(120,40)(120,33) \qbezier(120,33)(120,10)(24,0) % Max and mins \thicklines \put(120,8){\line(0,1){8}} \put(8,55){\line(1,0){8}} \thinlines \put(120,7){\makebox(0, 0)[t]{$\sigma^*$}} \put(120,12){\dashbox{3}(0,21){}} \put(7,55){\makebox(0, 0)[r]{$\mu^*$}} \put(12,55){\dashbox{3}(39,0){}} % Origin strategy \thicklines \put(24,8){\line(0,1){8}} \put(8,0){\line(1,0){8}} \thinlines \put(24,17){\makebox(0, 0)[b]{$\frac{|c^s|}{\lambda^p}$}} %\put(24,0){\dashbox{3}(0,12){}} \put(7,0){\makebox(0, 0)[r]{$-\frac{c^s}{\lambda^p}$}} %\put(12,0){\dashbox{3}(12,0){}} \end{picture} \label{fig:risk_sensitivity_sharpe_cycle} } \\ \subfloat[Graphical Interpretation]{ \begin{picture}(236,149)(-52,-22) \thicklines % Horizontal Axis \put(-10, 0){\vector(1, 0){155}} \put(147, 0){\makebox(0, 0)[l]% {$\sigma-\frac{|c^s|}{\lambda^p}$}} % Vertical Axis \put(24, -15){\vector(0, 1){125}} \put(24, 111){\makebox(0, 0)[b]% %{$\mu + \frac{c^s}{\lambda}$}} {$\overline{g^p} - \overline{c^p}$}} % Reward-to-Variability Line \thinlines \put(12, 85.2){\line(5,-2){133}} \put(110, 46){\circle*{5}} % Maximal Reward-to-Variability (Slope) \thinlines \put(31,77.6){\line(1,0){45}} \put(76,77.6){\line(0,-1){18}} \put(75,76.6){\makebox(0, 0)[rt]{$\rho^*$}} % Threshold Value %\thinlines %\put(0,85.2){\dashbox{3}(12,0){}} \thicklines \put(20,73.2){\line(1,0){8}} \put(12,-4){\line(0,1){8}} \thinlines %\put(-3,95.2){\makebox(0,0)[r]{$-\frac{c^s}{|c^s|}$}} \put(-3,95.2){\makebox(0,0)[r]{$-\sgn(c^s)$}} \put(-2,99.2){\line(0,-1){8}} \put(-2,91.2){\line(1,0){8}} \put(24,73.2){\circle*{1}} \put(22,75.2){\circle*{1}} \put(20,77.2){\circle*{1}} \put(18,79.2){\circle*{1}} \put(16,81.2){\circle*{1}} \put(14,83.2){\circle*{1}} \put(12,85.2){\circle*{3}} \put(10,87.2){\circle*{1}} \put(8,89.2){\circle*{1}} \put(6,91.2){\circle*{1}} \put(4,93.2){\circle*{1}} \put(2,95.2){\circle*{1}} \put(0,97.2){\circle*{1}} \put(-2,99.2){\circle*{1}} \put(-4,101.2){\circle*{1}} \put(-6,103.2){\circle*{1}} \put(-8,105.2){\circle*{1}} \put(-10,107.2){\circle*{1}} \put(-12,109.2){\vector(-1, 1){1}} \put(17,73.2){\makebox(0,0)[r]{$\frac{G^T}{N^p}$}} \put(12,-5){\makebox(0,0)[t]% {$-\frac{|c^s|}{\lambda^p}$}} % Curve, starting from upper left and moving CW \thinlines \qbezier(35,30)(35,55)(51,55) \qbezier(51,55)(80,55)(110,46) \qbezier(110,46)(120,40)(120,33) \qbezier(120,33)(120,10)(24,0) % Max and mins \thicklines \put(120,-4){\line(0,1){8}} \put(20,55){\line(1,0){8}} \thinlines \put(120,-5){\makebox(0, 0)[t]% {$\sigma^*-\frac{|c^s|}{\lambda^p}$}} \put(120,0){\dashbox{3}(0,33){}} \put(129,33){\circle*{1}} \put(126,33){\circle*{1}} \put(123,33){\circle*{1}} \put(120,33){\circle*{1}} \put(117,33){\circle*{1}} \put(114,33){\circle*{1}} \put(111,33){\circle*{1}} \put(108,33){\circle*{1}} \put(105,33){\circle*{1}} \put(102,33){\vector(-1, 0){1}} \put(19,55){\makebox(0, 0)[r]% {$\left( \overline{g^p} - \overline{c^p} \right)^*$}} \put(24,55){\dashbox{3}(27,0){}} \end{picture} \label{fig:risk_sensitivity_sharpe_cycle_visualization} } % \subfloat[Limiting Cases]{ % \begin{picture}(111,150)(-4,0) % \put(0,75){\scalebox{.5}{ % \begin{picture}(215,149)(-31,-22) % \thicklines % % Horizontal Axis % \put(-10, 0){\vector(1, 0){155}} % \put(147, 0){\makebox(0, 0)[l]% % {$\sigma-\frac{|c^s|}{\lambda^p}$}} % % Vertical Axis % \put(24, -15){\vector(0, 1){125}} % \put(24, 111){\makebox(0, 0)[b]% % {$\overline{g^p} - \overline{c^p}$}} % % Reward-to-Variability Line % \thinlines % \put(-10, 79){\line(1,-1){94}} % \put(-10, 11){\line(1,1){99}} % \put(24, 45){\circle*{10}} % % Threshold Value % \thicklines % %\put(20,73.2){\line(1,0){8}} % \thinlines % \put(-6,57)% % {\scalebox{0.75}{\makebox(0,0)[lb]{$-1$}}} % \put(-7,76){\line(0,-1){20}} % \put(-7,56){\line(1,0){20}} % \put(0,32)% % {\scalebox{0.75}{\makebox(0,0)[lt]{$1$}}} % \put(-1,20){\line(0,1){14}} % \put(-1,34){\line(1,0){14}} % %\put(21,73.2){\scalebox{0.75}{% % %\makebox(0,0)[rb]{$\frac{G^T}{N^p}$}}} % \linethickness{3.0\unitlength} % \put(24,55){\line(0,-1){55}} % % Max and mins % %\thicklines % %\put(20,55){\line(1,0){8}} % \thinlines % %\put(19,55){\makebox(0, 0)[r]% % %{$\left(\overline{g^p}% % %-\overline{c^p}\right)^*$}} % \put(34,57){\rotatebox{45}{ % \makebox(0,0)[lb]{$c^s < 0$}}} % \put(34,33){\rotatebox{315}{ % \makebox(0,0)[lt]{$c^s > 0$}}} % \put(84,52.5)% % {\makebox(100,15) % {\shortstack{Optimal Anywhere\\for\\% % $\frac{|c^s|}{\lambda^p} \to % \infty$\\[17\unitlength]% % $\rho^* = -\sgn(c^s)$}}} % \end{picture}} % } % End of \put\scalebox for upper figure % % % \put(0,0){\scalebox{.5}{ % \begin{picture}(215,149)(-31,-22) % \thicklines % % Horizontal Axis % \put(-10, 0){\vector(1, 0){155}} % \put(147, 0){\makebox(0, 0)[l]% % {$\sigma-\frac{|c^s|}{\lambda^p}$}} % % Vertical Axis % \put(24, -15){\vector(0, 1){125}} % \put(24, 111){\makebox(0, 0)[b]% % {$\overline{g^p} - \overline{c^p}$}} % % Reward-to-Variability Line % \thinlines % \put(120, 0){\line(0,1){110}} % \put(126,52.5){\rotatebox{90}{\makebox(0,0)[t]% % {\shortstack% % {Scale maximizer\\for\\$G^T \to \infty$}}}} % \put(22,52.5){\rotatebox{90}{\makebox(0,0)[b]% % {\shortstack% % {Scale minimizer\\for\\$G^T \to -\infty$}}}} % \put(120, 33){\circle*{10}} % \put(24, 0){\circle*{10}} % \put(72,80){\makebox(0,0)% % {$\lim\limits_{G^T\to\pm\infty} \rho^*% % =\mp \infty$}} % % Threshold Value % \thinlines % % Curve, starting from upper left and moving CW % \thinlines % \qbezier(35,30)(35,55)(51,55) % \qbezier(51,55)(80,55)(110,46) % \qbezier(110,46)(120,40)(120,33) % \qbezier(120,33)(120,10)(24,0) % % Max and mins % \thicklines % \put(120,-4){\line(0,1){8}} % \thinlines % \put(120,-5){\makebox(0, 0)[t]% % {$\sigma^*-\frac{|c^s|}{\lambda^p}$}} % \end{picture}} % End of \put\scalebox for lower figure % } % \end{picture} % \label{fig:risk_sensitivity_sharpe_cycle_limit_case} % } \caption[Visualization of Reward-to-Variability Maximization]{Reward-to-variability maximization. Curves show \aimention{Vilfredo Pareto}Pareto frontiers for maximizing mean net gain and minimizing net gain scale. $G^T$ denotes the net gain survival threshold. $\rho^*$ denotes the maximal per-cycle reward-to-variability ratio, and the solutions that achieve this ratio are shown as large dots at points of tangency with a line of slope $\rho^*$. The dotted arrows in \subref{fig:risk_sensitivity_sharpe_cycle_visualization} show how the figure changes as $|c^s|/\lambda^p$ increases. It is not necessary that $|\mu^*| < \infty$ and $\sigma^* < \infty$.} \label{fig:risk_sensitivity_sharpe} \end{figure} % The curve represents the \aimention{Vilfredo Pareto}Pareto frontiers for maximizing expected net gain and minimizing or maximizing net gain standard deviation. Notice that the shape of the curve shown in \longref{fig:risk_sensitivity_sharpe_lifetime} differs slightly from the shape of the curve in \longref{fig:risk_sensitivity_sharpe_cycle} due to different scaling of the axes. Even when $\lambda^p$ may be considered a parameter of the environment (\ie, not a function of decision variables), the shape of the \aimention{Vilfredo Pareto}Pareto frontier curve is still dependent upon it. Therefore, the graphical interpretation of maximization in \longref{fig:risk_sensitivity_sharpe_cycle_visualization} requires horizontally squeezing or stretching the curve with changes in $\lambda^p$. For this particular $\overline{g^p}$ and $\overline{c^p}$, several observations can be made. % \begin{description} \item\emph{Cycle Lifetime and Gain Threshold:} Similarly as with rate and efficiency maximization, the per-cycle gross gain threshold $G^T/N^p$ depends upon the number of lifetime cycles $N^p$. Therefore, the length of lifetime has an impact on the gain budget and therefore the optimal strategy. \item\emph{Negative Gain Budgets:} Assuming the gain threshold is high, the optimal behavior moves toward has a higher standard deviation and a lesser net gain as the net gain threshold is increased. In other words, the number of points \emph{lost} per unit standard deviation is actually lower at a higher standard deviation. Put another way, additional uncertainty in rewards is less costly because it comes with a higher probability that returns will be above the gain threshold. \end{description} % As with rate and efficiency maximization, an analysis of this optimization objective suggests explanations for behaviors that might normally be considered \index{ecological rationality}irrational.% \index{examples!graphical solutions!reward-to-variability ratio|)indexdef}% \index{reward-to-variability ratio!graphical optimization|)}% \index{reward-to-variability ratio|)}% \subsubsection{Reward-to-Variance Ratios}% \index{reward-to-variance ratio|(}% Optimization of portfolio performance based entirely on expectation and variance was originally developed by \citet{Markowitz52} and \citet{Tobin58}. \Citet{Sharpe66} makes the natural substitution of standard deviation for variance in the definition of the reward-to-variability ratio. While this substitution allows maximization of the performance index to have a risk minimization interpretation, it makes analytical derivation of the optimal behavior difficult. To compensate, we introduce a \emph{reward-to-variance} ratio that is motivated by the reward-to-variability ratio but provides greater analytical tractability.\index{reward-to-variance ratio|)}% \index{reward-to-variance ratio|(indexdef}% Let \index{gain success threshold}$G^T$ represent a (deterministic) net gain threshold of success for the $N^p$ cycles in an agent's lifetime. In our context\footnote{For simplicity, we assume \longrefs{eq:cycle_proc_gross_gain_2moment}--% \shortref{eq:cycle_net_gain_2moment}.}, the reward-to-variance ratio is expressed by % \begin{equation*} \frac{ \E(G^{N^p}) - G^T }{ \var( G^{N^p} ) } \quad \text{ or, equivalently, } \quad N^p \frac{ \E(G_1) - \frac{G^T}{N^p} }{ \var( G_1 ) } \end{equation*} % That is, this is a ratio of \emph{excess} returns to the \emph{variance} of those returns.% \index{reward-to-variance ratio|)indexdef}% \index{reward-to-variance ratio|(}% \index{reward-to-variance ratio!graphical optimization|(}% A \index{examples!graphical solutions!reward-to-variance ratio|(indexdef}graphical solution to this problem is shown in \longref{fig:risk_sensitivity_variance}. % %\begin{figure}[!ht]\centering \begin{figure}\centering \begin{picture}(168,68)(0,0) \put(0, 0){\scalebox{0.8}{\framebox(210,85){ \shortstack[c]% {$\mu \triangleq \E(G_1) = % \overline{g^p} - \overline{c^p} - \frac{c^s}{\lambda^p}% $\\% $\sigma^2 \triangleq \var(G_1) = % \var\left(g^p - c^p\right) % + \left(\frac{c^s}{\lambda^p}\right)^2$% \\% $\eta \triangleq \frac{\mu - % \frac{G^T}{N^p}}{\sigma^2}$\\% $\lozenge^* \triangleq \max\left\{\lozenge\right\}$}}}} \end{picture}\\ \subfloat[Lifetime Perspective]{ \begin{picture}(214,149)(-30,-22) % Horizontal Axis \thicklines \put(-10, 12){\vector(1, 0){145}} \put(137, 12){\makebox(0, 0)[l]{$\var(G^{N^p})$}} % Vertical Axis \thicklines \put(12, -15){\vector(0, 1){125}} \put(12, 111){\makebox(0, 0)[b]{$\E(G^{N^p})$}} % Reward-to-Variance Line \thinlines \put(12, 94){\line(2,-1){123}} \put(100, 50){\circle*{5}} % Maximal Reward-to-Variance (Slope) \thinlines \put(30,85){\line(1,0){34}} \put(64,85){\line(0,-1){17}} \put(63,84){\makebox(0, 0)[rt]{$\eta^*$}} % Threshold Value \thicklines \put(8, 94){\line(1,0){8}} \put(7, 94){\makebox(0, 0)[r]{$G^T$}} % Curve, starting from upper left and moving CW \thinlines \qbezier(35,35)(35,60)(51,60) \qbezier(51,60)(80,60)(100,50) \qbezier(100,50)(120,40)(120,33) \qbezier(120,33)(120,10)(29,0) % Max and mins \thicklines \put(120,8){\line(0,1){8}} \put(8,60){\line(1,0){8}} \thinlines \put(120,7){\makebox(0, 0)[t]{$N^p (\sigma^2)^*$}} \put(120,12){\dashbox{3}(0,21){}} \put(7,60){\makebox(0, 0)[r]{$N^p \mu^*$}} \put(12,60){\dashbox{3}(39,0){}} % Origin strategy \thicklines \put(29,8){\line(0,1){8}} \put(8,0){\line(1,0){8}} \thinlines \put(25,17){\makebox(0, 0)[lb]% {$N^p \left(\frac{c^s}{\lambda^p}\right)^2$}} \put(7,-3){\makebox(0, 0)[r]% {$-N^p \frac{c^s}{\lambda^p}$}} \end{picture} \label{fig:risk_sensitivity_variance_lifetime} } \subfloat[Per-Cycle Perspective]{ \begin{picture}(163,149)(-14,-22) \thicklines % Horizontal Axis \put(-10, 12){\vector(1, 0){145}} \put(137, 12){\makebox(0, 0)[l]{$\sigma^2$}} % Vertical Axis \put(12, -15){\vector(0, 1){125}} \put(12, 111){\makebox(0, 0)[b]{$\mu$}} % Reward-to-Variance Line \thinlines \put(12, 94){\line(2,-1){123}} \put(100, 50){\circle*{5}} % Maximal Reward-to-Variance (Slope) \thinlines \put(30,85){\line(1,0){34}} \put(64,85){\line(0,-1){17}} \put(63,84){\makebox(0, 0)[rt]{$\eta^*$}} % Threshold Value \thicklines \put(8, 94){\line(1,0){8}} \put(7, 94){\makebox(0, 0)[r]{$\frac{G^T}{N^p}$}} % Curve, starting from upper left and moving CW \thinlines \qbezier(35,35)(35,60)(51,60) \qbezier(51,60)(80,60)(100,50) \qbezier(100,50)(120,40)(120,33) \qbezier(120,33)(120,10)(29,0) % Max and mins \thicklines \put(120,8){\line(0,1){8}} \put(8,60){\line(1,0){8}} \thinlines \put(120,7){\makebox(0, 0)[t]{$(\sigma^2)^*$}} \put(120,12){\dashbox{3}(0,21){}} \put(7,60){\makebox(0, 0)[r]{$\mu^*$}} \put(12,60){\dashbox{3}(39,0){}} % Origin strategy \thicklines \put(29,8){\line(0,1){8}} \put(8,0){\line(1,0){8}} \thinlines \put(29,17){\makebox(0, 0)[b]% {$\left(\frac{c^s}{\lambda^p}\right)^2$}} \put(7,-3){\makebox(0, 0)[r]% {$-\frac{c^s}{\lambda^p}$}} \end{picture} \label{fig:risk_sensitivity_variance_cycle} } \\ \subfloat[Graphical Interpretation]{ \begin{picture}(239,149)(-30,-23) \thicklines % Horizontal Axis \put(-10, 0){\vector(1, 0){155}} \put(147, 0){\makebox(0, 0)[l]% {$\var\left(g^p - c^p\right)$}} % Vertical Axis \put(24, -15){\vector(0, 1){125}} \put(24, 111){\makebox(0, 0)[b]% {$\overline{g^p} - \overline{c^p}$}} % Reward-to-Variance Line \thinlines \put(7, 96.5){\line(2,-1){128}} \put(100, 50){\circle*{5}} % Maximal Reward-to-Variance (Slope) \thinlines \put(30,85){\line(0,-1){17}} \put(30,68){\line(1,0){34}} \put(31,70){\makebox(0, 0)[lb]{$\eta^*$}} % Threshold Value \thicklines \put(20,96.5){\line(1,0){8}} %\put(20,84.5){\line(1,0){8}} \put(7,-4){\line(0,1){8}} \thinlines \put(23,87.41){\circle*{1}} \put(22,88.62){\circle*{1}} \put(21,89.54){\circle*{1}} \put(20,90.32){\circle*{1}} \put(19,91){\circle*{1}} \put(17,92.2){\circle*{1}} \put(15,93.23){\circle*{1}} \put(13,94.15){\circle*{1}} \put(11,94.99){\circle*{1}} \put(9,95.77){\circle*{1}} \put(7,96.5){\circle*{3}} \put(5,97.19){\circle*{1}} \put(3,97.84){\circle*{1}} \put(1,98.46){\circle*{1}} \put(-1,99.05){\circle*{1}} \put(-3,99.62){\circle*{1}} \put(-5,100.17){\circle*{1}} \put(-7,100.71){\circle*{1}} \put(-9,101.22){\circle*{1}} \put(-11,101.72){\circle*{1}} \put(-13,102.20){\vector(-5, 1){1}} %\put(17, 84.5){\makebox(0, 0)[r]{$\frac{G^T}{N^p}$}} \put(29, 96.5){\makebox(0, 0)[l]% {$\frac{G^T}{N^p}+\frac{c^s}{\lambda^p}$}} \put(4,-5){\makebox(0,0)[t]% {$-\left(\frac{c^s}{\lambda^p}\right)^2$}} % Curve, starting from upper left and moving CW \thinlines \qbezier(35,35)(35,60)(51,60) \qbezier(51,60)(80,60)(100,50) \qbezier(100,50)(120,40)(120,33) \qbezier(120,33)(120,10)(29,0) % Max and mins \thicklines \put(120,-4){\line(0,1){8}} \put(20,60){\line(1,0){8}} \thinlines \put(120,-5){\makebox(0, 0)[t]% {$\left(\var\left(g^p-c^p\right)\right)^*$}} \put(120,0){\dashbox{3}(0,33){}} \put(19,60){\makebox(0, 0)[r]% {$\left( \overline{g^p} - \overline{c^p} \right)^*$}} \put(24,60){\dashbox{3}(27,0){}} \end{picture} \label{fig:risk_sensitivity_variance_cycle_visualization} } % \subfloat[Limiting Cases]{ % \begin{picture}(111,150)(-4,0) % \put(0,75){\scalebox{.5}{ % \begin{picture}(241,149)(-31,-22) % \thicklines % % Horizontal Axis % \put(-10, 0){\vector(1, 0){155}} % \put(147, 0){\makebox(0, 0)[l]% % {$\var\left(g^p-c^p\right)$}} % % Vertical Axis % \put(24, -15){\vector(0, 1){125}} % \put(24, 111){\makebox(0, 0)[b]% % {$\overline{g^p} - \overline{c^p}$}} % % Reward-to-Variance Line % \thinlines % \put(-10, 50){\line(1,0){155}} % \put(100, 50){\circle*{10}} % % Maximal Reward-to-Variance (Slope) % \thinlines % % Threshold Value % %\thicklines % %\put(20,84.5){\line(1,0){8}} % \thinlines % %\put(17,84.5){\scalebox{0.75}{% % %\makebox(0,0)[rb]{$\frac{G^T}{N^p}$}}} % % The curve % \thinlines % \qbezier(35,35)(35,60)(51,60) % \qbezier(51,60)(80,60)(100,50) % \qbezier(100,50)(120,40)(120,33) % \qbezier(120,33)(120,10)(29,0) % % Max and mins % %\thicklines % %\put(20,60){\line(1,0){8}} % \thinlines % %\put(19,61){\makebox(0, 0)[rb]% % %{$\left(\overline{g^p}% % %-\overline{c^p}\right)^*$}} % \put(145,90)% % {\makebox(0,0)[r] % {\shortstack{Optimal Anywhere\\for\\% % $\frac{|c^s|}{\lambda^p} \to % \infty$}}} % \put(96.5,25){\makebox(0,0)[rb]% % {$\eta^* = 0$}} % \end{picture}} % } % End of \put\scalebox for upper figure % % % \put(0,0){\scalebox{.5}{ % \begin{picture}(241,149)(-31,-22) % \thicklines % % Horizontal Axis % \put(-10, 0){\vector(1, 0){155}} % \put(147, 0){\makebox(0, 0)[l]% % {$\var\left(g^p-c^p\right)$}} % % Vertical Axis % \put(24, -15){\vector(0, 1){125}} % \put(24, 111){\makebox(0, 0)[b]% % {$\overline{g^p} - \overline{c^p}$}} % % Reward-to-Variance Line % \thinlines % \put(120, 0){\line(0,1){110}} % \put(126,52.5){\rotatebox{90}{\makebox(0,0)[t]% % {\shortstack% % {Variance maximizer\\for\\$G^T \to \infty$}}}} % \put(22,52.5){\rotatebox{90}{\makebox(0,0)[b]% % {\shortstack% % {Variance minimizer\\for\\$G^T \to -\infty$}}}} % \put(120, 33){\circle*{10}} % \put(24, 0){\circle*{10}} % \put(72,80){\makebox(0,0)% % {$\lim\limits_{G^T\to\pm\infty} \eta^*% % =\mp \infty$}} % % Threshold Value % \thinlines % % Curve, starting from upper left and moving CW % \thinlines % \qbezier(35,35)(35,60)(51,60) % \qbezier(51,60)(80,60)(100,50) % \qbezier(100,50)(120,40)(120,33) % \qbezier(120,33)(120,10)(29,0) % % Max and mins % \thicklines % \put(120,-4){\line(0,1){8}} % \thinlines % \put(120,-5){\makebox(0, 0)[t]% % {$\left(\var\left(g^p-c^p\right)\right)^*$}} % \end{picture}} % }% End of \put\scalebox for lower figure % \end{picture} % \label{fig:risk_sensitivity_variance_cycle_limit_case} % } \caption[Visualization of Reward-to-Variance Maximization]{Reward-to-variance maximization. Curves show \aimention{Vilfredo Pareto}Pareto frontiers for maximizing net gain mean and minimizing net gain variance. $G^T$ denotes the net gain survival threshold. $\eta^*$ denotes the maximal reward-to-variance ratio, and the solutions that achieve this ratio are shown as large dots at points of tangency with a line of slope $\eta^*$. The dotted parabolic arrow in \subref{fig:risk_sensitivity_variance_cycle_visualization} shows how the figure changes as $|c^s|/\lambda^p$ increases. It is not necessary that $|\mu^*| < \infty$ and $(\sigma^2)^* < \infty$.} \label{fig:risk_sensitivity_variance} \end{figure} % The curve represents the \aimention{Vilfredo Pareto}Pareto frontiers for maximizing expected net gain and minimizing or maximizing net gain variance. Notice that the shape of the the curve shown in \longref{fig:risk_sensitivity_variance_lifetime} is identical to the shape of the curve in \longref{fig:risk_sensitivity_variance_cycle}, which relates to the analytical tractability gained by using the reward-to-variance ratio over the reward-to-variability ratio. Observations made about the reward-to-variability ratio also hold for the reward-to-variance ratio, and so we omit a discussion of them here. \index{examples!graphical solutions!reward-to-variance ratio|)indexdef}% \index{reward-to-variance ratio!graphical optimization|)}% \index{reward-to-variance ratio|)}% \subsection{Generalized Pareto Tradeoffs}% \label{sec:alternate_pareto_tradeoffs}\aimention{Vilfredo Pareto}% \index{Pareto tradeoffs|(} As discussed, maximization of a rate has the ability to choose solutions that are \aimention{Vilfredo Pareto}Pareto efficient with respect to maximization of one objective and maximization or minimization of another objective, conditioned on the sign of the first objective. The particular \aimention{Vilfredo Pareto}Pareto efficient solution chosen depends on the shapes of the optimization functions. In other words, the relative importance of maximizing one objective and minimizing another objective is allowed to vary. Therefore, maximization of a rate is equivalent to picking a particular relative importance. In some cases, it may be useful to let the relative importance of maximization of one objective and minimization of a different objective be a parameter of the environment. For example, for objectives $A$ and $B$ and weight $w \in \R$, the maximization of $A - w B$ produces a \aimention{Vilfredo Pareto}Pareto efficient solution to maximization of $A$ and minimization of $B$, given that $w$ is the relative importance of minimizing $B$ over maximizing $A$. If $w < 0$, then $|w|$ represents the relative importance of \emph{maximizing} $B$ over maximizing $A$. Maximization of $A - w B$ is sufficient but not necessary for a solution to be \aimention{Vilfredo Pareto}Pareto efficient with respect to objectives $A$ and $B$; therefore, unless $A$ and $B$ are convex functions, there may exist additional \aimention{Vilfredo Pareto}Pareto efficient solutions that are not accessible by this method. Either way, collecting solutions to this maximization problem for all $w \in \R$ is one way of generating a \aimention{Vilfredo Pareto}Pareto efficient set. Therefore, here we recast the tradeoffs from \longref{sec:alternate_rate_tradeoffs} as optimization problems of relative importance. Keeping in mind that maximization of $A - w B$ is a sufficient but not necessary condition for \aimention{Vilfredo Pareto}Pareto efficient solutions, this process can be viewed as picking a particular point on the curves shown in \longrefs{fig:rate_maximization}--% \shortref{fig:risk_sensitivity_variance}. \subsubsection{Gain and Time} \index{Pareto tradeoffs!gain and time|(} Take some weight $w \in \R$ that represents the relative importance of decreasing total time over increasing net gain (\ie, if $w < 0$, then $|w|$ is the relative importance of \emph{increasing} total time over increasing total net gain). Solutions that maximize% \index{Pareto tradeoffs!gain and time!gain discounted by time|(indexdef}% % \begin{equation} \E( G_1 ) - w \E( T_1 ) \label{eq:payoff_pareto_gain_and_time} \end{equation} % \index{Pareto tradeoffs!gain and time!gain discounted by time|)indexdef}% will be \aimention{Vilfredo Pareto}Pareto efficient with respect to these two objectives. If all of the solutions for all $w \in \R$ are collected, the result will be a curve like the one shown in \longref{fig:rate_maximization}. This is analogous to optimizing $\E( \oft{G}_1 ) - w \E( \oft{T}_1 )$ in a classical \ac{OFT} context. \Citet{Schoener71} considered net gain maximization and time minimization as two extremes on a continuum of behaviors that would maximize reproductive success in a certain environment. If $w \in \R_{\geq0}$, then this problem captures this idea, where $w$ picks a particular \aimention{Vilfredo Pareto}Pareto efficient solution for a certain environment. As discussed by \citet{HouMc99}, when $w$ is set to the maximum long-term rate of gain (\ie, the maximum value of $\E(G_1)/\E(T_1)$), the behavior that maximizes \longref{eq:payoff_pareto_gain_and_time} will also be the behavior that maximizes the long-term rate of gain. \index{Pareto tradeoffs!gain and time|)} \subsubsection{Efficiency}% \index{Pareto tradeoffs!efficiency|(} Take some weight $w \in \R$ that represents the relative importance of decreasing total cost over increasing \emph{gross} gain (\ie, if $w < 0$, then $|w|$ is the relative importance of \emph{increasing} total cost over increasing total net gain). Solutions that maximize% \index{Pareto tradeoffs!efficiency!gain discounted by cost|(indexdef}% % \begin{equation} \E( G_1 + C_1 ) - w \E(C_1) \quad \text{or, equivalently,} \quad \E(G_1) + (1-w) \E(C_1) \label{eq:payoff_pareto_efficiency} \end{equation} % \index{Pareto tradeoffs!efficiency!gain discounted by cost|)indexdef}% will be \aimention{Vilfredo Pareto}Pareto efficient with respect to these two objectives. If all of the solutions for all $w \in \R$ are collected, the result will be a curve like the one shown in \longref{fig:efficiency_maximization}. \index{Pareto tradeoffs!efficiency|)} \subsubsection{Risk-Sensitivity: Mean and Standard Deviation}% \index{Pareto tradeoffs!mean and standard deviation|(}% \index{risk sensitivity|(} Take some weight $w \in \R$ that represents the relative importance of decreasing net gain standard deviation\footnote{Again, for simplicity, we assume \longrefs{eq:cycle_proc_gross_gain_2moment}--% \shortref{eq:cycle_net_gain_2moment}.} over increasing expected net gain (\ie, if $w < 0$, then $|w|$ is the relative importance of increasing net gain standard deviation over increasing expected net gain). Solutions that maximize% \index{Pareto tradeoffs!mean and standard deviation!mean discounted by standard deviation|(indexdef}% % \begin{equation} \E(G^{N^p}) - w \std(G^{N^p}) \quad \text{ or, equivalently, } \quad \E(G_1) - \frac{w}{\sqrt{N^p}} \std(G_1) \label{eq:payoff_pareto_stddev} \end{equation} % \index{Pareto tradeoffs!mean and standard deviation!mean discounted by standard deviation|)indexdef}% will be \aimention{Vilfredo Pareto}Pareto efficient with respect to these two objectives. If all of the solutions for all $w \in \R$ are collected, the result will be a curve like the one shown in \longref{fig:risk_sensitivity_sharpe}. \index{Pareto tradeoffs!mean and standard deviation|)} \subsubsection{Risk-Sensitivity: Mean, Variance, and Expected Utility} \index{Pareto tradeoffs!mean and variance|(} The analytical tractability of \longref{eq:payoff_pareto_stddev} can be low. However, substituting variance for standard deviation achieves a similar goal while allowing for a simpler analysis. That is, take some weight $w \in \R$ that represents the relative importance of decreasing net gain variance over increasing expected net gain (\ie, if $w < 0$, then $|w|$ is the relative importance of increasing net gain variance over increasing expected net gain). Solutions that maximize% \index{Pareto tradeoffs!mean and variance!mean discounted by variance|(indexdef}% % \begin{equation} \E(G^{N^p}) - w \var(G^{N^p}) \quad \text{ or, equivalently, } \quad \E(G_1) - w \var(G_1) \label{eq:payoff_pareto_variance} \end{equation} % \index{Pareto tradeoffs!mean and variance!mean discounted by variance|)indexdef}% will be \aimention{Vilfredo Pareto}Pareto efficient with respect to these two objectives. If all of the solutions for all $w \in \R$ are collected, the result will be a curve like the one shown in \longref{fig:risk_sensitivity_variance}. \index{utility theory|(}This approach was suggested by \citet{Real80} as a way of applying the mean-variance methods of \citet{Markowitz52} and \citet{Tobin58} to biology. \Citeauthor{Real80} calls this \emph{variance discounting}. The parameter $w$ reflects a shape parameter of the agent's utility function\footnote{A \emph{utility function} quantifies the preferences of an agent. For example, a utility function shaped one way may indicate that an agent is risk prone whereas a utility function shaped another way may indicate that an agent is risk avoiding. Economists conventionally assumed that agents make decisions that maximize future expected utility. The foundations of modern utility theory are due to \citet{VNM44}; these results are summarized by \citet{Marschak46}. Utility theory will be discussed further in \longref{sec:stochastic_dominance}.}. \Citeauthor{Real80} uses an approximation of a general utility function to argue that maximizing \longref{eq:payoff_pareto_variance} leads to maximal expected utility. \Citet{SK86} make this argument exact by using an approach of \citet{Caraco80} and some assumptions about the shape of the utility function and the distribution of returns. In either case, there must be some justification for the choice of $w$; information needs to be known about the shape of the agent's utility function. In an engineering context, this means that $w$ is a parameter of a utility function that captures the design preferences. The optimization of \longref{eq:payoff_pareto_variance} should on average lead to the most preferable outcome with respect to this utility function\footnote{The ratio-based risk-sensitivity methods in \longref{sec:alternate_rate_tradeoffs} may be viewed as choosing an agent's utility function and therefore its preferences based on its environment. Here, the preferences are a design variable.}.\index{utility theory|)}% \index{Pareto tradeoffs!mean and variance|)} \index{risk sensitivity|)}% \index{Pareto tradeoffs|)} \subsection{Constraints}% \index{constraints|see{optimization constraints}}% \index{optimization constraints|(} In \longrefs{sec:alternate_rate_tradeoffs} and \shortref{sec:alternate_pareto_tradeoffs}, multiple optimization objectives are traded off in a way to combine elements of both. In order to introduce threshold effects, the optimization problems are ultimately formulated in terms of \emph{excess}. These thresholds could be made more strict by acting on the means of the distributions rather than on the outcomes. Thus, for some threshold $\hat{X}$, instead of maximizing $X - \hat{X}$, we can maximize $X$ subject to the constraint that $\E(X) \geq \hat{X}$. These type of constraints can be viewed as drawing horizontal and vertical boundaries on the \aimention{Vilfredo Pareto}Pareto frontiers in \longrefs{fig:rate_maximization}--% \shortref{fig:risk_sensitivity_variance}. The optimal solution remains \aimention{Vilfredo Pareto}Pareto efficient with respect to all objectives in question; however, it will be optimal with respect to one particular objective when defined over the constrained domain. Unfortunately, analytically defining this constrained domain may be challenging. \subsubsection{Gain and Time} \index{optimization constraints!gain and time|(} Let $T$ be some threshold of time. Consider the objective % \begin{equation*} \text{maximize } \E(G^{N^p}) \quad \text{ subject to } \quad \E(T^{N^p}) \leq T \end{equation*} % This is identical to % \begin{equation*} \text{maximize } \E(G_1) \quad \text{ subject to } \quad \E(T_1) \leq \frac{T}{N^p} \end{equation*} % This is equivalent to drawing a vertical line at $\E(T_1)=T/N^p$ on \longref{fig:rate_maximization_cycle} and choosing the point on the \aimention{Vilfredo Pareto}Pareto efficient curve to the left of that line that has the highest $\E(G_1)$. Notice that as $N^p$ increases, the feasible set of behaviors decreases, which eventually drives down the maximum possible net gain. Similarly, let $G^T$ be some threshold of net gain. Consider the objective % \begin{equation*} \text{minimize } \E(T^{N^p}) \quad \text{ subject to } \quad \E(G^{N^p}) \geq G^T \end{equation*} % This is identical to % \begin{equation*} \text{minimize } \E(T_1) \quad \text{ subject to } \quad \E(G_1) \geq \frac{G^T}{N^p} \end{equation*} % This is equivalent to drawing a horizontal line at $\E(G_1)=G^T/N^p$ on \longref{fig:rate_maximization_cycle} and choosing the point on the \aimention{Vilfredo Pareto}Pareto efficient curve above that line that has the lowest $\E(T_1)$. Notice that as $N^p$ increases, the feasible set of behaviors increases, which eventually deactivates the constraint\footnote{Borrowing language from \aimention{Joseph-Louis Lagrange}Lagrange multiplier methods \citep{Bertsekas95}, a constraint is not \emph{active} when it is satisfied with strict inequality. Constraints form boundaries around solutions; when a solution falls on a boundary, the constraint is said to be active, which implies that the optimal solution would most likely fall outside of the boundary if it were removed.}. \index{optimization constraints!gain and time|)} \subsubsection{Efficiency} \index{optimization constraints!gain and cost|(} Let $C^T$ be some cost threshold. Consider the objective % \begin{equation*} \text{maximize } \E(G^{N^p})+\E(C^{N^p}) \quad \text{ subject to } \quad \E(C^{N^p}) \leq C^T \end{equation*} % This is identical to % \begin{equation*} \text{maximize } \overline{g^p} \quad \text{ subject to } \quad \E(C_1) \leq \frac{C^T}{N^p} \end{equation*} % This is equivalent to drawing a vertical line at $\E(C_1)=C^T/N^p$ on \longref{fig:efficiency_maximization_cycle} and choosing the point on the \aimention{Vilfredo Pareto}Pareto efficient curve to the left of that line that has the highest $\overline{g^p}$. Notice that as $N^p$ increases, the feasible set of behaviors decreases, which eventually drives down the maximum possible gross gain. Similarly, let $G_g^T$ be some threshold of \emph{gross} gain. Consider the objective % \begin{equation*} \text{minimize } \E(C^{N^p}) \quad \text{ subject to } \quad \E(G^{N^p})+\E(C^{N^p}) \geq G_g^T \end{equation*} % This is identical to % \begin{equation*} \text{minimize } \E(C_1) \quad \text{ subject to } \quad \overline{g^p} \geq \frac{G_g^T}{N^p} \end{equation*} % This is equivalent to drawing a horizontal line at $\overline{g^p}=G_g^T/N^p$ on \longref{fig:efficiency_maximization_cycle} and choosing the point on the \aimention{Vilfredo Pareto}Pareto efficient curve above that line that has the lowest $\E(C_1)$. Notice that as $N^p$ increases, the feasible set of behaviors increases, which eventually deactivates the constraint. \index{optimization constraints!gain and cost|)} \subsubsection{Risk-Sensitivity: Mean, Standard Deviation, and Variance}% \index{optimization constraints!mean and variance|(} Let $\hat{\sigma}^2$ be some threshold of net gain variance. Consider the objective % \begin{equation*} \text{maximize } \E(G^{N^p}) \quad \text{ subject to } \quad \var(G^{N^p}) \leq \hat{\sigma}^2 \end{equation*} % This is identical to % \begin{equation*} \text{maximize } \E(G_1) \quad \text{ subject to } \quad \var(G_1) \leq \frac{\hat{\sigma}^2}{N^p} \end{equation*} % This is equivalent to drawing a vertical line at $\var(T_1)=\hat{\sigma}^2/N^p$ on \longref{fig:risk_sensitivity_variance_cycle} and choosing the point on the \aimention{Vilfredo Pareto}Pareto efficient curve to the left of that line that has the highest $\E(G_1)$. Notice that as $N^p$ increases, the feasible set of behaviors decreases, which eventually drives down the maximum possible net gain. Similarly, let $G^T$ be some threshold of net gain. Consider the objective % \begin{equation*} \text{minimize } \var(G^{N^p}) \quad \text{ subject to } \quad \E(G^{N^p}) \geq G^T \end{equation*} % This is identical to % \begin{equation*} \text{minimize } \var(G_1) \quad \text{ subject to } \quad \E(G_1) \geq \frac{G^T}{N^p} \end{equation*} % This is equivalent to drawing a horizontal line at $\E(G_1)=G^T/N^p$ on \longref{fig:risk_sensitivity_variance_cycle} and choosing the point on the \aimention{Vilfredo Pareto}Pareto efficient curve above that line that has the lowest $\var(G_1)$. Notice that as $N^p$ increases, the feasible set of behaviors increases, which eventually deactivates the constraint.% \index{optimization constraints!mean and variance|)} \index{optimization constraints!mean and standard deviation|(}% \paragraph{Standard Deviation:} The optimization objectives here could be rewritten using standard deviation. That is, variance could be replaced with standard deviation and any constraints on variance could be replaced with its square root. The resulting optimization objective leads to the same solutions as the original objective. Therefore, we omit a special standard deviation case. Often, the use of variance will improve analytical tractability of these problems. \index{optimization constraints!mean and standard deviation|)}% \index{solitary agent model!processing-only analysis!optimization|)} \index{optimization constraints|)} \section{Future Directions Inspired by PMPT}% \label{sec:future_econ_directions}% \index{future directions}% \index{finance!post-modern portfolio theory (PMPT)|(} \index{MVA|see{mean-variance analysis}}As with classical \ac{OFT}, results here are influenced primarily by \ac{MPT}, which describes a set of methods for portfolio investment and capital budgeting that follows from the work of \citet{Markowitz52,Markowitz59} and \citet{Tobin58} and applications of that work by \citet{Lintner65}, \citet{Mossin66}, and \citet{Sharpe64}. \ac{MPT} is mostly concerned with \acro[\index{mean-variance analysis (MVA)}mean-variance analysis (MVA)]{MVA}{\index{mean-variance analysis (MVA)"|indexglo}mean-variance analysis}. Proponents of \ac{MPT} openly acknowledge that \ac{MVA} is often too naive \citep{Markowitz59,Sharpe64}; however, its use is justified because it is well-understood and has low computational demands. However, modern technological and theoretical advances may render these justifications invalid \citep{RF94}. Thus, \ac{PMPT} seeks to improve upon \ac{MPT} by using more advanced analytical approaches. We discuss two \ac{PMPT} topics here that may have potential applications in biology and engineering in the analysis and design of agents. \subsection{Lower Partial Moments} \index{stochasticity!statistics!semivariance|see{stochasticity, statistics, lower-partial variance}}\index{semivariance|see{stochasticity, statistics, lower-partial variance}}\index{LPV|see{stochasticity, statistics, lower-partial variance}}\index{lower-partial variance|see{stochasticity, statistics, lower-partial variance}}\index{LPM|see{stochasticity, statistics, lower-partial moment}}\index{lower-partial moment|see{stochasticity, statistics, lower-partial moment}}We have used the well-known constructs of standard deviation and variance to quantify the variability of a return. However, for a given gain threshold, minimization of the variation \emph{below} that threshold is much more important than the variation above the threshold. That is, rather than seeking certainty in future gains, it may be more useful to minimize the uncertainty in negative gain budgets. Therefore, \citet{Markowitz59} introduces the \emph{semivariance} which \citet{Bawa75} calls the \emph{\acro[\index{stochasticity!statistics!lower-partial variance (LPV)|(indexdef}lower-partial variance~(LPV)]{LPV}{\index{stochasticity"!statistics"!lower-partial variance (LPV)"|indexglo}lower partial variance}}. The \ac{LPV} is the expected value of the squared negative deviations of possible outcomes from some point of reference. That is, for a distribution $F$ and a reference point $t$, the \ac{LPV} is defined by % \begin{equation*} \LPV(t;F) \triangleq \int_{x}^t (t-x)^2 \total F(x) \end{equation*} % where the \index{mathematics!functions!integral ($\int$)}integral is the \aimention{Henri L. Lebesgue}Lebesgue integral.\index{stochasticity!statistics!lower-partial variance (LPV)|)indexdef} \Citet{BL77} generalize this idea with the notion of an $n$\th{}\ order \emph{\acro[\index{stochasticity!statistics!lower-partial moment (LPM)|(indexdef}lower-partial moment~(LPM)]{LPM}{\index{stochasticity"!statistics"!lower-partial moment (LPM)"|indexglo}lower-partial moment}} defined by % \begin{equation*} \LPM_n(t;F) \triangleq \int_{x}^t (t-x)^n \total F(x) \end{equation*} % \index{stochasticity!statistics!lower-partial moment (LPM)|)indexdef}Clearly, $\LPV(t;F) = \LPM_2(t;F)$ for all $t \in \R$. These \emph{lower-partial} moments\footnote{\Citet{Bawa78} shows that for all $n \in \N$ and $t \in \R$, $\LPM_n(t;F)$ is a convex function of $F$.} are generalized asymmetric notions of the familiar central moments (\eg, variance and skewness). Rather than integrating (\ie, summing) all variations around the mean of a distribution, the \ac{LPM} integrates all variations that fall beneath some arbitrary reference which might be viewed as a target benchmark (\eg, a net gain threshold). This is a type of \emph{downside risk} measure \citep{GH99,RF94}. \index{MLPM|see{mean-lower-partial-moment analysis}}\index{MSA|see{mean-lower-partial-moment analysis}}\index{mean-semivariance analysis|see{mean-lower-partial-moment analysis}}\index{mean-lower-partial-moment (MLPM) analysis|(indexdef}The $n$\th{}\ order \ac{LPM} can be substituted for variance in \ac{MVA} to yield a new method of analysis that trades off greater expected returns and shortfall uncertainty without putting any pressure on variances above return benchmarks. This is called \emph{\acro{MLPM}{mean-lower-partial-moment}} analysis \citep{Bawa78,LR88}. \index{mean-lower-partial-variance analysis|(indexdef}\index{mean-semivariance analysis|see{mean-lower-partial-variance analysis}}When $n=2$, this is called \emph{\acro{MLPV}{mean-lower-partial-variance}} analysis \citep{BL77} or \emph{\acro{MSA}{mean-semivariance analysis}} \citep{Mao70}. When \ac{MLPV} analysis is applied to normal distributions, the results are identical to \ac{MVA}\footnote{This is due to the symmetry of normal distributions.}.\index{mean-lower-partial-variance analysis|)indexdef} In fact, \ac{MLPM} analysis will always have results that ``do at least as well'' as \ac{MVA} \citep{BL77}. Thus, \citet{Mao70} claims that ``the time has come to shift'' from \ac{MVA} to \ac{MSA} in capital budgeting. We believe that optimization based on these new downside risk frameworks may also lead to advances in behavioral analysis and solitary agent design.\index{mean-lower-partial-moment (MLPM) analysis|)indexdef} \subsection{Stochastic Dominance} \label{sec:stochastic_dominance}% \index{finance!stochastic dominance (SD)|(indexdef} \index{SD|see{finance, stochastic dominance}}\index{stochastic dominance|see{finance, stochastic dominance}}\index{stochasticity!random variable|(}\index{stochasticity!distributions|(}\index{investment returns|see{finance, investment returns}}\index{finance!investment returns}Investment returns (\eg, net gain) are naturally random variables. Thus, \index{portfolios|see{finance, portfolios}}\index{finance!portfolios}portfolios or \index{capital budgeting|see{finance, capital budgets}}\index{finance!capital budgets}budgets (\eg, processing probability and time choices) correspond to random variables with different probability distributions. Portfolio selection is thus a decision problem of choosing the most preferable probability distribution. \index{maximal utility|see{utility theory}}\index{maximal expected utility|see{utility theory}}\index{utilitarianism|see{utility theory}}\index{expected utility|see{utility theory}}\index{von Neumann-Morgenstern utility|see{utility theory}}\index{utility theory|(}\Citet{VNM44} characterize preferences in terms of utility functions, which represent the effective value of certain returns to an agent. Different-shaped utility functions correspond to different preferences, and rational agents\footnote{\index{rational agent|see{utility theory}}\Citet{VNM44} define what \emph{rational agent} means with a number of axioms about preference.} will maximize their expected utility. That is, if $F$ and $G$ are two distributions of returns and $u$ is an agent's utility function, the distribution $F$ is preferred over the distribution $G$ if % \begin{equation} \int_{-\infty}^\infty u(x) \total F(x) - \int_{-\infty}^\infty u(x) \total G(x) > 0 \label{eq:maximal_exp_utility} \end{equation} % This rule can be used to pick the best distribution(s) from a set of distributions that correspond to portfolio choices.\index{stochasticity!distributions|)}\index{stochasticity!random variable|)} While each of these probability distributions may be known or at least knowable, it may not be possible to completely describe an agent's \index{utility function|see{utility theory}}utility function. Therefore, it is desirable to define rules like \longref{eq:maximal_exp_utility} that are a function of \emph{only} the probability distributions. Distributions that are preferable with respect to these rules will return maximal utility for \emph{all} utility functions that share certain very general characteristics. These are known as \emph{\acro[stochastic dominance~(SD)]{SD}{\index{finance"!stochastic dominance (SD)"|indexglo}stochastic dominance}} rules. \index{finance!stochastic dominance (SD)!first-order (FSD)|(indexdef}For example, as shown by \citet{Bawa75}, for any two distributions $F$ and $G$, $F$ is preferred (\ie, results in a greater expected utility) to $G$ for any increasing and continuously differentiable utility function $u(x)$ assumed to have finite values for finite values of $x$ \emph{if and only if} there exists some $x_0 \in \R$ such that % \begin{equation} F(x) \leq G(x) \text{ for all } x \in \R \quad \text{ and } \quad F(x_0) < G(x_0) \label{eq:fsd} \end{equation} % \longref{eq:fsd} is called \emph{\acro[first-order stochastic dominance~(FSD)]{FSD}{\index{finance"!stochastic dominance (SD)"!first-order (FSD)"|indexglo}first-order stochastic dominance}}. \ac{FSD} implies that for any return benchmark, the probability of failing to meet that benchmark will be less with the $F$ distribution\footnote{Of course, \ac{FSD} provides no guarantee that a particular outcome from the $G$ distribution will not succeed when the outcome from the $F$ distribution will fail. Dominance for all outcomes is the strongest form of \ac{SD} (and is rarely used).}. This class of utility functions for \ac{FSD} is very broad.\index{finance!stochastic dominance (SD)!first-order (FSD)|)indexdef} There are higher-order \ac{SD} rules that apply to smaller sets of utility functions (\eg, risk averse utility functions). In particular, \citet{Bawa75} shows that \emph{\acro[\index{finance!stochastic dominance (SD)!third-order (TSD)}third-order stochastic dominance~(TSD)]{TSD}{\index{finance"!stochastic dominance (SD)"!third-order (TSD)"|indexglo}third-order stochastic dominance}} has an important relationship to \ac{MLPM} analysis. An extensive categorized bibliography of articles on \ac{SD} and its application is given by \citet{Bawa82}, who also briefly discusses the foundations of \ac{SD}. Biologists have recognized the validity of utility functions \citep[\eg,][]{Caraco80,Real80,SK86}. However, the methods of \ac{SD} have not been applied to compare alternative behaviors in terms of maximal expected utility in a convincing way. That is, \ac{MVA} is used to approximate an \ac{SD} analysis; however, there seems to be little recognition that this approximation is only valid for a limited set of utility functions. \Citet{Bawa75} shows that for a large set of utility functions, \ac{MVA} meets neither necessary nor sufficient conditions for optimality and \ac{MLPV} analysis should be used instead. As design preferences are often rational, engineering can also benefit from an \ac{SD} approach. If nothing else, \ac{SD} provides a deeper understanding of the consequences of behavioral choice by avoiding heuristic arguments and approximations. Additionally, using \ac{SD} promises to allow both engineers and behavioral ecologists to benefit not only from classical works in economics but also from the modern-day research in the field.\index{utility theory|)}% \index{finance!stochastic dominance (SD)|)indexdef}% \index{finance!post-modern portfolio theory (PMPT)|)}