Dynamic allocation and stochastic control

Definition Fix a horizon $T > 0$ and a probability space with the physical measure $\mathbb{P}$, a Brownian motion $W$ under $\mathbb{P}$ and the filtration $(\mathcal{F}_t)$ generated by $W$ and completed with the null sets. A cash account $B_t = e^{rt}$ earns a constant continuously compounded rate $r$, and a stock that pays no dividends follows

with constants $\mu$ and $\sigma > 0$. An investor keeps the fraction $\pi_t$ of current wealth $X_t$ in the stock and the rest in the cash account, trades without costs, may borrow or sell short at the rate $r$, and neither consumes nor adds money. The self-financing condition then gives the controlled wealth equation

A control on $[t, T]$ is a real-valued process $\pi$ that is progressively measurable with respect to $(\mathcal{F}_s)$, so that each $\pi_s$ uses only the information available at time $s$, and that satisfies $\int_t^T \pi_s^2\,ds < \infty$ almost surely. For such a control and a starting wealth $X_t = x > 0$ the wealth equation has the unique solution

which is strictly positive. Because the control is a fraction of current wealth, the positive-wealth constraint holds automatically, while a strategy stated in currency amounts would need it as an explicit condition. The admissible controls form a class $\mathcal{A}(t)$ that is part of the problem. It contains only controls for which the expected utility below is well defined, it may impose constraints such as a cap on $\pi$, and the verification card fixes the class used for power utility. The stochastic control problem is to maximize the expected utility of terminal wealth, and its value function is

where $U$ is an increasing utility function on $(0, \infty)$, $\mathbb{E}^{\mathbb{P}}_{t,x}$ is the expectation under $\mathbb{P}$ for wealth started at $X_t = x$, and $J(T, x) = U(x)$. The expectation uses $\mathbb{P}$, because the problem is about preferences over actual outcomes, not about the price of a claim.

Definition The Bellman principle, also called the dynamic programming principle, states that for $0 \le t < t + h \le T$ and $x > 0$

with the supremum over admissible controls on $[t, t + h]$. The problem from $(t, x)$ splits into a choice over the first interval followed by the best continuation from the state $(t + h, X_{t+h})$ that is reached, and an optimal control remains optimal, with probability one, for the remaining problem from the state it reaches. The split works because the objective is an expectation of a function of terminal wealth and because the pair $(t, X_t)$ summarizes everything about the past that matters for the future, here since the coefficients are constant. Under standard conditions for controlled diffusions the principle is a theorem, but its proof needs measurability arguments beyond this lesson. Here it serves to derive the Hamilton-Jacobi-Bellman (HJB) equation, and for the explicit solution of the Merton card the verification argument confirms it directly. A criterion that is not of this form, such as a mean-variance criterion containing $\operatorname{Var}(X_T)$, does not satisfy the principle as stated, and its best strategy chosen at time $0$ need not remain best at later dates.

Definition Suppose $J$ is continuous on $[0, T] \times (0, \infty)$ and has continuous partial derivatives $J_t$, $J_x$ and $J_{xx}$ on $[0, T) \times (0, \infty)$. For a fraction $\pi \in \mathbb{R}$ the controlled generator is

By Itô’s formula, the drift of $J(s, X_s)$ under a constant fraction $\pi$ is $J_t + \mathcal{L}^{\pi} J$ evaluated at $(s, X_s)$, where the second-order term comes from the quadratic variation $d\langle X \rangle_s = \pi^2 \sigma^2 X_s^2\,ds$. If the stochastic integral in Itô’s formula has mean zero, the Bellman principle over $[t, t + h]$, divided by $h$ with $h \downarrow 0$, leads formally to the Hamilton-Jacobi-Bellman equation

for $0 \le t < T$ and $x > 0$. The derivation is heuristic. It assumes the smoothness of $J$ and the Bellman principle, and it exchanges a limit with an expectation. When the value function is not smooth, the equation is interpreted in the weaker viscosity sense, which lies beyond this lesson. When $x > 0$, $\sigma > 0$, $J_x > 0$ and $J_{xx} < 0$, the expression $\mathcal{L}^{\pi} J$ is a strictly concave quadratic in $\pi$. Setting its derivative $x(\mu - r) J_x + \pi \sigma^2 x^2 J_{xx}$ to zero gives the unique maximizer

and substituting it turns the HJB equation into the nonlinear PDE

If $J_{xx} > 0$, or if $J_{xx} = 0$ while $(\mu - r) J_x \ne 0$, the supremum is $+\infty$ and no maximizer exists. If the fraction is restricted to an interval $[a, b]$, the supremum is taken over that interval, and strict concavity puts the maximizer at $\min(\max(\pi^{*}, a), b)$, the point of the interval nearest to $\pi^{*}$.

Definition For a relative risk aversion $\gamma > 0$, the CRRA (constant relative risk aversion) or power utility on $(0, \infty)$ is

Its derivatives are $U'(x) = x^{-\gamma} > 0$ and $U''(x) = -\gamma x^{-\gamma - 1} < 0$, so $U$ is strictly increasing and strictly concave, and the coefficient of relative risk aversion is

at every wealth. For $\gamma \ne 1$ the utility is homogeneous of degree $1 - \gamma$, with $U(cx) = c^{1 - \gamma} U(x)$ for $c > 0$, and for log utility $\ln(cx) = \ln c + \ln x$. For $\gamma > 1$ the utility is negative, and larger wealth still gives a larger, less negative value. The log case is the limit of the shifted utility $(x^{1 - \gamma} - 1)/(1 - \gamma)$ as $\gamma \to 1$, and the shift is needed because the constant $1/(1 - \gamma)$ that separates it from $U$ diverges. Adding a constant to a utility or multiplying it by a positive constant changes no decision.

Definition In the market of the stochastic control card, with $\sigma > 0$ and CRRA utility with $\gamma > 0$ and $\gamma \ne 1$, look for a solution $J(t, x) = A(t)\,U(x)$ of the HJB equation with $A(T) = 1$ and $A > 0$. Then $J_x = A x^{-\gamma} > 0$ and $J_{xx} = -\gamma A x^{-\gamma - 1} < 0$, and

The factor $A(t)\, x^{1 - \gamma}$ is positive, so the supremum over $\pi$ is attained where $g$ is largest, at the Merton allocation

which agrees with the general formula of the HJB card, since $-J_x/(x J_{xx}) = 1/\gamma$. The largest value of $g$ is

The HJB equation becomes $A'(t)\, x^{1 - \gamma}/(1 - \gamma) + A(t)\, b\, x^{1 - \gamma} = 0$. Dividing by $x^{1 - \gamma} > 0$ and multiplying by the nonzero number $1 - \gamma$, which is negative when $\gamma > 1$, gives $A' + (1 - \gamma)\, b\, A = 0$ with $A(T) = 1$. Hence $A(t) = e^{(1 - \gamma) b (T - t)} > 0$ and

For $\gamma = 1$ the same steps with $J(t, x) = \ln x + a(t)$ give $\pi^{*} = (\mu - r)/\sigma^2$, the fraction that maximizes the expected log growth rate and the continuous-time counterpart of the fraction in Expected utility and Kelly growth, and $J(t, x) = \ln x + b\,(T - t)$ with $b = r + (\mu - r)^2/(2\sigma^2)$. In every case the certainty equivalent of $J(t, x)$, the sure terminal wealth with that utility, is $x\, e^{b (T - t)}$, so $b$ is the certainty-equivalent growth rate. The verification card proves that $\pi^{*}$ is optimal within a stated class of admissible controls. The fraction is constant because $r$, $\mu$ and $\sigma$ are constant and relative risk aversion is constant. The fraction exceeds $1$ when $\mu - r > \gamma \sigma^2$, and the investor then borrows. It is negative, a short position, when $\mu < r$, and it is $0$ when $\mu = r$. With random investment opportunities the value function depends on further state variables and the optimal fraction acquires hedging terms, and consumption or transaction costs change the problem. These extensions are beyond this lesson.

Definition Let $v$ be a function on $[0, T] \times (0, \infty)$ with continuous partial derivatives $v_t$, $v_x$ and $v_{xx}$ that satisfies the HJB equation on $[0, T) \times (0, \infty)$ with $v(T, x) = U(x)$. For an admissible control $\pi$ and $X_t = x$, Itô’s formula gives

where

The HJB equation makes the integrand of the $ds$ integral at most zero for every value of $\pi_s$. First, if $\mathbb{E}[M_T] = 0$, for example because $M$ is a square-integrable martingale, then $\mathbb{E}^{\mathbb{P}}_{t,x}[U(X_T)] \le v(t, x)$. Second, if a control $\hat\pi$ attains the supremum in the HJB equation at $(s, X_s)$ for almost every $s$, its $ds$ integral vanishes, and if also $\mathbb{E}[M_T] = 0$ for it, then $\mathbb{E}^{\mathbb{P}}_{t,x}[U(X^{\hat\pi}_T)] = v(t, x)$. On a class of admissible controls for which the first step applies and that contains $\hat\pi$, this proves $v = J$ and the optimality of $\hat\pi$. For a general control $M$ is only a local martingale. One then stops it at times $\tau_n$ increasing to $T$, obtains $\mathbb{E}[v(\tau_n, X_{\tau_n})] \le v(t, x)$, and needs uniform integrability of the stopped values to pass to the limit, so these integrability conditions are part of the optimality statement. For the Merton candidate the integrand of $M$ is $A(s)\, \sigma \pi_s X_s^{1 - \gamma}$, and the lesson takes as admissible the controls with

for which $M$ is a square-integrable martingale. This class contains every bounded progressively measurable fraction, because such fractions give wealth with finite moments of every order. Within it the Merton fraction is optimal and the candidate is the value function. For $0 < \gamma < 1$ the candidate is nonnegative, so $M$ is a local martingale bounded below by $-v(t, x)$, hence a supermartingale, and the upper bound extends to every control with $\int_t^T \pi_s^2\,ds < \infty$.