Project on Predictability and Data Assimilation

2006
Ioannou, P. J., & Farrell, B. F. (2006). Application of Generalized Stability Theory to Deterministic and Statistical Prediction., Predictability of Weather and Climate. In T. Palmer & R. Hagedorn (Ed.), (pp. 181-216) . Cambridge University Press, Cambridge.Abstract
Understanding of the stability of deterministic and stochastic dynamical systems 
has evolved recently from a traditional grounding in the system’s normal modes 
to a more comprehensive foundation in the system’s propagator and especially in 
an appreciation of the role of non-normality of the dynamical operator in deter-
mining the system’s stability as revealed through the propagator. This set of ideas, 
which approach stability analysis from a non-modal perspective, will be referred 
to as generalised stability theory (GST). Some applications of GST to determinis-
tic and statistical forecast are discussed in this review. Perhaps the most familiar 
of these applications is identifying initial perturbations resulting in greatest error 
in deterministic error systems, which is in use for ensemble and targeting appli-
cations. But of increasing importance is elucidating the role of temporally dis-
tributed forcing along the forecast trajectory and obtaining a more comprehensive 
understanding of the prediction of statistical quantities beyond the horizon of deter-
ministic prediction. The optimal growth concept can be extended to address error 
growth in non-autonomous systems in which the fundamental mechanism produc-
ing error growth can be identified with the necessary non-normality of the sys-
tem. The influence of model error in both the forcing and the system is examined 
using the methods of stochastic dynamical systems theory. In this review determin-
istic and statistical prediction, i.e. forecast and climate prediction, are separately 
discussed.
.pdf
Farrell, B. F., & Ioannou, P. J. (2006). Approximating Optimal State Estimation., Predictability of Weather and Climate. In T. Palmer & R. Hagedorn (Ed.), (pp. 181-216) . Cambridge University Press, Cambridge.Abstract

Minimising forecast error requires accurately specifying the initial state from which 
the forecast is made by optimally using available observing resources to obtain the 
most accurate possible analysis. The Kalman filter accomplishes this for linear sys-
tems and experience shows that the extended Kalman filter also performs well in 
non-linear systems. Unfortunately, the Kalman filter and the extended Kalman filter 
require computation of the time dependent error covariance matrix which presents 
a daunting computational burden. However, the dynamically relevant dimension of 
the forecast error system is generally far smaller than the full state dimension of the 
forecast model which suggests the use of reduced order error models to obtain near 
optimal state estimators. A method is described and illustrated for implementing a 
Kalman filter on a reduced order approximation of the forecast error system. This 
reduced order system is obtained by balanced truncation of the Hankel operator repre-
sentation of the full error system. As an example application a reduced order Kalman 
filter is constructed for a time-dependent quasi-geostrophic storm track model. The 
accuracy of the state identification by the reduced order Kalman filter is assessed 
and comparison made with the state estimate obtained by the full Kalman filter and 
with the estimate obtained using an approximation to 4D-Var. The accuracy assess-
ment is facilitated by formulating the state estimation methods as observer systems. 
A practical approximation to the reduced order Kalman filter that utilises 4D-Var 
algorithms is examined

.pdf
2005
Farrell, B. F., & Ioannou, P. J. (2005). Distributed Forcing of Forecast and Assimilation Error Systems. J. Atmos. Sci. , 62, 460–475 . J. Atmos. Sci.Abstract
Temporally distributed deterministic and stochastic excitation of the tangent linear forecast system gov-
erning forecast error growth and the tangent linear observer system governing assimilation error growth is 
examined. The method used is to determine the optimal set of distributed deterministic and stochastic 
forcings of the forecast and observer systems over a chosen time interval. Distributed forcing of an unstable 
system addresses the effect of model error on forecast error in the presumably unstable forecast error 
system. Distributed forcing of a stable system addresses the effect on the assimilation of model error in the 
presumably stable data assimilation system viewed as a stable observer. In this study, model error refers 
both to extrinsic physical error forcing, such as that which arises from unresolved cumulus activity, and to 
intrinsic error sources arising from imperfections in the numerical model and in the physical parameter-
izations.
.pdf
2004
Farrell, B. F., & Ioannou, P. J. (2004). Sensitivity of Perturbation Variance and Fluxes in Turbulent Jets to Changes in the Mean Jet. J. Atmos , Sci. , 61, 2644–2652 . J. Atmos , Sci.Abstract

Synoptic-scale eddy variance and fluxes of heat and momentum in midlatitude jets are sensitive to small changes in mean jet velocity, dissipation, and static stability. In this work the change in the jet producing the greatest increase in variance or flux is determined. Remarkably, a single jet structure change completely characterizes the sensitivity of a chosen quadratic statistical quantity to modification of the mean jet in the sense that an arbitrary change in the jet influences a chosen statistical quantity in proportion to the projection of the change on this single optimal structure. The method used extends previous work in which storm track statistics were obtained using a stochastic model of jet turbulence.

.pdf
2002
Farrell, B. F., & Ioannou, P. J. (2002). Optimal Perturbation of Uncertain Systems. Stochastics and Dynamics , Vol.2 (No. 3), 395-402 . Stochastics and Dynamics.Abstract

In studies of perturbation dynamics in physical systems, certain specification of the gov-
erning perturbation dynamical system is generally lacking, either because the perturba-
tion system is imperfectly known or because its specification is intrinsically uncertain, 
while a statistical characterization of the perturbation dynamical system is often avail-
able. In this report exact and asymptotically valid equations are derived for the ensemble 
mean and moment dynamics of uncertain systems. These results are used to extend the 
concept of optimal deterministic perturbation of certain systems to uncertain systems. 
Remarkably, the optimal perturbation problem has a simple solution: In uncertain sys-
tems there is a sure initial condition producing the greatest expected second moment 
perturbation growth.

.pdf
Farrell, B. F., & Ioannou, P. J. (2002). Perturbation Growth and Structure in UncertainFlows: Part II. J. Atmos. Sci. , 59, 2647-2664 . J. Atmos. Sci.Abstract

Perturbation growth in uncertain systems associated with fluid flow is examined concentrating on deriving, solving, and interpreting equations governing the ensemble mean covariance. Covariance evolution equations are obtained for fluctuating operators and illustrative physical examples are solved. Stability boundaries are obtained constructively in terms of the amplitude and structure of operator fluctuation required for existence of bounded second-moment statistics in an uncertain system. The forced stable uncertain system is identified as a primary physical realization of second-moment dynamics by using an ergodic assumption to make the physical connection between ensemble statistics of stable stochastically excited systems and observations of time mean quantities. Optimal excitation analysis plays a central role in generalized stability theory and concepts of optimal deterministic and stochastic excitation of certain systems are extended in this work to uncertain systems. Re- markably, the optimal excitation problem has a simple solution in uncertain systems: there is a pure structure producing the greatest expected ensemble perturbation growth when this structure is used as an initial condition, and a pure structure that is most effective in exciting variance when this structure is used to stochastically force the system distributed in time.

Optimal excitation analysis leads to an interpretation of the EOF structure of the covariance both for the case of optimal initial excitation and for the optimal stochastic excitation distributed in time that maintains the statistically steady state. Concepts of pure and mixed states are introduced for interpreting covariances and these ideas are used to illustrate fundamental limitations on inverting covariances for structure in stochastic systems in the event that only the covariance is known.

.pdf
Farrell, B. F., & Ioannou, P. J. (2002). Perturbation Growth and Structure in UncertainFlows: Part I. J. Atmos. Sci. , 59, 2629-2646 . J. Atmos. Sci.Abstract

Perturbation growth in uncertain systems is examined and related to previous work in which linear stability 
concepts were generalized from a perspective based on the nonnormality of the underlying linear operator. In 
this previous work the linear operator, subject to an initial perturbation or a stochastic forcing distributed in 
time, was either fixed or time varying, but in either case the operator was certain. However, in forecast and 
climate studies, complete knowledge of the dynamical system being perturbed is generally lacking; nevertheless, 
it is often the case that statistical properties characterizing the variability of the dynamical system are known. 
In the present work generalized stability theory is extended to such uncertain systems. The limits in which 
fluctuations about the mean of the operator are correlated over time intervals, short and long, compared to the 
timescale of the mean operator are examined and compared with the physically important transitional case of 
operator fluctuation on timescales comparable to the timescales of the mean operator. Exact and asymptotically 
valid equations for transient ensemble mean and moment growth in uncertain systems are derived and solved. 
In addition, exact and asymptotically valid equations for the ensemble mean response of a stable uncertain 
system to deterministic forcing are derived and solved. The ensemble mean response of the forced stable uncertain 
system obtained from this analysis is interpreted under the ergodic assumption as equal to the time mean of the 
state of the uncertain system as recorded by an averaging instrument. Optimal perturbations are obtained for 
the ensemble mean of an uncertain system in the case of harmonic forcing. Finally, it is shown that the remarkable 
systematic increase in asymptotic growth rate with moment in uncertain systems occurs only in the context of 
the ensemble.

.pdf
2001
Farrell, B. F., & Ioannou, P. J. (2001). State estimation using a reduced order Kalman filter. J. Atmos. Sci. , 58, 3666-3680 . J. Atmos. Sci.Abstract

Minimizing forecast error requires accurately specifying the initial state from which the forecast is made by 
optimally using available observing resources to obtain the most accurate possible analysis. The Kalman filter 
accomplishes this for a wide class of linear systems, and experience shows that the extended Kalman filter also 
performs well in nonlinear systems. Unfortunately, the Kalman filter and the extended Kalman filter require 
computation of the time-dependent error covariance matrix, which presents a daunting computational burden. 
However, the dynamically relevant dimension of the forecast error system is generally far smaller than the full 
state dimension of the forecast model, which suggests the use of reduced-order error models to obtain near-
optimal state estimators. A method is described and illustrated for implementing a Kalman filter on a reduced-
order approximation of the forecast error system. This reduced-order system is obtained by balanced truncation 
of the Hankel operator representation of the full error system and is used to construct a reduced-order Kalman 
filter for the purpose of state identification in a time-dependent quasigeostrophic storm track model. The accuracy 
of the state identification by the reduced-order Kalman filter is assessed by comparison to the true state, to the 
state estimate obtained by the full Kalman filter, and to the state estimate obtained by direct insertion.

.pdf
Farrell, B. F., & Ioannou, P. J. (2001). Accurate Low Dimensional Approximation of the Linear Dynamics of Fluid Flow. J. Atmos. Sci. , 58, 2771-2789 . J. Atmos. Sci.Abstract

Methods for approximating a stable linear autonomous dynamical system of lower order are examined. Reducing the order of a dynamical system is useful theoretically in identifying the irreducible dimension of the dynamics and in isolating the dominant spatial structures supporting the dynamics, and practically in providing tractable lower-dimension statistical models for climate studies and error covariance models for forecast analysis and initialization. Optimal solution of the model order reduction problem requires simultaneous representation of both the growing structures in the system and the structures into which these evolve. For autonomous operators associated with fluid flows a nearly optimal solution of the model order reduction problem with prescribed error bounds is obtained by truncating the dynamics in its Hankel operator representation. Simple model examples including a reduced-order model of Couette flow are used to illustrate the theory. Practical methods for obtaining approximations to the optimal order reduction problem based on finite-time singular vector analysis of the propagator are discussed and the accuracy of the resulting reduced models evaluated.

.pdf
1990
Farrell, B. F. (1990). Small Error Dynamics and the Predictability of Atmospheric Flows. In (Vol. 47, pp. 2409–2416) . J. Atmos. Sci.Abstract

forecast reliability is known to be the highly variable and this variability can be traced in part to differences in the innate predictability of atmospheric flow regimes. These differences in turn have traditionally been ascribed to variation in the growth rate of exponential instabilities supported by the flow. More recently, drawing on modern dynamical systems theory, the asymptotic divergence of trajectories in phase space of the nonlinear equations of motion has been cited to explain the observed loss of predictability. In this report it is shown that increase in error on synoptic forecast time scales is controlled by rapidly growing perturbations that are not of normal mode form. It is further noted that unpredictable regimes. moreover, model problems illustrating baroclinic and barotropic dynamics suggest that asymptotic measures of divergence in phase space, while applicable in the limit of infinite time, may not be appropriate over time intervals addressed by present synoptic forecast.

.pdf