Exploring Inexact Theories in Solving Inverse Problems

Lecture 9 Inexact Theories

Syllabus Lecture 01 Lecture 02 Lecture 03 Lecture 04 Lecture 05 Lecture 06 Lecture 07 Lecture 08 Lecture 09 Lecture 10 Lecture 11 Lecture 12 Lecture 13 Lecture 14 Lecture 15 Lecture 16 Lecture 17 Lecture 18 Lecture 19 Lecture 20 Lecture 21 Lecture 22 Lecture 23 Lecture 24 Describing Inverse Problems Probability and Measurement Error, Part 1 Probability and Measurement Error, Part 2 The L2 Norm and Simple Least Squares A Priori Information and Weighted Least Squared Resolution and Generalized Inverses Backus-Gilbert Inverse and the Trade Off of Resolution and Variance The Principle of Maximum Likelihood Inexact Theories Nonuniqueness and Localized Averages Vector Spaces and Singular Value Decomposition Equality and Inequality Constraints L1 , L Norm Problems and Linear Programming Nonlinear Problems: Grid and Monte Carlo Searches Nonlinear Problems: Newton s Method Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals Factor Analysis Varimax Factors, Empirical Orthogonal Functions Backus-Gilbert Theory for Continuous Problems; Radon s Problem Linear Operators and Their Adjoints Fr chet Derivatives Exemplary Inverse Problems, incl. Filter Design Exemplary Inverse Problems, incl. Earthquake Location Exemplary Inverse Problems, incl. Vibrational Problems

Purpose of the Lecture Discuss how an inexact theory can be represented Solve the inexact, linear Gaussian inverse problem Use maximization of relative entropy as a guiding principle for solving inverse problems Introduce F-test as way to determine whether one solution is better than another

Part 1 How Inexact Theories can be Represented

How do we generalize the case of an exact theory to one that is inexact?

exact theory case model,m 0 0.5 1 theory 1.5 dobs 2 d dpre 2.5 3 datum,d 3.5 4 d=g(m) 4.5 5 0 1 2 map 3 4 5 mest m

to make theory inexact ... must make the theory probabilistic or fuzzy model,m 0 0.5 1 1.5 dobs 2 d dpre 2.5 3 datum,d 3.5 4 d=g(m) 4.5 5 0 1 2 map 3 4 5 mest m

a prior p.d.f. theory combination model,m model,m model,m 0 0 0 1 1 1 dobs dobs dobs 2 2 2 d dpre datum,d d d 3 3 3 datum,d datum,d 4 4 4 5 5 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 map map mest map m m m

how do you combine two probability density functions ?

how do you combine two probability density functions ? so that the information in them is combined ...

desirable properties order shouldn t matter combining something with the null distribution should leave it unchanged combination should be invariant under change of variables

Answer

a priori , pA theory, pg total, pT model,m model,m model,m 0 0 0 1 1 1 dobs 2 2 2 dobs dobs d dpre d d 3 3 3 datum,d datum,d datum,d 4 4 4 5 5 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 map map mest map m m m (D) (E) (F) model,m model,m model,m 0 0 0 1 1 1 dobs 2 2 2 dobs dobs d dpre d d 3 3 3 datum,d datum,d datum,d 4 4 4 5 5 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 map map mest map m m m

solution to inverse problem maximum likelihood point of (with pN constant) simultaneously gives m mest and d dpre

probability that the estimated model parameters are near m m and the predicted data are near d d T probability that the estimated model parameters are near m m irrespective of the value of the predicted data

conceptual problem and T do not necessarily have maximum likelihood points at the same value of m

model,m 0 0 1 dobs dpre datum, 1 2 2 3 3 4 4 5 0 1 2 3 4 5 5 d 0 1 2 map 3 4 5 mest 1 1 0.8 0.8 0.6 p(m) p(m) 0.6 0.4 p(m) 0.4 0.2 0.2 0 0 1 2 3 4 5 model,m mest 0 m 0 1 2 3 4 5 m

illustrates the problem in defining a definitive solution to an inverse problem

illustrates the problem in defining a definitive solution to an inverse problem fortunately if all distributions are Gaussian the two points are the same

Part 2 Solution of the inexact linear Gaussian inverse problem

Gaussian a priori information

Gaussian a priori information a priori values of model parameters their uncertainty

Gaussian observations

Gaussian observations observed data measurement error

Gaussian theory

Gaussian theory linear theory uncertainty in theory

mathematical statement of problem find (m,d) that maximizes pT(m,d) = pA(m) pA(d) pg(m,d) and, along the way, work out the form of pT(m,d)

notational simplification group m m and d d into single vector x = [d dT, m mT]T group [cov m m]A and [cov d]A into single matrix write d d- -Gm=0 Gm=0 as Fx Fx=0 =0 with F=[I, F=[I, G] G]

after much algebra, we find pT(x) is a Gaussian distribution with mean and variance

after much algebra, we find pT(x) is a Gaussian distribution with mean solution to inverse problem and variance

after pulling m mest out of x x*

after pulling m mest out of x x* reminiscent of G GT(GG minimum length solution GGT)-1

after pulling m mest out of x x* error in theory adds to error in data

after pulling m mest out of x x* solution depends on the values of the prior information only to the extent that the model resolution matrix is different from an identity matrix

and after algebraic manipulation which also equals reminiscent of (G GTG G)-1 G GT least squares solution

interesting aside weighted least squares solution is equal to the weighted minimum length solution

what did we learn? for linear Gaussian inverse problem inexactness of theory just adds to inexactness of data

Part 3 Use maximization of relative entropy as a guiding principle for solving inverse problems

from last lecture

assessing the information content in pA(m m) Do we know a little about m m or a lot about m m ?

Information Gain, S -S called Relative Entropy

(A) pA(m) 0.3 p and q 0.2 pN(m) 0.1 0 -25 -20 -15 -10 -5 0 5 10 15 20 25 m m (B) 4 3 S( A) 2 S 1 0 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 sigmamp A

Principle of Maximum Relative Entropy or if you prefer Principle of Minimum Information Gain

find solution p.d.f. pT(m) that has the largest relative entropy as compared to a priori p.d.f. pA(m) or if you prefer find solution p.d.f. pT(m) that has smallest possible new information as compared to a priori p.d.f. pA(m)

properly normalized p.d.f. data is satisfied in the mean or expected value of error is zero

After minimization using Lagrange Multipliers process pT(m m) is Gaussian with maximum likelihood point m mest satisfying

After minimization using Lagrane Multipliers process pT(m m) is Gaussian with maximum likelihood point m mest satisfying just the weighted minimum length solution

What did we learn? Only that the Principle of Maximum Entropy is yet another way of deriving the inverse problem solutions we are already familiar with

Part 4 F-test as way to determine whether one solution is better than another