Understanding Matrix Operations and Properties

Chapter 2 Matrices 2.1 Operations with Matrices 2.2 Properties of Matrix Operations 2.3 The Inverse of a Matrix 2.4 Elementary Matrices 2.5 Markov Chains 2.6 Applications of Matrix Operations (Least Squares Regression) 2.7 Programming Homework 3 Since any real number can be expressed as a 1 1 matrix, all rules, operations, or theorems for matrices can be applied to real numbers, but not vice versa Only memorize those rules that are different for matrices and real numbers 2.1

2.1 Operations with Matrices Matrix: a a a a 11 12 13 1 n a a a a 21 22 23 2 n = = [ ] A a M a a a a 31 32 33 3 ij m n n a a a a 1 2 3 m m m mn m n (i, j)-th entry (or element): number of rows: m number of columns: n size: m n ij a Square matrix: m = n 2.2

Equal matrices: two matrices are equal if they have the same size (m n) and entries corresponding to the same position are equal = = For [ ] and [ ] ij m n b , A ij m n a B = = if and only if for 1 , 1 A B a b i m j n ij ij Ex: Equality of matrices 1 2 a b = = A B 3 4 c d = = = = = If , then B 1, 2, 3, and 4 A a b c d 2.3

Matrix addition: = = If [ ] , [ ] ij m n b , A ij m n a B A B + = + = + = = then [ ] [ ] ij m n b [ ] [ ] ij m n c ij m n a a ij m n b C ij Ex 2: Matrix addition + + 1 2 1 3 1 1 2 3 0 5 + = = + 0 1 1 2 0 1 1 2 1 3 1 1 0 1 1 + = + = 0 3 3 3 3 + 2 2 0 2 2 2.4

Scalar () multiplication: = If then [ ] [ and is a constant scalar, c ] ij m n ca A ij m n a cA = Matrix subtraction: B A = + ( ) 1 A B Ex 3: Scalar multiplication and matrix subtraction 1 2 4 2 0 0 = = 3 0 1 1 4 3 A B 2 1 2 1 3 2 Find (a) 3A, (b) B, (c) 3A B 2.5

Sol: (a) ( ) 3 ( ) ( ) ( ) 1 3 ( ) 2 3 1 2 4 3 1 3 2 3 4 3 6 12 ( ) ( ) = = = 3A 3 3 0 1 3 3 0 3 1 9 0 3 ( ) 2 3 ( ) 2 1 2 6 3 6 (b) 2 0 0 2 0 0 ( ) = = 1 4 3 1 1 4 3 B 1 3 2 1 3 2 (c) 1 6 12 3 6 12 2 0 0 = = 10 4 6 3 9 0 3 1 4 3 A B 7 0 4 6 3 6 1 3 2 2.6

Matrix multiplication: If [ A a = = ] and [ ] ij n p b = , B a [ ] ij m p c ij m n = = then [ ] [ ] ij n p b , AB C ij m n If equal, A and B are multipliable ( ) size of C=AB is m p n = = + + + c a b a b a b a b where 1 1 2 2 ij ik kj i j i j in nj = 1 k c c c c a a a 11 12 1 1 j n 11 12 1 n b b b b b b 11 1 1 j n 21 2 2 j n = c c c c a a a 1 2 i i ij in 1 2 i i in b b b 1 n nj nn c c c c a a a 1 2 n n nj nn 1 2 n n nn The entry cij is obtained by calculating the sum of the entry-by-entry product between the i-th row of A and the j-th column of B 2.7

Ex 4: Find AB 1 4 5 3 2 0 3 4 2 1 = = A B 2 2 3 2 Sol: + + ( 1)( 3) (4)( 3) (5)( 3) (3)( 4) ( 2)( 4) (0)( 4) + ( 1)(2) (4)(2) (5)(2) (3)(1) ( 2)(1) (0)(1) + = + + AB 3 2 9 4 15 10 1 6 = 3 2 Note: (1) BA is not multipliable (2) Even BA is multipliable, it could be that AB BA 2.8

Matrix form of a system of linear equations in n variables: + + + n n x a x a x a + + + = a x a x a x b 11 1 12 2 1 1 n n = b 21 1 22 2 2 2 linear equations m + + + = a x a x a x b 1 1 2 2 m m mn n m a a a x b 11 12 1 1 1 n a a a x b single matrix equation A = x 1 n n m 21 22 2 2 2 n = b m 1 a a a x b 1 2 m m mn n m = A = x = b 2.9

Partitioned matrices (): row vector ( ) r r r a a a a a a a a a a a a 11 12 13 14 1 = = A 21 22 23 24 2 31 32 33 34 3 a a a a a a a a a a a a column vector ( ) 11 12 13 14 = = c c c c A 21 22 23 24 1 2 3 4 31 32 33 34 submatrix a a a a 11 12 13 14 A A Partitioned matrices can be used to simplify equations or to obtain new interpretation of equations (see the next slide) 11 12 = = A a a a a 21 22 23 24 A A 21 22 a a a a 31 32 33 34 2.10

The result of Ax can be interpreted as a linear combination ( ) of the column vectors of matrix A: a a a a a a x x 11 12 1 1 n 21 22 2 2 n = = = c c c x A and 1 2 n a a a x 1 2 m m mn n + + + + + + 11 1 a x a x 12 2 a x a x a x a x a a a 11 21 1 1 n n n a a a 21 22 2 21 1 22 2 2 n n n = = + + + x A x x x 1 2 n + + + 1 1 m a x a x a x a = a = a = 1 2 2 2 m m mn m mn n 1 m nc 1c 2c = + + + 1 1 x c 2 2 x c n n x c Ax can be viewed as the linear combination (refer to Ch 4 for its definition) of the column vectors of A with coefficients x1, x2, , xn expressed in column vectors and x directly x x 1 2 = c c c You can derive the same result if you perform the matrix multiplication for matrix A 1 2 n x 2.11 n

To practice the exercises in Sec. 2.1, we need to know the trace operation and the notion of diagonal matrices Trace ( ) operation: = + + + If [ ] , then ( ) A ij n n a Tr A a a a 11 22 nn Diagonal matrix ( ): a square matrix in which nonzero elements are found only in the principal diagonal 0 0 d 0 1 0 d = 2 = ( , , , ) A diag d d n d M 1 2 n n 0 0 d n A usual notation for a diagonal matrix 2.12

Keywords in Section 2.1: equality of matrices: matrix addition: scalar multiplication: matrix multiplication: partitioned matrix: row vector: column vector: trace: diagonal matrix: 2.13

2.2 Properties of Matrix Operations Three basic matrix operators, introduced in Sec. 2.1: (1) matrix addition (2) scalar multiplication (3) matrix multiplication 0 0 0 0 0 0 = Zero matrix ( ): 0 m n 0 0 0 m n 1 0 1 = 0 0 0 Identity matrix of order n ( ): (Note that identity matrices must be squared matrices) I n 0 0 1 n n 2.14

Properties of matrix addition and scalar multiplication: If , , A B C , and , are scalars, c d M m n then (1) A+B = B+A (Commutative property ( ) of matrix addition) (2) A+(B+C) = (A+B)+C (Associative property ( ) of matrix addition) (3) (cd) A = c (dA) (Associative property of scalar multiplication) (Multiplicative identity property ( ), and 1 is the multiplicative identity ( ) for all matrices) (4) 1A = A (5) c(A+B) = cA + cB (Distributive ( ) property of scalar multiplication over matrix addition) (6) (c+d) A = cA + dA (Distributive property of scalar multiplication over real-number addition) Notes: All above properties are very similar to the counterpart properties for real numbers 2.15

Properties of zero matrices: If , and is a scalar, c A M m n A + = 0 then (1) A m n So, 0m n is also called the additive identity ( ) for the set of all m n matrices + = 0 (2) ( ) A A m n Thus , A is called the additive inverse ( ) of A = = = 0 0 (3) 0or cA c A m n m n Notes: All above properties are very similar to the counterpart properties for the real number 0 The role of zero matrices is analogous to the real number 0. Note that 0 is unique in real numbers and there could be many zero matrices with different sizes 2.16

Properties of matrix multiplication: (1) A(BC) = (AB)C (Associative property of matrix multiplication) (Distributive property of LHS matrix multiplication over matrix addition) (Distributive property of RHS matrix multiplication over matrix addition) (Associative property of scalar and matrix multiplication) (2) A(B+C) = AB + AC (3) (A+B)C = AC + BC (4) c(AB) = (cA)B = A(cB) For real numbers, the properties (2) and (3) are the same since the order for the multiplication of real numbers is irrelevant The real-number multiplication can satisfy the above properties and there is an additional commutative property for the real-number multiplication, i.e., cd = dc Properties of the identity matrix: If (2) = = , then (1) A M AI I A A A m n n m The role of the real number 1 is similar to the identity matrix. However, 1 is unique in real numbers and there could be many identity matrices with different sizes 2.17

Ex 3: Matrix Multiplication is Associative Calculate (AB)C and A(BC) for 1 0 1 2 2 1 1 3 0 2 = = = , , and 3 2 1 . 4 A B C 2 1 Sol: 1 0 1 2 2 1 1 0 2 = ( ) 3 2 1 4 AB C 3 2 1 1 0 5 1 4 2 0 3 17 13 14 4 = = 3 2 1 4 2.18

1 0 1 2 2 1 1 3 0 2 = ( ) 3 2 1 4 A BC 2 1 1 2 2 1 3 8 2 17 13 14 4 = = 7 2.19

Equipped with the four properties of matrix multiplication on Slide 2.17, we can prove a statement on Slide 1.35: if a homogeneous system has any nontrivial solution, this system must have infinitely many nontrivial solutions x Suppose there is a nonzero solution for this homegeneous system such that . Then it is straightforward to show that solution, i.e., ( ) ( A t t A = x x1) t = 1 = x 0 x must be another A t 1 1 = ( ) 0 0 1 The fourth property of matrix multiplication on Slide 2.17 Finally, since can be any real number, it can be concluded that there are infinitely many solutions for this homogeneous system t 2.20

Definition of Ak :repeated multiplication of a square matrix A: = = = 1 2 k , , , A A A AA A AA A matrices of k A Properties for Ak: (1) AjAk = Aj+k (2) (Aj)k = Ajk where j and k are nonegative integers and A0 is defined to be I (with the same size as A) For diagonal matrices: 0 d 0 0 k d 0 0 0 d 1 1 0 k 0 d 2 = = k 2 D D 0 0 k n d 0 0 d n 2.21

Transpose of a matrix (): a a a a a a 11 12 1 n 21 22 2 n = If , A M m n a a a 1 2 m m mn a a a a a a 11 21 1 m 12 22 2 m = T then A M n m a a a 1 2 n n mn The transpose operation is to move the entry aij (original at the position (i, j)) to the position (j, i) Note that after performing the transpose operation, AT is with the size n m 2.22

Ex 8: Find the transpose of the following matrix 1 2 3 0 1 2 = = 4 5 6 2 4 A A = A (a) (b) (c) 8 7 8 9 1 1 Sol: (a) 2 8 = = T A 2 A 8 1 (b) 2 3 1 4 7 = = T 4 5 6 2 5 8 A A 7 8 9 3 6 9 (c) 0 1 0 2 1 = = 2 4 A T A 1 4 1 1 1 2.23

Properties of transposes: = T T ( ) 1 ( ) A A + ) = + T T T ( ) 2 ( A B A B = T T ( ) 3 ( ) ( ) cA c A = T T T ( ) 4 ( ) AB B A Properties (2) and (4) can be generalized to the sum or product of multiple matrices. For example, (A+B+C+D)T = AT+BT+CT+DT and (ABCD)T = DTCTBTAT Since a real number also can be viewed as a 1 1 matrix, the transpose of a real number is itself, that is, for , aT = a. In other words, the transpose operation has actually no effect on real numbers a R 2.24

Ex 9: Verify that (AB)T and BTAT are equal 2 1 0 1 0 2 2 3 1 3 2 3 1 1 0 = = B A Sol: T T 2 1 0 1 0 2 2 3 1 3 2 3 1 1 0 2 6 1 1 1 2 2 1 6 1 = = = T ( ) AB 1 2 2 1 2 1 0 3 0 2 1 3 1 2 3 2 1 6 1 = = T T B A 1 0 1 2 2.25

Symmetric matrix (): A square matrix A is symmetric if A = AT Skew-symmetric matrix ( ): A square matrix A is skew-symmetric if AT = A Ex: 1 a 2 3 = is symmetric, find a, b, c? If 4 c 5 A 6 b Sol: 1 a 2 3 1 a b A = T A = = 4 c 5 A AT 2 4 c = , 3 = = , 2 5 a b c 6 b 3 5 6 2.26

Ex: 0 a 1 2 = is a skew-symmetric, find a, b, c? If 0 c 3 A 0 b Sol: 0 a b 0 a 1 2 = = 0 c 3 A AT 1 0 c 0 b 2 3 0 = , 1 = , 2 = = 3 a b c T A A Note: must be symmetric T AA The matrix A could be with any size, i.e., it is not necessary for A to be a square matrix. Pf: = = T T T T T T ( ) ( ) AA A A AA However, AAT must generate a square matrix. T symmetric is AA 2.27

Before finishing this section, two properties are discussed, which hold for real numbers, but not for matrices: the first is the commutative property of matrix multiplication and the second is the cancellation property Real number: ab = ba (Commutative property of real-number multiplication) Matrix: AB n n Three possibilities for BA: BA n p m n m p = (1) If (2) If , then , p m is defined, but , then AB is undefined BA M m m p AB BA , n M (Sizes are not the same) m m n n = = , (3) If , then m p n AB M BA M m m m m (Sizes are the same, but resulting matrices may not be equal) 2.28

Ex 4: Show that AB and BA are not equal for the following matrices. 1 3 2 1 = = B A and 0 2 2 1 Sol: 1 3 2 1 2 5 = = AB 2 1 0 2 4 4 2 1 1 3 0 7 = = BA 0 2 2 1 4 2 AB BA (Noncommutativity of matrix multiplication) 2.29

Notes: (1) A+B = B+A (commutative property of matrix addition) AB (2) (the matrix multiplication is not with the commutative property) (so the order of matrix multiplication matters) BA This property is different from the property for the multiplication operations of real numbers, for which the order of multiplication makes no difference 2.30

For real number: = a = and b 0 ac bc c (Cancellation property for real numbers) For matrix: AC = 0 and ( is not a zero matrix) C BC C (1) If C is invertible, then A = B A (2) If C is not invertible, then B (Cancellation property is not necessary to be valid) Here I skip to introduce the definition of invertible ( ) because it will be studied soon in the next section 2.31

Ex 5: An example in which the cancellation property is not valid Show that AC=BC 1 3 2 4 1 2 = = = , , A B C 0 1 2 3 1 2 Sol: 1 3 1 2 2 4 = = AC 0 2 1 4 1 2 2 1 2 2 4 1 = = BC 2 3 1 2 1 2 A AC = So, although , B BC 2.32

Keywords in Section 2.2: zero matrix: identity matrix: commutative property: associative property: distributive property: cancellation law: transpose matrix: symmetric matrix: skew-symmetric matrix: 2.33

2.3 The Inverse of a Matrix Inverse matrix ( ): A M Consider , n n = = if there exists a matrix such that , B M AB BA I n n n then (1) A is invertible ( ) (or nonsingular ( )) (2) B is the inverse of A Note: A square matrix that does not have an inverse is called noninvertible (or singular ( )) The definition of the inverse of a matrix is similar to that of the inverse of a scalar, i.e., c (1/c) = 1 Since there is no inverse (or said multiplicative inverse ( )) for the real number 0, you can imagine that noninvertible matrices act a similar role to the real number 0 in some sense 2.34

Theorem 2.7: The inverse of a matrix is unique If B and C are both inverses of the matrix A, then B = C. = ) ( (associative property of matrix multiplication and the property for the identity matrix on Slide 2.17) AB I Pf: = C AB ) CI = CA ( B C = IB C = B C Consequently, the inverse of a matrix is unique. Notes: (1) The inverse of A is denoted by 1 A = = 1 1 ) 2 ( AA A A I 2.35

Find the inverse of a matrix by the Gauss-Jordan elimination: A Gauss-Jordan elimination 1 | | A I I Ex 2: Find the inverse of the matrix A 1 4 = A 1 3 Sol: = AX I 1 4 1 0 x x 11 12 = + x 1 3 0 1 x x 21 22 + 4 4 1 0 x x x 11 21 12 22 = 3 3 0 1 x x x x 11 21 12 22 2.36

by equating corresponding entries + = 4 1 x x This two systems of linear equations share the same coefficient matrix, which is exactly the matrix A 0 (1) 11 21 = 3 x x 11 21 + = 4 0 x x 1 (2) 12 22 = 3 x x 12 22 1 1 4 3 1 0 1 0 0 3 1 A (1) 1,2 A ( 4) 2,1 = , 3 = , 1 x x (1) 11 21 1 1 1 4 3 0 1 1 0 0 4 1 A (1) 1,2 A ( 4) 2,1 = , 4 = , 1 x x (2) 12 22 1 Perform the Gauss- Jordan elimination on the matrix A with the same row operations Thus 3 4 = = 1 X A 1 1 2.37

Note: Instead of solving the two systems separately, one can solve them simultaneously by appending ( ) the identity matrix to the right of the coefficient matrix 1 1 4 3 1 0 0 1 0 0 3 1 4 1 Gauss-Jordan elimination (1) 1,2 A ( 4) 2,1 A 1 1 , 1 A I I A x x x x 11 12 solution for solution for 21 22 If A cannot be row reduced to I, then A is singular 2.38

Ex 3: Find the inverse of the following matrix 1 1 0 = 1 0 1 A 6 2 3 Sol: 1 1 0 1 0 0 = 1 0 1 0 1 0 A I 6 2 3 0 0 1 1 1 1 4 0 1 3 1 0 1 6 0 1 0 6 1 1 2 0 1 3 1 0 1 0 0 (6) 1,3 A A ( 1) 1,2 0 1 0 0 1 0 0 0 1 1 1 1 1 0 0 1 1 1 1 2 0 1 4 0 0 1 1 1 1 0 0 1 1 1 0 1 2 0 ( 1) 3 (4) 2,3 A M 0 0 1 0 4 0 0 1 2.39

1 0 0 0 0 2 3 1 3 3 4 1 1 1 1 1 0 1 0 0 1 3 2 0 3 4 0 1 1 (1) 2,1 (1) 3,2 A A 1 0 0 0 1 0 1 [ = 1 ] I A So the matrix A is invertible, and its inverse is 2 3 1 = 1 3 3 1 A 2 4 1 Check it by yourselves: = = 1 1 AA A A I 2.40

Matrix Operations in Excel TRANSPOSE: calculate the transpose of a matrix MMULT: matrix multiplication MINVERSE: calculate the inverse of a matrix MDETERM: calculate the determinant of a matrix SUMPRODUCT: calculate the inner product of two vectors For Excel 2019 or earlier version: For TRANSPOSE, MMULT, and MINVSRSE, since the output should be a matrix or a vector, we need to input the formula in a cell first, then identify the output range (the cell with the input formula located at the position a11), and finally put focus on the formula description cell and press Ctrl+Shift+Enter to obtain the desired result. See Matrix operations in Excel for Ch2.xlsx on my website for illustration For Excel 2021: using TRANSPOSE, MMULT, and MINVSRSE is like using other functions, since Excel helps you to complete everything 2.41

Theorem 2.8:Properties of inverse matrices If A is an invertible matrix, k is a positive integer, and c is a scalar, then = 1 1 1 (1) invertible is and ( ) A A A = 1 1 k k k (2) is invertible and ( ) ( ) A A A 1 c = 1 1 (3) is invertible given 0 and ( ) cA c cA A = 1 1 T T T (4) invertible is and ( ) ( ) A A A T is not the number of power. It denotes the transpose operation. Pf: (1) ? 1? = ? ; (2) ??(? 1)?= ?; (3) (??)(1 (4) ??(? 1)?= ? 1??= ??= ? ?? 1) = ?; (See Property (4) on Slide 2.24) 2.42

Theorem 2.9: The inverse of a product If A and B are invertible matrices of order n, then AB is invertible and 1 ) ( B AB Pf: 1 1 1 1 ( )( ) ( ) AB B A A BB A = = = 1 1 A = = = 1 1 1 ( ) ( ) A I A AI A AA I (associative property of matrix multiplication) 1 1 Thus, if is invertible, then its inverse is AB B A Note: (1) It can be generalized to the product of multiple matrices ( ) 3 2 1 A A A A n = 1 n 1 1 1 1 A A A A 3 2 1 (2) It is similar to the results of the transpose of the products of multiple matrices (see Slide 2.24) ( ) 1 2 3 n A A A A = T T n T T T A A A A 3 2 1 2.43

Theorem 2.10: Cancellation properties for matrix multiplication If C is an invertible matrix, then the following properties hold: (1) If AC=BC, then A=B (right cancellation property) (2) If CA=CB, then A=B (left cancellation property) Pf: = AC BC ( = 1 1 ( C ) C ) AC BC -1 ( is invertible, so C exists) C = 1 1 CC ( ) CC ( ) A B (Associative property of matrix multiplication) = AI BI = A B Note: IfC is not invertible, then cancellation property is not valid 2.44

Theorem 2.11: Systems of equations with a unique solution If A is an invertible matrix, then the system of linear equations Ax = bhas a unique solution given by Pf: A = = = = x b A = 1 x b x x x b 1 1 b b A A A ( A is nonsingular) 1 I A 1 A A = x If and were two solutions of equation x x b , 1 2 = = = x b x x x then A A (left cancellation property) 1 2 1 2 So, this solution is unique 2.45

Ex 8: Use an inverse matrix to solve each system (a) (b) 2 3 3 3 2 4 x y z + + + + + + = = = + + + + + + = = = 1 x x y y z z 2 3 2 3 3 4 4 8 5 x x x y y y z z z 1 2 (c) + + + + + + = = = 2 3 2 3 3 4 0 0 0 x x x y y y z z z Sol: 2 3 2 3 1 3 1 4 1 1 1 1 0 0 1 A = Gauss-Jordan elimination = 1 A 6 2 3 2.46

(a) = = 1 1 1 0 0 1 1 2 This technique is very convenient when you face the problem of solving several systems with the same coefficient matrix A = 1 x b 1 1 2 6 2 3 2 (b) Because once you have A-1, you simply need to perform the matrix multiplication to solve the unknown variables 1 1 1 0 0 1 4 8 5 4 1 A = = = 1 x b 6 2 3 7 (c) If you solve only one system, the computational effort is less for the G. E. plus the back substitution or the G. J. E. 1 1 1 0 0 1 0 0 0 0 0 0 A = = = 1 x b 6 2 3 2.47

Keywords in Section 2.3: inverse matrix: invertible: nonsingular: singular: 2.48

2.4 Elementary Matrices Elementary matrix ( ): An n n matrix is called an elementary matrix if it can be obtained from the identity matrix I by a single elementary row operation Three row elementary matrices: (1) ( ) i j i j E I = = I (interchange two rows) , , ( ) k i ( ) k i (2) ( ) ( I 0) E M k (multiply a row by a nonzero constant) = ( ) , i j ( ) , i j k k (3) ( ) I E A (add a multiple of a row to another row) Note: 1. Perform only a single elementary row operation on the identity matrix 2. Since the identity matrix is a square matrix, elementary matrices must be square matrices 2.49

Ex 1: Elementary matrices and nonelementary matrices 1 0 1 0 0 0 0 1 0 3 0 0 0 1 1 0 0 1 0 0 (c) 0 (b) (a) 0 0 0 No (row multiplication must be with a nonzero constant) (3) 2 Yes ( ( )) I M No (not square) 3 1 0 0 1 0 1 0 1 0 2 0 0 0 1 2 0 1 (d) 0 (f) 0 (e) 0 0 1 No (use two elementary row operations, M Yes ( ( )) I I (2) 1,2 Yes ( ( )) I A 2,3 3 2 (2) 2 ( 1) 3 and ) M 2.50