Second hour

1 files changed, 93 insertions, 12 deletions

diff --git a/lecture_1.typ b/lecture_1.typ
index 38aa602..dcade12 100644
--- a/lecture_1.typ
+++ b/lecture_1.typ
@@ -26,6 +26,14 @@
 
 #let colourmaths(x, color) = text(fill: color)[$#x$]
 
+// author: eric1102 - may slightly interfere with maths blocks
+#show math.equation: it => {
+  if it.body.fields().at("size", default: none) != "display" {
+    return math.display(it)
+  }
+  it
+}
+
 = Introduction
 
 Since everyone comes with a different background in mathematics to their masters in CompSci, this course exists to
@@ -39,9 +47,8 @@ skills that we may use in our research.
 We will be focusing on slightly less abstract concepts than in the bachelors courses. We will only work over the field
 $RR$, real matrices, and vector spaces, and so on, rather than abstract fields, spotlighting applicative concepts.
 
-== Reminder
 
-=== Matrices
+= Matrices
 
 Let there be a matrix $M in "Mat"_(n times m) (RR)$. The vectors will all be column vectors (unless stated otherwise) $v
 in RR^(n)$. We may transpose our matrix $M : M^T in "Mat"_(m times n) (RR)$ defined such that $M^T_(i, j) = M_(j, i)$.
@@ -49,7 +56,7 @@ in RR^(n)$. We may transpose our matrix $M : M^T in "Mat"_(m times n) (RR)$ defi
 If $M$ is a square matrix: $M in "Mat"_(n times n) (RR)$, it is called *invertible* if $exists M^(-1) in M_(n) (RR) : M
 dot M^(-1) = I_n$.
 
-=== Eigenvalues and Eigenvectors
+== Eigenvalues and Eigenvectors
 
 Let there be $M in "Mat"_(n) (RR)$.
 
@@ -73,17 +80,17 @@ such that $M = P D P^(-1)$, where $P, D in M_(n)$.
 
 Let $v_i$ be the $i$-th column of $P$. Then
 $
-              M dot v_i & = P D P^(-1) dot v_i \
-  P dot P^(-1) ==> quad & = P D e_i \
-                        & = P dot D_(i i) dot e_i \
-                        & = D_(i, i) P e_i \
-                        & = D_(i, i) v_i \
+             M dot v_i & = P D P^(-1) dot v_i \
+  P dot P^(-1) => quad & = P D e_i \
+                       & = P dot D_(i i) dot e_i \
+                       & = D_(i, i) P e_i \
+                       & = D_(i, i) v_i \
 $
 This implies that $D_(i, i)$ is an eigenvalue, with eigenvector $v_i$. From this we may conclude:
 
 *Theorem*: $M$ is diagonalisable $<==>$ there is a basis $v_1, dots, v_n$ of $RR^n$ of eigenvectors of $M$.
 
-=== Orthogonal Diagonalisation
+== Orthogonal Diagonalisation
 
 We will firstly discuss the standard inner product. There are many different inner products, but we will focus on the
 standard one:
@@ -91,11 +98,11 @@ $
   x, y in RR^n ==> lr(chevron.l x, y chevron.r) = sum_(i = 1)^(n) x_i dot y_i
 $
 
-Let $0 != v_1, dots, v_n in RR^n$. 
+Let $0 != v_1, dots, v_n in RR^n$.
 
 *Definition* (orthogonal): $v_1, dots, v_n$ are _orthogonal_ if $forall i != j quad lr(chevron.l v_i, v_j chevron.r) = 0$
 
-*Definition* (orthonormal): $v_1, dots, v_n$ are _orthonormal_ if they are orthogonal, and \ 
+*Definition* (orthonormal): $v_1, dots, v_n$ are _orthonormal_ if they are orthogonal, and \
 $forall i quad norm(v_i) = sqrt(lr(chevron.l v_i, v_i chevron.r)) = 1$ (unit vectors)
 
 *Definition* (orthogonal matrix): An $n times n$ matrix $V$ is called _orthogonal_ if $V^T dot V = I$. Equivalent to:
@@ -108,5 +115,79 @@ $forall i quad norm(v_i) = sqrt(lr(chevron.l v_i, v_i chevron.r)) = 1$ (unit vec
 $V$, diagonal matrix $D$ such that $M = V dot D dot V^T$.
 
 *Theorem* (Spectral theorem for symmetric matrices): If $M$ is symmetric ($M = M^T$), then $M$ is orthogonally
-diagonalisable, i.e. there is an orthonormal basis of eigenvectors. In other words $M = V dot D dot V ^ (T)$, where $V$
+diagonalisable, i.e. there is an orthonormal basis of eigenvectors. In other words $M = V dot D dot V^(T)$, where $V$
 is orthogonal, and $D$ is diagonal.
+
+*Examples*:
+$
+   "No eigenvalues: " & mat(0, -1; 1, 0) \
+  "Two eigenvalues: " & mat(1, 1; 0, 2) \
+                      & lambda_1 = 1, vec(1, 0) \
+                      & lambda_2 = 2, vec(1, 1)
+$
+The second matrix is diagonalisable, but not orthogonal diagonalisable.
+
+Why is orthonormality so useful? Let us suppose that $v_1, dots, v_n$ is an orthonormal basis of $RR^n$. This means that
+there is a unique way to write the vector $x$ as a linear combination of these:
+$
+  x = sum_(i = 1)^(n)alpha_i v_i
+$
+If we suppose
+$
+                                 y & = sum_(i = 1)^(n)beta_i v_i \
+  ==> lr(chevron.l x, y chevron.r) & = sum_(i = 1)^(n)alpha_i beta_i
+$
+Similarly: $norm(x)^2 = sum_(i = 1)^(n) alpha_i^2$. \
+Additionally, assuming that $v_1, dots, v_n$ are eigenvectors, with eigenvalues $lambda_1, dots, lambda_n$, for matrix
+$M$:
+$
+          M x & = sum_(i = 1)^(n)alpha_i lambda_i v_i \
+  norm(M x)^2 & = sum_(i = 1)^(n) alpha_i^2 lambda_i^2
+$
+
+== Singular Value Decomposition (SVD)
+
+*Definition* (SVD): Let $M in M_(n times m) (RR)$.
++ $V$ is an $m times m$ orthogonal matrix
++ $U$ is an $n times n$ orthogonal matrix
++ $Sigma$ is diagonal: $Sigma_(i, j) = 0 "if" i != j$, where $Sigma_(1, 1) >= Sigma_(2, 2) >= dots >= Sigma_(k, k) >= 0
+  : k = min(m, n)$
+$
+  "s.t." M & = U dot Sigma dot V^T
+$
+We denote $sigma_i = Sigma_(i,i) : 1 <= i <= k$, and call $sigma_1, dots, sigma_k$ the singular values of $M$.
+
+Let $u_1, dots, u_n$ be the columns of $U$, and $v_1, dots, v_n$ be the columns of $V$. This allows the following:
+$
+  M dot v_i & = U dot Sigma dot overbrace(V^T dot v_i, e_i) \
+            & = U dot Sigma dot e_i \
+            & = U dot sigma_i dot e_i \
+            & = sigma_i dot U dot e_i \
+            & = sigma_i dot u_i
+$
+
+*Remark*: If $m > n and i > n => M v_i = 0$
+
+Let us now consider finding $M^T$:
+$
+  M^T = (U Sigma V^T)^T = V Sigma^T U^T
+$
+
+*Definition*: $U = u_1, dots, u_n, V = v_1, dots, v_n$ are called the _singular vectors_ of $M$.
+
+*Definition* (operator norm): $M in M_(n times m) (RR), quad norm(M)_"op" = max{norm(M dot v) : v in RR^m and norm(v) =
+  1}$ (may be considered the maximum increase that $M$ can do to a vector).
+
+*Claim*: $norm(M)_"op" = sigma_1$, and $v_1$ is a vector that attains the maximum. \
+_Proof_: Let $v in RR^m : norm(v) = 1$. We will write $v = sum_(i = 1)^(m)alpha_i v_i$. So:
+$
+                                  M dot v & = sum_(i = 1)^(m)alpha_i (M dot v_i) \
+                                          & = sum_(i = 1)^(k) alpha_i dot sigma_i u_i \
+                            1 = norm(v)^2 & = sum_(i = 1)^(m)alpha_i^2 \
+                                norm(M v) & = sum_(i = 1)^(k) alpha_i^2 dot sigma_i^2 \
+                                          & <= sigma_1^2 dot sum_(i = 1)^(k)alpha_i^2 \
+                                          & <= sigma_1^2
+$
+We showed that $norm(M v) <= sigma_1$, and this may be attained by taking $v = v_1$: $M v_1 = sigma_1 u_1 => norm(M v_1)
+= sigma_1$.
+