path: root/lecture_1.typ

#set document(
  title: [Lecture 1],
  author: "Gidon Rosalki",
  date: auto,
)
#set heading(numbering: "1.")
#set text(font: "New Computer Modern")
#set page(margin: (
  bottom: 0.5cm,
  right: 1.5cm,
  left: 1.5cm,
  top: 1.5cm,
))
#set list(indent: 10pt)
#set enum(indent: 10pt)

#show title: set align(center)
#show link: set text(rgb("005eff"))
#show link: underline
#show math.equation.where(block: true): set block(breakable: true)
#title()
#align(center)[
  Gidon Rosalki \
  2026-10-13
]

#let colourmaths(x, color) = text(fill: color)[$#x$]

// author: eric1102 - https://sitandr.github.io/typst-examples-book/book/typstonomicon/math_display.html
// may slightly interfere with maths blocks
#show math.equation: it => {
  if it.body.fields().at("size", default: none) != "display" {
    return math.display(it)
  }
  it
}

= Introduction

Since everyone comes with a different background in mathematics to their masters in CompSci, this course exists to
ensure everyone knows a standard foundation. \
The course is split into 4 main parts, Linear Algebra, Infinitesimal Calculus, Optimisation, and Probability. Aside from
the optimisation section, it will all be content already taught in one's bachelors degree. The purpose is to give us
skills that we may use in our research.

= Linear Algebra

We will be focusing on slightly less abstract concepts than in the bachelors courses. We will only work over the field
$RR$, real matrices, and vector spaces, and so on, rather than abstract fields, spotlighting applicative concepts.


= Matrices

Let there be a matrix $M in "Mat"_(n times m) (RR)$. The vectors will all be column vectors (unless stated otherwise) $v
in RR^(n)$. We may transpose our matrix $M : M^T in "Mat"_(m times n) (RR)$ defined such that $M^T_(i, j) = M_(j, i)$.

If $M$ is a square matrix: $M in "Mat"_(n times n) (RR)$, it is called *invertible* if $exists M^(-1) in M_(n) (RR) : M
dot M^(-1) = I_n$.

== Eigenvalues and Eigenvectors

Let there be $M in "Mat"_(n) (RR)$.

*Definition* (eigenvalue): $lambda in RR$ is called an *eigenvalue* of $M$ if $exists v in RR^n != 0 : M dot v = lambda
dot v$. The eigenvalues of a matrix are called its _spectrum_.

*Definition* (eigenvector): Given an eigenvalue $lambda$ of matrix $M$, where $M dot v = lambda dot v$, then $v in RR^n
!= 0$ is called an *eigenvector*.

*Definition* (eigenspace): The *eigenspace* of $lambda$ is defined as: $V_lambda = {v in RR^n : M v = lambda v}$. \
It is easy to show that $V_lambda$ is a linear subspace of $RR^n$. \
$dim (V_lambda)$ is the (geometric) multiplicity of $lambda$.

*Fact*:
$
  sum_(lambda "eigenvalue of" M)^() dim (V_lambda) <= n
$

*Definition* (diagonalisable): $M$ is diagonalisable if there is an invertible matrix $P$, and a diagonal matrix $D$
such that $M = P D P^(-1)$, where $P, D in M_(n)$.

Let $v_i$ be the $i$-th column of $P$. Then
$
             M dot v_i & = P D P^(-1) dot v_i \
  P dot P^(-1) => quad & = P D e_i \
                       & = P dot D_(i i) dot e_i \
                       & = D_(i, i) P e_i \
                       & = D_(i, i) v_i \
$
This implies that $D_(i, i)$ is an eigenvalue, with eigenvector $v_i$. From this we may conclude:

*Theorem*: $M$ is diagonalisable $<==>$ there is a basis $v_1, dots, v_n$ of $RR^n$ of eigenvectors of $M$.

== Orthogonal Diagonalisation

We will firstly discuss the standard inner product. There are many different inner products, but we will focus on the
standard one:
$
  x, y in RR^n ==> lr(chevron.l x, y chevron.r) = sum_(i = 1)^(n) x_i dot y_i
$

Let $0 != v_1, dots, v_n in RR^n$.

*Definition* (orthogonal): $v_1, dots, v_n$ are _orthogonal_ if $forall i != j quad lr(chevron.l v_i, v_j chevron.r) = 0$

*Definition* (orthonormal): $v_1, dots, v_n$ are _orthonormal_ if they are orthogonal, and \
$forall i quad norm(v_i) = sqrt(lr(chevron.l v_i, v_i chevron.r)) = 1$ (unit vectors)

*Definition* (orthogonal matrix): An $n times n$ matrix $V$ is called _orthogonal_ if $V^T dot V = I$. Equivalent to:
+ The columns of $V$ are orthonormal
+ The rows of $V$ are orthonormal
+ $V^(-1) = V^T$
+ $forall x in RR^n quad norm(V x) = norm(x)$

*Definition* (orthogonally diagonalisable): A matrix $M$ is _orthogonally diagonalisable_ if there exist orthogonal matrix
$V$, diagonal matrix $D$ such that $M = V dot D dot V^T$.

*Theorem* (Spectral theorem for symmetric matrices): If $M$ is symmetric ($M = M^T$), then $M$ is orthogonally
diagonalisable, i.e. there is an orthonormal basis of eigenvectors. In other words $M = V dot D dot V^(T)$, where $V$
is orthogonal, and $D$ is diagonal.

*Examples*:
$
   "No eigenvalues: " & mat(0, -1; 1, 0) \
  "Two eigenvalues: " & mat(1, 1; 0, 2) \
                      & lambda_1 = 1, vec(1, 0) \
                      & lambda_2 = 2, vec(1, 1)
$
The second matrix is diagonalisable, but not orthogonal diagonalisable.

Why is orthonormality so useful? Let us suppose that $v_1, dots, v_n$ is an orthonormal basis of $RR^n$. This means that
there is a unique way to write the vector $x$ as a linear combination of these:
$
  x = sum_(i = 1)^(n)alpha_i v_i
$
If we suppose
$
                                 y & = sum_(i = 1)^(n)beta_i v_i \
  ==> lr(chevron.l x, y chevron.r) & = sum_(i = 1)^(n)alpha_i beta_i
$
Similarly: $norm(x)^2 = sum_(i = 1)^(n) alpha_i^2$. \
Additionally, assuming that $v_1, dots, v_n$ are eigenvectors, with eigenvalues $lambda_1, dots, lambda_n$, for matrix
$M$:
$
          M x & = sum_(i = 1)^(n)alpha_i lambda_i v_i \
  norm(M x)^2 & = sum_(i = 1)^(n) alpha_i^2 lambda_i^2
$

== Singular Value Decomposition (SVD)

*Definition* (SVD): Let $M in M_(n times m) (RR)$.
+ $V$ is an $m times m$ orthogonal matrix
+ $U$ is an $n times n$ orthogonal matrix
+ $Sigma$ is diagonal: $Sigma_(i, j) = 0 "if" i != j$, where $Sigma_(1, 1) >= Sigma_(2, 2) >= dots >= Sigma_(k, k) >= 0
  : k = min(m, n)$
$
  "s.t." M & = U dot Sigma dot V^T
$
We denote $sigma_i = Sigma_(i,i) : 1 <= i <= k$, and call $sigma_1, dots, sigma_k$ the singular values of $M$.

Let $u_1, dots, u_n$ be the columns of $U$, and $v_1, dots, v_m$ be the columns of $V$. This allows the following:
$
  M dot v_i & = U dot Sigma dot overbrace(V^T dot v_i, e_i) \
            & = U dot Sigma dot e_i \
            & = U dot sigma_i dot e_i \
            & = sigma_i dot U dot e_i \
            & = sigma_i dot u_i
$

*Remark*: If $m > n and i > n => M v_i = 0$

Let us now consider finding $M^T$:
$
  M^T = (U Sigma V^T)^T = V Sigma^T U^T
$

*Definition*: $U = u_1, dots, u_n, V = v_1, dots, v_m$ are called the _singular vectors_ of $M$.

*Definition* (operator norm): $M in M_(n times m) (RR), quad norm(M)_"op" = max{norm(M dot v) : v in RR^m and norm(v) =
  1}$ (may be considered the maximum increase that $M$ can do to a vector).

*Claim*: $norm(M)_"op" = sigma_1$, and $v_1$ is a vector that attains the maximum. \
_Proof_: Let $v in RR^m : norm(v) = 1$. We will write $v = sum_(i = 1)^(m)alpha_i v_i$. So:
$
        M dot v & = sum_(i = 1)^(m)alpha_i (M dot v_i) \
                & = sum_(i = 1)^(k) alpha_i dot sigma_i u_i \
  1 = norm(v)^2 & = sum_(i = 1)^(m)alpha_i^2 \
      norm(M v) & = sum_(i = 1)^(k) alpha_i^2 dot sigma_i^2 \
                & <= sigma_1^2 dot sum_(i = 1)^(k)alpha_i^2 \
                & <= sigma_1^2
$
We showed that $norm(M v) <= sigma_1$, and this may be attained by taking $v = v_1$: $M v_1 = sigma_1 u_1 => norm(M v_1)
= sigma_1$.

*Reminder*: $v bot v' => lr(chevron.l v, v' chevron.r) = 0$ \
*Exercise*: $max{norm(M v) : v in RR^m, norm(v) = 1, v bot v_1} = sigma_2$, $v_2$ attains the maximum.

#pagebreak()

=== SVD Proof

Our strategy is that we will construct an orthonormal basis $v_1, dots, v_m$ for $RR^m$, such that $M v_1, dots, M v_m$
are orthogonal: $lr(chevron.l M v_i, M v_j chevron.r) = 0$. The idea is that
$
  u_1 = (M v_1) / (norm(M v_1)), wide u_2 = (M v_2) / norm(M v_2), wide dots
$
as long as $M v_i != 0$, and complete it to an orthonormal basis. In particular, $sigma_i := norm(M v_i)$.

_Proof_: We will construct recursively:
+ Let $v_1$ be a unit vector that maximises $ {norm(M v) : v in RR^(m), norm(v) = 1} $
+ Given $v_1, dots, v_(i - 1)$, we will let $v_i$ be a unit vector that maximises $ {norm(M v) : v in RR^m, norm(v) = 1,
    v bot v_1, dots, v_(i - 1)} $
We will now prove that $forall i < j #h(0.5em) lr(chevron.l M v_i, M v_j chevron.r) = 0$. Consider $f : RR -> RR,
#h(0.5em) f(t) = norm(M (v_i + t v_j))^2$.
+ We will observe that $t = 0$ is a local maximum of $f(t)$ since $x_t = (v_i + t v_j) / norm(v_i + t v_j)$ is a unit
  vector orthogonal to $v_1, dots, v_(i - 1)$. Therefore $norm(M x_t)^2 <= norm(M v_i)^2 = norm(M underbrace(x_0, t =
  0))^2$

Now:
$
       f(t) & = norm(M((v_i + t v_j) / norm(v_i + t v_j)))^2 \
            & = (norm(M(v_i + t v_j))^2) / norm(v_i + t v_j)^2 \
            & = norm(M (v_i) + t M (v_j))^2 / (norm(v_i)^2 + 2 lr(chevron.l v_i, t v_j chevron.r) + norm(t v_j)^2) \
            & = (norm(M v_i)^2 + 2 lr(chevron.l M v_i, t M v_j chevron.r) + norm(t M v_j)^2) / (1 + 0 + t^2) \
            & = (norm(M v_i)^2 + 2 t lr(chevron.l M v_i, M v_j chevron.r) + t^2 norm(M v_j)^2) / (1 + t^2) \
      f'(t) & = ((2 lr(chevron.l M v_i, M v_j chevron.r) + 2 norm(M v_j)^2 t) dot (1 + t^2) - 2 t dot (dots)) / ((1 +
                t^2)^2) \
  ==> f'(0) & = 2 lr(chevron.l M v_i, M v_j chevron.r) \
            & = 0 "since " t = 0 " is a local maximum"
$