1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
|
#set document(
title: [Lecture 2 - Matrix Norms: Low rank Approximation],
author: "Gidon Rosalki",
date: auto,
)
#set heading(numbering: "1.")
#set text(font: "New Computer Modern")
#set page(margin: (
bottom: 0.5cm,
right: 1.5cm,
left: 1.5cm,
top: 1.5cm,
))
#set list(indent: 10pt)
#set enum(indent: 10pt)
#show title: set align(center)
#show link: set text(rgb("005eff"))
#show link: underline
#show math.equation.where(block: true): set block(breakable: true)
#title()
#align(center)[
Gidon Rosalki \
2026-10-20
]
#let colourmaths(x, color) = text(fill: color)[$#x$]
// author: eric1102 - https://sitandr.github.io/typst-examples-book/book/typstonomicon/math_display.html
// may slightly interfere with maths blocks
#show math.equation: it => {
if it.body.fields().at("size", default: none) != "display" {
return math.display(it)
}
it
}
#let Mat = [Mat]
= Reminder
*Definition* (SVD): Let there be $M in MM_(n times m) (RR)$. The SVD of this matrix is:
$
M = U dot Sigma dot V^T
$
Where $U, V$ are orthogonal. The columns $u_1, dots, u_n$ of $U$ are orthonormal basis for $RR^n$, and the columns $v_1,
dots, v_m$ of $V$ are an orthonormal basis for $RR^m$. \
$Sigma$ is a diagonal matrix where $forall i != j, Sigma_(i j) = 0$. Note that $Sigma$ is not necessarily square. We
denote the values of the diagonal $sigma_1 >= sigma_2 >= dots >= sigma_k : k = min(m, n)$.
*Definition* trace:
$
tr(A) & = sum_(i = 1)^(n)A_(i, i) \
tr(A B) & = tr(B A)
$
*IEP*: $E, F subset.eq R^m$, then $dim(E + F) = dim(E) + dim(F) - dim(E inter F)$ where $E + F = "span"(E union F)$
= Matrix Norms
*Definition* (norm): A norm on a (real) vector space $V$ is a function $norm(dot.c) : V -> [0, inf)$ such that:
+ $norm(x) = 0 <==> x = 0 quad forall x in V$
+ Triangle inequality: $norm(x + y) <= norm(x) + norm(y) quad forall x, y in V$.
+ $norm(alpha x) = abs(alpha) norm(x) quad forall x in V, alpha in RR$
== Examples
For $V = RR^n$
$
norm(x) = & norm(x)_2 && = sqrt(sum_(i = 1)^(n)x_i^2) \
1 <= p < infinity : & norm(x)_p && = (sum_(i = 1)^(n) abs(x_i)^(p))^(1 / p) \
& norm(x)_infinity && = max_(i in {1, dots, n}) abs(x_i)
$
== Matrix Norms
This is a fancy name where we take $V$ to be the vector space of matrices: $V = MM_(n times m) (RR) tilde.equiv R^(n dot
m)$. For example, for $M in V$:
+ The operator norm / spectral norm is a measure of the size of a matrix, how much it can increase a vector $v$:
$
norm(M)_"op" = max {norm(M v) : v in RR^m, norm(v) = 1}
$
Recall that $norm(M)_"op" = sigma_1$, the largest singular value of $M$
+ Frobenius norm:
$
norm(M)_F = (sum_(i = 1)^(n) sum_(j = 1)^(m)MM_(i j)^(2))^(1 / 2)
$
#pagebreak()
*Claim*: $norm(M)_F^2 = tr(M dot M^T)$ \
_Proof_:
$
(M dot M^T)_(i i) & = sum_(j = 1)^(m) MM_(i j) dot MM_(j i)^(T) \
& = sum_(j = 1)^(m)MM_(i j)^(2)
$
*Claim*: $norm(M)_F = (sum_(i = 1)^(k)sigma_i^2)^(1 / 2)$\
_Proof_:
$
norm(M)_F^2 & = tr(M M^T) \
& = tr(underbrace(U Sigma V^T, M) dot underbrace(V Sigma^T U^T, M^T)) \
& = tr(underbrace(U Sigma Sigma^T, A) underbrace(U^T, B)) \
& = tr(U^T U Sigma Sigma^T) \
& = tr(Sigma Sigma^T) \
& = norm(Sigma)_F^2 \
& = sum_(i = 1)^(k)sigma_i^2
$
= Low Rank Approximation
#let rk = [rk]
#let rank = [rank]
#let Im = [Im]
*Definition* $rk(M) = rank(M) = dim(Im(M))$
*Claim*: Let $M in MM_(n times m) (RR)$. The rank of $M$ is the minimum number $r$ such that
$
exists A in MM_(n times r) (RR), B in MM_(r times m) (RR) : M = A dot B
$
It is clear that $rk(A) <= r and rk(A dot B) <= rk(A) <= r$. Therefore, if $M = A dot B => rk(M) <= r$.
*Claim*: Suppose that the singular values of $M$ are $sigma_1 >= sigma_2 >= dots >= sigma_k >= 0$. Then
$rk(M)$ is the number of non-zero singular values.
$
rk(M) = max{i : i < k and sigma_i > 0}
$
_Proof_:
$
M & = U Sigma V^T \
M v_i & = cases(
sigma_i u_i && "if" 1 <= i <= k,
0 && "if" i > k
)
$
Therefore, $Im(M)$ is spanned by $u_i : sigma_i > 0$.
*Question*: Given $M in MM_(n times m)(RR)$, find $B in MM_(n times m) (RR)$ of rank $<= r$ that minimises
$norm(M - B)_"op"$ and $norm(M - B)_F$.
*Definition*: Given diagonal matrix $Sigma$, and $r <= k$, replace the last $k - r$ singular values on the diagonal with 0.
#pagebreak()
*Theorem*: For every matrix $B$ of rank $<= r$, it holds that $norm(M - B) >= norm(M - U Sigma_((r)) V^T)$, for both the
Frobenius, and the operator norm:
$
norm(M - B)_F & >= norm(M - U Sigma_((r)) V^T)_F \
& = (sum_(i = r + 1)^(k)sigma_i^2)^(1 / 2) \
norm(M - B)_"op" & >= norm(M - U Sigma_((r)) V^T)_"op" \
& = sigma_(r + 1)
$
_Proof_: This proof requires the Min-Max Theorem for singular values (below).
*Theorem* (Min-Max Theorem for singular values): Let $M in MM_(n times m) (RR)$, with singular value decomposition $M = U
Sigma V^T$.
$
forall 1 <= i <= k & = min(m, n) \
sigma_i & = max_(E subset.eq RR^m \ dim(E) = i) (min_(v in E \ norm(v) = 1) norm(M v)) \
& = min_(E subset.eq R^m \ dim(E) = m - i + 1) (max_(v in E \ norm(v) = 1) norm(M v))
$
The second equality will not be proven, but is left as an exercise. \
_Proof_ (of first equality):
+ We need to show that there is $E subset.eq RR^m, dim(E) = i : min_(v in E \ norm(v) = 1) norm(M v) = sigma_i$
+ For every $E subset.eq RR^m, dim(E) = i$, it holds that $min_(v in E \ norm(v) = 1) <= sigma_i$, that s $exists v in
E, #h(0.4em) norm(v) = 1 : norm(M v) <= sigma_i$
#let sp = [sp]
1. _Proof_: We will choose $E = sp{v_1, dots, v_i} => dim(E) = i$. Let $v in E, norm(v) = 1$. Let us write
$
v & = sum_(j = 1)^(i)alpha_j v_j \
1 & = norm(v) = sum_(j = 1)^(i)alpha_j^2 \
M v & = sum_(j = 1)^(i)alpha_j M v_j \
& = sum_(j = 1)^(i)alpha_j sigma_j u_j \
==> norm(M v)^2 & = sum_(j = 1)^(i)alpha_j^2 sigma_j^2 \
norm(M v) & >= sigma_i quad forall v in E, norm(v) = 1 \
norm(M v)^2 & = sum_(j = 1)^(i)alpha_j^2 sigma_j^2 \
& >= sigma_i^2 sum_(j = 1)^(i)alpha_j^2 \
& = sigma_i^2 \
"Choose" v = v_i & : v in E, norm(v) = 1 and \
norm(M v) & = norm(M v_i) \
& = norm(sigma_i u_i) \
& = sigma_i
$
+ _Proof_: Given $E$, we need to find the vector, rather than choosing $E$: \
Let $E subset.eq R^m$ of $dim(E) = i$. Consider $F = sp{v_i, v_(i + 1), dots, v_m}, #h(0.5em) dim(F) = m - i + 1$.
Therefore
$
dim(E) + dim(F) & = m + 1 \
dim(E inter F) & > 0
$
Let $v in E inter F$ of $norm(v) = 1$.
+ $v in E, norm(v) = 1$
+ $v = sum_(j = i)^(m)alpha_j v_j$ since $v in F$.
$
1 = norm(v) & = sum_(j = i)^(m)alpha_j^2 \
M v & = sum_(j = i)^(m) alpha_j M v_j \
& = sum_(j = i)^(k)alpha_i sigma_j u_j \
==> norm(M v)^2 & = sum_(j = i)^(k)alpha_j^2 sigma_j^2 \
& <= sigma_i^2 dot underbrace(sum_(j = i)^(k)alpha_i^2, <= norm(v)^2 = 1) \
& <= sigma_i^2
$
|
