diff options
Diffstat (limited to 'lecture_2.typ')
| -rw-r--r-- | lecture_2.typ | 79 |
1 files changed, 78 insertions, 1 deletions
diff --git a/lecture_2.typ b/lecture_2.typ index b765f74..c655762 100644 --- a/lecture_2.typ +++ b/lecture_2.typ @@ -56,6 +56,8 @@ $ *IEP*: $E, F subset.eq R^m$, then $dim(E + F) = dim(E) + dim(F) - dim(E inter F)$ where $E + F = "span"(E union F)$ +For $B : RR^m -> RR^n, dim(ker(B) + "rk"(B)) = m$ + = Matrix Norms *Definition* (norm): A norm on a (real) vector space $V$ is a function $norm(dot.c) : V -> [0, inf)$ such that: @@ -150,7 +152,7 @@ $ norm(M - B)_"op" & >= norm(M - U Sigma_((r)) V^T)_"op" \ & = sigma_(r + 1) $ -_Proof_: This proof requires the Min-Max Theorem for singular values (below). +_Proof_: This proof requires the Min-Max Theorem for singular values (below). Proof given below. *Theorem* (Min-Max Theorem for singular values): Let $M in MM_(n times m) (RR)$, with singular value decomposition $M = U @@ -202,3 +204,78 @@ _Proof_ (of first equality): & <= sigma_i^2 $ +*Theorem* (Interlacing of singular values): For a matrix $M_1$, we denote by $sigma_i (M)$ the $i$-th singular value of +$M$. $M, B in MM_(n times m) (RR)$ and $rk(B) <= r$. Then +$ + sigma_(i - r) (M) underbrace(>=, i > r) sigma_i (M + B) >= sigma_(i + r) (M) +$ +_Proof_ (of second inequality, uses Min-Max): We will construct $E subset.eq RR^m, dim(E) = i$ such that +$ + forall v in E, norm(v) = 1 : norm((M + B)v) >= sigma_(i + r)(M) +$ +Note that $dim(underbrace(ker, {v : B v = 0})(B)) >= m - r$. Let us take +$ + F & = sp{v_1, dots, v_(i + r)} \ + E & = F inter ker(B) \ + dim(E) & = dim(F inter ker(B)) \ + & >= dim(F) + dim(ker(B)) - m \ + & >= (i + r) + (m - r) - m \ + & = i +$ +We may show $forall v in E : norm((M + B) v) >= sigma_(i + r)(m)$ +- $(M_B)v = M v$ since $v in E subset.eq ker(B)$ +- $norm(M v) >= sigma_(i + r)$ since $v in sp{v_1, dots, v_(i + r)}$ + + +#pagebreak() +*Theorem*: $M = U Sigma V^T$. For every matrix $B$ of rank $<= r$, it holds that $norm(M - B) >= norm( + M - U Sigma_((r)) + V^T +)$, for both the Frobenius, and the operator norm: +$ + norm(M - B)_F & >= norm(M - U Sigma_((r)) V^T)_F \ + & = (sum_(i = r + 1)^(k)sigma_i^2)^(1 / 2) \ + norm(M - B)_"op" & >= norm(M - U Sigma_((r)) V^T)_"op" \ + & = sigma_(r + 1) +$ +Both inequalities are sharp (meaning that we may find $B$ such that they are equivalent, where $B = U Sigma_r V^T$, and +is left as an exercise) \ +_Proof_: +$ + sigma_i (M - B) & >= sigma_(i + r) (M) \ + norm(M - B)_"op" & = sigma_1 (M - B) \ + & >= sigma_(r + 1) (M) \ + norm(M - B)_F^2 & = sum_(i = 1)^(k)sigma_i (M - B)^2 \ + & >= sum_(i = 1)^(k)sigma_(i + r) (M)^2 \ + & = sum_(i = r + 1)^(k)sigma_i^2 +$ + +#pagebreak() += Linear PCA + +Let there be vectors $x_1, dots, x_n in RR^n$. Let there be linear maps $B : RR^n -> RR^r$, and $A : RR^r -> RR^n$. We +want to find $A, B$, that minimise +$ + sum_(i = 1)^(m)norm(A B x_i - x_i) +$ +The best way to do this is to perform SVD on the data. Let +$ + X = mat(x_1, dots, x_m) = U Sigma V^T +$ +I.e., create a matrix $X$, where column $i$ is the vector $x_i$. Additionally, let $P = A B$, which is of rank $<= r$. +So +$ + sum_(i = 1)^(m)norm(A B x_i - x_i) & = norm(X - P X)_F^2 \ + "Solution:" P X & = U Sigma_((r)) V^T \ + P U Sigma V^T & = U Sigma_((r)) V^T \ + P U Sigma & = U Sigma_((r)) \ + U^T P U Sigma & = Sigma_((r)) \ + U^T A B U Sigma & = Sigma_((r)) \ + ==> A = mat(u_1, dots, u_r) quad & quad B = mat(u_1^T; dots.v; u_r^T) \ +$ +So +$ + P & = A dot B & = mat(u_1, dots, u_r) mat(u_1^T; dots.v; u_r^T) \ +$ +which is the projection on $sp{u_1, dots, u_r}$ + |