Lecture 1-2: Mathematical Preliminaries - Part 2

Lecture 1-2: Mathematical Preliminaries - Part 2#

Download the original slides: CMSE382-Lec1_2.pdf

Warning

This is an AI-generated transcript of the lecture slides and may contain errors or inaccuracies. Please refer to the original course materials for authoritative content.

Eigenvalues and Eigenvectors#

Definition: Given a square matrix \(A \in \mathbb{R}^{n \times n}\), a scalar \(\lambda \in \mathbb{R}\) is an eigenvalue of \(A\) if there exists a non-zero vector \(v \in \mathbb{R}^n\) such that

\[Av = \lambda v.\]

The vector \(v\) is called an eigenvector corresponding to the eigenvalue \(\lambda\).

Solving for eigenvalues and eigenvectors:

  1. Rewrite as \((A - \lambda I)v = 0\).

  2. For non-trivial solutions: \(\det(A - \lambda I) = 0\).

  3. Solve for \(\lambda\) (characteristic polynomial).

  4. Plug \(\lambda\) back in to solve for \(v\).

Other values:

  • The spectrum of \(A\) is the set of all eigenvalues of \(A\).

  • \(\text{trace}(A) = \sum_{i=1}^n \lambda_i\).

  • \(\det(A) = \prod_{i=1}^n \lambda_i\).

Singular Values#

Definition (Singular values): Given a real matrix \(A \in \mathbb{R}^{m \times n}\), the singular values of \(A\) are defined as the square roots of the eigenvalues of \(A^TA\).

Properties of singular values:

  • If \(\sigma\) is a singular value of \(A\), then there exists a non-zero vector \(v \in \mathbb{R}^n\) such that \(A^TA v = \sigma^2 v\).

  • All singular values are non-negative, i.e., \(\sigma \geq 0\).

  • The number of non-zero singular values is equal to the rank of \(A\).

  • Singular values provide information about the scaling and stretching properties of the linear transformation represented by \(A\).

  • For a symmetric real matrix (\(A = A^T\)), the singular values are the absolute values of the eigenvalues.

Norms of Matrices#

Definition: A norm on the set of real matrices \(\mathbb{R}^{m \times n}\) is a function \(\|\cdot\| : \mathbb{R}^{m \times n} \to \mathbb{R}\) that satisfies for all \(\alpha \in \mathbb{R}\) and \(A, B \in \mathbb{R}^{m \times n}\) the following:

  • \(\|A\| \geq 0\) (positive-valued)

  • \(\|A\| = 0\) if and only if \(A = \mathbf{0}_{m,n}\)

  • \(\|\alpha A\| = |\alpha| \|A\|\) (positive homogeneity)

  • \(\|A + B\| \leq \|A\| + \|B\|\) (triangle inequality)

Induced Matrix Norm#

Definition (Induced matrix norm): Given a real matrix \(A \in \mathbb{R}^{m \times n}\), as well as the norms \(\|\cdot\|_a\) and \(\|\cdot\|_b\) defined on \(\mathbb{R}^n\) and \(\mathbb{R}^m\), respectively, define the induced matrix norm

\[\|A\|_{a,b} = \max_{\|x\|_a = 1} \|Ax\|_b.\]

Notes:

  1. For \(L_p\) norms, if \(a=b=p\), we write \(\|A\|_p\) instead of \(\|A\|_{p,p}\).

  2. This norm returns the maximum of all linear transformations (measured using \(b\)) of input vectors of unit length (as measured by \(a\)).

  3. It measures the maximum “stretching” of unit vectors (as measured by \(a\)) in \(\mathbb{R}^n\) due to the action of \(A\).

Examples for \(p\)-Norms#

Definition: The matrix \(p\)-norm for \(A \in \mathbb{R}^{m \times n}\) is induced by the corresponding \(p\)-norm for vectors in \(\mathbb{R}^n\) and \(\mathbb{R}^m\):

\[\|A\|_p = \max_{\|x\|_p = 1} \|Ax\|_p.\]

Examples:

  • \(p = 1\): max absolute column sum, \(\displaystyle\|A\|_1 = \max_{1 \leq j \leq n} \sum_{i=1}^m |a_{ij}|.\)

  • \(p = 2\): largest singular value of \(A\), i.e., square root of largest eigenvalue of \(A^TA\): \(\displaystyle\|A\|_2 = \sigma_{\max}(A) = \sqrt{\lambda_{\max}(A^TA).}\)

    • Called the spectral norm; default assumption for \(\|A\|\) if no subscript is given.

  • \(p = \infty\): max absolute row sum, \(\displaystyle\|A\|_\infty = \max_{1 \leq i \leq m} \sum_{j=1}^n |a_{ij}|.\)

Non-Induced Matrix Norms: Frobenius Norm#

Definition (Frobenius norm): The Frobenius norm for \(A \in \mathbb{R}^{m \times n}\) is defined as

\[\|A\|_F = \sqrt{\sum_{i=1}^m \sum_{j=1}^n |a_{ij}|^2}.\]

Other versions:

  • \(\|A\|_F = \sqrt{\text{trace}(A^TA)}\)

  • \(\|A\|_F = \sqrt{\sum_{i=1}^{\min\{m,n\}} \sigma_i^2}\), where \(\sigma_i\) are the singular values of \(A\).

Non-Induced Matrix Norms: Nuclear Norm#

Definition (Nuclear norm): The nuclear norm (also called trace norm) for \(A \in \mathbb{R}^{m \times n}\) is defined as

\[\|A\|_* = \sum_{i=1}^{\min\{m,n\}} \sigma_i,\]

where \(\sigma_i\) are the singular values of \(A\).

Other versions:

  • \(\|A\|_* = \text{trace}(\sqrt{A^TA})\), where \(\sqrt{A^TA}=B\) is the positive semi-definite square root of \(A^TA\), so that \(BB = A^TA\).

Contents