Mathematical Derivation: Cholesky Whitening Implementation

This document details the Cholesky Whitening method implemented in Cholesky.R. This method is computationally efficient and results in a lower-triangular transformation matrix (in the context of the whitening operation), which induces a specific ordering dependency among features.

1. Core Concept

Cholesky decomposition factorizes a Symmetric Positive Definite (SPD) matrix into the product of a triangular matrix and its transpose. It is generally faster than eigendecomposition but is not scale-invariant or rotation-invariant (permutation of channels affects the result).

2. Mathematical Formulation

Step 1: Decomposition

Given the covariance matrix $\Sigma$, we compute the Cholesky factorization. In R, the chol() function returns an Upper Triangular matrix $R$ such that:

Note: In standard math texts, Cholesky is often defined as $\Sigma = L L^\top$ where $L$ is lower triangular. R returns the upper triangular factor $R = L^\top$.

R Code Correspondence:

chol_Sigma <- chol(Sigma)  # R, upper triangular

Step 2: Constructing the Whitening Matrix

With $L = R^\top$, the whitening matrix is the inverse of the lower-triangular factor:

Then $\text{Cov}(Z) = L^{-1} \Sigma L^{-\top} = L^{-1} L L^\top L^{-\top} = I$. Because $L^{-1}$ is also lower triangular, the $j$-th whitened component depends only on channels $1, \dots, j$: the first channel is simply rescaled, and each later channel has the earlier ones regressed out before being rescaled. This is the ordering dependency mentioned above.

R Code Correspondence:

# Solve R^T W = I by forward substitution
W <- backsolve(chol_Sigma, diag(ncol(Sigma)), transpose = TRUE)