ICA Objective Functions in One Place: Kurtosis, Negentropy, MI, and MLE
6 minute read
Published:
This note consolidates my 2023-2024 group-meeting derivations into a single map:
data model,
objective functions,
and optimization choices.
Starting Point
Classical ICA assumes:
\[x = As,\quad y = Wx\]
where \(x\) is observed mixture, \(s\) is latent independent source, \(A\) is mixing matrix, and \(W\) is separating matrix.
Whitening already does half of the work. If \(z = Vx\) is whitened, so that \(\operatorname{Cov}(z) = I\), then every separating matrix whose outputs are uncorrelated with unit variance has the form \(W = RV\) with \(R\) orthogonal. Second-order statistics cannot tell these rotations apart. The objectives below are different ways of choosing \(R\).
source square2σ contourtarget: Cov = Icolumns of A
Cov of the points shown
1.000.000.001.00
\(\hat W A\), where \(\hat W\) is the transform so far
1.000.000.001.00
Each dot is a sample, coloured by its position in the source square so you can follow it. Whitening (steps 1–2) makes Cov(z) = I but leaves the square tilted: the dashed covariance contour is already a circle and cannot see the last rotation. That rotation has to come from a higher-order objective; here it is computed in the page by symmetric FastICA with the logcosh contrast. The second readout ends close to a signed permutation rather than I (here the two outputs come out swapped, one with flipped sign; the small residue is sampling error), because ICA recovers sources only up to order and sign.
where \(\nu \sim N(0, 1)\) is a standard Gaussian variable. This is the basis of the fixed-point FastICA family.
Objective 2: Minimize Mutual Information
For outputs \(y_i\):
\[I(y_1,\dots,y_m)=\sum_i H(y_i)-H(y).\]
Independence is equivalent to minimizing mutual information. This gives the information-theoretic view behind many ICA updates.
After whitening, this is the negentropy objective again. For an orthogonal \(R\), \(H(Rz) = H(z)\), so the joint term does not change with the rotation. If the outputs are uncorrelated with unit variance, every \(J(y_i)\) uses the same Gaussian reference, the standard Gaussian \(\nu\) above, and
\[I(y_1,\dots,y_m) = C - \sum_i J(y_i),\]
where \(C = m H(\nu) - H(z)\) does not depend on \(R\). Minimizing mutual information over rotations is maximizing total negentropy.
Gaussian sources are the exception. A rotated white Gaussian vector is still a white Gaussian vector, so every rotation gives the same distribution and no objective can prefer one. This is why ICA allows at most one Gaussian source.
In two dimensions the rotation is one angle. The figure below turns one direction \(w\) through whitened data and evaluates the objectives at every angle.
colour: angle in source spaceunit circletrue separating directionsw(θ)w⊥×outlier (wedge: off view)
Histogram of \(y = w^\top z\)
θ0°kurt(y)–Ĵ logcosh (×10−3)–Ĵ spacing (nats)–Î(y1, y2), pair with w⊥–
One output, \(y = w(\theta)^\top z\), \(\theta \in [0^\circ, 180^\circ)\)
Toy data: 4000 samples of two independent, unit-variance sources (600 drawn), mixed by the same \(A\) as the figure above and whitened the same way, \(z = \Lambda^{-1/2}U^\top x\). So \(w(\theta) = (\cos\theta, \sin\theta)\) is one row of a rotation \(R\), and with uniform sources the peaks of \(\lvert\mathrm{kurt}\rvert\), logcosh and spacing sit, up to sampling error, at the rows of the \(R\) that FastICA found above. Dashed lines mark the true separating directions, the rows of \((VA)^{-1}\). Dots mark the clear peaks of each curve, and the clear minimum in the pair panel: those that stand out from their surroundings by at least a tenth of the axis. Every curve has a fixed y-axis, so with Gaussian sources and no outliers they all stay flat: a rotated white Gaussian looks the same from every angle. With two uniform sources the signed kurtosis is negative everywhere and largest at the 45° mixtures, which is why ICA maximizes \(\lvert\mathrm{kurt}\rvert\). The logcosh curve is the approximation \((\mathbb{E}\,G(y) - \mathbb{E}\,G(\nu))^2\) with \(G = \log\cosh\) and \(\nu \sim N(0, 1)\), not negentropy itself. The spacing curve is Vasicek's entropy estimate (\(m = \sqrt N\)) subtracted from the same estimate on Gaussian samples of the same size, which removes its bias for Gaussian data. In the pair panel \(\hat H(z)\) does not depend on \(\theta\). Because the sources are simulated, it is estimated from them with the same estimator, \(\hat H(s_1) + \hat H(s_2) + \log\lvert\det VA\rvert\), and the two curves differ only by the constant \(C\). The outlier toggle replaces five samples with large values along one sensor direction and redoes the whitening. The largest \(\lvert\mathrm{kurt}\rvert\) moves away from the source directions. The logcosh peaks stay near them, but a uniform source near the outlier direction loses most of its logcosh peak, and what is left shifts. With a uniform source the spacing estimate drops as well, because the outliers widen the spacings in the tails.
Extended Infomax can be read as a likelihood-based ICA method with adaptive nonlinearities.
Algorithm Layer
A useful mental model:
Objective decides what independence means.
Algorithm decides how fast and stable we can optimize it.
Common choices:
gradient/natural-gradient updates,
FastICA fixed-point updates,
deflationary vs symmetric orthogonalization for multi-component extraction.
Practical Takeaway
In high-dimensional settings, algorithmic convenience can dominate statistical quality. FastICA is often a strong baseline, but the surrogate contrast function and orthogonalization strategy should be treated as design choices, not defaults.
