Small Calibration Cohorts Do Not Reliably Select Motor-Imagery EEG Systems for New Participants
Under Review
Abstract
Motor-imagery EEG systems can use several decoders. Picking the most accurate one is not enough. It does not tell us when a second model is worth running, and a choice made from a few calibration participants may not hold for new ones. We studied this problem on three public datasets. Training, calibration, and test participants never overlapped. We compared three single decoders, six directed routes that called a second decoder on uncertain trials, and three ensembles. We scored each complete configuration by final accuracy and measured execution time. When we looked only at pairwise gain, the ranking sometimes favored a weaker complete system because it ignored the accuracy of the first decoder. Selections made from small calibration cohorts did not beat a fixed routed configuration on held-out participants. This result was clearest in the two larger datasets. EEGNet alone gave the highest accuracy. Routed systems cost less but were less accurate. Complementarity by itself does not identify which system should be deployed. System designers should change a configuration only when evidence shows that its gain carries over to unseen participants.
Why it matters
- Cross-participant nonstationarity: the same generalization problem studied in my dissertation across sessions appears across participants; a configuration ranking estimated on a few calibration users can reverse on new ones.
- Deployment decisions, not leaderboards: decoders are compared as complete execution paths (CSP+LDA, Riemannian tangent-space, EEGNet, uncertainty-routed pairs, ensembles) under final accuracy and measured compute cost, the way a system designer must choose between them.
- Practical rule: keep a fixed, well-tested configuration unless there is evidence that a switch’s gain transfers to unseen participants.
Keywords: motor-imagery EEG, brain-computer interface, cross-participant evaluation, system selection, conditional computation, calibration transfer