LOSO versus Chronological Validation in Longitudinal BCI
Under Review · IEEE Access
Research question
When a decoder is updated from a participant’s earlier recordings, its evaluation should use the same training history. Leave-one-session-out (LOSO) validation trains on every other session, including later recordings. Chronological, expanding-window validation trains only on sessions preceding the test. These protocols answer different questions about cross-session performance.
Study design
The study compares both protocols on identical test sessions across six public motor-imagery EEG datasets and 138 participants. Common spatial pattern and tangent-space decoders use fixed configurations. Additional comparisons match the number of training sessions and examine decoder rankings and score margins.
Main findings
- Where LOSO had access to later sessions, its participant-weighted mean area under the ROC curve (AUC) exceeded chronological validation by 0.035, with a 95% confidence interval of 0.030–0.040.
- The difference was positive in each dataset, with variation in its size. Matching the number of training sessions reduced the difference substantially.
- The dataset-level order of the two decoders stayed the same. Participant-level ranking changes occurred mainly near ties.
The AUC difference reflects the training histories used by the two protocols. It is not a gain from a new decoding method.
Research software
The companion repository includes a chronological cross-validator for the Mother of All BCI Benchmarks (MOABB), saved results, and scripts for reconstructing figures and tables. It connects evaluation methodology to a reusable experiment workflow.