Study Reveals Time Series Foundation Models Hide Critical Failures in Traffic Forecasting

Researchers have identified a significant blind spot in how time series foundation models (TSFMs) are evaluated for traffic speed prediction. A new study submitted to arXiv demonstrates that standard aggregate benchmark metrics mask severe performance degradation during critical operating conditions, particularly when traffic transitions between free-flow and congested states.

The research applies regime-stratified evaluation to three different TSFMs tested on two standard traffic speed benchmarks. The findings reveal dramatic accuracy losses during regime transitions: mean absolute error (MAE) reaches 11 mph during transitions compared to just 3 mph overall, while the empirical coverage of 90% prediction intervals drops to as low as 55%. These failures remain invisible in traditional aggregate metrics because observations during free-flow conditions dominate the dataset, masking the poor performance during critical transitions.

Traffic dynamics exhibit abrupt regime switching between free-flow and congested states, producing bimodal speed distributions during transitions. Notably, a simple historical conditional baseline that samples from per-sensor training distributions achieves better transition coverage than any of the tested TSFMs, though it performs far worse on overall accuracy. This trade-off highlights a fundamental challenge in current evaluation methodologies.

To address this gap, the researchers propose bimodal mixture augmentation (BMA), a post-hoc method that combines TSFM forecasts with historical distributional knowledge. BMA approaches the historical baseline's superior transition coverage while preserving the TSFM's accuracy on aggregate metrics. The study concludes that existing TSFM benchmarks should incorporate regime-aware evaluation frameworks to expose failures that traditional aggregate metrics fail to capture, ensuring more robust model assessment for real-world applications.