Building Foundation Models for Time Series: Lessons from NLP and What's Different

Why Foundation Models for Time Series?

The success of large language models in NLP has inspired a compelling question: can we build analogous general-purpose pre-trained models for time series data? The appeal is obvious — a single model that can forecast, classify, detect anomalies, and impute missing values across diverse domains, with minimal task-specific fine-tuning.

What We Borrowed from NLP

TSFounder borrows the masked pre-training paradigm from BERT: we randomly mask time windows and train the model to reconstruct them from surrounding context. We also borrow the patching idea from PatchTST, treating contiguous temporal segments as "tokens" to reduce sequence length while preserving local temporal structure.

What Makes Time Series Harder

Unlike text, time series data is continuous, multi-scale, and domain-heterogeneous. A model pre-trained on ECG signals and financial price data must somehow develop shared representations that generalize to weather forecasting and industrial sensor data. We address this through domain-adaptive normalization layers and curriculum training across heterogeneous corpora.

Visual Forecasting: An Unexpected Finding

One surprising result from our lab: converting time series into structured image encodings — essentially plot images with consistent formatting — allows vision transformers to perform competitive long-horizon forecasting. Our VisForecast system explores this direction systematically.

What's Next: Text-Based Time Series

An emerging direction we are actively exploring is text-based time series understanding: can language models reason directly over numerical sequences described in text, or over time series paired with textual metadata? Early experiments suggest substantial untapped potential.