Google's New Time-Series Foundation Model
A new method for time series prediction with a decoder only transformer!
Created on February 5|Last edited on February 5
Comment
Time-series forecasting plays a critical role across multiple sectors, including retail, finance, and healthcare, where it drives decisions by predicting future trends and demands. Historically, this task has been challenging due to the complex nature of time-series data, which can be rich, multivariate, and variable across different domains.
NLP Advances
Recent advances in large foundation models for natural language processing tasks, such as translation and code completion, have demonstrated the potential of applying similar approaches to time-series forecasting. These NLP models are trained on extensive textual datasets, enabling them to identify patterns and make predictions, often without further training when applied to new tasks. This ability is particularly appealing for time-series forecasting, where DL models typically require extensive training and validation, making them less accessible for immediate application to new data.
Large Scale Training
Addressing this challenge, Google Research has introduced TimesFM, a new forecasting model specifically designed for time-series data. Unlike the larger language models that often require billions of parameters, TimesFM is a more compact model with 200 million parameters, pre-trained on a vast dataset of 100 billion real-world time-points. Despite its smaller size, TimesFM demonstrates remarkable zero-shot forecasting capabilities, closely matching or even surpassing state-of-the-art methods that require explicit training on target datasets.
The Architecture
TimesFM is based on a decoder-only architecture, similar to those used in NLP for tasks like text generation. It translates time-series data into a format that can be processed by transformer layers, which then predict future data points. A key innovation of TimesFM is its ability to handle variable input and output lengths, making it versatile across different forecasting horizons and domains. This flexibility is crucial for applications ranging from retail demand planning to financial forecasting, where the requirements can vary widely.
The model's architecture is designed to efficiently encode time-series patterns, using techniques like residual connections and positional encodings to enhance performance. It is trained to predict longer sequences of future data points from shorter inputs, a method that improves accuracy and reduces the potential for error accumulation in longer forecasts.
The Dataset
For its training, TimesFM leverages both synthetic data, to grasp the basic patterns of time-series, and real-world data, to understand more complex, real-world dynamics. This combination enables the model to perform well across a wide range of forecasting tasks, even when applied to data it has not seen during training. Additionally, large amounts of Google Search Trends and Wikipedia Pageviews data is used.
High Performance
In evaluations using benchmark datasets, TimesFM has shown to outperform traditional statistical methods and match or exceed the capabilities of advanced DL models, all without the need for dataset-specific training. This is particularly notable in domains like traffic, weather, and demand forecasting, where the model's zero-shot capabilities allow for immediate application to new challenges.
Closed Source
Planned for release on Google Cloud Vertex AI, TimesFM is a huge step forward in making advanced forecasting capabilities more accessible. By reducing the need for extensive model training and tuning, TimesFM enables organizations to more quickly and efficiently leverage their data for forecasting purposes, potentially transforming how decisions are made across a variety of fields.
Add a comment
Tags: ML News
Iterate on AI agents and models faster. Try Weights & Biases today.