6 ms·
Interesting, but I'm very skeptical. There are over a dozen transformers-based foundation time series model released in the past year and without fail, every on
by longdog 2y ago
Interesting, but I'm very skeptical. There are over a dozen transformers-based foundation time series model released in the past year and without fail, every one of them claims to be at or near SOTA. For example:
- Time-LLM (https://arxiv.org/abs/2310.01728 https://arxiv.org/abs/2310.01728)
- Lag-Llama (https://arxiv.org/abs/2310.08278 https://arxiv.org/abs/2310.08278)
- UniTime (https://arxiv.org/abs/2310.09751 https://arxiv.org/abs/2310.09751)
- TEMPO (https://arxiv.org/abs/2310.04948 https://arxiv.org/abs/2310.04948)
- TimeGPT (https://arxiv.org/abs/2310.03589 https://arxiv.org/abs/2310.03589)
- TimesFM (https://arxiv.org/html/2310.10688v2 https://arxiv.org/html/2310.10688v2)
- GPT4TS (https://arxiv.org/pdf/2308.08469.pdf https://arxiv.org/pdf/2308.08469.pdf)
Yet not a SINGLE transformer-based model I've managed to successfully run has beaten gradient boosted tree models on my use case (economic forecasting). To be honest I believe these foundational models are all vastly overfit. There's basically only 2 benchmarking sets that are ever used in time series (the Monash set and the M-competition set), so it'd be easy to overtune a model just to perform well on these.
I would love to see someone make a broader set of varied benchmarks and have an independent third party do these evaluations like with LLM leaderboards. Otherwise I assume all published benchmarks are 100% meaningless and gamed.
- dcl 2y agoWhy would you expect anything to work well for economic forecasting :p
- donbreo 2y agoJamie pull up the article that proves none of the published models work well with economic forecasting
- rokkitmensch 2y agoI'm so sad. This hilarious comment is languishing in the doldrums.
- idiotsecant 2y agoNot reddit.
- ImHereToVote 2y agoThere is always Gary Stevensons Economics model. Works without fail.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- hackerlight 2y agoNeural nets are known to struggle with tabular data. Have you tried fine tuning or attaching a decoder somewhere that you train on your task? Zero-shot inference might be asking for too much.
- boredemployee 2y ago>> Neural nets are known to struggle with tabular data. Not disagreeing with you, and I'm not a specialist, but it's funny that lot of papers seem to claim exactly the opposite.
- hackerlight 2y agoWhat paper says the opposite? This is what I can find: https://arxiv.org/abs/2207.08815 https://arxiv.org/abs/2207.08815 https://arxiv.org/abs/2305.02997 https://arxiv.org/abs/2305.02997
- logicchains 2y agoPretty much any real-world time series prediction task is going to involve more data than just the time series itself, and some of this data will probably be tabular, so it's not surprise gradient boosted trees perform better.
- Tarq0n 2y agoHonestly the best part of this paper is they've put together a large new set of time series for benchmarking.
- tudorw 2y agohttps://facebook.github.io/prophet/ https://facebook.github.io/prophet/ "Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects. It works best with time series that have strong seasonal effects and several seasons of historical data. Prophet is robust to missing data and shifts in the trend, and typically handles outliers well."
- refulgentis 2y ago?