5 ms·
So they used a LLM with knowledge cut in mid 2023 to evaluate 2023? Seems like a classic leakage problem. From paper: "testing set: January 1, 2023, to Decembe
by bguberfain 1y ago
So they used a LLM with knowledge cut in mid 2023 to evaluate 2023? Seems like a classic leakage problem.
From paper: "testing set: January 1, 2023, to December 31, 2023"
From the Llama 2 doc: "(...) some tuning data is more recent, up to July 2023."
- mhmmmmmm 1y agoRemoving the "Market expert" which uses OHLCV (Open, High, Low, Close, Volume) also drops the sharpee from 5.01 to 1.88 while also increasing the max draw down to 13.29% (v.s. 9.70% for the index). I'd be very surprised if the pre training of the base model was the only source of leakage...