10 ms·

Figure 3 on p.40 of the paper seems to show that their LLM based model does not statistically significantly outperform a 3 layer neural network using 59 variabl

Figure 3 on p.40 of the paper seems to show that their LLM based model does not statistically significantly outperform a 3 layer neural network using 59 variables from 1989. This figure compares the prediction performance of GPT and quantitative models based on machine learning. Stepwise Logistic follows Ou and Penman (1989)’s structure with their 59 financial predictors. ANN is a three-layer artificial neural network model using the same set of variables as in Ou and Penman (1989). GPT (with CoT) provides the model with financial statement information and detailed chain-of-thought prompts. We report average accuracy (the percentage of correct predictions out of total predictions) for each method (left) and F1 score (right). We obtain bootstrapped standard errors by randomly sampling 1,000 observations 1,000 times and include 95% confidence intervals.

infecto 2y ago
Was going to point out the same. Glad to have the paper to read but I don't think the findings are significant.
- foota 2y ago
  I agree this isn't earth shattering, but I think the benefit here is that it's a general solution instead of one trained on financial statements specifically.
  - _se 2y ago
    That is not a benefit. If you use a tool like this to try to compete with sophisticated actors (e.g. all major firms in the capital markets space) you will lose every time.
    - foota 2y ago
      We come up with all sorts of things that are initially a step backwards, but that lead to eventual improvement. The first cars were slower than horses. That's not to suggest that Renaissance is going to start using Chat GPT tomorrow, but maybe in a few years they'll be using fine tuned versions of LLMs in addition to whatever they're doing today. Even if it's not going to compete with the state of the art models for something, a single model capable of many things is still useful, and demonstrating domains where they are applicable (if not state of the art) is still beneficial.
      - ecjhdnc2025 2y ago
        Far too much in the way of "maybe in a few years" LLM prediction relies on the unspoken assumption that there will not be any gains in the state of the art in the existing, non-LLM tools. "In a few years" you'd have the benefit of the current, bespoke tools, plus all the work you've put into improving them in the meantime. And the LLM would still be behind, unless you believe that at some point in the future, a radically better solution will simply emerge from the model. That is, the bet is that at some point, magic emerges from the machine that renders all domain-specialist tooling irrelevant, and one or two general AI companies can hoover up all sorts of areas of specialism. And in the meantime, they get all the investment money. Why is it that we wouldn't trust a generalist over a specialist in any walk of life, but in AI we expect one day to be able to?
        z7 2y ago
        >Why is it that we wouldn't trust a generalist over a specialist in any walk of life, but in AI we expect one day to be able to? The specialist is a result of his general intelligence though.
        Terr_ 2y ago
        > That is, the bet is that at some point, magic emerges from the machine that renders all domain-specialist tooling irrelevant, and one or two general AI companies I have a slightly more cynical take: Those LLMs are not actually general models, but niche specialists on correlated text-fragments. This means human exuberance is riding on the (questionable) idea that a really good text-correlation specialist can effectively impersonate a general AI. Even worse: Some people assume an exceptional text-specialist model will effectively meta-impersonate a generalist model impersonating a different kind of specialist!
        ecjhdnc2025 2y ago
        > Even worse: Some people assume an exceptional text-specialist model will effectively meta-impersonate a generalist model impersonating a different kind of specialist! Eloquently put :-)
    - fennecbutt 2y ago
      If you don't look, you will never see.
  - yaj54 2y ago
    agreed. most people can't create a custom tailored finance statement model. but many people can write the following sentence: "analyze this financial statement and suggest a market strategy." and if that sentence performs as well as an (albeit old) custom model, and is likely to have compound improvements in its performance over time with no changes to the instruction sentence...
    - TechDebtDevin 2y ago
      No
    - ecjhdnc2025 2y ago
      But it can't come up with a particularly imaginative strategy; it can only come up with a mishmash of existing stuff it has seen, equivocate, or hallucinate a strategy that looks clever but might not be. So it all needs checking. It's the classic LLM situation. If you're trained enough to spot the errors, the analysis wouldn't take you much time in the first place. And if you're not trained enough to spot the errors... And let's say it does work. It's like automated exchange betting robots. As soon as everyone has access to a robot that can exploit some hidden pattern in the data for a tiny marginal gain, the price changes and the gain collapses. So if everyone has the same access to the same banal, general analysis tools, you know what's going to happen: the advantage disappears. All in all, why would there be any benefits from a generalised model?
    - jimbokun 2y ago
      "buy and hold the S&P 500 until you're ready to retire"
      - chronic94057 2y ago
        > "buy and hold the S&P 500 until you're ready to retire" That is bad advice. VGT Vanguard Technology ETF has outperformed S&P 500 over the past 20 years. All the people who say “VTSAX and chill” disappeared in the past 3-4 years because their cherished total passive index fund is no longer the best over long horizons. And no, the markets are not efficient.
flourpower471 2y ago
Not to mention, as somebody who works in quant trading doing ml all day on this kind of data. That ann benchmark is nowhere near state of the art. People didn't stop working on this in 1989 - they realised they can make lots of money doing it and do it privately.
- bethekind 2y ago
  Do you use llama 3 for your work?
  - posting_mess 2y ago
    No hedge fund registered before the last 2 weeks will use Llama3 for their "prod work" beyond "experiments". Quant trading is about "going fast" or "being super right", so either you'd need to be sitting on some huge llama.cpp/transformer improvement (possible but unlikely) or its more likely just some boring math applied faster than others. Even if they are using a "LLM", they wont tell you or even hint at it - "efficient market" n all that. Remember all quants need to be "the smartest in the world" or their whole industry falls apart, wait till you find out its all "high school math" based on algo's largely derived 30/40 years ago (okay not as true for "quants" but most "trading" isn't as complex as they'd like you/us to believe).
    - qeternity 2y ago
      It’s impressive how incorrect so much of this information is. High frequency trading is about going fast. There is a huge mid and low freq quant industry. Also most quant strategies are absolutely not about being “super right”…that would be the province of concentrated discretionary strategies. Quant is almost always about being slightly more right than wrong but at large scale. What algos are you referring to derived 30 or 40 years ago? Do you understand the decay for a typical strategy? None of this makes any sense.
      - posting_mess 2y ago
        Quantitative trading is simply the act of trading on data, fast or slowly, but I'll grant you for the more sophisticated audience there is a nuance between "HFT" and "Quant" trading. To be "super right" you just have to make money over a timeline, you set, according to your own models. If I choose a 5 year timeline for a portfolio, I just have to show my portfolio outperforming "your preferred index here" over that timeline - simple (kind of, I ignore other metrics than "make me money" here). Depending on what your trading will depend on which algo's you will use, the way to calculate the price of an Option/Derivative hasn't changed in my understanding for 20/30 years - how fast you can calculate, forecast, and trade on that information has. My statement wont hold true in a conversation with an "investing legend", but to the audiance who asks "do you use llama3" its clearly an appropriate response.
tempodox 2y ago
But I bet it uses way more energy.
richrichie 2y ago
The infamous 1/N portfolio comparison is missing. 1/N puts to shame many strategies.

infecto 2y ago

Was going to point out the same. Glad to have the paper to read but I don't think the findings are significant.

foota 2y ago
I agree this isn't earth shattering, but I think the benefit here is that it's a general solution instead of one trained on financial statements specifically.
- _se 2y ago
  That is not a benefit. If you use a tool like this to try to compete with sophisticated actors (e.g. all major firms in the capital markets space) you will lose every time.
  - foota 2y ago
    We come up with all sorts of things that are initially a step backwards, but that lead to eventual improvement. The first cars were slower than horses. That's not to suggest that Renaissance is going to start using Chat GPT tomorrow, but maybe in a few years they'll be using fine tuned versions of LLMs in addition to whatever they're doing today. Even if it's not going to compete with the state of the art models for something, a single model capable of many things is still useful, and demonstrating domains where they are applicable (if not state of the art) is still beneficial.
    - ecjhdnc2025 2y ago
      Far too much in the way of "maybe in a few years" LLM prediction relies on the unspoken assumption that there will not be any gains in the state of the art in the existing, non-LLM tools. "In a few years" you'd have the benefit of the current, bespoke tools, plus all the work you've put into improving them in the meantime. And the LLM would still be behind, unless you believe that at some point in the future, a radically better solution will simply emerge from the model. That is, the bet is that at some point, magic emerges from the machine that renders all domain-specialist tooling irrelevant, and one or two general AI companies can hoover up all sorts of areas of specialism. And in the meantime, they get all the investment money. Why is it that we wouldn't trust a generalist over a specialist in any walk of life, but in AI we expect one day to be able to?
      - z7 2y ago
        >Why is it that we wouldn't trust a generalist over a specialist in any walk of life, but in AI we expect one day to be able to? The specialist is a result of his general intelligence though.
      - Terr_ 2y ago
        > That is, the bet is that at some point, magic emerges from the machine that renders all domain-specialist tooling irrelevant, and one or two general AI companies I have a slightly more cynical take: Those LLMs are not actually general models, but niche specialists on correlated text-fragments. This means human exuberance is riding on the (questionable) idea that a really good text-correlation specialist can effectively impersonate a general AI. Even worse: Some people assume an exceptional text-specialist model will effectively meta-impersonate a generalist model impersonating a different kind of specialist!
        ecjhdnc2025 2y ago
        > Even worse: Some people assume an exceptional text-specialist model will effectively meta-impersonate a generalist model impersonating a different kind of specialist! Eloquently put :-)
  - fennecbutt 2y ago
    If you don't look, you will never see.
- yaj54 2y ago
  agreed. most people can't create a custom tailored finance statement model. but many people can write the following sentence: "analyze this financial statement and suggest a market strategy." and if that sentence performs as well as an (albeit old) custom model, and is likely to have compound improvements in its performance over time with no changes to the instruction sentence...
  - TechDebtDevin 2y ago
    No
  - ecjhdnc2025 2y ago
    But it can't come up with a particularly imaginative strategy; it can only come up with a mishmash of existing stuff it has seen, equivocate, or hallucinate a strategy that looks clever but might not be. So it all needs checking. It's the classic LLM situation. If you're trained enough to spot the errors, the analysis wouldn't take you much time in the first place. And if you're not trained enough to spot the errors... And let's say it does work. It's like automated exchange betting robots. As soon as everyone has access to a robot that can exploit some hidden pattern in the data for a tiny marginal gain, the price changes and the gain collapses. So if everyone has the same access to the same banal, general analysis tools, you know what's going to happen: the advantage disappears. All in all, why would there be any benefits from a generalised model?
  - jimbokun 2y ago
    "buy and hold the S&P 500 until you're ready to retire"
    - chronic94057 2y ago
      > "buy and hold the S&P 500 until you're ready to retire" That is bad advice. VGT Vanguard Technology ETF has outperformed S&P 500 over the past 20 years. All the people who say “VTSAX and chill” disappeared in the past 3-4 years because their cherished total passive index fund is no longer the best over long horizons. And no, the markets are not efficient.

flourpower471 2y ago

Not to mention, as somebody who works in quant trading doing ml all day on this kind of data. That ann benchmark is nowhere near state of the art. People didn't stop working on this in 1989 - they realised they can make lots of money doing it and do it privately.

bethekind 2y ago
Do you use llama 3 for your work?
- posting_mess 2y ago
  No hedge fund registered before the last 2 weeks will use Llama3 for their "prod work" beyond "experiments". Quant trading is about "going fast" or "being super right", so either you'd need to be sitting on some huge llama.cpp/transformer improvement (possible but unlikely) or its more likely just some boring math applied faster than others. Even if they are using a "LLM", they wont tell you or even hint at it - "efficient market" n all that. Remember all quants need to be "the smartest in the world" or their whole industry falls apart, wait till you find out its all "high school math" based on algo's largely derived 30/40 years ago (okay not as true for "quants" but most "trading" isn't as complex as they'd like you/us to believe).
  - qeternity 2y ago
    It’s impressive how incorrect so much of this information is. High frequency trading is about going fast. There is a huge mid and low freq quant industry. Also most quant strategies are absolutely not about being “super right”…that would be the province of concentrated discretionary strategies. Quant is almost always about being slightly more right than wrong but at large scale. What algos are you referring to derived 30 or 40 years ago? Do you understand the decay for a typical strategy? None of this makes any sense.
    - posting_mess 2y ago
      Quantitative trading is simply the act of trading on data, fast or slowly, but I'll grant you for the more sophisticated audience there is a nuance between "HFT" and "Quant" trading. To be "super right" you just have to make money over a timeline, you set, according to your own models. If I choose a 5 year timeline for a portfolio, I just have to show my portfolio outperforming "your preferred index here" over that timeline - simple (kind of, I ignore other metrics than "make me money" here). Depending on what your trading will depend on which algo's you will use, the way to calculate the price of an Option/Derivative hasn't changed in my understanding for 20/30 years - how fast you can calculate, forecast, and trade on that information has. My statement wont hold true in a conversation with an "investing legend", but to the audiance who asks "do you use llama3" its clearly an appropriate response.

tempodox 2y ago

But I bet it uses way more energy.

richrichie 2y ago

The infamous 1/N portfolio comparison is missing. 1/N puts to shame many strategies.