5 ms·

Most software innovations seem to be made to make the developer's life easier instead of creating more performant software. I'm not complaining but user focused

Most software innovations seem to be made to make the developer's life easier instead of creating more performant software. I'm not complaining but user focused software innovations like LLMs do seem to come very few and far between compared to hardware.

wahahah 3y ago
I wouldn't consider LLMs a software innovation.
- karlshea 3y ago
  I feel like if you're gonna drop a take like that you should have at least one more sentence of justification to follow it up with.
  - hkt 3y ago
    I kind of admire the gumption, personally. Like writing them off as fancy markov bots.
  - Joeri 3y ago
    I’m not sure I agree, but I’ll do my best to argue the position. The neural net software algorithms have been around for decades. What made LLM’s feasible are the hardware advances to achieve unprecedented scale, just barely providing the ability (at great cost) to train today’s LLM models. Transformer architecture might be called a software innovation, but RWKV Raven gets similar performance to transformers and is built on decades-old RNN technology. So it is the hardware that was far more instrumental than the software in achieving LLM’s. Counter to that argument: had google not done neural net research for google translate and proved the transformer approach scaled and performed well in their “attention is all you need” paper, people wouldn’t have spent the money to train foundation LLM models and we would not be having this discussion, so the software really mattered more than the hardware. In reality I think it’s a little bit of both.
    - Shorel 3y ago
      Great take. It is really a mix of several factors, each one leveraging the other, and your arguments are great.
    - NicoJuicy 3y ago
      That's not necessarily true. The algorithm ( transformers) was done by Google in 2017. Even while they missed the opportunity, Gpt-1 was released in 2018 by OpenAI. Then they incrementally added parameters in the next versions.
    - karlshea 3y ago
      I don’t think I totally agree either but your argument for the position definitely has some merit.

wahahah 3y ago

I wouldn't consider LLMs a software innovation.

karlshea 3y ago
I feel like if you're gonna drop a take like that you should have at least one more sentence of justification to follow it up with.
- hkt 3y ago
  I kind of admire the gumption, personally. Like writing them off as fancy markov bots.
- Joeri 3y ago
  I’m not sure I agree, but I’ll do my best to argue the position. The neural net software algorithms have been around for decades. What made LLM’s feasible are the hardware advances to achieve unprecedented scale, just barely providing the ability (at great cost) to train today’s LLM models. Transformer architecture might be called a software innovation, but RWKV Raven gets similar performance to transformers and is built on decades-old RNN technology. So it is the hardware that was far more instrumental than the software in achieving LLM’s. Counter to that argument: had google not done neural net research for google translate and proved the transformer approach scaled and performed well in their “attention is all you need” paper, people wouldn’t have spent the money to train foundation LLM models and we would not be having this discussion, so the software really mattered more than the hardware. In reality I think it’s a little bit of both.
  - Shorel 3y ago
    Great take. It is really a mix of several factors, each one leveraging the other, and your arguments are great.
  - NicoJuicy 3y ago
    That's not necessarily true. The algorithm ( transformers) was done by Google in 2017. Even while they missed the opportunity, Gpt-1 was released in 2018 by OpenAI. Then they incrementally added parameters in the next versions.
  - karlshea 3y ago
    I don’t think I totally agree either but your argument for the position definitely has some merit.