5 ms·

Out of curiosity, what do you work on where you don’t have to experiment with different solutions to see what works best?

FridgeSeal 1y ago
Usually when we’re doing it in practice there’s _somewhat_ more awareness of the mechanics than just throwing random obstructions in and hoping for the best.
- RussianCow 1y ago
  LLMs are still very young. We'll get there in time. I don't see how it's any different than optimizing for new CPU/GPU architectures other than the fact that the latter is now a decades-old practice.
  - girvo 1y ago
    > I don't see how it's any different than optimizing for new CPU/GPU architectures I mean that seems wild to say to me. Those architectures have documentation and aren't magic black boxes that we chuck inputs at and hope for the best: we do pretty much that with LLMs. If that's how you optimise, I'm genuinely shocked.
    - swyx 1y ago
      i bet if we talked to a real low level hardware systems/chip engineer they'd laugh and take another shot at how we put them on a pedestal
      - girvo 1y ago
        Not really, in my experience. There's still fundamental differences between designed systems and trained LLMs.
  - th0ma5 1y ago
    Not to pick on you, but this is exactly the objectionable handwaving. What makes you think we'll get there? The kinds of errors that these technologies make have not changed, and anything that anyone learns about how to make them better changes dramatically from moment to moment and no one can really control that. It is different because those other things were deterministic ...
    - Closi 1y ago
      In comp sci it’s been deterministic, but in other science disciplines (eg medicine) it’s not. Also in lots of science it looks non-deterministic until it’s not (eg medicine is theoretically deterministic, but you have to reason about it experimentally and with probabilities - doesn’t mean novel drugs aren’t technological advancements). And while the kind of errors hasn’t changed, the quantity and severity of the errors has dropped dramatically in a relatively short span of time.
      - th0ma5 1y ago
        The problem has always been that every token is suspect.
        Closi 1y ago
        It's the whole answer being correct that's the important thing, and if you compare GPT 3 vs where we are today only 5 years later the progress in accuracy, knowledge and intelligence is jaw dropping.
        th0ma5 1y ago
        I have no idea what you're talking about because they still screw up in the exact same way as gpt3.
        Closi 1y ago
        The hallucination quantity and severity is way less in new frontier models.
greenchair 1y ago
most people are building straightforward crud apps. no experimentation required.
- RussianCow 1y ago
  [citation needed] In my experience, even simple CRUD apps generally have some domain-specific intricacies or edge cases that take some amount of experimentation to get right.
- brokencode 1y ago
  Idk, it feels like this is what you’d expect versus the actual reality of building something. From my experience, even building on popular platforms, there are many bugs or poorly documented behaviors in core controls or APIs. And performance issues in particular can be difficult to fix without trial and error.
  - karn97 1y ago
    Not helpful when the llm knowledge cutoff is a year out of date and api and lib has been changed since

FridgeSeal 1y ago

Usually when we’re doing it in practice there’s _somewhat_ more awareness of the mechanics than just throwing random obstructions in and hoping for the best.

RussianCow 1y ago
LLMs are still very young. We'll get there in time. I don't see how it's any different than optimizing for new CPU/GPU architectures other than the fact that the latter is now a decades-old practice.
- girvo 1y ago
  > I don't see how it's any different than optimizing for new CPU/GPU architectures I mean that seems wild to say to me. Those architectures have documentation and aren't magic black boxes that we chuck inputs at and hope for the best: we do pretty much that with LLMs. If that's how you optimise, I'm genuinely shocked.
  - swyx 1y ago
    i bet if we talked to a real low level hardware systems/chip engineer they'd laugh and take another shot at how we put them on a pedestal
    - girvo 1y ago
      Not really, in my experience. There's still fundamental differences between designed systems and trained LLMs.
- th0ma5 1y ago
  Not to pick on you, but this is exactly the objectionable handwaving. What makes you think we'll get there? The kinds of errors that these technologies make have not changed, and anything that anyone learns about how to make them better changes dramatically from moment to moment and no one can really control that. It is different because those other things were deterministic ...
  - Closi 1y ago
    In comp sci it’s been deterministic, but in other science disciplines (eg medicine) it’s not. Also in lots of science it looks non-deterministic until it’s not (eg medicine is theoretically deterministic, but you have to reason about it experimentally and with probabilities - doesn’t mean novel drugs aren’t technological advancements). And while the kind of errors hasn’t changed, the quantity and severity of the errors has dropped dramatically in a relatively short span of time.
    - th0ma5 1y ago
      The problem has always been that every token is suspect.
      - Closi 1y ago
        It's the whole answer being correct that's the important thing, and if you compare GPT 3 vs where we are today only 5 years later the progress in accuracy, knowledge and intelligence is jaw dropping.
        th0ma5 1y ago
        I have no idea what you're talking about because they still screw up in the exact same way as gpt3.
        Closi 1y ago
        The hallucination quantity and severity is way less in new frontier models.

greenchair 1y ago

most people are building straightforward crud apps. no experimentation required.

RussianCow 1y ago
[citation needed] In my experience, even simple CRUD apps generally have some domain-specific intricacies or edge cases that take some amount of experimentation to get right.
brokencode 1y ago
Idk, it feels like this is what you’d expect versus the actual reality of building something. From my experience, even building on popular platforms, there are many bugs or poorly documented behaviors in core controls or APIs. And performance issues in particular can be difficult to fix without trial and error.
- karn97 1y ago
  Not helpful when the llm knowledge cutoff is a year out of date and api and lib has been changed since