8 ms·

O3 is multiple orders of magnitude more expensive to realize a marginal performance gain. You could hire 50 full time PhDs for the cost of using O3. You're wi

O3 is multiple orders of magnitude more expensive to realize a marginal performance gain. You could hire 50 full time PhDs for the cost of using O3. You're witnessing the blowoff top of the scaling hype bubble.

fspeech 2y ago
That's a very static view of the affairs. Once you have a master AI, at a minimum you can use it to train cheaper slightly less capable AIs. At the other end the master AI can train to become even smarter.
whynotminot 2y ago
What they’ve proven here is that it can be done. Now they just have to make it cheap. Tell me, what has this industry been good at since its birth? Driving down the cost of compute and making things more efficient. Are you seriously going to assume that won’t happen here?
- Jensson 2y ago
  > What they’ve proven here is that it can be done. No they haven't, these results do not generalize, as mentioned in the article: "Furthermore, early data points suggest that the upcoming ARC-AGI-2 benchmark will still pose a significant challenge to o3, potentially reducing its score to under 30% even at high compute" Meaning, they haven't solved AGI, and the task itself do not represent programming well, these model do not perform that well on engineering benchmarks.
  - whynotminot 2y ago
    Sure, AGI hasn’t been solved today. But what they’ve done is show that progress isn’t slowing down. In fact, it looks like things are accelerating. So sure, we’ll be splitting hairs for a while about when we reach AGI. But the point is that just yesterday people were still talking about a plateau.
    - peepeepoopoo97 2y ago
      About 10,000 times the cost for twice the performance sure looks like progress is slowing to me.
      - whynotminot 2y ago
        Just to be clear — your position is that the cost of inference for o3 will not go down over time (which would be the first time that has happened for any of these models).
        peepeepoopoo97 2y ago
        Even if compute costs drop by 10X a year (which seems like a gross overestimate IMO), you're still looking at 1000X the cost for a 2X annual performance gain. Costs outpacing progress is the very definition of diminishing returns.
        whynotminot 2y ago
        From their charts, o3 mini outperforms o1 using less energy. I don’t see the diminishing returns you’re talking about. Improvement outpacing cost. By your logic, perhaps the very definition of progress? You can also use the full o3 model, consume insane power, and get insane results. Sure, it will probably take longer to drive down those costs. You’re welcome to bet against them succeeding at that. I won’t be.
        peepeepoopoo97 2y ago
        [dead]
- peepeepoopoo97 2y ago
  Yes, that's exactly what I'm implying, otherwise they would have done it a long time ago, given that the fundamental transformer architecture hasn't changed since 2017. This bubble is like watching first year CS students trying to brute force homework problems.
  - whynotminot 2y ago
    > Yes, that's exactly what I'm implying, otherwise they would have done it a long time ago They’ve been doing it literally this entire time. O3-mini according to the charts they’ve released is less expensive than o1 but performs better. Costs have been falling to run these models precipitously.
- YeGoblynQueenne 2y ago
  >> Now they just have to make it cheap. Like they've been making it all this time? Cheaper and cheaper? Less data, less compute, fewer parameters, but the same, or improved performance? Not what we can observe. >> Tell me, what has this industry been good at since its birth? Driving down the cost of compute and making things more efficient. No, actually the cheaper compute gets the more of it they need to use or their progress stalls.
  - whynotminot 2y ago
    > Like they've been making it all this time? Yes exactly like they’ve been doing this whole time, with the cost of running each model massively dropping sometimes even rapidly after release.
    - YeGoblynQueenne 2y ago
      No, the cost of training is the one that isn't dropping any time soon. When data, compute and parameters increase, then the cost increases, yes?
      - whynotminot 2y ago
        Do you understand the difference between training and inference? Yes, it costs a lot to train a model. Those costs go up. But once you trained it, it’s done. At that point inference — the actual execution/usage of the model — is the cost you worry about. Inference cost drops rapidly after a model is released as new optimizations and more efficient compute comes online.
        feznyng 2y ago
        That’s precisely what’s different about this approach. Now the inference itself is expensive because the system spends far more time coming up with potential solutions and searching for the optimal one.
        2y ago
MVissers 2y ago
I would agree if the cost of AI compute over performance hasn't been dropping by more than 90-99% per year since GPT3 launched. This type of compute will be cheaper than Claude 3.5 within 2 years. It's kinda nuts. Give these models tools to navigate and build on the internet and they'll be building companies and selling services.
Bolwin 2y ago
The high efficiency version got 75% at just $20/task. When you count the time to fill in the squares, that doesn't sound far off from what a skilled human would charge

fspeech 2y ago

That's a very static view of the affairs. Once you have a master AI, at a minimum you can use it to train cheaper slightly less capable AIs. At the other end the master AI can train to become even smarter.

whynotminot 2y ago

What they’ve proven here is that it can be done. Now they just have to make it cheap. Tell me, what has this industry been good at since its birth? Driving down the cost of compute and making things more efficient. Are you seriously going to assume that won’t happen here?

Jensson 2y ago
> What they’ve proven here is that it can be done. No they haven't, these results do not generalize, as mentioned in the article: "Furthermore, early data points suggest that the upcoming ARC-AGI-2 benchmark will still pose a significant challenge to o3, potentially reducing its score to under 30% even at high compute" Meaning, they haven't solved AGI, and the task itself do not represent programming well, these model do not perform that well on engineering benchmarks.
- whynotminot 2y ago
  Sure, AGI hasn’t been solved today. But what they’ve done is show that progress isn’t slowing down. In fact, it looks like things are accelerating. So sure, we’ll be splitting hairs for a while about when we reach AGI. But the point is that just yesterday people were still talking about a plateau.
  - peepeepoopoo97 2y ago
    About 10,000 times the cost for twice the performance sure looks like progress is slowing to me.
    - whynotminot 2y ago
      Just to be clear — your position is that the cost of inference for o3 will not go down over time (which would be the first time that has happened for any of these models).
      - peepeepoopoo97 2y ago
        Even if compute costs drop by 10X a year (which seems like a gross overestimate IMO), you're still looking at 1000X the cost for a 2X annual performance gain. Costs outpacing progress is the very definition of diminishing returns.
        whynotminot 2y ago
        From their charts, o3 mini outperforms o1 using less energy. I don’t see the diminishing returns you’re talking about. Improvement outpacing cost. By your logic, perhaps the very definition of progress? You can also use the full o3 model, consume insane power, and get insane results. Sure, it will probably take longer to drive down those costs. You’re welcome to bet against them succeeding at that. I won’t be.
        peepeepoopoo97 2y ago
        [dead]
peepeepoopoo97 2y ago
Yes, that's exactly what I'm implying, otherwise they would have done it a long time ago, given that the fundamental transformer architecture hasn't changed since 2017. This bubble is like watching first year CS students trying to brute force homework problems.
- whynotminot 2y ago
  > Yes, that's exactly what I'm implying, otherwise they would have done it a long time ago They’ve been doing it literally this entire time. O3-mini according to the charts they’ve released is less expensive than o1 but performs better. Costs have been falling to run these models precipitously.
YeGoblynQueenne 2y ago
>> Now they just have to make it cheap. Like they've been making it all this time? Cheaper and cheaper? Less data, less compute, fewer parameters, but the same, or improved performance? Not what we can observe. >> Tell me, what has this industry been good at since its birth? Driving down the cost of compute and making things more efficient. No, actually the cheaper compute gets the more of it they need to use or their progress stalls.
- whynotminot 2y ago
  > Like they've been making it all this time? Yes exactly like they’ve been doing this whole time, with the cost of running each model massively dropping sometimes even rapidly after release.
  - YeGoblynQueenne 2y ago
    No, the cost of training is the one that isn't dropping any time soon. When data, compute and parameters increase, then the cost increases, yes?
    - whynotminot 2y ago
      Do you understand the difference between training and inference? Yes, it costs a lot to train a model. Those costs go up. But once you trained it, it’s done. At that point inference — the actual execution/usage of the model — is the cost you worry about. Inference cost drops rapidly after a model is released as new optimizations and more efficient compute comes online.
      - feznyng 2y ago
        That’s precisely what’s different about this approach. Now the inference itself is expensive because the system spends far more time coming up with potential solutions and searching for the optimal one.
        2y ago

MVissers 2y ago

I would agree if the cost of AI compute over performance hasn't been dropping by more than 90-99% per year since GPT3 launched. This type of compute will be cheaper than Claude 3.5 within 2 years. It's kinda nuts. Give these models tools to navigate and build on the internet and they'll be building companies and selling services.

Bolwin 2y ago

The high efficiency version got 75% at just $20/task. When you count the time to fill in the squares, that doesn't sound far off from what a skilled human would charge