5 ms·

Are these prominent events? Some of these models are just trained a little longer or cooked with different data? I think it would be better to list pivotable e

by bArray 1y ago

Are these prominent events? Some of these models are just trained a little longer or cooked with different data? I think it would be better to list pivotable events, i.e. significant changes in architecture or approach, or a benchmark being surpassed. Otherwise this is just a list of models released by big tech companies irrespective of their importance. Also I think I would prefer it be renamed from "Artificial Intelligence Timeline" when it's only since 2022 and AI encapsulates a hell of a lot of important work other than LLMs. I've been around long enough to see a few AI bubbles now. Important events of AI are definitely missing anyway, for example when in 1956 some of the greatest minds in the field of the time set about trying to solve AI [1], only to realise it was far more complex than they imagined. It's only now we even approach addressing some of those original aims, some 70 years later. [1] https://www-formal.stanford.edu/jmc/history/dartmouth/dartmouth.html https://www-formal.stanford.edu/jmc/history/dartmouth/dartmo...

NHLOCAL 1y ago
I’m familiar with the field of machine learning — I’m not an expert, but it’s quite clear that the developments in recent years have been very significant and very rapid. There’s actually a tendency among experts in the field to underestimate the power and potential of current AI progress.
NitpickLawyer 1y ago
> Are these prominent events? Some of these models are just trained a little longer or cooked with different data? >I think it would be better to list pivotable events, i.e. significant changes in architecture or approach, or a benchmark being surpassed. Otherwise this is just a list of models released by big tech companies irrespective of their importance. I don't think it's that easy to dismiss changes in recipes, data and training regime, tbh. There have been plenty of "huh, that's strange" moments with LLMs even without major breakthroughs in the overall architecture. The case of one model being really strong at chess while others from the same family (larger to boot) aren't is such a "huh" moment for me. Or the new "reasoning" thing, where it's obvious the new models are better at some tasks than the previous gen. Training steps is another interesting one, where two teams - one fine-tuning on 100k+ samples and one fine-tuning on 1k samples w/ 15 epochs and getting similar performance is also a good example. Or the (extremely readable) simple paper coming out of meta fair ~6 mo ago where they found that repeating some (5-10%) of the samples leads to much better generalisation. All of these are "huh" worthy, and I don't think we thoroughly understand why they happen, or why they sometimes happen and not other times (i.e. w/ 20+T training sets on massively scaled LLMs like the purported gpt5).
NHLOCAL 1y ago
As the well known saying goes: 'Garbage in, garbage out.' I believe the same logic applies in reverse when extremely high-quality data goes in, an exceptionally good model comes out. At the end of the day, it's all about data, training time, and all those so-called 'details'that’s exactly what really matters

by bArray 1y ago

NHLOCAL 1y ago

I’m familiar with the field of machine learning — I’m not an expert, but it’s quite clear that the developments in recent years have been very significant and very rapid. There’s actually a tendency among experts in the field to underestimate the power and potential of current AI progress.

NitpickLawyer 1y ago

> Are these prominent events? Some of these models are just trained a little longer or cooked with different data? >I think it would be better to list pivotable events, i.e. significant changes in architecture or approach, or a benchmark being surpassed. Otherwise this is just a list of models released by big tech companies irrespective of their importance. I don't think it's that easy to dismiss changes in recipes, data and training regime, tbh. There have been plenty of "huh, that's strange" moments with LLMs even without major breakthroughs in the overall architecture. The case of one model being really strong at chess while others from the same family (larger to boot) aren't is such a "huh" moment for me. Or the new "reasoning" thing, where it's obvious the new models are better at some tasks than the previous gen. Training steps is another interesting one, where two teams - one fine-tuning on 100k+ samples and one fine-tuning on 1k samples w/ 15 epochs and getting similar performance is also a good example. Or the (extremely readable) simple paper coming out of meta fair ~6 mo ago where they found that repeating some (5-10%) of the samples leads to much better generalisation. All of these are "huh" worthy, and I don't think we thoroughly understand why they happen, or why they sometimes happen and not other times (i.e. w/ 20+T training sets on massively scaled LLMs like the purported gpt5).

NHLOCAL 1y ago

As the well known saying goes: 'Garbage in, garbage out.' I believe the same logic applies in reverse when extremely high-quality data goes in, an exceptionally good model comes out. At the end of the day, it's all about data, training time, and all those so-called 'details'that’s exactly what really matters