7 ms·

Technical report is available here https://www.amazon.science/publications/the-amazon-nova-family-of-models-technical-report-and-model-card https://www.amazon.s

Technical report is available here https://www.amazon.science/publications/the-amazon-nova-family-of-models-technical-report-and-model-card https://www.amazon.science/publications/the-amazon-nova-fami...

kajecounterhack 2y ago
TL;DR comparison of models vs frontier models on public benchmarks here https://imgur.com/a/CKMIhmm https://imgur.com/a/CKMIhmm
- brokensegue 2y ago
  So looks like they are trying to win on speed over raw metric performance
  - azinman2 2y ago
    Either that, or that’s just where they landed.
  - Calimax 2y ago
    [dead]
- SparkyMcUnicorn 2y ago
  This doesn't include all the benchmarks. The one that really stands out is GroundUI-1K, where it beats the competition by 46%. Nova Pro looks like it could be a SOTA-comparable model at a lower price point.
  - oblio 2y ago
    SOTA?
    - camel_Snake 2y ago
      "State of the Art", if that's what you were asking.
  - maeil 2y ago
    Just means it's better at one specific task than the others, which has always been the case. For each of Sonnet, GPT and Gemini I can readily name a task they are individually the best at. At the same time the consensus that Sonnet 3.5 is overall the currently strongest model remains correct, and that's what most people care about. Additionally most people do tasks that all of the models perform similarly at, or they can't be bothered to optimize every task by using the best model for that one task. Which makes sense since not a single cloud provider has all three of them. Now this one will likely be AWS-exclusive too.
  - retinaros 2y ago
    in the berkeley function calling it is similar than 4-o for multi turn while being way faster
  - int_19h 2y ago
    Benchmarks are way too easy to game. There's no shortage of models that "beat GPT-4" according to some benchmark or another, that are obviously nowhere even close when you try them on novel tasks.
  - attentive 2y ago
    on https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ Nova Pro is on par with Yi Coder 9B Chat. Which is not very inspiring.

kajecounterhack 2y ago

TL;DR comparison of models vs frontier models on public benchmarks here https://imgur.com/a/CKMIhmm https://imgur.com/a/CKMIhmm

brokensegue 2y ago
So looks like they are trying to win on speed over raw metric performance
- azinman2 2y ago
  Either that, or that’s just where they landed.
- Calimax 2y ago
  [dead]
SparkyMcUnicorn 2y ago
This doesn't include all the benchmarks. The one that really stands out is GroundUI-1K, where it beats the competition by 46%. Nova Pro looks like it could be a SOTA-comparable model at a lower price point.
- oblio 2y ago
  SOTA?
  - camel_Snake 2y ago
    "State of the Art", if that's what you were asking.
- maeil 2y ago
  Just means it's better at one specific task than the others, which has always been the case. For each of Sonnet, GPT and Gemini I can readily name a task they are individually the best at. At the same time the consensus that Sonnet 3.5 is overall the currently strongest model remains correct, and that's what most people care about. Additionally most people do tasks that all of the models perform similarly at, or they can't be bothered to optimize every task by using the best model for that one task. Which makes sense since not a single cloud provider has all three of them. Now this one will likely be AWS-exclusive too.
- retinaros 2y ago
  in the berkeley function calling it is similar than 4-o for multi turn while being way faster
- int_19h 2y ago
  Benchmarks are way too easy to game. There's no shortage of models that "beat GPT-4" according to some benchmark or another, that are obviously nowhere even close when you try them on novel tasks.
- attentive 2y ago
  on https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ Nova Pro is on par with Yi Coder 9B Chat. Which is not very inspiring.