5 ms·
They've reported surpassing GPT4 on several benchmarks. Does anyone know of these are hand picked examples or is this the new SOTA?
by thatcherthorn 3y ago
They've reported surpassing GPT4 on several benchmarks. Does anyone know of these are hand picked examples or is this the new SOTA?
- williamstein 3y agoThey certainly claim it is SOTA for multimodal tasks: “Gemini surpasses SOTA performance on all multimodal tasks.”
- xiphias2 3y agoIt will be SOTA maybe when Gemini Ultra is available. GPT-4 is still SOTA.
- philomath_mn 3y agoUsually SOTA status is established when the benchmark paper is released (probably after some review). But GPT4 is the current generally-available-SOTA
- silveraxe93 3y agoThey also compare to RLHFed GPT-4, which reduces capabilities, while their model seems to be pre-RLHF. So I'd expect those numbers to be a bit inflated compared to public release.