6 ms·
One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subag
by nsingh2 14d ago
One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup.
Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something like Pi.
- randomblock1 14d agoI think it's still a useful data point. For example, omp, which is pi with some default extensions, scores worse. I do agree that adding more configurations of Pi would help though.
- dfltr 14d agoIt does seem a bit odd to say "We tested all of these bicycles with the same rider" when one of the test cases is actually a bare high-end frame with no components on it. It's even weirder given that another of the test cases is essentially "We put parts on the frame (OhMyPi) and it went faster!"
- jdthedisciple 14d agoI'm not sure your analogy holds, because here the optimization metric is clear and unanimous: every one wants max pass rate at min costs. With the bicycle, some may prefer comfort, others speed, others offroad, etc., so it would not be obvious which one is "best".
- infecto 14d agoNot weird or odd in the least. I think it clearly shows that Pi barebones at least in this set of tests, preformed better and cheaper cost that OMP.
- infecto 14d agoI use Pi barebones.
- Edward40 14d agoI also prefer using vanilla Pi over Oh My Pi.