6 ms·
New Computer Language Benchmarks Game metric: time + source code size
- NeutralForest 4y agoThis presentation is pretty bad, there should be more context, some kind of color scheme or labels instead of text in the background, spacing between the languages represented, other benchmarks than the geometric mean, etc.
- gus_massa 4y ago> other benchmarks than the geometric mean The text is not clear enough, but "geometric mean" is not the benchmark. The 11 problems are listed in https://benchmarksgame-team.pages.debian.net/benchmarksgame/description/summary.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... The results of the 11 problems are combined using the "geometric mean" into a single number. Some people prefer the "geometric mean", other people prefer the "arithmetic mean" to combine the numbers, other people prefer the maximum, and there rare many other methods (like the average excluding both borders).
- NeutralForest 4y ago>The text is not clear enough, but "geometric mean" is not the benchmark. Thanks that makes more sense, that's another issue for context then. I don't have anything against geometric means but there should be basic statistics like average, max, min,... available as well.
- igouy 4y ago> … basic statistics like… median, quartiles https://benchmarksgame-team.pages.debian.net/benchmarksgame/box-plot-summary-charts.html#chart-fastest https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- NeutralForest 4y agoCould you guide me to the ones I mentioned, I'm not seeing them.
- igouy 4y agoThe bar in the middle of the box is an "average" - the median. https://www.merriam-webster.com/dictionary/average https://www.merriam-webster.com/dictionary/average https://www.itl.nist.gov/div898/handbook/eda/section3/boxplot.htm https://www.itl.nist.gov/div898/handbook/eda/section3/boxplo...
- gus_massa 4y agoIt would be nice to be able to see the numbers in a table (perhaps in a auxiliary page, instead of the main page). Sometimes people want to rearrange the data or use another representation. (log scale? sort by 75% quartile? ...)
- igouy 4y agoFor people who want to rearrange the data or use another representation, there are data files — https://salsa.debian.org/benchmarksgame-team/benchmarksgame/-/tree/master/public/data https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...
- stonemetal12 4y agoThe linked page has box and whisker plots. On a box and whisker plot the lower bar is the min, the upper bar is the max. The box goes from 25th percentile to 75th percentile while the bar in the middle of the box is the 50th percentile. Therefore the stats you mentioned are all there min, max, and average with two different definitions of average given (geometric mean, and 50th percentile).
- Thaxll 4y agoThe thing they should change is to forbid the nonsense like: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/nbody-csharpcore-9.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Actually if you look at all the top net core submissions the only one fast are the one using low level intrinsics etc ...
- spullara 4y agoAll of the languages now have that trash in them. I'd like a "naive" benchmarks game where you write the code straight forwardly in a normal style for the language.
- igouy 4y ago"simple" (2nd link on the homepage.) https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/simple.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- weberer 4y ago>Java: 40 seconds >Python 3: 1h 09 minutes Well damn.
- UncleEntity 4y agoNot hard to do, I once took some C code, naively converted it to python [0] and it took hours instead of seconds to run. [0] I wanted to output an image and the C code only ran statistics so I would have had to figure out some random C image library which wasn’t how I wanted to spend my day.
- igouy 4y ago:-) "Or even acknowledge — Anyone else a bit shocked by how well Javascript on V8 performs? I might need to rethink my assumptions…" https://benchmarksgame-team.pages.debian.net/benchmarksgame/why-measure-toy-benchmark-programs.html#why-measure https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- IshKebab 4y agoWith a totally arbitrary conversion of 1 second = 1 gzipped byte. This is basically meaningless. I don't see why you'd even need to do this. You can easily show code size and performance on the same graph.
- Shadonototra 4y agothese pseudo benchmarks should be banned
- deleted 4y ago[deleted]
- _b 4y agoI'd be interested to see "C compiled with Clang" added as another language to the benchmark games. In part, digging into Clang vs gcc benchmarks is always interesting, and in part, as Rust & Clang share the same LLVM backend, it would shed light on how much of the C vs Rust difference is from frontend language stuff vs backend code gen stuff.
- igouy 4y agoAlready done: https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/c.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/rust-clang.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- kibwen 4y agoFor comparing multiple implementations of a single benchmark in a single language, this sort of data would be interesting as a 2D plot, to see how many lines it takes to improve performance by how much. But for cross-language benchmarking this seems somewhat confounding, as the richness of standard libraries varies between languages (and counting the lines of external dependencies sounds extremely annoying, not only because you have to decide whether to include standard libraries (including libc), you also need to find a way not to penalize those for having many lines devoted to tests).
- simion314 4y agoAnd when you want to make the code readable you try to space things out, split things in small functions, use longer and clear variables name. I guess they are asking for running the code trough a minifier so their implementation gains some points.
- kibwen 4y agoOn other benchmarks they measure the size of source code after it's been run through compression, as a way to normalize that. Not sure if that's been done here, but it should be.
- igouy 4y agoYes, they're the same measurements.
- benstrumental 4y agoI can't find the documentation for it, but you can see here that they measure the size of the source file after gzip compression, which reduces advantage of code-golf solutions: https://salsa.debian.org/benchmarksgame-team/benchmarksgame/-/blob/master/bencher/bin/bencher.py#L706 https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...
- igouy 4y ago
- sidkshatriya 4y agoGeometric mean of (time + gzipped source code size in bytes) seems statistically wrong. What if you shifted time to nanoseconds ? Or source code size in terms of Megabytes. The rankings could change. The culprit is the '+' I would think Geometric mean of (time x gzipped source code size) is the correct way to compare languages together. It would not matter what the units of time or size are in that case. [Here the geometric mean is the geometric mean of (time x gzipped size) of all benchmark programs of a particular language.]
- ntoskrnl 4y agoYep this is correct. Adding disparate units is almost always nonsensical. You can confirm with a scientific calculator like insect: $ insect '5s + 10MB' Conversion error: Cannot convert unit MB (base units: bit) to unit s $ insect '5s * 10MB' 50 s·MB
- smegsicle 4y agounits, frink, insect oh my
- dwattttt 4y agoIt's not necessarily wrong to add disparate units like this. It's implicitly weighting one unit to the other. Changing to nanoseconds just gives more weight to the time metric in the unified benchmark. You could instead explicitly weight them without changing units, if you cared about the size more you could add a multiplier to it.
- sidkshatriya 4y agoYou really don’t know what weight is the right weight to balance time and gripped size. Multiplying them together sidesteps the whole issue and puts time and size on par with each other regardless of the individual unit scaling. The whole point of benchmarks is to protect against accidental bias in your calculations. Adding them seems totally against my intuition. If you did want to give time more weight then I would raise it to some power. Example: geometric mean of (time x time x source size) would give time much more importance in an arguably more principled way.
- agentgt 4y agoI really wish they aggregated the metric of build time (+ whatever). That is a huge metric I care about. You can figure out it somewhat by clicking on each language benchmark but it is not aggregated. BTW as biased guy in the Java world I can tell you this is one area Java is actually mostly the winner even beating out many scripting languages apparently.
- cpurdy 4y agoPredictably, "The Computer Language Benchmarks Game" once again proves the worthlessness of "The Computer Language Benchmarks Game". This thing has been a long running joke in the software industry, exceeded only by the level of their defensiveness. SMH.
- hexo 4y agoI dont buy these results at all. Julia at second place looks like plain lie and complete nonsense, to the point I'm gonna look into this and run it myself. After trying hard to use julia for about a year and I came to conclusion it's one of the slowest things around. Maybe the stuff changed? Maybe, but julia code still remains incorrect. I hope they fix both things, speed (including start up speed, it counts A LOT) and correctness.
- ChrisRackauckas 4y agoNote that these benchmarks include compilation time for Julia, while it does not include compilation time for C, Rust, etc.
- igouy 4y agoJulia is presented like this — “Julia features optional typing, multiple dispatch, and good performance, achieved using type inference and just-in-time (JIT) compilation, implemented using LLVM.” Julia 1.7 Documentation, Introduction https://docs.julialang.org/en/v1/ https://docs.julialang.org/en/v1/
- ChrisRackauckas 4y agoYes, because it's all set for prime time in the next release. "Julia features optional typing, multiple dispatch, and good performance, achieved using type inference and just-in-time (JIT) compilation (and optional ahead-of-time compilation), implemented using LLVM." https://docs.julialang.org/en/v1.9-dev/ https://docs.julialang.org/en/v1.9-dev/ So it'll be updated when v1.9 comes out? Anyways, it's a somewhat interesting thing that Julia still gets 3rd even though it's measuring compilation time.
- igouy 4y agoTell us when it becomes achieved using ahead-of-time compilation (and optional just-in-time (JIT) compilation).
- arunc 4y agoJust curious, why does this benchmark not include D language? I remember seeing it a few years ago. Was it removed recently?
- igouy 4y agohttps://benchmarksgame-team.pages.debian.net/benchmarksgame/sometimes-people-just-make-up-stuff.html#maintenance-burden https://benchmarksgame-team.pages.debian.net/benchmarksgame/... 2009
- guenthert 4y agoI think APL had already shown, that brevity in itself is not desirable.