cjsaltlake

6 ms·

1.
by cjsaltlake 5mo ago
Normal people definitely will not benefit from this by default.
2.
by cjsaltlake 5mo ago
SWE-bench was created to replace olympiad coding benchmarks. I think past olympiad coding benchmarks were much worse representative of real-world coding than something like SWE-bench, which is derived from real units of labor. Further, olym
3.
by cjsaltlake 5mo ago
But, that's an enormous source of coding productivity, and it's why Anthropic is worth billions... The reason SWE-bench has been so successful and useful for coding is that software engineering has a ton of tradition and infrastru
4.
by cjsaltlake 5mo ago
code clash I think would be quite hard to game or contaminate unintentionally; considering that models need to compete against one another
5.
by cjsaltlake 5mo ago
I suggest reading the Mythos report's discussion on SWE-bench and contamination. I think it's fairly convincing that you can account for contamination and still trust SWE-bench numbers on models that aren't over-optimized for
6.
by cjsaltlake 5mo ago
If you read the mythos report, in which they discuss and account for contamination substantially, it still suggests that performance on SWE-bench verified is meaningful. Benchmarks, including SWE-bench can absolutely be gamed, but if you&#x
7.
by cjsaltlake 5mo ago
Labor replacing devices means nobody works in those fields anymore. If AI can do this for every field, nearly no one will need to work in any field. We'll have a giant fully automated resource-extraction machine.
8.
by cjsaltlake 1y ago
We absolutely did not.
9.
by cjsaltlake 1y ago
I'm also on the SWE-bench team. This was simply a classic bug. We had code before that we believed was sufficient to hide / remove future GitHub history and it turns out it was not. We've patched it.
10.
SWE-Bench: Can Language Models Resolve Real-World GitHub Issues?(swebench.com)
2 points by cjsaltlake 3y ago|0 comments
11.
by cjsaltlake 4y ago
If trends continue, there will be 10 billion Amish in 100 years.
12.
by cjsaltlake 4y ago
I think one should be consequentialist about all things.
13.
by cjsaltlake 4y ago
The only people I hate more than woke people are anti-woke people.
14.
by cjsaltlake 4y ago
Companies are entities with political interests like any other. The only way you can curtail their influence politically is through regulation. It is reasonable for entities to pursue their goals through any legal means.
15.
by cjsaltlake 6y ago
The community for straight people is too disparate to be meaningful. .bi is interesting though..
16.
by cjsaltlake 6y ago
India is a single country, with a relatively well established national identity (obviously not perfect!, but enough for people to work and live in the same area), and a relatively stable government. Most African countries are still in flux
17.
by cjsaltlake 6y ago
Additionally, there's north/south discrimination, with greater antipathy from southern Indians against northern Indians.
18.
by cjsaltlake 6y ago
It's not at all highly correlated, and skin color certainly wouldn't trump real caste affiliation.
19.
by cjsaltlake 7y ago
~19:50