ofirpress

8 ms·

1.
by ofirpress 5mo ago
I'm a co-creator of SWE-bench: 1. SWE-bench Verified is now saturated at 93.9% (congrats Anthropic), but anyone who hasn't reached that number yet still has more room for growth. 2. SWE-bench Multilingual and SWE-bench Multimodal
2.
by ofirpress 8mo ago
This is a good way to benchmark models. We [the SWE-bench team] took the meta-version of this and implemented it as a new benchmark called CodeClash - We have agents implement agents that play games against each other- so Claude isn't
3.
by ofirpress 8mo ago
Benchmarks can get costly to run- you can reach out to frontier model creators to try and get them to give you free credits, but usually they'll only agree to that once your benchmark is pretty popular.
4.
by ofirpress 8mo ago
[SWE-bench co-author here] It seems like they run this test on a subset of 50 tasks, and that they only run the test once per day. So a lot of the movement in accuracy could be attributed to that. I would run on 300 tasks and I'd run
5.
by ofirpress 8mo ago
We (the SWE-bench team) have a 100 line of code agent that is now pretty popular in both academic and industry labs: https://github.com/SWE-agent/mini-swe-agent I think it's a great way to dive into the agent worl
6.
by ofirpress 9mo ago
As John says in that thread, we've fixed this issue in SWE-bench: https://xcancel.com/jyangballin/status/2006987724637757670 If you run SWE-bench evals, just make sure to use the most up-to-date code from our
7.
by ofirpress 9mo ago
> There are certain tasks, like improving a given program for speed, for instance, where in theory the model can continue to make progress with a very clear reward signal for a very long time. Yup, this will absolutely be a big driver of
8.
by ofirpress 1y ago
[I'm on the SWE-bench team] Multiple people have looked into this, for example right in that thread: https://github.com/SWE-bench/SWE-bench/issues/465#issuecomme... This issue had affected a tiny fractio
9.
by ofirpress 1y ago
We (the Princeton SWE-bench team) have a 100 line of code agent that does pretty well, you can read the code here: https://github.com/SWE-agent/mini-swe-agent
10.
by ofirpress 1y ago
We (the Princeton SWE-bench team) built an agent in ~100 lines of code that does pretty well on SWE-bench, you might enjoy it too: https://github.com/SWE-agent/mini-swe-agent
11.
VideoGameBench from Princeton: Can vision-language models play 90s video games?(vgbench.com)
6 points by ofirpress 1y ago|1 comments
12.
by ofirpress 1y ago
Not sure what you mean by benchmaxxing but we think there's still a lot of useful signals you can infer from SWE-bench-style benchmarking. We also have SWE-bench Multimodal which adds a twist I haven't seen elsewhere: https:&#x2F
13.
by ofirpress 1y ago
[I'm one of the co-creators of SWE-bench] The team managed to improve on the already very strong o3 results on SWE-bench, but it's interesting that we're just seeing an improvement of a few percentage points. I wonder if gett
14.
VideoGameBench: Benchmarking video games for Vision Language Models(vgbench.com)
4 points by ofirpress 1y ago|0 comments
15.
by ofirpress 2y ago
Good time to re-read The Bitter Lesson: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...
16.
by ofirpress 2y ago
I'm one of the co-authors of SWE-bench. We just created a Javascript (+visual) SWE-bench: https://www.swebench.com/multimodal.html We're going to release the eval suite for this soon so that people can start makin
17.
Why and How ChatGPT Works: Building 5 LMs at Increasing Complexity Levels [video](youtube.com)
26 points by ofirpress 3y ago|0 comments
18.
by ofirpress 3y ago
Thanks for posting this! I'm here if you have any questions.
19.
by ofirpress 3y ago
The ALiBi paper shows that our method beats the sinusoidal PE you refer to across many benchmarks. https://arxiv.org/abs/2108.12409
20.
by ofirpress 3y ago
(I wrote ALiBi) Thanks for posting this! You can view a video where I explain what we did and why it's useful at: https://www.youtube.com/watch?v=Pp61ShI9VGc
21.
by ofirpress 3y ago
> so if some prediction task requires long range dependencies you might expect one of the less-penalized heads to end up specializing Exactly. You have heads that focus on content nearby and ones that focus on stuff that is far away. &gt
22.
by ofirpress 3y ago
(I wrote ALiBi) You can read the paper here https://arxiv.org/abs/2108.12409 While intuitively it does seem like ALiBi would make it hard for the model to attend to things that are far away, in many scenarios we'v
23.
by ofirpress 4y ago
Cool new efficient inference method that saves 2x memory and does not degrade performance for large language models! More from the author about this at: https://twitter.com/Tim_Dettmers/status/1559892888326049792
24.
LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale(arxiv.org)
7 points by ofirpress 4y ago|1 comments
25.
by ofirpress 6y ago
Thank you! We spent a lot of time on making this as easy to understand as possible.
26.
by ofirpress 6y ago
Thanks for posting our paper! If anyone has any questions, I'll stick around this thread for a bit. There's a summary of our paper on twitter: https://twitter.com/OfirPress/status/1344387959563325442 And
27.
Neural Language Modeling from Scratch(ofir.io)
97 points by ofirpress 9y ago|10 comments
28.
by ofirpress 10y ago
Thats awesome, I wasn't aware of this. I slightly disagree with Greg though about the "one main resource". While the DL book is amazing, its not really aimed at newbies, and so I think CS231n is a much better starting point.
29.
by ofirpress 10y ago
Lots of people have heard about deep learning but dont really know where to start. When I was starting a year or so ago I was in the same place. I just took the resources that I used to learn about the field and put them all in one place. A
30.
How to Start Learning Deep Learning(ofir.io)
264 points by ofirpress 10y ago|29 comments

More ›