nielstron

7 ms·

1.
Coding Agents Are "Fixing" Correct Code(sri.inf.ethz.ch)
3 points by nielstron 6mo ago|1 comments
2.
by nielstron 7mo ago
Yes that's a great summary and I agree broadly. Note with different prompt types I refer to different types of meta-prompts to generate the AGENTS.md. All of these are quite useless. Some additional experiments not in the paper showed
3.
by nielstron 7mo ago
It could... but as pointed out by other the significance is unclear and per-model results have even less samples than the benchmark average. So: maybe :)
4.
by nielstron 7mo ago
Hey thanks for your review, a paper author here. Regarding the 4% improvement for human written AGENTS.md: this would be huge indeed if it were a _consistent_ improvement. However, for example on Sonnet 4.5, performance _drops_ by over 2%.
5.
by nielstron 7mo ago
This is life of an LLM researcher. We literally ran the last experiments only a month ago on what were the latest models back then...
6.
by nielstron 7mo ago
Exactly my thoughts... the model should just auto ingest README and CONTRIBUTING when started.
7.
by nielstron 7mo ago
Hey, paper author here. We did try to get an even sample - we include both SWE-bench repos (which are large, popular and mostly human-written) and a sample of smaller, more recent repositories with existing AGENTS.md (these tend to contain
8.
by nielstron 7mo ago
Hey, a paper author here :) I agree, if you know well about LLMs it shouldn't be too surprising that autogenerated context files are not helping - yet this is the default recommendation by major AI companies which we wanted to scrutini
9.
Transcribe your aunts post cards with Gemini 3 Pro(leserli.ch)
1 points by nielstron 7mo ago|0 comments
10.
by nielstron 1y ago
Debunking the Claims of K2-Think https://www.sri.inf.ethz.ch/blog/k2think
11.
by nielstron 1y ago
Debunking the Claims of K2-Think https://www.sri.inf.ethz.ch/blog/k2think
12.
Debunking the Claims of K2-Think(sri.inf.ethz.ch)
6 points by nielstron 1y ago|0 comments
13.
by nielstron 1y ago
noted. we'll make sure to critizise turing complete type systems more thoroughly next time :))
14.
by nielstron 1y ago
Yes this work is super cool too! Note that LSPs can not guarantee resolving the necessary types that we use to ensure the prefix property, which we leverage to avoid backtracking and generation loops.
15.
by nielstron 1y ago
thank you!
16.
by nielstron 1y ago
re detecting and switching language: you could run several constraint systems in parallel and switch as soon as one of them rejects the input and another accepts it re backtracking: a core part of this paper is ensuring a prefix property. t
17.
by nielstron 1y ago
the problem with LSPs is that they don't guarantee generating a type annotation that we can use for constraints, i.e. we can not ensure the prefix property using LSPs. so we had to roll our own :) Pulling in more features to help the s
18.
by nielstron 1y ago
The downside is that you need to properly preprocess code, have less non-code Training Data, and can not adapt easily to new programming languages
19.
by nielstron 1y ago
we were thinking about doing exactly this, the closest current work is probably the amazing "Learning Formal Mathematics from Intrinsic Motivation" by Poesia et al (they use constraints too increase the likelihood of generating co