tadamcz

7 ms·

1.
OEIS Open: A benchmark of 492 unsolved math conjectures, formalized in Lean(arxiv.org)
2 points by tadamcz 1mo ago|0 comments
2.
by tadamcz 1mo ago
You're in luck! https://news.ycombinator.com/item?id=49254706
3.
by tadamcz 1mo ago
We studied this question pretty systematically in the MirrorCode paper [1], comparing Python, C, Rust, Go, OCaml, and Ada across 19 very long-horizon tasks, for Claude Opus 4.7 and GPT-5.5. > In our results, there was little sign of inte
4.
MirrorCode: What's the largest software project AI can complete on its own?(epoch.ai)
4 points by tadamcz 3mo ago|1 comments
5.
by tadamcz 3mo ago
Hi HN! I'm the creator of MirrorCode, a benchmark of long-horizon SWE tasks. In a MirrorCode task, AI models are tasked with reimplementing an entire program end-to-end, without access to the original source code. AI-generated solution
6.
by tadamcz 4mo ago
I agree it's a potentially big problem, affecting almost any benchmark out there. We discuss it briefly in "Appendix A: Contamination and memorization" https://epoch.ai/blog/mirrorcode-preliminary-results
7.
by tadamcz 4mo ago
I think we're talking past each other here...
8.
by tadamcz 4mo ago
I should say one big difference is ProgramBench has 200 target programs while MirrorCode has about 30. We did many manual things to ensure task quality, that would have required huge resources to do at ProgramBench scale.
9.
by tadamcz 4mo ago
There's a more detailed description in "Appendix B: Qualitative discussion of the gotree task" https://epoch.ai/blog/mirrorcode-preliminary-results#appendi...
10.
by tadamcz 4mo ago
> Eg cal is totally routine. I would expect most sophomores to be able to write a perfectly good cal. This is incidental to the main disagreement, but btw I also doubt this. Let's try and make the claim more precise. e.g. are you sa
11.
by tadamcz 4mo ago
I don't think so. ProgramBench authors say no LLMs fully resolve any task, i.e. even the easiest tasks in their benchmark are unsolved. Whereas we found Opus 4.6 successfully reimplements almost every program up to gotree’s size (aro
12.
by tadamcz 4mo ago
Nice work once again from Ofir Press and team; this seems to be an idea that's in the air. > Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and f
13.
by tadamcz 5mo ago
Cool! I couldn't reproduce your results, btw. 4 attempts, it answers "Sarah Constantin" in all 4. Is it possible your setup somehow leaks your name (or other identifying information) into the context?
14.
by tadamcz 5mo ago
I tried it on my writing, and it failed every time (I'm extremely obscure but have had a blog for 10 years). My verdict is that it guesses almost entirely based on the content/topic, not style. https://bayes.net/pr
15.
MirrorCode: Evidence that AI can do some weeks-long coding tasks(epoch.ai)
3 points by tadamcz 5mo ago|0 comments
16.
by tadamcz 1y ago
Using a security key instead of TOTP would have prevented this.
17.
by tadamcz 1y ago
Using a security key as 2FA instead of TOTP would have prevented this attack, right? If you maintain popular open source packages for the love of God get yourself a couple of security keys.
18.
Claude 4 Sonnet hacked SWE-bench by peeking at future commits(bayes.net)
3 points by tadamcz 1y ago|1 comments
19.
by tadamcz 1y ago
In July, I predicted future AI models would someday learn to cheat on SWE-bench by accessing future git history. Turns out, they were already doing it!
20.
by tadamcz 2y ago
This doesn't explain how to get API access
21.
by tadamcz 2y ago
The launch post for Gemma 3 says: > use Gemma 3 with the Google GenAI SDK https://blog.google/technology/developers/gemma-3/ Does this mean (serverless) API access? I haven't been able to do so or fin
22.
by tadamcz 2y ago
Other discussion: https://news.ycombinator.com/item?id=42972581
23.
Open database of AI benchmark results with raw evaluation logs(epoch.ai)
1 points by tadamcz 2y ago|1 comments
24.
by tadamcz 2y ago
Hi, I'm the maintainer of the Epoch AI Benchmarking Hub. We're building a transparent public dataset of AI model performance. We log and publish every prompt and response -- not just aggregate scores. We even store and display the
25.
by tadamcz 2y ago
Generally, we'll use the API provider's defaults. For models we run ourselves from the weights, at the moment we'd use vLLM's defaults, but this may warrant more thought and adjustment. Other things being equal, I prefer
26.
by tadamcz 2y ago
Hi! I'm Tom, a machine learning engineer at the nonprofit research institute Epoch AI [0]. I've been working on building infrastructure to: * run LLM evaluations systematically and at scale * share the data with the public in a ri
27.
Reanalyzing survey forecasts of AI timelines(bayes.net)
1 points by tadamcz 2y ago|0 comments
28.
by tadamcz 3y ago
Hi -- I'm the author of this tool, glad to see it on Hacker News I found this post after I noticed that some users were coming to the site from HN even though I've never posted it here :) Let me know if you have any questions!
29.
Show HN: Probly – a Python-like language for quick Monte Carlo simulation(usedagger.com)
3 points by tadamcz 3y ago|0 comments