bluecoconut

6 ms·

1.
by bluecoconut 3mo ago
Very cool! one subtle consistency bug that made it hard for me to interpret when I was clicking around: the small thumbnail plot vs the full plot often (always?) seem to use different colors. The blue / orange gets assigned to the oppo
2.
by bluecoconut 3mo ago
For those who like these types of puzzles, i made a benchmark called pencil puzzle bench Testing AI model's ability to solve puzzles like these. https://ppbench.com/ Can play the puzzles and compare your timing and acc
3.
Show HN: Pencil Puzzle Bench – LLM Benchmark for Multi-Step Verifiable Reasoning(ppbench.com)
5 points by bluecoconut 7mo ago|0 comments
4.
by bluecoconut 7mo ago
Fun read. One upside of the deterministic schemes is they include provenance/lineage. Can literally "trace up" the path the history back to the original ID giver. Kinda has me curious about how much information is required to
5.
by bluecoconut 8mo ago
I almost feel like this goes opposite to what attention is good at. This would be good at approximating all the places where attention is low / not sharp. Where attention/the exponential is key is when it selects out / needle
6.
by bluecoconut 9mo ago
Bytedance is publishing pretty aggressively. Recently, my favorite from them was lumine: https://arxiv.org/abs/2511.08892 Here's their official page: https://seed.bytedance.com/en/research
7.
by bluecoconut 11mo ago
Previous paper from DeepSeek has mentioned Anna’s Archive. > We cleaned 860K English and 180K Chinese e-books from Anna’s Archive (Anna’s Archive, 2024) alongside millions of K-12 education exam questions. https://arxiv.org&#x
8.
by bluecoconut 1y ago
After maintaining my own agents library for a while, I’ve switched over to pydantic ai recently. I have some minor nits, but overall it's been working great for me. I’ve especially liked combining it with langfuse. Towards coding agent
9.
by bluecoconut 1y ago
The first time I got off at and heard Komagome's tune I mistakenly thought it was some halloween special because it was late October at the time, and the song felt so distinct and unique.
10.
by bluecoconut 1y ago
Interestingly this one seems it is from before 高輪ゲートウェイ (Takanawa Gateway) station which opened in 2020, but the numbering shows the gap (JY 25 -> JY 27). That led me to looking it up, and turns out that they introduced the numbering in
11.
by bluecoconut 1y ago
Not getting around it, just benefiting from parallel compute / huge flops of GPUs. Fundamentally, it's just that prefill compute is itself highly parallel and HBM is just that much faster than LPDDR. Effectively H100s and B100s ca
12.
by bluecoconut 1y ago
I was able to get gpt-oss:20b wired up to claude code locally via a thin proxy and ollama. It's fun that it works, but the prefill time makes it feel unusable. (2-3 minutes per tool-use / completion). Means a ~10-20 tool-use inter
13.
by bluecoconut 1y ago
I've been working on something very similar as a tool for my own AI research -- though I don't have the success they claim. Mine often plateaus on the optimization metric. I think there's secret sauce in the meta-prompting an
14.
by bluecoconut 1y ago
I’ve been using whisky to play Elden ring on my M4 MBP and it’s been great! I love that the Game porting toolkit and wine all work so well. I did have to do some pinning of steam to an older version to keep it working recently. I guess I’ll
15.
by bluecoconut 2y ago
I tried to do this myself about ~1.5 years ago, but ran into issues with capturing state for sockets and open files (which started to show up when using some data science packages, jupyter widgets, etc.) What are some of the edge cases wher
16.
by bluecoconut 2y ago
I don’t quite understand this - by 30k pages, is this the number of entries in your index? Did you mean 30M? At the <100k scale I just full compute / inner product directly, and I don’t mess with vector stores or added complexity. N
17.
by bluecoconut 2y ago
By my estimates, for this single benchmark, this is comparable cost to training a ~70B model from scratch today. Literally from 0 to a GPT-3 scale model for the compute they ran on 100 ARC tasks. I double checked with some flop estimates (P
18.
by bluecoconut 2y ago
3400 came from counting pixels on the plot. Also its $20 on for the o3-low via the table for the semi-private, which x172 is 3440, also coming in close to the 3400 number
19.
by bluecoconut 2y ago
they use some poor language. "High Efficiency" is O3 Low "Low Efficiency" is O3 High They left the "Low efficiency" (O3 High) values as `-` but you can infer them from the plot at the top. Note the $20 and $17
20.
by bluecoconut 2y ago
some other imporant quotes: "Average human off the street: 70-80%. STEM college grad: >95%. Panel of 10 random humans: 99-100%" -@fchollet on X So, considering that the $3400/task system isn't able to compete with STE
21.
by bluecoconut 2y ago
That's the low-compute mode. In the plot at the top where they score 88%, O3 High (tuned) is ~3.4k
22.
by bluecoconut 2y ago
Efficiency is now key. ~=$3400 per single task to meet human performance on this benchmark is a lot. Also it shows the bullets as "ARC-AGI-TUNED", which makes me think they did some undisclosed amount of fine-tuning (eg. via the A
23.
by bluecoconut 2y ago
I like writing to myself / talking to myself... but I'm trying to convert that internal captured thought / language into brand value and useful information in a wider contexts. Good points about recognizing the fantasy of doi
24.
by bluecoconut 2y ago
I write for myself a lot, roughly ~5000 words a day in notes, messages to self, etc. I have no problem writing and talking to myself. It's the editing process and formalizing it for public consumption. Either, the actual work of doing
25.
by bluecoconut 2y ago
Worrying what others think resonates with me a lot. Every few weeks I try to motivate myself to write more online (HN, X, blogs) and consistently get “self sabotage” stuck. (Been going on for >2 years) The article just says they pushed
26.
by bluecoconut 2y ago
oh, they do talk about it On the 2024 AIME exams, GPT-4o only solved on average 12% (1.8/15) of problems. o1 averaged 74% (11.1/15) with a single sample per problem, 83% (12.5/15) with consensus among 64 samples, and 93%
27.
by bluecoconut 2y ago
Some other follow up reflections 1. I wish that Y-axes would switch to be logit instead of linear, to help see power-law scaling on these 0->1 measures. In this case, 20% -> 80% it doesn't really matter, but for other papers (eg.
28.
by bluecoconut 2y ago
Super hand-waving rough estimate: Going off of five points of reference / examples that sorta all point in the same direction. 1. looks like they scale up by about ~100-200 on the x axis when showing that test time result. 2. Based on
29.
by bluecoconut 2y ago
This power law behavior of test-time improvement seems to be pretty ubiquitous now. In more agents is all you need [1], they start to see this as a function of ensemble size. It also shows up in: Large Language Monkeys: Scaling Inference Co
30.
by bluecoconut 2y ago
Nice~ Glad to see this published / confirmed by others. Next I hope to see some of this symmetry used to improve MoE / dynamic compute / adaptive style models! Context: I found the same structure: early - middle - end layers

More ›