robrenaud

5 ms·

1.
by robrenaud 4d ago
Neal Nanda's walkthrough of it is pretty good. https://youtu.be/KV5gbOmHbjU?is=GSiv0rSHBocNdU8l Also long, but it's detailed and complicated, so there is no escaping that.
2.
by robrenaud 6d ago
The part where he chooses his inputs to hit worst case behavior in Python's hash function.
3.
by robrenaud 7d ago
Experts vary per token in MoE, there is maximum flexibility. Good for driving down loss, bad for locality/gpu memory/bandwidth. If expert selection were more constrained, inference systems could take advantage of it. Keeping expe
4.
by robrenaud 8d ago
Should LLMs be designed to be modular, so that instead of needing access to the whole model, for a given prompt, only a small subset of the model would be used? If knolwedge was sufficiently modularized, most of it could be ignored. Maybe
5.
by robrenaud 10d ago
It's cool that it proves that a bunch of vectorized outputs from an unknown embedder on an unknown dataset is in no way private, because of this ability to reverse engineer the embedder. I talked to the author at his poster session at
6.
by robrenaud 10d ago
I think a less stringent, but still workable assumption is that for very similair objects, their distances will be small. This is much easier to accomplish than agreement across all pairs.
7.
by robrenaud 13d ago
The number of steps between token outputs is increasing. I don't actually think this is a problem, but it is a further step towards inscrutability.
8.
by robrenaud 20d ago
If black and white had different permutations of the back row, you could find more balanced starting positions. Adopt more flexible starting conditions in the computer chess age.
9.
by robrenaud 2mo ago
Does the model quality become degraded in other ways?
10.
by robrenaud 2mo ago
Do consider the incentives of those developing the model that made those predictions. Afaict, it was not selected for purpose other than testimony.
11.
by robrenaud 2mo ago
How do you get personalization without surveillance?
12.
by robrenaud 3mo ago
I suspect idiomatic TypeScript or idiomatic F# are both way better solutions in the real world than abstruse Typescript emulating idiomatic F#.
13.
by robrenaud 3mo ago
If a good SWE is $150/hour, does the model cost actually matter? Surely you'd be willing to spend $10/hour to make that SWE 20% more productive? The model cost is still much less than the salary.
14.
by robrenaud 3mo ago
They use a lightweight adapter to silently degrade the performance. Usually these adaptors are made to improve the performance for a given domain/task.
15.
by robrenaud 3mo ago
3 blue 1 brown has a great visual introduction to transformers, the heart of LLMs. It's chapter 5. Start at chapter 1 if you want more background on neural nets and backprop. https://youtu.be/wjZofJX0v4M?si=HFXbrB-5cAr
16.
by robrenaud 3mo ago
"The reasoning is the weights." The reasoning is in a process that uses the weights. Sorting algorithms are just bytes. Those bytes don't sort by themselves. They do instruct a computer on how to sort though.
17.
by robrenaud 4mo ago
There is some recent work on modularizing knowledge in LLMs. https://arxiv.org/html/2605.06663v1 It might be possible to train a big generalist that is a composition of modules, some of which can be dropped dynamically
18.
by robrenaud 5mo ago
Is every American tax payer morally compromised?
19.
by robrenaud 5mo ago
My big gripe with unions is the unwavering protection of their worst performing members. Eg, that they necessitated so called "rubber rooms" like these in the NYC public schools, where teachers got paid to do nothing while waiting
20.
by robrenaud 5mo ago
The flat earthers are why I hate astronomy. Afaict, the grand parent poster is just very wrong. You do want to cause acute stresses to your heart (cardiovascular exercise) to get it work better.
21.
by robrenaud 5mo ago
Yeah, it's different. Anthropic profits when it delivers tokens. Hosting providers pay when Anthropic scrapes them.
22.
by robrenaud 6mo ago
Yeah, my big problem with the paper is it just might be an artifact of qwen's training process.
23.
by robrenaud 6mo ago
Was Alphago's move 37 original? In the last step of training LLMs, reinforcement learning from verified rewards, LLMs are trained to maximize the probability of solving problems using their own output, depending on a reward signal akin
24.
by robrenaud 6mo ago
Recursive self improvement. It's when AI speeds up the development of the next AI.
25.
by robrenaud 7mo ago
Location: SF (current). NYC/Philly general area acceptable. Remote okay. email: rrenaud@gmail.com Resume: 16 year SWE -> MLE @ Google, MS from NYU with focus on ML. Retired. Now I hack on data analysis for video game projects f
26.
by robrenaud 7mo ago
What do y'all think about the latency/quality tradeoff with LLMs? Human voices don't take 30 seconds to think, retrieve, research, and summarize a high quality answer. Humans are calibrated in their knowledge, they know what
27.
Ask HN: Is there something like Google style guide for AI-only coded apps?
1 points by robrenaud 7mo ago|2 comments
28.
by robrenaud 7mo ago
Please serve well quantized models. If you can get 99 percent of the quality for 50 percent of the cost, that is most times a good tradeoff.
29.
by robrenaud 7mo ago
Cite a source. Your concrete claim is that, on average, for every $1 of subscription revenue on a monthly subscription, OpenAI and Anthropic were losing $11.50? It seems completely implausible. I could believe that if a $20 sub used every
30.
by robrenaud 7mo ago
I used to play very competitively, but I've been more chill recently. I just think it's a nice problem/dataset to work with, because of the depth of my understanding of the game.

More ›