leogao

29 ms·

1.
My hobby: running deranged surveys(nablatheta.substack.com)
3 points by leogao 6mo ago|0 comments
2.
by leogao 8mo ago
> The purpose here is not to responsibly warn us of a real threat. If that were the aim there would be a lot more shutting down of data centres and a lot less selling of nuclear-weapon-level-dangerous chatbots. you're lumping togeth
3.
by leogao 8mo ago
39% just doesn't pass basic muster. in the past year, my rent hasn't near-doubled. it doesn't cost anywhere near twice as much as last year to buy food or clothing or transportation. 39% inflation over the past year would mea
4.
by leogao 9mo ago
the converse is that good hiring managers who can spot actual undervalued talent have a huge advantage.
5.
by leogao 9mo ago
in the grand scheme of things, this is a very small amount of plastic waste, and as far as resources go, one of the less scarce ones. at some point, the cost of the hand wringing to avoid waste is more of a drag on society than the actual w
6.
by leogao 9mo ago
where are you getting this 39% figure? inflation in 2025 was only 2.7%.
7.
by leogao 9mo ago
this kind of cynicism hurts young people who actually do good work on their own though.
8.
by leogao 10mo ago
I'm not an expert at hardware, so take this with a grain of salt, but there are two main reasons: - Discrete optimisation is always going to be harder than continuous optimization. Learning the right sparsity mask is fundamentally a ve
9.
by leogao 10mo ago
I'm also very excited about SAE/Transcoder based approaches! I think the big tradeoff is that our approach (circuit sparsity) is aiming for a full complete understanding at any cost, whereas Anthropic's Attribution Graph appr
10.
by leogao 10mo ago
There's been some work (e.g RASP - https://arxiv.org/abs/2106.06981 ) on taking logical computations and compiling them into transformer weights.
11.
by leogao 10mo ago
Mixture of experts sparsity is very different from weight sparsity. In a mixture of experts, all weights are nonzero, but only a small fraction get used on each input. On the other hand, weight sparsity means only very few weights are nonze
12.
by leogao 10mo ago
For what it's worth, we think it's unfortunately quite unlikely that frontier models will ever be trained with extreme unstructured sparsity, even with custom sparsity optimized hardware. Our main hope is that understanding sub-fr
13.
by leogao 1y ago
you can always have an LVM layer for atomic snapshots
14.
by leogao 1y ago
btrfs has eaten my data within the last decade. (not even because of the broken erasure coding, which I was careful to avoid!) not sure I'm willing to give it another chance. I'd much rather use zfs.
15.
by leogao 2y ago
> Sentences like the opening line of the Declaration of Independence simply do not occur in conversation. maybe not in your conversations
16.
by leogao 2y ago
The GPT-4 pretraining set included chess games in PGN notation from 1800+ ELO players. I can't comment on any other models.
17.
by leogao 2y ago
Nvidia SASS has a similar instruction too (LOP3.LUT)
18.
by leogao 2y ago
I looked into the question of what counts as reasonable care and wrote up my conclusions here: https://www.lesswrong.com/posts/kBg5eoXvLxQYyxD6R/my-takes-o...
19.
by leogao 2y ago
In domains like ML, people care way more about the half precision FLOPs than single precision.
20.
by leogao 2y ago
Is there a better test than the one that is normally prescribed, to distinguish the false positive cases you describe?
21.
by leogao 2y ago
The paper cites Anthropic's work extensively.
22.
by leogao 2y ago
Note that we focus on random positive activations, which are less susceptible to interpretability illusions than top activations (but also look less impressive as a result). We also provide access to random uncherrypicked features, whereas
23.
by leogao 2y ago
The paper introduces substantial improvements over the methodology in the Anthropic SAE paper, and the research was done concurrently.
24.
by leogao 2y ago
We were planning to release the paper around this time independent of the other events you mention. I think it is still predominantly accurate to say that we have no idea how LLMs work. SAEs might eventually change that, but there's st
25.
by leogao 2y ago
This project has been in the works for about a year. The initial commit to the public repo was not really closely related to this project, it was part of the release of the Transformer debugger, and the repo was just reused for this release
26.
by leogao 5y ago
Thankfully, there already exist evaluation tasks like that, and Eleuther actually has a project collecting a handful of them together; see https://github.com/EleutherAI/lm-evaluation-harness/
27.
by leogao 5y ago
To date, EleutherAI as an "organization" (read: basically a Discord server) has not really attempted any kind of marketing. It has no PR dept, just individuals tweeting about the work that Eleuther does.
28.
by leogao 5y ago
Could the title of this post be change to emphasize that the model sizes released were 1.3B and 2.7B? Something like "EleutherAI releases 1.3B and 2.7B parameter GPT-like language models". The current title implies that a full siz
29.
by leogao 6y ago
My impression from reading the paper is most of the other optimizations (custom kernels, contiguous memory, checkpointing, etc) are orthogonal to the partitioning stuff. That seems to imply that ZeRO is model+pipeline parallel plus a bunch
30.
by leogao 6y ago
I have a reasonable amount of experience with distributed machine learning (and transformers in particular, too) and I have to 100% agree that this blog post (and even the ZeRO paper) is largely technobabble. I don't doubt that this mi

More ›