sdpmas

5 ms·

1.
by sdpmas 6mo ago
oh ensemble can be distilled to a single model easily.
2.
by sdpmas 6mo ago
i think evolution meta-learns the architecture, hyperparams. some domain knowledge too (for ex, we all perceive the world as 3d) but not much. if you compare the text consumed by human vs AI (and i think this is fair b/c even with evol
3.
by sdpmas 6mo ago
the T stands for tea :)
4.
by sdpmas 6mo ago
continual learning is LLMs :) ultimately everything will be/already is data bottlenecked.
5.
by sdpmas 6mo ago
thanks!
6.
by sdpmas 6mo ago
> you can simply generate more, and higher quality, artificial data this is simply not true. and it's very clear if you look at continual learning, robotics, biology, etc. each has enough economic incentives to spend 1000x compute i
7.
NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute(qlabs.sh)
168 points by sdpmas 6mo ago|46 comments
8.
by sdpmas 7mo ago
absolutely!
9.
by sdpmas 7mo ago
thanks, here's another one: https://arxiv.org/abs/2511.03276
10.
by sdpmas 7mo ago
diffusion is promising, but still an open question how much data efficient they are compared to AR. in practice, you can also train AR forever with high enough regularization, so let's see.
11.
by sdpmas 7mo ago
yes! typically the optimizer that trains faster also get better data efficiency. it maybe not be absolutely true, but that has been my observation so far. also see https://arxiv.org/pdf/2510.09378 for second-order meth
12.
by sdpmas 7mo ago
no ensembling means train 8 models and during inference avg logits of all 8 models to make a prediction.
13.
by sdpmas 7mo ago
yeah, we do incorporate some of the findings from the paper in our repo! like aggressive regularization and ensembling.
14.
by sdpmas 7mo ago
yes, agreed, modded-nanogpt is already a data-efficient variant of original nanogpt. just that the kinds of algorithms it allows are somewhat constrained because it optimizes for wall clock time.
15.
by sdpmas 7mo ago
yes, good point. right now, it's somewhat hard to overfit because the meta-optimization extracts tiny bits of information. but over time, we will switch the validation set to some other random subset of the FineWeb or even entirely OOD
16.
by sdpmas 7mo ago
hey, it's Samip (behind the Slowrun repo). yeah that's a fair point, we will mention them in the blog. but there are a couple of major differences: 1. our emphasis is on using more compute to get better data efficiency. this is im
17.
NanoGPT Slowrun: Language Modeling with Limited Data, Infinite Compute(qlabs.sh)
185 points by sdpmas 7mo ago|35 comments
18.
Fastest way to search for code in GitHub from your IDE(sdpmas.github.io)
1 points by sdpmas 5y ago|0 comments