edmara

4 ms·

1.
by edmara 2y ago
https://arxiv.org/abs/2310.02207 https://transformer-circuits.pub/2024/scaling-monosemanticit...
2.
by edmara 2y ago
Of course. A trademark exists to mutually protect consumers and businesses from deceptive advertising. When a term referring to a specific product becomes a term for a product category etc, trademark protections then becomes harmful to cons
3.
by edmara 2y ago
If you have an evaluation function which does this accurately and generalizes, you pretty much already have have AGI.
4.
by edmara 2y ago
Gary Marcus was arguing in 2020 that scaling up GPT-2 wouldn't result in improvements in common sense or reasoning. He was wrong, and he continues to be wrong. It's called the bitter lesson for a reason. Nobody likes seeing their
5.
by edmara 2y ago
From reporting GPT-5 finished pre-training while ago and was in the process of red-teaming.
6.
by edmara 2y ago
Assuming there is a development that makes GPUs obsolete, I think it's safe to assume that what will replace them at scale will still take the form dedicated AI card/rack 1. Tight integration necessary for fundamental compute cons
7.
by edmara 2y ago
Because attention is all you need. I.E. the KV cache is 'just' a time saving measure because an LLM goes back and calculates those values anyway. (Which is why per-token compute increases exponentially otherwise) You're not w
8.
by edmara 2y ago
> Why it's maximal is not in the model at all, nor the data >It replays the data to us and we suppose the LLM must have the property that generates this data originally. So to clarify, what you're saying is that under the h
9.
by edmara 2y ago
The modelling is advanced enough that you can't fundamentally distinguish it from (lossy, limited) planning in the way you're describing. If the KQV doesn't encode information about likely future token sequences then a transf
10.
by edmara 2y ago
Transformers are still stateless, KV cache is just a compute-saving measure (but otherwise correctly described)
11.
by edmara 2y ago
In the case of 3sum, because the LLM has been fine tuned to use 'blank' token key values as a register to represent the sums of specific integer triplets.