apstroll

8 ms·

1. 1.▲
   by apstroll 1y ago

   Extremely doubtful that it boils down to quadratic scaling of attention. That whole issue is a leftover from the days of small bert models with very few parameters. For large models, compute is very rarely dominated by attention. Take, for
2. 2.▲
   by apstroll 2y ago

   Under a crossentropy loss the output activations do absolutely represent a probability distribution, since that is what we're modeling.
3. 3.▲
   by apstroll 2y ago

   The output distribution is deterministic, the output token is sampled from the output distribution, and is therefore not deterministic. Temperature modulates the output distribution, but sitting it to 0 (i.e. argmax sampling) is not the nor
4. 4.▲
   by apstroll 2y ago

   This paper is doing exactly that though, handwaving with a couple of floats. The paper is just a collection of observations about what their implementation of shapley value analysis gives for a few variations of a prompt.
5. 5.▲
   by apstroll 3y ago

   Cosine Similarity is very much about similarity, but it's quite fickle and indirect. Given a function f(l, r) that measures, say, the logprobability of observing both l and r, and that the function takes the form f(l, r) = <L(l), R(