6 ms·

This trick “they found” is part of the standard torch implementation of multi head attention, namely it is called, add_zero_attention. They add a zero to the lo

This trick “they found” is part of the standard torch implementation of multi head attention, namely it is called, add_zero_attention. They add a zero to the logits, resulting in a one in the denominator as e^0=1 https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html https://pytorch.org/docs/stable/generated/torch.nn.Multihead...

civilized 3y ago
It's an option which is set to false by default. Does that mean people have tried it and it's not usually helpful...?
- mlyle 3y ago
  Yes.
  - quickthrower2 3y ago
    Can you elaborate? (It wouldn't be the first time there was an extraneous feature that no one has every used in some code!)
    - Q6T46nT668w6i3m 3y ago
      It’s useful but it’s less used than dummy tokens.
      - thomasahle 3y ago
        Are dummy tokens just tokens that don't have an associated input/output token? Like, a way to give more computational power to the model without splitting the text into more actual tokens?
        dijksterhuis 3y ago
        TL;DR sort of yes. But they're also useful for reasons not related to computational "power". An example here with an actual algorithm, although it's been a couple of years so my explanation might be a bit wrong in places. and/or i might have gotten the completely wrong end of the stick with the current thread. -- The CTC (Connectionist Temporal Classification [0]) algorithm maps a sequence x with length X -> sequence y with length Y. i.e. in speech to text we might have some audio features that correspond to the following class predictions (post softmax classification) x -> hellllloooooooooo wwwooorrrllld we want to get this as the output y -> hello world we have the alphabet as classes we try to predict for each sequence item in x. we could just removed all the duplicate in the first long sequence, but we would end up with `helo world` ... we need to preserve one of the early `l` characters in `hello` somehow CTC uses a blank token (aka dummy) token to handle potentially deliberately repeated items in sequence x. By adding the blank token to the classes predictions, we can get the model to predict something like this (post softmax classification) y* -> hel~l~~oooo~~~~~~ w~~o~~r~~l~~d The CTC decoder (non-ML decoding algo) heuristically removes repeated tokens. Turning the above into ... y -> hello world ... the duplicate `o` and `~` characters are removed. It was a decent enough algorithm for speech-to-text prior to attention/transformers etc. However, it makes CTC vulnerable to well designed adversarial example attacks because there is a massive bias within models to predict the blank token -- meaning it's very easy to modify input sequence x to switch the output sequence y to include blank tokens for nefarious purposes (the subject of my unfinished phd). [0]: www.cs.toronto.edu/~graves/preprint.pdf
        thomasahle 3y ago
        > By adding the blank token to the classes predictions, we can get the model to predict something like this (post softmax classification) > y* -> hel~l~~oooo~~~~~~ w~~o~~r~~l~~d This is a great solution. Though that's a dummy token in the output rather than the input. I guess you could do something inverse to do text to speech, but it might be hard to say where to insert the dummy tokens in that case.
    - thomasahle 3y ago
      If you take the inner product between a lot of more or less random vectors (the key and query vectors in attention) most values are going to be close to 0. This means they contribute by e^0 to the denominator. Now, if you have a context length of say 2000, your denominator is already ~ 2000. Increasing it to 2001 doesn't really make a difference. Adding 1 to the denominator can be useful if you have softmax with just a few options. Not in self-attention where you have thousands.
      - quickthrower2 3y ago
        That simple comment is a strong counterpoint to the entire blog post? Except with the +1 denominator, it might be that the model trains all of the inputs to become very negative so softmax chucks out close to zeros, whereas it wouldn't bother before because making one prob bigger makes another smaller.
        thomasahle 3y ago
        > it might be that the model trains all of the inputs to become very negative It still can't do this because of L2 regularization / weight decay. If two vectors are norm 1, their inner product is at least -1, so with 2000 vectors that's still 2000 * e^(-1) =~ 735. Not saying it's theoretically impossible that it could happen. But you would have to try _really_ hard to make it happen.
        redox99 3y ago
        I guess you could add a sort of gating operation with a learnable parameter that sends the value to -inf if doesn't reach the threshold. Of course it might have some other serious repercussions.
- blackkettle 3y ago
  It probably means they have tried it for _some_ purpose, but not necessarily the one described in OP's post here. The claim is that this is specifically useful for quantization. It's seems reasonable to assume that this would have initially been tried and potentially discarded for having little or impact on general accuracy. But that's a different issue. I suppose we'll here something definitive in a month or so.
janalsncm 3y ago
Nice catch! Hopefully OP will see this.
lovelearning 3y ago
I find its documentation quite poor though: "If specified, adds a new batch of zeros to the key and value sequences at dim=1." Doesn't describe the implications even briefly. If they add just your second sentence to that description, it'll immediately become so much more useful.
fstokesman 3y ago
https://en.wikipedia.org/wiki/Multiple_discovery https://en.wikipedia.org/wiki/Multiple_discovery

civilized 3y ago

It's an option which is set to false by default. Does that mean people have tried it and it's not usually helpful...?

mlyle 3y ago
Yes.
- quickthrower2 3y ago
  Can you elaborate? (It wouldn't be the first time there was an extraneous feature that no one has every used in some code!)
  - Q6T46nT668w6i3m 3y ago
    It’s useful but it’s less used than dummy tokens.
    - thomasahle 3y ago
      Are dummy tokens just tokens that don't have an associated input/output token? Like, a way to give more computational power to the model without splitting the text into more actual tokens?
      - dijksterhuis 3y ago
        TL;DR sort of yes. But they're also useful for reasons not related to computational "power". An example here with an actual algorithm, although it's been a couple of years so my explanation might be a bit wrong in places. and/or i might have gotten the completely wrong end of the stick with the current thread. -- The CTC (Connectionist Temporal Classification [0]) algorithm maps a sequence x with length X -> sequence y with length Y. i.e. in speech to text we might have some audio features that correspond to the following class predictions (post softmax classification) x -> hellllloooooooooo wwwooorrrllld we want to get this as the output y -> hello world we have the alphabet as classes we try to predict for each sequence item in x. we could just removed all the duplicate in the first long sequence, but we would end up with `helo world` ... we need to preserve one of the early `l` characters in `hello` somehow CTC uses a blank token (aka dummy) token to handle potentially deliberately repeated items in sequence x. By adding the blank token to the classes predictions, we can get the model to predict something like this (post softmax classification) y* -> hel~l~~oooo~~~~~~ w~~o~~r~~l~~d The CTC decoder (non-ML decoding algo) heuristically removes repeated tokens. Turning the above into ... y -> hello world ... the duplicate `o` and `~` characters are removed. It was a decent enough algorithm for speech-to-text prior to attention/transformers etc. However, it makes CTC vulnerable to well designed adversarial example attacks because there is a massive bias within models to predict the blank token -- meaning it's very easy to modify input sequence x to switch the output sequence y to include blank tokens for nefarious purposes (the subject of my unfinished phd). [0]: www.cs.toronto.edu/~graves/preprint.pdf
        thomasahle 3y ago
        > By adding the blank token to the classes predictions, we can get the model to predict something like this (post softmax classification) > y* -> hel~l~~oooo~~~~~~ w~~o~~r~~l~~d This is a great solution. Though that's a dummy token in the output rather than the input. I guess you could do something inverse to do text to speech, but it might be hard to say where to insert the dummy tokens in that case.
  - thomasahle 3y ago
    If you take the inner product between a lot of more or less random vectors (the key and query vectors in attention) most values are going to be close to 0. This means they contribute by e^0 to the denominator. Now, if you have a context length of say 2000, your denominator is already ~ 2000. Increasing it to 2001 doesn't really make a difference. Adding 1 to the denominator can be useful if you have softmax with just a few options. Not in self-attention where you have thousands.
    - quickthrower2 3y ago
      That simple comment is a strong counterpoint to the entire blog post? Except with the +1 denominator, it might be that the model trains all of the inputs to become very negative so softmax chucks out close to zeros, whereas it wouldn't bother before because making one prob bigger makes another smaller.
      - thomasahle 3y ago
        > it might be that the model trains all of the inputs to become very negative It still can't do this because of L2 regularization / weight decay. If two vectors are norm 1, their inner product is at least -1, so with 2000 vectors that's still 2000 * e^(-1) =~ 735. Not saying it's theoretically impossible that it could happen. But you would have to try _really_ hard to make it happen.
        redox99 3y ago
        I guess you could add a sort of gating operation with a learnable parameter that sends the value to -inf if doesn't reach the threshold. Of course it might have some other serious repercussions.
blackkettle 3y ago
It probably means they have tried it for _some_ purpose, but not necessarily the one described in OP's post here. The claim is that this is specifically useful for quantization. It's seems reasonable to assume that this would have initially been tried and potentially discarded for having little or impact on general accuracy. But that's a different issue. I suppose we'll here something definitive in a month or so.

janalsncm 3y ago

Nice catch! Hopefully OP will see this.

lovelearning 3y ago

I find its documentation quite poor though: "If specified, adds a new batch of zeros to the key and value sequences at dim=1." Doesn't describe the implications even briefly. If they add just your second sentence to that description, it'll immediately become so much more useful.

fstokesman 3y ago

https://en.wikipedia.org/wiki/Multiple_discovery https://en.wikipedia.org/wiki/Multiple_discovery