6 ms·

Can anyone help me understand how this works without special bitnet precision-specific hardware? Is special hardware unnecessary? Maybe it just doesn't reach th

• hansvm 2y ago
  I haven't checked this one out yet, but a common trick is using combinations of instructions and data invariants allowing you to work in "lanes". The easiest example is xor, which can trivially be interpreted as either xoring one large integer or xoring a vector of smaller integers. Take a look at the SWAR example here [0] as a pretty common/easy example of that technique being good for something in the real world. Dedicated hardware is almost always better, but you can still get major improvements with a little elbow grease. [0] https://nimrod.blog/posts/algorithms-behind-popcount/ https://nimrod.blog/posts/algorithms-behind-popcount/
  • 15155 2y ago
    This is extremely easy to implement in-FPGA.
• eightysixfour 2y ago
  While fancy hardware would make it faster, what you are comparing it to is a bunch of floating point and large number multiplication. I believe in this case they just use a look up table: If one value is 0, it is 0. If the signs are different, it is -1. If the signs are the same, it is 1. I’m sure those can be done with relatively few instructions using far less power hungry hardware.
• summerlight 2y ago
  The major benefit would be its significant decrease in memory consumption, rather than the compute itself. The major bottleneck of the current LLM infra is typically memory bandwidth and that's the reason why those chip industries are going crazy on HBM. Surely compute optimization helps but this is useful even without any hardware changes.
  • az226 2y ago
    Inference speeds go brrrr as well.