fheinsen

13 ms·

1.
by fheinsen 7mo ago
The github repository's first toy example is with 8 Taylor terms, applied to a context with 1B tokens, each token with 1K heads: https://github.com/glassroom/sata_attention That toy example is not practical with t
2.
by fheinsen 7mo ago
Unlike previous efforts, which typically stop at a low-order (e.g., quadratic) term of the Taylor expansion, this work derives a succinct, efficient, parallel general method for approximating attention with any number of Taylor terms, to ar
3.
by fheinsen 7mo ago
As far as I know, there is no widely accepted explanation for context rot. Numerical error in long sequences of query-key dot-products may be a key factor.
4.
by fheinsen 8mo ago
Yes, there must be a connection. While adaptive truncation may prove impractical, it should be possible to measure spectral statistics on sample data, and specify a different fixed truncation order per layer, per head, etc. The github repos
5.
by fheinsen 8mo ago
[3] is linear attention, https://arxiv.org/abs/2006.16236 , a well-known result with ~3K citations: https://scholar.google.com/scholar_lookup?arxiv_id=2006.1623...
6.
by fheinsen 8mo ago
This is a form of linear attention ( https://arxiv.org/abs/2006.16236 ) that approximates standard scaled dot-product attention to arbitrary precision, by adding Taylor terms in an efficient manner. Each additional Tay
7.
by fheinsen 8mo ago
The method is more general. The github repository's first example is with eight Taylor terms (P = 8).
8.
by fheinsen 8mo ago
As the error via linear approximation approaches similar magnitude as numerical error via quadratic computation, don’t the two start becoming comparable in practice ? I ask because in practice , for inference, attention is typically com
9.
Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation(arxiv.org)
164 points by fheinsen 8mo ago|96 comments
10.
by fheinsen 11mo ago
Yes. See this comment for context: https://news.ycombinator.com/item?id=45611863
11.
by fheinsen 11mo ago
To the best of our knowledge, this is the first time anyone has successfully trained a non-diagonal RNN computed in parallel, via prefix scan, without requiring any form of stabilization. We abstained from claiming as much out of an abund
12.
by fheinsen 11mo ago
The manuscript formally defines GOOMs as a set of mathematical objects, shows that floating-point formats are a special case of GOOMs, and notes that they extend prior work on logarithmic number systems (LNSs), which go back to at least the
13.
A non-diagonal SSM RNN computed in parallel without requiring stabilization(github.com)
9 points by fheinsen 11mo ago|1 comments
14.
by fheinsen 11mo ago
The paper formally defines GOOMs as the subset of the complex plane that elementwise exponentiates to the real line. We'll update the README to make that clearer. Thank you for pointing that out!
15.
by fheinsen 11mo ago
> GOOMs generalize the concept of "order of magnitude" to incorporate complex numbers that exponentiate to real ones. As with ordinary orders of magnitude, GOOMs are more stable than the real numbers to which they exponentiate.
16.
Generalized Orders of Magnitude (GOOMs)(github.com)
2 points by fheinsen 11mo ago|4 comments
17.
by fheinsen 11mo ago
repo: https://github.com/glassroom/generalized_orders_of_magnitude
18.
by fheinsen 7y ago
I'm the author of that paper. Happy to answer questions about it here.
19.
by fheinsen 7y ago
Hi HN, I'm the author of this paper. As it turns out, I posted it and answered a few questions about it on HN a few days ago, on this thread: https://news.ycombinator.com/item?id=21397444 Please feel free to ask questi
20.
by fheinsen 7y ago
Thanks. Will do (work permitting!). FWIW, a while back I reimplemented and tinkered a bit with the Sabour et al. version too... and did not see much promise in it. Note that the routing algorithm I've proposed generalizes to vectors (
21.
by fheinsen 7y ago
FYI, I reached out to two of those individuals (one is a CVPR reviewer, it turns out) and both suggested I first upload this to arXiv, so I did that yesterday. The paper is now stamped with a date, on the queue for site-wide notification. T
22.
by fheinsen 7y ago
Thank you for saying that. Sometimes I forget how petty and small people can be, especially when they are under pressure, academic and otherwise. I'll take a look at submitting it to CVPR. In the meantime, please circulate my work
23.
by fheinsen 7y ago
Thank you! Yes, I think this has legs. Maximizing "bang per bit" (a) seems truly a new idea , as opposed to some minor tweak on the same old thing, and (b) the evidence so far shows it works better than previous methods . (FWIW,
24.
by fheinsen 7y ago
> You’re comparing sentence classification done using transformer embeddings to older results which use inferior embeddings. How do regular convnets do when you feed them transformer embeddings? Actually, I'm comparing it to recent
25.
by fheinsen 7y ago
> How does a regular convnet do on another domain? As far as I know, regular convnets have failed to outperform query-key-value self-attention models (i.e., transformers based on Vaswani et al.'s work) on pretty much every sequence
26.
by fheinsen 7y ago
I’m surprised you did not comment on the fact that my version of EM routing also achieves SOTA on another domain, natural language. Same code. Here are the answers to your questions: 1. The final, published version is stamped “ICLR 2018,
27.
by fheinsen 7y ago
Great questions. Happy to answer them here. First of all, this work builds on Hinton et al.’s second paper, the one about EM routing of matrix capsules , from last year: https://ai.google/research/pubs/pub46653
28.
Show HN: Beating Hinton et al.'s capsule net with fewer params and less training
14 points by fheinsen 7y ago|16 comments