5 ms·
SubQ: a sub-quadratic LLM with 12M-token context
- tuandin 5mo ago[flagged]
- mohsen1 5mo ago- magic.dev claimed 200M context window and it's been two years since and no real product yet. - They are admitting that this is built on top of a Chinese model[1] - They committed a huge chart crime with the Y axis of a chart comparing to Opus on their website that I can't find anymore (Too embarrassing to keep?). The delta between their score (81%) vs. Opus (87%) on SWE bench was hugely minimized - They named the company subquadratic but in parts they said O(1) linear scaling. At O(1) you could do much more than 12M tokens context window. At O(log n) even. I hope this is real but I doubt...
- artisin 5mo agoAh, I nearly forgot about magic.dev. I took a quick peek to check up on them. Welp, last social/blog activity was in... 2024. But hey, their careers page still says they're hiring! So they must be doing just fine.
- shdh 5mo agoThey did raise over $500M
- alexsubq 4mo agoThe chart crime was not intentional! We will not make you wait two years. We are O(n), not O(1). O(1) would unfortunately be an impossibility. We may as well do infinite context at that point!
- esafak 4mo agoGood luck.
- alexsubq 4mo agoThanks!
- bbctr1 4mo agoWhat’s keeping you from releasing paper and access to the model?
- alexsubq 4mo agoModel: - making sure it has been properly red-teamed, meets user preferences, etc. - it depends on what folks want in the model. Our original papers was mostly the technical blog post, but we decided to wait a little longer to see what else folks wanted and share more benchmarks
- pvtmert 4mo ago> not affiliated with subq, i see in the linked post they mention O(n) not O(1). O(1) would basically be impossible and instant. Something like no compute required, constant results... The name subquadratic is actually good and makes sense to me. Because today's models are usually O(n^2) or worse. Anything equals or less than O(n^1) is basically sub-quadratic. Meanwhile O(log n) would be logarithmic as the log name indicates. But we have a long way to go there. Maybe with double tokenizer plus extensive caching it may be possible... What I mean here is tokenizing the user input; then capturing intent; caching intent -> response. So that next time once you get the intent, you don't need to do full transformer inference compute. This can be logarithmic complexity in terms of time complexity.
- kovek 5mo ago> The core idea is content-dependent selection. For each query, the model selects which parts of the sequence are worth attending to, and computes attention exactly over those positions. I don't know if this will help for things like understanding code, where the all relevant parts can be the file of 1000 lines that we are analyzing, and where every token is relevant in understanding recursion, loops, function calls, etc. This sounds like it would be great to do SSA before passing things along to a code model like claude code. Let me know if I misunderstood
- alexsubq 4mo agoYeah, tokens are excluded, only pairwise relationships between tokens. Coding is something we are looking at carefully!
- in-silico 5mo agoI wonder how different their method actually is from other sub-quadratic sparse attention methods like Reformer [1] and Routing Transformer [2]. [1]: https://arxiv.org/abs/2001.04451 https://arxiv.org/abs/2001.04451 [2]: https://arxiv.org/abs/2003.05997 https://arxiv.org/abs/2003.05997
- _burner256 5mo agoFunny how they claim a 12M context window, yet all benchmarks are cherry picked with a 1M context window. Also, nobody has questioned how they did a training run before receiving funding. SoTA training runs cost well above $10M, yet no mention of funding prior to yesterday, interesting.
- nicomontuschi 4mo ago[flagged]
- nicomontuschi 4mo ago[flagged]
- lostmsu 4mo agoNo API access for independent verification - vaporware. See also comment about astroturfing accounts in this thread.
- noashavit 4mo agoAn architecture where compute grows linearly with context length seems dangerous. It can get very expensive as context grows and performance degrades
- roflcopter69 4mo agoI'm usually okay with most LLM-assisted writing, but the amount of "it's not X. it's Y" style of phrases in https://subq.ai/how-ssa-makes-long-context-practical https://subq.ai/how-ssa-makes-long-context-practical is disturbing. Also, holy moly, the astroturfing. But I'll still keep an eye on what they'll show up with in the next months. Sounds intriguing.
- charliecs 4mo agoDon't let a C-suite marketing video blow your mind. They are trying to discover the new Transformer, that's not easy. 12 million token context with worse quality means this isn't going anywhere. Want to bet me bitcoin that we won't be talking about them in 1 year? Heck, they may have found something great, but the prior should be one of skepticism.
- remaximize 5mo agoThis is pretty remarkable. We've spent a lot of time finding workarounds for LLMs reading long docs. Now that's gone.
- wilddolphin 5mo ago[flagged]
- deleted 5mo ago[deleted]
- williamimoh 5mo agoLooks like long context isn’t a problem anymore
- tamarru 5mo agoNeither is cost, and latency, in the long-term. LLMs ultimately become more economically viable than they are now, and broaden the scope of every existing LLM-driven application (particularly STS, conversational AI, etc, etc.)
- pstorm 5mo agoI’m very surprised this isn’t getting more attention. Am I missing something? It seems at or above SOTA on the given benchmarks, doesn’t have context rot, is orders of magnitude faster, and uses less compute that current transformer models. I suppose it’s just an announcement and we can’t test it ourselves yet.
- amw-zero 5mo agoYes you're missing something: the snake oil.
- dvfjsdhgfv 5mo ago> Am I missing something? Yes, this product doesn't exist. And the last time a company claimed something similar it disappeared after taking money from investors.
- jakevoytko 5mo agoThe proof is in the pudding. At this point, there have been plenty of models that overperformed on benchmarks and underperformed on real work. So my stance is that I'm curious, I'm excited to see where it goes, and I don't believe it until I can try it.
- remaximize 5mo agoI agree, it's a real architectural breakthrough if true
- alexsubq 5mo agoWe are SOTA in some ways and not in others, continuously working to make it better! We need a little more time to scale, as we are working on things like disaggregated prefill, etc., the norms of large-scale model infra. I am happy to answer any questions!
- dirtyalt 5mo agoI have questions. Can you back up your claims? Why did you not release the white paper in parallel with the product? Feels really fishy.
- thlt 5mo ago[dead]
- creamyhorror 5mo agoWhether this is real or not, multiple commenters here look like astroturfers - created in the past year (or hours) with very low karma
- GorbachevyChase 5mo agoThere are some comments which are identical to comments on X as well. That is not the say the frontier labs do not engage in highly unethical marketing, but this is a little bit too obvious.
- 2001zhaozhao 5mo agoAssuming this is real and much better than existing linear attention methods as advertised, not launching with a technical report is a big miss. Edit: their blog post (https://subq.ai/how-ssa-makes-long-context-practical https://subq.ai/how-ssa-makes-long-context-practical) does go pretty in-depth about it Edit 2: the fact that they're going straight for an end-to-end coding product on day 1 is very ambitious. Other speed/efficiency-oriented AI companies (Cerebras and Inception come to mind) still don't have a first-party coding product after years. IMO this is absolutely the right way to go if they really do have the big breakthrough they're claiming.
- avrilfanomar 5mo agoyou really call this 1-minute blog post "in-depth"?