thunderbird120

6 ms·

1.
by thunderbird120 1mo ago
Lots of chiplet interconnects hurt your max performance pretty severely and drives up your complexity a lot. There's a reason they want to keep it monolithic. I already harped on Cerebras, but their approach of just copy/pasting a
2.
by thunderbird120 1mo ago
It makes fairly little sense for consumer hardware, unless you're embedding an absolutely tiny model. Since you're embedding the entire model in the chip itself, required chip size scales directly with model size. The reason they
3.
by thunderbird120 2mo ago
This is a near perfect encapsulation of the exact type of person you should not take financial advice from.
4.
by thunderbird120 2mo ago
Kimi Delta Attention (KDA), despite having "Attention" in the name, isn't really attention at all in any conventional sense. It's more like an RNN which can be efficiently parallelized during training. It's a very s
5.
by thunderbird120 3mo ago
Medical expenses would be a combination of cost of insurance / treatment (PPP relevant) and government transfers in a country with partially or fully public healthcare, which is why it's so important to do both.
6.
by thunderbird120 3mo ago
I see. That paper is looking a median income from a PPP perspective but without accounting for taxes and transfers, which is a tiny bit unorthodox in this exact context which tries to get into the weeds slightly more than just standard PPP
7.
by thunderbird120 3mo ago
>median is higher in Germnay I cannot think of any standard by which this is true, certainly not by nominal or PPP income for either personal or household income. [1] https://en.wikipedia.org/wiki/Median_income [2] h
8.
by thunderbird120 4mo ago
Yes, since the weights being updated are a small subset of the overall total it's manageable. Just like how each separate conversation currently requires you to store a separate KV cache, you'd need to store the fast weights separ
9.
by thunderbird120 4mo ago
The idea of periodically stopping to write blocks of recent context into a fast-weight state is interesting, but I think it liked it better when E2E-TTT[1] did it. It's a more flexible and elegant continuous learning approach. Essentia
10.
by thunderbird120 4mo ago
You can merge it into OpenWhip https://old.reddit.com/r/ClaudeAI/comments/1sek7gk/someone_m...
11.
by thunderbird120 4mo ago
>Its turn around time is ridiculous, it has to be maintained with specialized equipment/hangers, along with external contractor assistance. >Compared to the Gripen, as an example, which can land on a freeway and be up in the air
12.
by thunderbird120 7mo ago
Cerebras has effectively 100% yield on these chips. They have an internal structure made by just repeating the same small modular units over and over again. This means they can just fuse off the broken bits without affecting overall functio
13.
by thunderbird120 7mo ago
That's what it's running on. It's optimized for very high throughput using Cerebras' hardware which is uniquely capable of running LLMs at very, very high speeds.
14.
by thunderbird120 9mo ago
You prevent enemy air defenses from shooting down your aircraft by blowing them up as part of SEAD/DEAD missions, which is exactly what the US did.
15.
by thunderbird120 9mo ago
Takes much longer to build, requires a much larger up-front investment, and requires a lot more land. The footprint needed when trying to generate this much power from solar or wind necessitates large-scale land acquisition plus the transmi
16.
by thunderbird120 1y ago
Precision bombing during WW2 was not possible at the required scale. To put a bomb precisely on target back then you needed something like a dive bomber, a tactic which is incompatible with strategic-scale bombing. Even "precise"
17.
by thunderbird120 1y ago
Nice to see that they added that, but that section wasn't in the article when I wrote that comment.
18.
by thunderbird120 1y ago
This article doesn't mention TPUs anywhere. I don't think it's obvious for people outside of google's ecosystem just how extraordinarily good the JAX + TPU ecosystem is. Google several structural advantages over other ma
19.
by thunderbird120 2y ago
People would correctly identify that their standard of living is being reduced for ideological reasons without tangible individual benefits and would likely not respond well to that, resulting in a loss of political power for whatever movem
20.
by thunderbird120 2y ago
Yeah, that's the normal outcome for papers like this. Papers which claim to be groundbreaking improvements on Transformers universally aren't. Same story roughly once a month for the past 5 years.
21.
by thunderbird120 2y ago
If that were totally true you would expect to see more or less uniform ratios of HP/HD cells mixes across different product types, but that's very much not the case. Dennard scaling may be dying but it's not dead yet. You can
22.
by thunderbird120 2y ago
If it were that simple fabs wouldn't offer a standard cell libraries in both high performance and high density varieties. TSMC continues to provide both for their 2nm process. A tradeoff between power efficiency and raw performance con
23.
by thunderbird120 2y ago
>Intel on 18A is literally TSMC's 3nm process + backside power delivery, which means more power efficiency, performance also less heat. That's a pretty serious abuse of the word "literally" given that they have nothin
24.
by thunderbird120 2y ago
The cost for both training and inference is vaguely quadratic while, for the vast majority of users, the marginal utility of additional context is sharply diminishing. For 99% of ChatGPT users something like 8192 tokens, or about 20 pages o
25.
by thunderbird120 2y ago
No, additional context does not cause exponential slowdowns and you absolutely can use FlashAttention tricks during training, I'm doing it right now. Transformers are not RNNs, they are not unrolled across timesteps, the backpropagatio
26.
by thunderbird120 2y ago
Frankly, yes. Many creative works these days require the effort and input of so many people, so much time, and so much money that they can't have a specific creative vision. Mediums like book, comics, indie movies, and very low budget
27.
by thunderbird120 2y ago
The APU offerings from both AMD and Intel have been improving pretty rapidly recently but they're still pretty low end by dGPU standards. I can certainly see them causing the death of dGPUs in laptops but it's difficult to imagine
28.
by thunderbird120 2y ago
The 18A naming convention is as simple as not wanting to put a decimal point in their node name. Simple as that. It's not like the the "nanometer" node names actually mean anything at this point anyways. Neither TSMC N3 or In
29.
by thunderbird120 2y ago
Intel 3 is an economically interesting node. Performance seems to be somewhere between TSMC N4P and N3B which is respectable but Intel 18A is due very soon after it debuts and incorporates GAA transistors and backside power delivery. It see
30.
by thunderbird120 2y ago
The ISA is much less important than many people seem to think. The RISC vs CISC debate is beyond outdated at this point because no modern architecture actually works strictly like either under the hood. Organizations who did x86 architectur

More ›