f_devd

8 ms·

91.
by f_devd 2y ago
I was responding to > a Transformer layer can look back see O(X^2) numbers, while an RNN can only see O(X) numbers The thing is RNN can look back infinitely if you don't exceed the state capacity. For transformers the state it is de
92.
by f_devd 2y ago
Transformers actually have an quantifiable state size (see https://hazyresearch.stanford.edu/static/posts/2024-06-22-ac... ) although it's anywhere between 200k and 2M floats (for 360M and 1.33B respectively i
93.
by f_devd 2y ago
Seems they did "introduce" it in 20A this year but implies no commercial use yet, that would be with 18A in 2025[0]. [0]: https://www.intel.com/content/www/us/en/newsroom/opinion/con..
94.
by f_devd 2y ago
I don't think the rust team would claim writing idiomatic safe rust code is always zero-overhead. Indexing into an array adds bounds checks which takes non-zero time. So by default you trade-off safety for performance to some degree; b
95.
by f_devd 2y ago
If you read the article I linked they show that it's entirely based on 16x16 matrices (or "tiles") which is fairly standard across gpus.
96.
by f_devd 2y ago
> How much is the flash attention algorithm tied to the hardware? The original FA, almost none. For the latest versions depends on your abstraction, ThunderKittens[0] provides about the same speed up over FA2 (1.3x-2x%) as the article bu
97.
by f_devd 2y ago
Only for the reference implementation, flash attention is just based on optimization of the memory bandwidth between caches. The ThunderKittens[0] library has what is effectively flash attention v3, and they are working on supporting AMD ar
98.
by f_devd 2y ago
It is possible, mainly depends on LLVM/clang support as rust ASM is very easy to do
99.
Deps.rs: Keep your dependencies up-to-date(deps.rs)
2 points by f_devd 2y ago|0 comments
100.
by f_devd 2y ago
Not an expert, but I've always understood it as (LI5): You may have heard the effect of light is like a wave, higher frequency wave-lengths have a shorter wavelength, which means it can 'reach' smaller features without impact
101.
by f_devd 2y ago
ISPs do give enough where I am the issue is usually when you are behind 2 firewalls past that or have VMs that you want to DHCP, things like that. A real world example my ISP provides /56 at the router level but if you put a firewall b
102.
by f_devd 2y ago
My only gripe with IPv6 is that clients use an entire /64, I get it's useful for SLAAC, but I wish I could make clients receive a /68 or more with DHCPv6 so that the address space can be subdivided even when you are already b
103.
by f_devd 2y ago
(disclaimer: I've not looked at mamba specifically yet) but state spaces as used current are different from traditional RNNs in a very simple way: the state is linearly (often associatively) accumulated while in RNN the state is passed
104.
ReScript: Fast, Simple Typed JavaScript from the Future(rescript-lang.org)
2 points by f_devd 2y ago|0 comments
105.
by f_devd 2y ago
Doesn't fix ranking unfortunately
106.
by f_devd 2y ago
Having written some esp-hal rust, you can also just use crates which use embedded-hal (for the I2C part) and embedded-graphics (for the display driver part). If they don't exist it's usually quite easy to port the C/C++ code
107.
by f_devd 2y ago
I mean, if they somehow ported google analytics (or some other brokered PII network) I think they technically would need consent and disclosure.
108.
by f_devd 2y ago
That's what I've observed, I gave it a task for a PoC on something I've been thinking about for a while and it's answer while syntactically correct is entirely useless (in the literal sense) due to it ignoring parts of t
109.
by f_devd 2y ago
Not for YouTube, but Netflix DRM possibly?
110.
by f_devd 2y ago
This is one of the main features I miss in rust, build.rs kind of gets you there (if you really need it) but it's always jank codegen.
111.
by f_devd 2y ago
JMAP exists
112.
by f_devd 2y ago
I would expect the price to decrease over time given perovskite solar cells are currently not yet mass produced and still actively being researched.
113.
by f_devd 2y ago
I recommend checking out XXHash[0], FNV is simple but not really optimized and relatively low quality (often still good enough). From the readme page: Hash Name, Bandwidth, Small Velocity, Quality, Comment XXH3 (SSE2), 31.5 GB/s,
114.
by f_devd 2y ago
> Conversely, your brain does not contain infinite storage. But you can address it and use it, you can think of a system which can make use of infinite memory and then execute that system. LLMs they cannot use something like `set(0x808
115.
by f_devd 2y ago
The argument isn't finite memory but rather the lack of it being expandable memory; as in it cannot learn to read/write to a hypothetical infinite tape. (Although I'm maybe not the right person to word it, I recommend the pri
116.
by f_devd 2y ago
Interesting, I thought it was proven Transformers cannot do recursion[0] and aren't actually able to be equivalent to a turing machine due to lack of expandable memory, and are rather finite state automata. [0]: https://yout
117.
"App Store Levels Playing Field and Supports Developers"(boehs.org)
3 points by f_devd 2y ago|0 comments
118.
by f_devd 2y ago
I'm not too familiar with SHARP but I've implemented SophiaH before, it is effectively mSGD but with the momentum being scaled by inverse of a separate hessian moment with some additional clipping. It works surprisingly well if yo
119.
by f_devd 2y ago
I'm unfamiliar with a bare-metal thread model that doesn't do preemption outside of a Runtime. I imagine you'd need to effectively inject code to do a cooperative switch as there aren't many ways for a cpu to exit it&#x2
120.
by f_devd 2y ago
I think you might be confusing Runtime, OS and bare-metal primitives. Java virtual threads are possible because there is always the runtime which code will return to, and since it's already executing in a VM the concept of Stack/H

More ›