taykolasinski

6 ms·

1.
Show HN: Mapping where a quadruped RL policy fails, with a live probe(poissonlabs.ai)
2 points by taykolasinski 28d ago|0 comments
2.
Where V-JEPA 2.1's Dense Features Hold Up (and Where They Don't)(poissonlabs.ai)
2 points by taykolasinski 4mo ago|0 comments
3.
by taykolasinski 8mo ago
OP here. This is Part 2 of my reproduction series. I scaled the experiment from 10M params (MacBook) to 1.7B params (8x H100s) to test DeepSeek's instability claims. The paper reported 3,000x signal amplification. I found 10,924x. The
4.
10,924x: The Instability Bomb at 1.7B Scale(taylorkolasinski.com)
4 points by taykolasinski 8mo ago|1 comments
5.
by taykolasinski 8mo ago
That's interesting. I suspect your intuition about scale is correct. The theoretical benefit of mHC is that it acts as a sort of relief valve/router for information flow in very deep/wide networks where the standard residual
6.
by taykolasinski 8mo ago
This is a fantastic catch. I hadn't realized Gemma 3n was already shipping with a variant of this in production. It feels like we are entering the era of residual stream engineering. For a long time, the standard x + F(x) additive back
7.
by taykolasinski 8mo ago
Thanks! Manually created Astro components with inline SVG and CSS animations.
8.
by taykolasinski 8mo ago
That initialization strategy (effectively starting as identity to match the standard residual stream) is clever. It would let you surgery an existing model like Llama-3 and fine-tune it into an mHC architecture. The main risk I see is that
9.
by taykolasinski 8mo ago
That's a fair point. We don't have the weights or code for the closed models, so we can't be 100% certain. However, transformer-based (which their technical reports confirm they are) implies the standard pre-norm/post-nn
10.
by taykolasinski 8mo ago
I’m referring specifically to the fundamental residual connection backbone that defines the transformer architecture (x_{l+1} = x_l + F(x_l)). While the sub-modules differ (MHA vs GQA, SwiGLU vs GeLU, Mixture-of-Depths, etc.), the core sign
11.
by taykolasinski 8mo ago
OP here. I spent the last few days reproducing the mHC architecture from the recent DeepSeek paper (2512.24880). Two key takeaways from the reproduction: Unconstrained Hyper-Connections really do explode (7x amplification even at 10M scale)
12.
Reproducing DeepSeek's MHC: When Residual Connections Explode(taylorkolasinski.com)
121 points by taykolasinski 8mo ago|30 comments