cgadski

10 ms·

1.
by cgadski 20d ago
Yep. Using the same kind of calculus ideas, I can also think about a vector field that has a Dirac mass of divergence at some point and zero divergence everywhere else. Then you get an expression that you can sum over faces to determine if
2.
by cgadski 5mo ago
Does anyone understand how/why old HN accounts become mouthpieces for language models?
3.
by cgadski 6mo ago
No marketing bots on HN please.
4.
by cgadski 8mo ago
https://cgad.ski
5.
by cgadski 1y ago
I was a little informal with my argument. It's not strictly true that we only see C = 0.2 when K = 2. I was reading what the graph says about the case when N is much greater than k. I'll try to clarify. C is meant to be the smalle
6.
by cgadski 1y ago
There's a lot of beautiful writing on these topics on the "pure math" side, but it's hard to figure out what results are important for deep learning and to put them in a form that doesn't take too much of an investm
7.
by cgadski 1y ago
> The implications of these geometric properties are staggering. Let's consider a simple way to estimate how many quasi-orthogonal vectors can fit in a k-dimensional space. If we define F as the degrees of freedom from orthogonality
8.
by cgadski 1y ago
The technical report says (page 7): > Our architectural choices are closely aligned with principles observed in biological brains. How? They point out three design choices: linear attention, MoE layers, and spike coding. Apparently linea
9.
Bayes, Bits and Brains(bayesbitsbrains.github.io)
96 points by cgadski 1y ago|6 comments
10.
by cgadski 1y ago
It's not really about the language. If someone doesn't speak English well and wants to use a model to translate it, that's cool. What I'm picking up on is the dishonesty and vapidness. The article _doesn't_ explore
11.
by cgadski 1y ago
> This blog post has explored the most critical equations in machine learning, from foundational probability and linear algebra to advanced concepts like diffusion and attention. With theoretical explanations, practical implementations,
12.
by cgadski 1y ago
To summarize: we're making a series of i.i.d. draws from a distribution and asking how many draws N we need to make until we get something larger than our first draw. Conditional on the value of the first draw, N is geometrically distr
13.
by cgadski 1y ago
It's server-side! I render the latex with katex as part of my static generation. See: https://katex.org/docs/api#server-side-rendering-or-renderin... .
14.
Entropy of a Mixture(cgad.ski)
54 points by cgadski 1y ago|5 comments
15.
by cgadski 1y ago
In little bits of free time I get here and there, I've been working on using reinforcement learning to build some better bots for my favorite multiplayer game. Project is up here: https://github.com/cgadski/autotud
16.
by cgadski 1y ago
Where x is the final hidden layer of the base model, the idea here is to steer outputs in some direction by adding a vector y. More specifically, y is an exponential moving average over a sequence of vectors W(z_t), where z_t are some sort
17.
by cgadski 2y ago
Yeah, I think it's really important to understand how to coax non-equivariant models into being equivariant when needed. I don't think purely equivariant architectures are the way forward. One example that comes to mind (I don&#x2
18.
by cgadski 2y ago
One way to understand why without writing down the CDF/PDF: When X is an exponential variable and c is a constant, X + c has the same distribution as X after conditioning on large outcomes. In other words, these two variables have same
19.
by cgadski 2y ago
I'm very excited that we're figuring out how to use deep learning on small numbers of data points! I'm curious about the focus on information compression, though. The classical view of inference as compression is beautiful an
20.
by cgadski 2y ago
Hm, so maybe the following is one answer. In this article, we fix a mereology and a kind of quantity Q that "decomposes" over it---in the sense that Q(p) = sum_{r <= p} q(r) for some function q(r)---and then see that Mobius inv
21.
by cgadski 2y ago
Super incomplete thought: how does this point of view relate to Euler characteristic? Can I get to Euler characteristic by asking how to solve an equation for some quantities q in terms of some quantities Q?
22.
by cgadski 2y ago
It's a little funny to think of a polyhedron (like the set of doubly stochastic matrices) as a manifold. The point here is that we're equipping the interior of this set with a certain "Riemannian metric" and using it to
23.
Decomposing the Dark Matter of Sparse Autoencoders(arxiv.org)
3 points by cgadski 2y ago|0 comments
24.
GPU Puzzles(github.com)
356 points by cgadski 2y ago|40 comments
25.
by cgadski 2y ago
Yeah, also wanted to suggest Stylus labs' app. Don't use a tablet much at present, but two years ago this was the best software I found. Besides selecting/manipulating with rectangle/lasso/intersection selections, h
26.
How efficient are distributed codes?(cgad.ski)
4 points by cgadski 2y ago|0 comments
27.
Distributed Representations (1986) [pdf](stanford.edu)
1 points by cgadski 2y ago|0 comments
28.
When numbers are bits: distributed representations for sets(cgad.ski)
1 points by cgadski 2y ago|0 comments
29.
by cgadski 2y ago
B is the number of data vectors going on. You can erase the line labeled by B without much loss. (You just get the diagram for the feed-forward of a single vector.)
30.
by cgadski 2y ago
After learning about tensor diagrams a few months ago, they're my default notation for tensors. I liked your chart and also Jordan Taylor's diagram for multi-head attention. Some notes for other readers seeing this for the first t

More ›