jsenn

7 ms·

1.
by jsenn 6d ago
This is probably a harness problem rather than a model problem. GitHub Copilot will happily and effectively use Powershell while Claude Code struggles in my experience.
2.
by jsenn 2mo ago
I don’t see Tao suggesting what you have suggested there. Instead he suggests that humans responsibly disclose AI use, and that mathematicians develop a set of norms to deal with an overabundance of AI generated results. For example, he sug
3.
by jsenn 2mo ago
I can see this being important if you only care about the results as evaluations of AI progress, but if what you care about is the math itself why should you care about the prompt or anything other than the proof?
4.
What Is Enlightenment (Kant)(columbia.edu)
1 points by jsenn 2mo ago|0 comments
5.
The Most Iconic American Artwork Is the Hardest to See(nytimes.com)
1 points by jsenn 3mo ago|0 comments
6.
by jsenn 3mo ago
This looks cool, but I wonder how well their trained compiler generalizes to new task families . They trained on 29 specific types of tasks, with 800 sub tasks and many rephrasings of each one (the specs). They hold out some specs for vali
7.
by jsenn 3mo ago
Their demo is almost unbelievably fast, but as I understand it, the limitation of Taalas's strategy is KV-cache. This grows with context length, so either needs to be stored in SRAM (small) or streamed in (slow). Even for a tiny model
8.
by jsenn 3mo ago
> isn't the verification code going to be sloppy as well The beauty of formal methods is it doesn't matter if your proof is sloppy. As long as it passes verification, it is correct. And unlike in pure math, the proof that a sof
9.
Modeling Snakes and Ladders: The Board(john.senneker.ca)
1 points by jsenn 4mo ago|0 comments
10.
by jsenn 5mo ago
The article you are responding to showed that a strange LLM behaviour was caused by a training signal that was explicitly designed to produce that type of behaviour. They were able to isolate it, clearly demonstrate what happened, and roll
11.
by jsenn 7mo ago
Is this parlour trick so different from useful tasks like “implement this feature while following the naming conventions of my project”?
12.
by jsenn 8mo ago
> Section 2.6 gives the hidden state size per token, which, on first read, is strictly larger than the hidden state in normal attention This is where you’ve gone off track. The “hidden state” for their model is a fixed size thing, like i
13.
by jsenn 8mo ago
You can find papers discussing "cubic" attention, i.e. each token gets to interact with each pair of other tokens, but always in very theoretical settings with single-layer transformers on contrived synthetic tasks. Keep in mind
14.
by jsenn 9mo ago
If you remove the terms "self", "agency", and "trivially reducible", it seems to me that a classical robot/game AI planning algorithm, which no one thinks is conscious, matches these criteria. How do you d
15.
by jsenn 1y ago
I don’t know the post you’re referring to but I highly recommend How the Immune System Works by Lauren Sompayrac. It explains the interesting parts without getting bogged down in the details of every signalling pathway, but without dumbing
16.
by jsenn 1y ago
A string fixed at both ends produces harmonic sounds because of its particular structure. In order to have a non-integer overtone the ends would have to move up and down, which by construction they can't. Similarly for wind instruments
17.
by jsenn 1y ago
> It will have overtones that are integer multiples of the fundamental that give it its characteristic sound. What I’m wondering is why would the overtones go in integer multiples (I.e. be harmonic) for a fan? A flute and a saxophone hav
18.
by jsenn 1y ago
Is the drone of a fan harmonic? I would’ve thought it’s more like a repetition pitch so its overtones would not be harmonic and would not exhibit a missing fundamental. Agree with the broader point, just curious if there’s some interesting
19.
An analytic theory of creativity in convolutional diffusion models(arxiv.org)
2 points by jsenn 1y ago|0 comments
20.
by jsenn 1y ago
Apple's implementation of this is discussed here: https://news.ycombinator.com/item?id=43685714
21.
Differentially Private Synthetic Data via Foundation Model APIs(arxiv.org)
3 points by jsenn 1y ago|1 comments
22.
by jsenn 1y ago
yeah, the linked paper [1] has more detail--basically they seem to start with a seed set of "class labels" and subcategories (e.g. "restaurant review" + "steak house"). They ask an LLM to generate lots of rando
23.
by jsenn 1y ago
> This approach works by randomly polling participating devices for whether they’ve seen a particular fragment, and devices respond anonymously with a noisy signal. By noisy, we mean that devices may provide the true signal of whether a
24.
by jsenn 1y ago
I think the main advantage is that you can compute the extra parameters (the PRNG seeds) from the network weights alone, whereas most other quantization methods require simulating the quantization procedure at training time (Quantization-Aw
25.
by jsenn 1y ago
> What makes this technique particular to LLM weights This is my understanding as a non-expert. LLM activations tend to be relatively sparse with large outliers. With linear quantization, this means you either have to clip off the outlie
26.
by jsenn 1y ago
This doesn’t answer your question, but one thing to keep in mind is that past the very first layer, every “token” position is a weighted average of every previous position, so adjacency isn’t necessarily related to adjacent input tokens. A
27.
by jsenn 2y ago
Are there any technical innovations here over Moshi, which invented some of the pieces they use for their model? The only comparison I see is they split the temporal and depthwise transformers on the zeroth RVQ codebook, whereas Moshi has a
28.
by jsenn 2y ago
> To train DeepSeek-R1-Zero, we adopt a rule-based reward system that mainly consists of two types of rewards: > Accuracy rewards: The accuracy reward model evaluates whether the response is correct. For example, in the case of math p
29.
by jsenn 2y ago
You might be interested in HippoRAG [1] which takes a graph-based approach similar to what you’re suggesting here. [1]: https://arxiv.org/abs/2405.14831
30.
by jsenn 2y ago
Same experience embedding random alphanumeric strings or strings of digits with smaller embedding models—very important to filter those out.

More ›