in-silico

6 ms·

1.
by in-silico 7d ago
A lot of current diffusion LLMs don't convert tokens to continous space before noising them. They add discrete "noise", which is often as simple as replacing some tokens with [MASK]. The real problem is that when using fewer
2.
Predictions for the Era of Continual Learning(dwarkesh.com)
3 points by in-silico 1mo ago|0 comments
3.
by in-silico 1mo ago
Does the free tier really not have access to reasoning models? That would explain a lot of the terrible AI/LLM takes online.
4.
SWE-Touch: Benchmarking Coding Agents When Users Touch the Code(trae1oung.github.io)
1 points by in-silico 1mo ago|0 comments
5.
by in-silico 2mo ago
This algorithm: sample a bunch of latents, train the model using the one with the lowest error. IWAE: sample a bunch of latents, weight the loss of training the model using each one by softmax(-error). For images and text where the errors h
6.
by in-silico 2mo ago
The training method is very similar to Importance Weighted Autoencoders: https://arxiv.org/abs/1509.00519
7.
by in-silico 2mo ago
Did the model really need to hack huggingface to get access to ExploitGym data? I'd imagine that once it had full internet access it could have just used the HF API or website (but the heavy prompting/nudging towards hacking mad
8.
by in-silico 2mo ago
What would evidence of "intelligence" or "creativity" look like for you?
9.
by in-silico 2mo ago
One "solution" to these problems is to have the probes land on planets instead of asteroids, and build the necessary infrastructure there.
10.
by in-silico 2mo ago
The difference is that artistic sensibility is largely subjective. This means that: 1. It's hard to measure (and people can disagree about it) 2. It can't really be improved using RL without a human in the loop (which is how math
11.
by in-silico 2mo ago
I feel like the data should have been generated by a much less predictable policy. It often feels like the model is ignoring my inputs and just doing what it would expect the bot to do (which is unsurprising if the model could predict what
12.
A global workspace in language models(anthropic.com)
467 points by in-silico 2mo ago|201 comments
13.
by in-silico 3mo ago
This post misses a very important point: humans aren't 100% correct either. This means that the bar for being useful (at tasks that humans usually do) isn't perfection, it's human-level correctness. If we can create AI models
14.
A New Look at AI's Impact on Jobs(ramp.com)
5 points by in-silico 3mo ago|0 comments
15.
by in-silico 3mo ago
Why can't a next token predictor do math? Humans aren't calculators either, but we can do math. If you want proof just look at the benchmarks. Modern frontier models can get basically perfect accuracy on American Invitational Math
16.
by in-silico 3mo ago
Additionally, maybe it's easier for a model to realize that it doesn't know the answer when the question is easier. If Opus gets all but the hardest questions right, it might have a higher hallucination rate because the questions
17.
by in-silico 3mo ago
Neither of these strike me as particularly groundbreaking. The first idea (as I understand it as retrieving token ids rather than hidden states) is going to really struggle to do useful compositional reasoning and contextual recall. The sec
18.
by in-silico 3mo ago
> LLMs cannot do math This is plainly not true anymore
19.
by in-silico 3mo ago
These types of ablation studies are always good. However, I'm not sure how generalizable the language model findings here are. Their 1.2B model was trained on only 10B tokens, which is less than half of the chinchilla compute optimal n
20.
by in-silico 3mo ago
> The problem is "public schools". The idea itself is wrong, and it can't be made to work. Do you have an alternative idea in mind?
21.
by in-silico 4mo ago
Care to elaborate?
22.
by in-silico 4mo ago
This is really semantics, but I wouldn't call attending to the KV cache re-reading the context. The model takes in the context, encodes it into a "memory" (the KV cache), and accesses that memory later. That fact doesn't
23.
by in-silico 4mo ago
> It's not changed by the experience The entire file is not changed, but the KV cache is. > It doesn't remember anything The model definitely remembers previous exchanges within the same conversation.
24.
by in-silico 4mo ago
> Would be interesting whether it is possible to write a LLM-like program just using compression and function interpolation algoritms. gzip can be used as a (not very good) LLM-like text and image generator: https://arxiv.org&
25.
by in-silico 4mo ago
We know how the models are built and trained, but we have a very limited understanding of how the final products work . That is to say, we don't know why they give the outputs that they do. If we did know how they worked, AI interpret
26.
by in-silico 4mo ago
> But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers A bold claim given that the current top post on HN is "An OpenAI model has disproved a c
27.
by in-silico 4mo ago
> Can you elaborate on this? I'm seen estimates of ~1.5bit per English letter The reference I always go back to is the GPT-3 paper. The cross-entropy loss (an upper bound for entropy) got down to 1.75 nats (2.5 bits). I took 2.1 bec
28.
by in-silico 4mo ago
I think you underestimate just how much information 100M words-ish of information is. It's like a 300,000 page novel. That's a 50 foot (~15 meter) thick book. Surely with (much less than) 300K pages you could describe every meanin
29.
by in-silico 4mo ago
While there is a limit to the amount of information you can fit in a fixed-size state, the theoretical ceiling is pretty high. A Hebbian associative matrix (one of the simplest and weakest memory constructions) can store about 0.7 bits of i
30.
by in-silico 4mo ago
They basically just added DeltaNet hypernetworks to existing LLMs. Nothing super novel or groundbreaking, but a moderately interesting read.

More ›