libraryofbabel

7 ms·

1.
by libraryofbabel 4d ago
This is one of those papers that ought to have several textbook chapters unpacking some of the insights... I'll just pick a couple things I love: 1) There's a kind of "rabbit–duck illusion" moment where they show how you
2.
by libraryofbabel 6d ago
This book (and the book author's blog) is the best source for learning LLM internals for someone just getting started. It's clear and well-written (by a human!) and goes into much more explanatory detail than this does: https:&#x
3.
by libraryofbabel 7d ago
Thanks for clarifying. You're right, I skipped over talking about dynamic looping, since it adds another level of complexity to the discussion, and OpenAI's claim (quoted in TFA) that the compute graph depth of Astra is "with
4.
by libraryofbabel 7d ago
> that effectively moves the CoT inside the architecture This may be a bit of a nitpick, but... does it? I agree that giving the model decisions on looping certainly makes interpretability harder, because it adds more transient internal
5.
by libraryofbabel 7d ago
Well sure, that's the possible weak point in Sebastian's article: it could be true that there's some more sophisticated stuff going on in Astra around looping, because OpenAI haven't specified their architecture. But i
6.
by libraryofbabel 7d ago
Everyone interested in LLM internals should read Sebastian. He's great. The tldr here is that the recent "The Information" article[0] reporting GPT 6 Astra was using “recurrent depth” or “looped transformers" made it sou
7.
by libraryofbabel 8d ago
> This makes the total effort linear over the entire context (or constant per forward pass). This is incorrect. The compute required per forward pass to generate each additional token during decode will scales as O(N), even with a KV c
8.
by libraryofbabel 14d ago
I once heard someone describe Admiral Cloudberg's air crash investigation blog as "the best thing on the Internet" and I still think that might be true. She is a great researcher and a superb writer, which stands out all the
9.
by libraryofbabel 23d ago
> there’s no way to “crack open” an LLM and see precisely where each skill or tendency lives Mechanistic Interpretability has entered the chat. For a classic example, see https://www.anthropic.com/research/tracing-th
10.
by libraryofbabel 3mo ago
I am saying this probably is "silly behavior by a government" and it is a milestone that points towards what the future may look like. Why can't it be both? It's easy to wave this aside as the current administration pl
11.
by libraryofbabel 3mo ago
You may be right, and I actually agree with you: I think that in this case the most likely outcome is that Fable becomes available again at some point, albeit possibly only to a restricted set of users within the US. But I think my larger p
12.
by libraryofbabel 3mo ago
So many comments here missing the big picture, and just gleefully pointing out that Anthropic got what they deserved, or that this is the natural culmination of some kind of marketing stunt. The real story here is that this may be the begin
13.
by libraryofbabel 4mo ago
Where? Please point it out! All he says is there will be more desperate people on the market in future and because they're desperate they'll have to accept trial work. But that's not answering the objection that people who d
14.
by libraryofbabel 4mo ago
Well yes, there is tons of AI bullshit about and all sorts of scammy behavior , but I don’t think that says anything at all either way about whether the core technology is a “scam”, theranos-style. In fact I’m not sure how it could be ot
15.
by libraryofbabel 4mo ago
People that know more about nuclear physics than I do already answered, but I’ll just say that: 1) It’s easy to think about the past in terms of what we now know, and it involves a real effort to put yourself in the shoes of the people livi
16.
by libraryofbabel 4mo ago
You’re thinking of the other bomb, the U-235 one, which they didn’t test at Trinity and which was dropped on Hiroshima. That is two separate pieces of Uranium that are slammed together to create a critical mass. The Pu-239 core was a single
17.
by libraryofbabel 4mo ago
That’s the one I meant. It’s the core, but in a box, which makes it look even more innocuous, like he is indeed just lugging a piece of industrial equipment around. There’s lots of photos of the actual (unboxed) cores online if you search.
18.
by libraryofbabel 4mo ago
I used to teach a class on the history of contemporary science (WW2-present) and I started the class with Trinity. There’s no other moment better. We know how it turned out, but the people there waiting for the test did not know how it woul
19.
by libraryofbabel 4mo ago
> We haven't seen a significant increase in the quality of LMM output since 2023 that hasn't been the result of throwing even more energy and compute at it. This is completely false. Most of the dramatic improvements in LLM qua
20.
by libraryofbabel 4mo ago
> Looking away shall be my only negation. I’ve been thinking of building myself my own frontend to HN that makes it impossible to view comments, for this reason. Yet sometimes there are still really interesting discussions and it’s hard
21.
by libraryofbabel 4mo ago
Well yes, but there is a choice being made here and I would love to believe we can do better. The rational response to being afraid about your livelihood isn’t to spend time filling every HN thread on LLMs with embittered negativity. Not to
22.
by libraryofbabel 4mo ago
This HN thread depressed me. I’m still thinking about why. Look past the press-releasey gushing from OpenAI and there are all sorts of interesting and subtle questions here about the role for LLMs in mathematical research. I urge folks to c
23.
by libraryofbabel 4mo ago
This is a good point, and there’s some deep philosophical questions there about the extent to which mathematics is invented or discovered. I personally hedge: it’s a bit of both. That said. I think it’s worth saying that “LLMs just interpol
24.
by libraryofbabel 4mo ago
It's fast. Two skilled pressmen working together could do 200 to 250 impressions per hour or about one every 15 seconds (which might be 4, 8, 16 pages on each impression depending on page size). That was the speed text was put to paper
25.
by libraryofbabel 4mo ago
Well, yes, but as the other commenter says, that’s a very broad general statement akin to something like “AI will change knowledge work“. That’s certainly true, but how? What are the details? What kind of companies are going to be the winne
26.
by libraryofbabel 4mo ago
Thanks for the summary. I do love Benedict‘s work; I find he’s one of the few commentators who consistently strikes a balance between taking the transformative potential of AI seriously while not falling over into hype. Some things that sta
27.
by libraryofbabel 4mo ago
You didn’t answer my question. You just restated your claims. Specific examples? Specific tasks?
28.
by libraryofbabel 4mo ago
> When you train LLMs on large volumes of text that describe logically consistent facts in a million different ways, the "logic" sort of becomes part of the grammer that the model learns. That is logic becomes a higher kind of
29.
by libraryofbabel 4mo ago
I did read it. She doesn’t mention mathematics or RLVR training once, so I assume you’re referring to my point about empirical testability. Well, I think her statement that the claim “LLMs are stochastic parrots” is not an empirical claim i
30.
by libraryofbabel 4mo ago
Maybe, but a claim about what and LLM is not is still a claim about what it can or cannot do. And specifically: > without any reference to meaning is vague, but I read it as actually quite a strong claim about the limitations of LLMs.

More ›