nsingh2

6 ms·

1.
by nsingh2 9d ago
What's with slop-factories forgetting what a README is supposed to be? A quick intro to the project, not a gigantic abomination with irrelevant details.
2.
by nsingh2 12d ago
I don't know how good of a marketing ploy it is, tbh. Considering it paints them as incompetent, and that they cannot be trusted with developing this technology safely. > It's so stupid it can't follow the basic spirit of
3.
by nsingh2 13d ago
Also note that Opus 5 (High) has an index value of 62, vs Fable 5 (Max) has 61. So some strangeness going on with that index.
4.
by nsingh2 13d ago
I have some doubts about AA-index. For example Opus 5 (High) is at the same index value as Fable 5 (Max), that doesn't seem right.
5.
by nsingh2 14d ago
One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative o
6.
by nsingh2 15d ago
I would expect the benchmark scores to be nonlinear near the top, as the easier tasks get solved and the harder ones are left over. So going from 10 to 15 would be easier than going from 60 to 65. I only take the Intelligence Index value ro
7.
by nsingh2 15d ago
From Artificial Analysis cost per task, it looks like Fable 5.1 (max) is more expensive per task than Fable 5 (max)? Cache hit price went down, but the other components still add up to more. Edit: 5.1-xhigh seems to be cheaper than 5-max, a
8.
by nsingh2 16d ago
All of this stuff seems like a band-aid solution. These things need to be trained ground-up to maintain and update persistent memory (maybe outside the context window?). Also seems like a requirement for any sort of continual learning capab
9.
by nsingh2 25d ago
What's going on with sol here? The note says it spends a lot of time waiting, did it just not effectively use time (i.e. something like parallel runs) so it's graph ends up being stretched in the time axis? I'm also seeing no
10.
by nsingh2 1mo ago
Also with Qwen 3.8 being more token hungry than Luna, using around 2.3x tokens. Which hurts for local deployment.
11.
by nsingh2 1mo ago
One line of evolution seems to be toward some form of latent-space reasoning, as in [1]. Natural language seems like a relatively low-bandwidth channel for intermediate reasoning. [1] https://github.com/sapientinc/HRM-T
12.
by nsingh2 1mo ago
I've never found these discussions to be all that useful, because it's hard to define what conditions are sufficient to say something is "thinking" or "reasoning". It just ends up being circular and metaphysica
13.
by nsingh2 1mo ago
You are making two different claims, and then treating them as one. 1. Mimicking a process may not prove that the system is truly preforming the same process. In this case, a "reasoning" process. 2. A system that simulates a proce
14.
by nsingh2 1mo ago
Work doesn't start with a PR description though. I'm assuming most people that are using LLMs start with some sort of document (plan, spec, intent, etc) which captures intent. I guess you could also use all the session rollouts sa
15.
by nsingh2 2mo ago
Their models don't consume encrypted text, it would be absurd to train them to do so. Surely they decrypt the text before feeding it into the LLM, so the contents could get leaked out by asking it.
16.
by nsingh2 2mo ago
My own experience: much less context bloat, and hidden system prompts. Can tune it to your own workflow, instead of dealing with whatever hidden system prompt Claude Code forces on you.
17.
by nsingh2 2mo ago
To be fair, Pi only had to be useful. Claude Code had to solve the difficult unsolved problem of making a terminal print text, so naturally they built a small game engine [1] [1] https://x.com/trq212/status/2014051
18.
by nsingh2 2mo ago
I think it might affect real work if part of it requires a lot of thinking, i.e. something similar in nature to a puzzle. There seems to be something wrong with the "commentary" channel related intermediate updates, maybe the mode
19.
by nsingh2 2mo ago
This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early to provide updates. Removi
20.
by nsingh2 2mo ago
This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early to provide updates. Removi
21.
by nsingh2 2mo ago
From some of the numbers I'm seeing in the GitHub issue, the codex desktop app has the same 516 spikes. So most likely it is affected.
22.
by nsingh2 2mo ago
Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and return the wrong resul
23.
by nsingh2 3mo ago
Codex apparently added this too, I only noticed a few days ago: https://github.com/openai/codex/issues/28969 I'm usually at my desk, and get a notification when it stops to ask a question, so I've n
24.
by nsingh2 3mo ago
A highly detailed specification is not what I mean here. It's closer to plugging in a few sentence descriptions (or a totally cluttered brain dump) and having the model interview you to help pin down critical details before continuing.
25.
by nsingh2 3mo ago
Refusing to sufficiently specify a task and hoping the model guesses correctly is not being productive. Again, these models still don't really ask questions when they should. You have to explicitly tell them to. Specifying the problem
26.
by nsingh2 3mo ago
Why supply underspecified requirements in the first place? Both models are good at challenging assumptions/edge cases and asking questions to clarify, but seemingly only when explicitly asked (i.e. something like a "brainstorm&quo
27.
by nsingh2 3mo ago
Cost per task is shockingly high. More expensive than Opus 4.8, second in place to Fable. Cost per task data is only available for max effort though, might just be very inefficient at that effort level.
28.
by nsingh2 3mo ago
That seems to only be true for the "Agentic Search" benchmark. That benchmark in particular is a bit weird, because Sonnet 4.6 effort levels had a relatively small effect, so Sonnet 5 med is basically comparable to all effort leve
29.
by nsingh2 3mo ago
> Don't use it to write code One of the things I've started doing is to write the high level scaffold myself, create stubs (with comments) and have it fill out the details. I haven't done this for long, but I feel like my
30.
by nsingh2 3mo ago
This is false. A process needs read permission on the relevant `/dev/input/` device, typically by running as root or as a user in a group like `input`. Normal desktop users generally should not be in the input group. Regular

More ›