kgeist

5 ms·

1.
by kgeist 5d ago
Judging by the leaks, OpenAI and Anthropic already train reasoning traces to use fewer tokens (they deliberately omit articles and prepositions, use very short sentences, etc.), even though you pay per token. So it wouldn't make sense
2.
by kgeist 5d ago
The technique looked dubious from the start, because LLMs were trained to expect certain outputs from common bash tools. If the output is not what it expects, an LLM may issue more tool calls than before, because it will assume the tool is
3.
by kgeist 7d ago
>What stops them <..> simply use cheaper model for every Nth request. That would trigger a full prefill (context recompute) every Nth request because cached tokens aren't interchangeable between models, and that would require
4.
by kgeist 7d ago
The web UI's system prompt is also probably in Chinese
5.
by kgeist 8d ago
They mention it here: https://github.com/argonautlabsai/deltafin/blob/main/k3-publ... >device bytes read during the prefill window, all four drives (arm csv) 8,977 GB at 24.1 GB/s aggregate I.e.
6.
by kgeist 8d ago
There's a tendency to cite only decode speeds, but in practice, an LLM generates far fewer tokens than it has to read (unless you ask general knowledge questions). So the effective performance is much slower than the decode rate sugges
7.
by kgeist 8d ago
Do those reports require Kimi K3 though? Qwen3.6+ could probably do the same in a few seconds with similar quality.
8.
by kgeist 13d ago
On their Agentic Index, GPT-6 Astra (both max/xhigh) has the same result as Qwen3.8-27b. Weird.
9.
by kgeist 15d ago
I'm currently trying to write an inference engine that combines the benefits of llama.cpp (one binary deployment, good support for heterogenous non-datacenter compute, wide quantization support) with the benefits of vLLM/SGlang (t
10.
by kgeist 16d ago
>but also of other people. Yeah, there's this thing called the curse of knowledge. If an engineer has a deep understanding of something, it's not a given that they can explain it well. For them, the topic feels so simple, and t
11.
by kgeist 17d ago
How about: "Type the seahorse emoji to solve the CAPTCHA" :) Something that triggers infinite loops in LLMs or trips the guardrails.
12.
by kgeist 19d ago
In agentic scenarios, an LLM has to read far more tokens than it outputs. I think focusing only on the decode speed is somewhat misleading. 14 tok/sec for decode is actually okayish. 93 tok/sec prefill is what's abysmal, my R
13.
by kgeist 19d ago
Qwen 3.8 has the same architecture and the same parameter count as Qwen 3.6. Something is not right with the GGUF if it's 2 times slower. The post says "The hybrid attention architecture is new" and says the author's old
14.
by kgeist 23d ago
AI is as deterministic as a human programmer. A programmer who is burned out and has not slept well produces code of a different quality than a programmer who has slept well and is highly motivated. That's hardly deterministic. AI can
15.
by kgeist 27d ago
Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's
16.
by kgeist 29d ago
>Tiny ~6mb binary I wonder why it's so large for a program written in Zig. It's basically just a loop that accepts user input, prepares the context, sends it to the LLM, parses the output, invokes the tools, and presents it all
17.
by kgeist 1mo ago
It's basically a variation of HyDE (Hypothetical Document Embeddings), and the rationale is that the embedding of the query is not necessarily close to the embedding of the answer. If you generate a hallucinated answer, it can line up
18.
by kgeist 1mo ago
A common case I have is when you don't have classifications to begin with. For example, you need to find what users complain about most. I take embeddings of all records, then cluster the embeddings into semantic groups, then ask an LL
19.
by kgeist 1mo ago
I wonder how it impacts code generation. It shouldn't impact prose in general because of synonyms and whatnot, but code requires exact reproduction. That is, what happens if you ask an LLM to recite a large, human-written excerpt as is
20.
by kgeist 1mo ago
More compute/larger datasets during training != larger models. The Bitter Lesson was that just scaling things up beats custom hand-crafted optimizations. Up until 2024, we thought that meant scaling up the parameter count, but then tha
21.
by kgeist 1mo ago
I agree with the article, but there's one thing Go has that doesn't help LLMs: structural typing. An LLM has to grep a little more to understand which interfaces a struct implements.
22.
by kgeist 1mo ago
In the BlackHat presentation on the HuggingFace incident, OpenAI showed some excerpts from the reasoning traces, and they had that grug speak too (skipped articles, etc.). So the OP's method must have indeed found the actual reasoning
23.
by kgeist 1mo ago
LLMs already think in latent space. The generated reasoning tokens are only the surface of what's happening internally. An LLM may write one thing in the traces but decide differently in the latent space. The whole token-based "re
24.
by kgeist 1mo ago
The article is AI-written as well, with all the "honest problems" and "real weak spots". The code must be AI-generated too, so without a human properly verifying it, I can't take the project seriously. It may just b
25.
by kgeist 1mo ago
The social network VK internally uses highly specialized database engines per business domain. They don't use stock DBs. They have a DB engine for posts, a DB engine for likes, etc. They have a team of DB engineers. Their DB load was a
26.
by kgeist 1mo ago
I'm not sure it's a fair test either to compare the "low" setting of one model with the "low" setting of another. They're completely different settings that just happen to have the same name.
27.
by kgeist 1mo ago
It's an old, small model, so that's expected. It's more of a prototype.
28.
by kgeist 1mo ago
>avoid extra detail when it does not help I wonder if they actually do it to optimize inference. I maintain a corporate AI server and one of the tricks to reduce the load was to modify the system prompt to be as terse as possible so the
29.
by kgeist 2mo ago
Transformers lack recursion and are limited by the network's fixed depth, so "reasoning", IMHO, is basically a way to emulate deeper recursion. As we go through the layers, concepts are pattern-matched and refined, but at som
30.
by kgeist 2mo ago
In my experience, almost all the problems that microservices advertise solving can also be solved with a modular monolith plus some tooling to enforce certain rules (say, one module shouldn't be able to peek into another module's

More ›