5 ms·

The context window has nothing to do with RAM usage and even if it did, a million tokens of context is maybe 5mb.

vlovich123 4mo ago
It has nothing to do with local RAM usage. But a million tokens of LLM context is decidedly not 5mb. The rough estimate is 2 * L * H_kv * D * bytes per element Where: * L = number of layers * H_kv = # of KV heads * D = head dimension * factor of 2 = keys + values The dominant factor here is typically 2 * H_kv * D since it’s usually at least 2048 bytes. Per token. For Llama3 7B youre looking at 128gib if you’re context is really 1M (not that that particular model supports a context so big). DeepSeek4 uses something called sparse attention so the above calculus is improved - 1M of context would use 5-10GiB. But regardless of the details, you’re off by several orders of magnitude.
- tujux 4mo ago
  Pretty sure we're talking about the output text, not the tensors.
  - m00x 4mo ago
    These LLM replies are really getting annoying.
    - vlovich123 4mo ago
      Mine? I literally wrote what I wrote because “context window” as a term of art refers to the LLM’s context window. I guess get better at detecting LLMs instead of accusing everything of being an LLM reply?
bluegatty 4mo ago
'A million tokens of context' is literally Terrabytes of KV cache VRAM on very expensive Nvidia silicon - on the model. On the Agent, yes, the context window does relate to RAM, because the 'entire conversational history' is generally kept in memory. So ballpark 1M 'words' across a bunch of strings. It's not that-that much. Claude Code is not inneficient because 'it's not Rust' - it's just probably not very efficiently designed. Rust does not bestow magical properties that make memory more efficient really. A bit more, but it's not going to change this situation. 'Dong it in Rust' might yield amazing returns just because the very nature of the activity is 'optimization'.
- rixed 4mo ago
  Rust "denialism" is as annoying as rust evangelism. Of course any seemingly idiomatic rust is going to run circles around TS transpiled into JIT-compiled JS.
  - bluegatty 4mo ago
    Lamenting any 'not even criticism' of Rust as 'denialism' is just evidence of the insane cult that is Rust. Rebuilding Claude Code in Rust will make almost no difference in terms of real world performance. V8 is 'relatively fast', and there wouldn't be any noticeable improvements there, and probably not memory footprint either. The source for Claude Code was leaked and it's a vibe-coded mess, there's not much thought given to clean architecture, it's unlikely they've just cleaned up a bit and given thought to memory consumption etc, if they did, they'd get by far most of the way there and likely abnegate and real want to 'do it in rust', unless there are other architectural considerations.
    - imtringued 4mo ago
      You're the delusional one for bringing up the memory usage of the inference server that clearly isn't running inside the coding agent. The problem with your comments is that you're showing off a fundamental lack of understanding between managed languages and unmanaged languages. The vast majority of GCs are optimized for throughput and allocate big chunks of memory. They also tend to never release it if there was a temporary memory spike. The most advanced GCs also tend to have either read or write barriers, which slow down basic object accesses. Just in time compilation and managed languages in general need to retain a runtime representation of the source code to perform JIT compilation and then they have to store the compiled code in memory as well. JavaScript uses references against dynamic objects, which means you have to pay the indirection cost of a pointer but you also need to store type information as well to monomorphize the object literals and classes at runtime and fall back to a regular hashmap when fields are added dynamically. All of these things will add up and increase the amount of memory the application uses and how slow it runs. Sure Claude Code has severe architectural issues causing it to leak hundreds of gigabytes of RAM, but if those were not there you could easily build a C++ based alternative that runs circles around a hypothetical JavaScript based Claude Code that got its act together.

It has nothing to do with local RAM usage. But a million tokens of LLM context is decidedly not 5mb. The rough estimate is 2 * L * H_kv * D * bytes per element Where: * L = number of layers * H_kv = # of KV heads * D = head dimension * factor of 2 = keys + values The dominant factor here is typically 2 * H_kv * D since it’s usually at least 2048 bytes. Per token. For Llama3 7B youre looking at 128gib if you’re context is really 1M (not that that particular model supports a context so big). DeepSeek4 uses something called sparse attention so the above calculus is improved - 1M of context would use 5-10GiB. But regardless of the details, you’re off by several orders of magnitude.

tujux 4mo ago
Pretty sure we're talking about the output text, not the tensors.
- m00x 4mo ago
  These LLM replies are really getting annoying.
  - vlovich123 4mo ago
    Mine? I literally wrote what I wrote because “context window” as a term of art refers to the LLM’s context window. I guess get better at detecting LLMs instead of accusing everything of being an LLM reply?

bluegatty 4mo ago

'A million tokens of context' is literally Terrabytes of KV cache VRAM on very expensive Nvidia silicon - on the model. On the Agent, yes, the context window does relate to RAM, because the 'entire conversational history' is generally kept in memory. So ballpark 1M 'words' across a bunch of strings. It's not that-that much. Claude Code is not inneficient because 'it's not Rust' - it's just probably not very efficiently designed. Rust does not bestow magical properties that make memory more efficient really. A bit more, but it's not going to change this situation. 'Dong it in Rust' might yield amazing returns just because the very nature of the activity is 'optimization'.

rixed 4mo ago
Rust "denialism" is as annoying as rust evangelism. Of course any seemingly idiomatic rust is going to run circles around TS transpiled into JIT-compiled JS.
- bluegatty 4mo ago
  Lamenting any 'not even criticism' of Rust as 'denialism' is just evidence of the insane cult that is Rust. Rebuilding Claude Code in Rust will make almost no difference in terms of real world performance. V8 is 'relatively fast', and there wouldn't be any noticeable improvements there, and probably not memory footprint either. The source for Claude Code was leaked and it's a vibe-coded mess, there's not much thought given to clean architecture, it's unlikely they've just cleaned up a bit and given thought to memory consumption etc, if they did, they'd get by far most of the way there and likely abnegate and real want to 'do it in rust', unless there are other architectural considerations.
  - imtringued 4mo ago
    You're the delusional one for bringing up the memory usage of the inference server that clearly isn't running inside the coding agent. The problem with your comments is that you're showing off a fundamental lack of understanding between managed languages and unmanaged languages. The vast majority of GCs are optimized for throughput and allocate big chunks of memory. They also tend to never release it if there was a temporary memory spike. The most advanced GCs also tend to have either read or write barriers, which slow down basic object accesses. Just in time compilation and managed languages in general need to retain a runtime representation of the source code to perform JIT compilation and then they have to store the compiled code in memory as well. JavaScript uses references against dynamic objects, which means you have to pay the indirection cost of a pointer but you also need to store type information as well to monomorphize the object literals and classes at runtime and fall back to a regular hashmap when fields are added dynamically. All of these things will add up and increase the amount of memory the application uses and how slow it runs. Sure Claude Code has severe architectural issues causing it to leak hundreds of gigabytes of RAM, but if those were not there you could easily build a C++ based alternative that runs circles around a hypothetical JavaScript based Claude Code that got its act together.