8 ms·

How large are large language models?

• OtherShrezzing 1y ago
  >None of this document was not written by AI I think in these scenarios, articles should include the prompt and generating model.
  • oc1 1y ago
    You are absolutely right! The AI slop is getting out of control.
  • WesolyKubeczek 1y ago
    I don’t think the author knows that double negatives in English in a sentence like this cancel, not reinforce, each other.
  • kylecazar 1y ago
    I thought this was an accidental double negative by the author -- trying to declare they wrote it themselves. There are some signs it's written by possibly a non-native speaker.
  • rain1 1y ago
    I have corrected that. It was supposed to say "None of this document was written by AI." Thank you for spotting the error.
    • OtherShrezzing 1y ago
      Understood, thanks for updating it!
• unwind 1y ago
  Meta: The inclusion of the current year ("(2025)") in the title is strange, even though it's in the actual title of the linked-to post, repeating it here makes me look around for the time machine controls.
• dale_glass 1y ago
  How big are those in terms of size on disk and VRAM size? Something like 1.61B just doesn't mean much to me since I don't know much about the guts of LLMs. But I'm curious about how that translates to computer hardware -- what specs would I need to run these? What could I run now, what would require spending some money, and what I might hope to be able to run in a decade?
  • mjburgess 1y ago
    At 1byte/param that's 1.6GB (f8), at 2 bytes (f16) that's 2.3GB -- but there's other space costs beyond loading the parameters for the GPU. So a rule of thumb is ~4x parameter count. So round up, 2B -> 2*4 = 8GB VRAM
    • 1718627440 1y ago
      That sounds about the size of a modern browser (aka. any Electron et al. application)
  • loudmax 1y ago
    Most of these models have been trained using 16-bit weights. So a 1 billion parameter model takes up 2 gigabytes. In practice, models can be quantized to smaller weights for inference. Usually, the performance loss going from 16 bit weights to 8 bit weights is very minor, so a 1 billion parameter model can take 1 gigabyte. Thinking about these models in terms of 8-bit quantized weights has the added benefit of making the math really easy. A 20B model needs 20G of memory. Simple. Of course, models can be quantized down even further, at greater cost of inference quality. Depending on what you're doing, 5-bit weights or even lower might be perfectly acceptable. There's some indication that models that have been trained on lower bit weights might perform better than larger models that have been quantized down. For example, a model that was trained using 4-bit weights might perform better than a model that was trained at 16 bits, then quantized down to 4 bits. When running models, a lot of the performance bottleneck is memory bandwidth. This is why LLM enthusiasts are looking for GPUs with the most possible VRAM. You computer might have 128G of RAM, but your GPU's access to that memory is so constrained by bandwidth that you might as well run the model on your CPU. Running a model on the CPU can be done, it's just much slower because the computation is so parallel. Today's higher end consumer grade GPUs have up to 24G of dedicated VRAM (an Nvidia RTX 5090 has 32G of VRAM and they're like $2k). The dedicated VRAM on a GPU has a memory bandwidth of about 1 Tb/s. Apple's M-series of ARM-based CPU's have 512 Gb/s of bandwidth, and they're one of the most popular ways of being able to run larger LLMs on consumer hardware. AMD's new "Strix Halo" CPU+GPU chips have up to 128G of unified memory, with a memory bandwidth of about 256 Gb/s. Reddit's r/LocalLLaMA is a reasonable place to look to see what people are doing with consumer grade hardware. Of course, some of what they're doing is bonkers so don't take everything you see there as a guide. And as far as a decade from now, who knows. Currently, the top silicon fabs of TSMC, Samsung, and Intel are all working flat-out to meet the GPU demand from hyperscalers rolling out capacity (Microsoft Azure, AWS, Google, etc). Silicon chip manufacturing has traditionally followed a boom/bust cycle. But with geopolitical tensions, global trade barriers, AI-driven advances, and whatever other black swan events, what the next few years will look like is anyone's guess.
• mjburgess 1y ago
  Deepseek v1 is ~670Bn which is ~1.4TB physical. All digitized books ever written/encoded compress to a few TB. The public web is ~50TB. I think a usable zip of all english electronic text publicly available would be on O(100TB). So we're at about 1% of that in model size, and we're in a diminishing-returns area of training -- ie., going to >1% has not yielded improvements (cf. gpt4.5 vs 4o). This is why compute spend is moving to inference time with "reasoning" models. It's likely we're close to diminshing returns on inference-time compute now too, hence agents whereby (mostly,) deterministic tools are supplementing information /capability into the system. I think to get any more value out of this model class, we'll be looking at domain-specific specialisation beyond instruction fine-tuning. I'd guess targeting 1TB inference-time VRAM would be a reasonable medium-term target for high quality open source models -- that's within the reach of most SMEs today. That's about 250bn params.
  • layer8 1y ago
    Just a nitpick, but please don’t misuse big O notation like that. Any fixed storage amount is O(100TB).
  • account-5 1y ago
    > All digitized books ever written/encoded compress to a few TB. The public web is ~50TB. I think a usable zip of all english electronic text publicly available would be on O(100TB). Where you getting these numbers from? Interested to see how that's calculated. I read somewhere, but cannot find the source anymore, that all written text prior to this century was approx 50MB. (Might be misquoted as don't have source anymore).
    • WesolyKubeczek 1y ago
      Maybe prior to the prior century, and even then I smell a lot of bullshit. I mean, just look at the Project Gutenberg. Even plaintext only, even compressed.
      • bravesoul2 1y ago
        Even Shakespeare alone needs 4 floppy disks.
• christianqchung 1y ago
  This is a bad article. Some of the information is wrong, and it's missing lots of context. For example, it somehow merged Llama 4 Maverick's custom Arena chatbot version with Behemoth, falsely claiming that the former is stopping the latter from being released. It also claims 40B of internet text data is 10B tokens, which seems a little odd. Llama 405B was also trained on more than 15 trillion tokens[1], but the post claims only 3.67 trillion for some reason. It also doesn't mention Mistral large for some reason, even though it's the first good European 100B+ dense model. >The MoE arch. enabled larger models to be trained and used by more people - people without access to thousands of interconnected GPUs You still need thousands of GPUs to train a MoE model of any actual use. This is true for inference in the sense that it's faster I guess, but even that has caveats because MoE models are less powerful than dense models of the same size, though the trade-off has apparently been worth it in many cases. You also didn't need thousands of GPUs to do inference before, even for the largest models. The conclusion is all over the place, and has lots of just weird and incorrect implications. The title is about how big LLMs are, why is there such a focus on token training count? Also no mention of quantized size. This is a bad AI slop article (whoops, turns out the author accidentally said it was AI generated, so it's a bad human slop article). [1] https://ai.meta.com/blog/meta-llama-3-1/ https://ai.meta.com/blog/meta-llama-3-1/
  • rain1 1y ago
    I can correct mistakes. > it somehow merged Llama 4 Maverick's custom Arena chatbot version with Behemoth I can clarify this part. I wrote 'There was a scandal as facebook decided to mislead people by gaming the lmarena benchmark site - they served one version of llama-4 there and released a different model' which is true. But it is inside the section about the llama 4 model behemoth. So I see how that could be confusing/misleading. I could restructure that section a little to improve it. > Llama 405B was also trained on more than 15 trillion tokens[1], You're talking about Llama 405B instruct, I'm talking about Llama 405B base. Of course the instruct model has been traiend on more tokens. > why is there such a focus on token training count? I tried to include the rough training token count for each model I wrote about - plus additional details about training data mixture if available. Training data is an important part of an LLM.
• fossa1 1y ago
  It’s ironic: for years the open-source community was trying to match GPT-3 (175B dense) with 30B–70B models + RLHF + synthetic data—and the performance gap persisted. Turns out, size really did matter, at least at the base model level. Only with the release of truly massive dense (405B) or high-activation MoE models (DeepSeek V3, DBRX, etc) did we start seeing GPT-4-level reasoning emerge outside closed labs.
• stared 1y ago
  If you want it visually, here's a chart of total parameters as a function of year: https://app.charts.quesma.com/s/rmyk38 https://app.charts.quesma.com/s/rmyk38
  • rain1 1y ago
    This is really awesome. Thank you for creating that. I included a screenshot and link to the chart with credit to you in a comment to my post.
    • stared 1y ago
      I am happy you like it! If you like darker color scheme, here it is: https://app.charts.quesma.com/s/f07qji https://app.charts.quesma.com/s/f07qji And active vs total: https://app.charts.quesma.com/s/4bsqjs https://app.charts.quesma.com/s/4bsqjs
  • rain1 1y ago
    I think that one thing that this chart makes visually very clear is the point I about GPT-3 being such a huge leap, and there being a long gap before anybody was able to match it.
• ljoshua 1y ago
  Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t perfect, but holy cow the breadth of information that is embedded in an 8.1 GB file is incredible! Lossy, sure, but a pretty amazing way of compressing all of human knowledge into something incredibly contained.
  • ljlolel 1y ago
    How big is Wikipedia text? Within 3X that size with 100% accuracy
    • phkahler 1y ago
      Google AI response says this for compressed size of wikipedia: "The English Wikipedia, when compressed, currently occupies approximately 24 GB of storage space without media files. This compressed size represents the current revisions of all articles, but excludes media files and previous revisions of pages, according to Wikipedia and Quora." So 3x is correct but LLMs are lossy compression.
  • rain1 1y ago
    It's extremely interesting how powerful a language model is at compression. When you train it to be an assistant model, it's better at compressing assistant transcripts than it is general text. There is an eval which I have a lot of interested in and respect for https://huggingface.co/spaces/Jellyfish042/UncheatableEval https://huggingface.co/spaces/Jellyfish042/UncheatableEval called UncheatableEval, which tests how good of a language model an LLM is by applying it on a range of compression tasks. This task is essentially impossible to 'cheat'. Compression is a benchmark you cannot game!
    • MPSimmons 1y ago
      Agreed. It's basically lossy compression for everything it's ever read. And the quantization impacts the lossiness, but since a lot of text is super fluffy, we tend not to notice as much as we would when we, say, listen to music that has been compressed in a lossy way.
• simonw 1y ago
  > There were projects to try to match it, but generally they operated by fine tuning things like small (70B) llama models on a bunch of GPT-3 generated texts (synthetic data - which can result in degeneration when AI outputs are fed back into AI training inputs). That parenthetical doesn't quite work for me. If synthetic data always degraded performance, AI labs wouldn't use synthetic data. They use it because it helps them train better models. There's a paper that shows that if you very deliberately train a model in its own output in a loop you can get worse performance. That's not what AI labs using synthetic data actually do. That paper gets a lot of attention because the schadenfreude of models destroying themselves through eating their own tails is irresistible.
  • rybosome 1y ago
    Agreed, especially when in this context of training a smaller model on a larger model’s outputs. Distillation is generally accepted as an effective technique. This is exactly what I did in a previous role, fine-tuning Llama and Mistral models on a mix of human and GPT-4 data for a domain-specific task. Adding (good) synthetic data definitely increased the output quality for our tasks.
    • rain1 1y ago
      Yes but just purely in terms of entropy, you can't make a model better than GPT-4 by training it on GPT-4 outputs. The limit you would converge towards is GPT-4.
      • deleted 1y ago
        [deleted]
      • simonw 1y ago
        A better way to think about synthetic data is to consider code. With code you can have an LLM generate code with tests, then confirm that the code compiles and the tests pass. Now you have semi-verified new code you can add to your training data, and training on that will help you get better results for code even though it was generated by a "less good" LLM.
• 1vuio0pswjnm7 1y ago
  1. "raw text continuation engine" https://gist.github.com/rain-1/cf0419958250d15893d8873682492c3e https://gist.github.com/rain-1/cf0419958250d15893d8873682492... 2. "superintelligence" https://en.m.wikipedia.org/wiki/Superintelligence https://en.m.wikipedia.org/wiki/Superintelligence "Meta is uniquely positioned to deliver superintelligence to the world." https://www.cnbc.com/2025/06/30/mark-zuckerberg-creating-meta-superintelligence-labs-read-the-memo.html https://www.cnbc.com/2025/06/30/mark-zuckerberg-creating-met... Is there any difference between 1 and 2 Yes. One is purely hypothetical
• angusturner 1y ago
  I wish people would stop parroting the view that LLMs are lossy compression. There is kind of a vague sense in which this metaphor holds, but there is a much more interesting and rigorous fact about LLMs which is that they are also _lossless_ compression algorithms. There are at least two senses in which this is true: 1. You can use an LLM to losslessly compress any piece of text at a cost that approaches the log-likelihood of that text under the model, using arithmetic coding. A sender and receiver both need a copy of the LLM weights. 2. You can use an LLM plus SGD (I.e the training code) as an lossless compression algorithm, where the communication cost is area under the training curve (and the model weights don’t count towards description length!) see: Jack Rae “compression for AGI”
  • actionfromafar 1y ago
    Re 1 - classical compression is also extremely effective if both sender and receiver have access to the same huge dictionary.
• kamranjon 1y ago
  This is somehow missing the Gemma and Gemini series of models from Google. I also think that not mentioning the T5 series of models is strange from a historical perspective because they sort of pioneered many of the concepts in transfer learning and kinda kicked off quite a bit of interest in this space.
  • rain1 1y ago
    The Gemma models are too small to be included in this list. You're right the T5 stuff is very important historically but they're below 11B and I don't have much to say about them. Definitely a very interesting and important set of models though.
    • tantalor 1y ago
      > too small Eh? * Gemma 1 (2024): 2B, 7B * Gemma 2 (2024): 2B, 9B, 27B * Gemma 3 (2025): 1B, 4B, 12B, 27B This is the same range as some Llama models which you do mention. > important historically Aren't you trying to give a historical perspective? What's the point of this?
    • kamranjon 1y ago
      Since you included GPT-2, everything from Google including T5 would qualify for the list I would think.
• lukeschlather 1y ago
  This is a really nice writeup. That said, there's an unstated assumption here that these truly large language models are the most interesting thing. The big players have been somewhat quiet but my impression from the outside is that OpenAI let a little bit leak with their behavior. They built an even larger model and it turned out to be disappointing so they quietly discontinued it. The most powerful frontier reasoning models may actually be smaller than the largest publicly available models.
• bobsmooth 1y ago
  There's got to be tons of books that remain undigitized that can be mined for training data, hasn't there?