8 ms·
Show HN: I built a tiny LLM to demystify how language models work
Built a ~9M param LLM from scratch to understand how they actually work. Vanilla transformer, 60K synthetic conversations, ~130 lines of PyTorch. Trains in 5 min on a free Colab T4. The fish thinks the meaning of life is food.
Fork it and swap the personality for your own character.
- Morpheus_Matrix 6mo ago[flagged]
- aditya7303011 6mo ago[dead]
- AndrewKemendo 6mo agoI love these kinds of educational implementations. I want to really praise the (unintentional?) nod to Nagel, by limiting capabilities to representation of a fish, the user is immediately able to understand the constraints. It can only talk like a fish cause it’s very simple Especially compared to public models, thats a really simple correspondence to grok intuitively (small LLM > only as verbose as a fish, larger LLM > more verbose) so kudos to the author for making that simple and fun.
- dvt 6mo ago> the user is immediately able to understand the constraints Nagel's point was quite literally the opposite[1] of this, though. We can't understand what it must "be like to be a bat" because their mental model is so fundamentally different than ours. So using all the human language tokens in the world can't get us to truly understand what it's like to be a bat, or a guppy, or whatever. In fact, Nagel's point is arguably even stronger: there's no possible mental mapping between the experience of a bat and the experience of a human. [1] https://www.sas.upenn.edu/~cavitch/pdf-library/Nagel_Bat.pdf https://www.sas.upenn.edu/~cavitch/pdf-library/Nagel_Bat.pdf
- AndrewKemendo 6mo agoDifferent argument I’m not going to argue other than to say that you need to view the point from a third party perspective evaluating “fish” vs “more verbose thing,” such that the composition is the determinant of the complexity of interaction (which has a unique qualia per nagel) Hence why it’s a “unintentional nod” not an instantiation
- deleted 6mo ago[deleted]
- deleted 6mo ago[deleted]
- Terr_ 6mo agoIMO we're a step before that: We don't even have a real fish involved, we have a character that is fictionally a fish. In LLM-discussions, obviously-fictional characters can be useful for this, like if someone builds a "Chat with Count Dracula" app. To truly believe that a typical "AI" is some entity that "wants to be helpful" is just as mistaken as believing the same architecture creates an entity that "feels the dark thirst for the blood of the living." Or, in this case, that it really enjoys food-pellets.
- andoando 6mo agoId highly disagree with that. Were all living in the same shared universe, and underlying every intelligence must be precisely an understanding of events happening in this space-time.
- vixen99 6mo agoWhat does 'precisely' mean? Everyone has the same understanding of events - a precise one?
- andoando 6mo agoNo I am saying the basis of intelligence must be shared, not that we have the same exact mental model. I might for example say a human entered a building, a bat might on the other hand think "some big block with two sticks moved through a hole", but both are experiencing a shared physical observation, and there is some mapping between the two. Its like when people say, if there are aliens they would find the same mathematical constants thet we do
- nullbyte808 6mo agoAdorable! Maybe a personality that speaks in emojis?
- armanified 6mo agoOMG! You just gave me the next idea..
- SilentM68 6mo agoWould have been funny if it were called "DORY" due to memory recall issues of the fish vs LLMs similar recall issues :)
- armanified 6mo agoOMG! Why didn't I thought fo this first :P
- weiyong1024 6mo ago[flagged]
- ordinarily 6mo agoIt's genuinely a great introduction to LLMs. I built my own awhile ago based off Milton's Paradise Lost: https://www.wvrk.org/works/milton https://www.wvrk.org/works/milton
- cbdevidal 6mo ago> you're my favorite big shape. my mouth are happy when you're here. Laughed loudly :-D
- vunderba 6mo agoThis is a direct output from the synthetic training data though - wonder if there is a bit of overfitting going on or it’s just a natural limitation of a much smaller model.
- deleted 6mo ago[deleted]
- gnarlouse 6mo agoI... wow, you made an LLM that can actually tell jokes?
- murkt 6mo agoWith 9M params it just repeats the joke from a training dataset.
- martmulx 6mo ago[flagged]
- ethanmacavoy 6mo ago[flagged]
- NyxVox 6mo agoHm, I can actually try the training on my GPU. One of the things I want to try next. Maybe a bit more complex than a fish :)
- LeonTing1010 6mo ago[flagged]
- secabeen 6mo agoTraining data is here: https://huggingface.co/datasets/arman-bd/guppylm-60k-generic https://huggingface.co/datasets/arman-bd/guppylm-60k-generic
- agenexus 6mo ago[flagged]
- aesopturtle 6mo ago[flagged]
- dinkumthinkum 6mo agoI think this is a nice project because it is end to end and serves its goal well. Good job! It's a good example how someone might do something similar for a specific purpose. There are other visualizers that explain different aspects of LLMs but this is a good applied example.
- aditya7303011 6mo agoDid something similar last year https://github.com/aditya699/EduMOE https://github.com/aditya699/EduMOE
- ankitsanghi 6mo agoLove it! I think it's important to understand how the tools we use (and will only increasingly use) work under the hood.
- oyebenny 6mo agoNeat!
- kaipereira 6mo agoThis is so cool! I'd love to see a write-up on how made it, and what you referenced because designing neural networks always feel like a maze ;)
- kubrador 6mo agohow's it handle longer context or does it start hallucinating after like 2 sentences? curious what the ceiling is before the 9M params
- zwaps 6mo agoI like the idea, just that the examples are reproduced from the training data set. How does it handle unknown queries?
- deleted 6mo ago[deleted]
- armanified 6mo agoIt mostly doesn't, at 9M it has very limited capacity. The whole idea of this project is to demonstrate how Language Models work.
- mudkipdev 6mo agoThis is probably a consequence of the training data being fully lowercase: You> hello Guppy> hi. did you bring micro pellets. You> HELLO Guppy> i don't know what it means but it's mine.
- functional_dev 6mo agoGreat find! It appears uppercase tokens are completely unknonw to the tokenizer. But the character still comes through in response :)
- brcmthrowaway 6mo agoWhy are there so many dead comments from new accounts?
- AlecSchueler 6mo agoThey all seem to be slop comments.
- loveparade 6mo agoIt really seems it's mostly AI comments on this. Maybe this topic is attractive to all the bots.
- armanified 6mo agoThis title might have triggered something in those bots; most of them have sneaky AI SaaS links in their bio. Honestly, I never expected this post to become so popular. It was just the outcome of a weekend practice session.
- 59nadir 6mo agoBecause despite what HN users seem to think, HN is a LLM-infested hellscape to the same degree as Reddit, if not more.
- wiseowise 6mo agoYou’re absolutely right! HN isn’t just LLM-infested hellscape, it’s a completely new paradigm of machine assisted chocolate-infused information generation.
- toyg 6mo agoJust let me know which type of information goo you'd like me to generate, and I'll tailor the perfect one for you.
- siva7 6mo agoBut what should we do? The parent company isn't transparent about communicating the seriousness of this problem
- Alexzoofficial 6mo ago[flagged]
- monksy 6mo agoIs this a reference from the Bobiverse?
- rclkrtrzckr 6mo agoI could fork it and create TrumpLM. Not a big leap, I suppose.
- search_facility 6mo agoprobably 8M params are too much even :)
- danparsonson 6mo agoAs long as you use the best parameters then it doesn't matter
- wiseowise 6mo agoGrab her by the pointer.
- jiusanzhou 6mo ago[flagged]
- peifeng07 6mo ago[dead]
- zephyrwhimsy 6mo ago[flagged]
- _q4yj 6mo ago[dead]
- cpldcpu 6mo agoLove it! Great idea for the dataset.
- techpulselab 6mo ago[dead]
- _2fnr 6mo ago[flagged]
- amelius 6mo agoIt's arguably even better than the most famous answer to that question.
- zkmon 6mo agoMeaning/goal of life is to reproduce. Food (and everything else) is only a means to it. Reproduction is the only root goal given by nature to any life form. All resources and qualities are provided are only to help mating.
- amelius 6mo agoThen why are reproductive rates so low in western countries? https://en.wikipedia.org/wiki/List_of_countries_by_total_fertility_rate#/media/File:Total_Fertility_Rate_Map_by_Country.svg https://en.wikipedia.org/wiki/List_of_countries_by_total_fer...
- darepublic 6mo agoThe western lifestyle is an evolutionary dead end?
- vixen99 6mo ago
- bblb 6mo agoCould it be possible to train LLM only through the chat messages without any other data or input? If Guppy doesn't know regular expressions yet, could I teach it to it just by conversation? It's a fish so it wouldn't probably understand much about my blabbing, but would be interesting to give it a try. Or is there some hard architectural limit in the current LLM's, that the training needs to be done offline and with fairly large training set.
- gdzie-jest-sol 6mo ago* How creating dataset? I download it but it is commpresed in binary format. * How training. In cloud or in my own dev * How creating a gguf
- gdzie-jest-sol 6mo ago``` uv run python -m guppylm chat Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/user/gupik/guppylm/guppylm/__main__.py", line 48, in <module> main() File "/home/user/gupik/guppylm/guppylm/__main__.py", line 29, in main engine = GuppyInference("checkpoints/best_model.pt", "data/tokenizer.json") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/gupik/guppylm/guppylm/inference.py", line 17, in __init__ self.tokenizer = Tokenizer.from_file(tokenizer_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Exception: No such file or directory (os error 2) ```
- gdzie-jest-sol 6mo agomeybe add training again (read best od fine) and train again ``` # after config device checkpoint_path = "checkpoints/best_model.pt" ckpt = torch.load(checkpoint_path, map_location=device, weights_only=False) model = GuppyLM(mc).to(device) if "model_state_dict" in ckpt: model.load_state_dict(ckpt["model_state_dict"]) else: model.load_state_dict(ckpt) start_step = ckpt.get("step", 0) print(f"Encore {start_step}") ```
- freetonik 6mo agoYou sound like Guppy. Nice touch.
- nate8bit 6mo agoThis is really great! I've been wanting to do something similar for a while.
- Propelloni 6mo agoGreat work! I still think that [1] does a better job of helping us understand how GPT and LLM work, but yours is funnier. Then, some criticism. I probably don't get it, but I think the HN headline does your project a disservice. Your project does not demystify anything (see below) and it diverges from your project's claim, too. Furthermore, I think you claim too much on your github. "This project exists to show that training your own language model is not magic." and then just posts a few command line statements to execute. Yeah, running a mail server is not magic, just apt-get install exim4. So, code. Looking at train_guppylm.ipynb and, oh, it's PyTorch again. I'm better off reading [2] if I'm looking into that (I know, it is a published book, but I maintain my point). So, in short, it does not help the initiated or the uninitiated. For the initiated it needs more detail for it to be useful, the uninitiated more context for it to be understood. Still a fun project, even if oversold. [1] https://spreadsheets-are-all-you-need.ai/ https://spreadsheets-are-all-you-need.ai/ [2] https://github.com/rasbt/LLMs-from-scratch https://github.com/rasbt/LLMs-from-scratch
- jadengeller 6mo agothis comment seems to be astroturfing to sell a course
- Propelloni 5mo agoWhat do you mean, the LLM from Scratch book?
- Morpheus_Matrix 6mo ago[dead]
- totetsu 6mo agohttps://bbycroft.net/llm https://bbycroft.net/llm has 3d Visualization of tiny example LLM layers that do a very good job at showing what is going on (https://news.ycombinator.com/item?id=38505211 https://news.ycombinator.com/item?id=38505211)
- maverickxone 6mo agohave little to do with this, but i have to say your project are indeed pretty cool! Consider adding some more UI?
- armanified 6mo agoPretty neat! I'll definitely take a deeper look into this.
- skramzy 6mo agoNeat!
- devsteru 6mo agoThanks for sharing
- Elengal 6mo agoCool
- fg137 6mo agoHow does this compare to Andrej Karpathy's microgpt (https://karpathy.github.io/2026/02/12/microgpt/ https://karpathy.github.io/2026/02/12/microgpt/) or minGPT (https://github.com/karpathy/minGPT https://github.com/karpathy/minGPT)?
- armanified 6mo agoI haven't compared it with anything yet. Thanks for the suggestion; I'll look into these.
- BrokenCogs 6mo agoWho cares how it compares, it's not a product it's a cool project
- tantalor 6mo agoEven cool projects can learn from others. Maybe they missed something that could benefit the project, or made some interesting technical choice that gives a different result. For the readers/learners, it's useful to understand the differences so we know what details matter, and which are just stylistic choices. This isn't art; it's science & engineering.
- BrokenCogs 6mo agoBut it isn't the OP's responsibility to compare their project to all other projects. The GP could themselves perform the comparison and post their thoughts instead of asking an open ended question.
- tantalor 6mo ago100% agree, I didn't mean to imply that OP is responsible for that, or that the (lack of) comparison detracts in any way from the work.
- philipallstar 6mo ago> it isn't the OP's responsibility to compare their project to all other projects No one, including the GP, said it was.
- ananandreas 6mo agoGreat and simple way to bridge the gap between LLMs and users coming in to the field!
- fawabc 6mo agohow did you generate the synthetic data?
- abkolan 5mo agoIt's here https://github.com/arman-bd/guppylm/blob/main/guppylm/generate_data.py https://github.com/arman-bd/guppylm/blob/main/guppylm/genera...
- Duplicake 6mo agoI love this! Seems like it can't understand uppercase letters though
- armanified 6mo agoUppercase letters were intentionally ignored.
- amelius 6mo ago> A 9M model can't conditionally follow instructions How many parameters would you need for that?
- armanified 6mo agoMy initial idea was to train a navigation decision model with 25M parameters for a Raspberry Pi, which, in testing, was getting about 60% of tool calls correct. IMO, it seems like around 20M parameters would be a good size for following some narrow & basic language instructions.
- amelius 6mo agoOk. This makes me wonder about a broader question. Is there a scientific approach showing a pyramid of cognitive functions, and how many parameters are (minimally) required for each layer in this pyramid?
- drincanngao 6mo agoI was going to suggest implementing RoPE to fix the context limit, but realized that would make it anatomically incorrect.
- armanified 6mo agoI intentionally removed all optimizations to keep it vanilla.
- novachen 6mo ago[dead]
- algoth1 6mo agoThis really makes me think if it would be feasible to make an llm trained exclusively on toki pona (https://en.wikipedia.org/wiki/Toki_Pona https://en.wikipedia.org/wiki/Toki_Pona)
- MarkusQ 6mo agoThere isn't enough training data though, is there? The "secret sauce" of LLMs is the vast amount of training data available + the compute to process it all.
- algoth1 6mo agoI think you could probably feed a copy of a toki pona grammar book to a big model, and have it produce ‘infinite’ training data
- eden-u4 6mo agoThere are not enough samples in that book to generate new "infinite" data.
- MarkusQ 6mo agoThis is essentially a distillation on the bigger model; you'd wind up surfacing a lot of artifacts from the host model, amplifying them in the same way repeated photocopying introduces errors. https://dailyai.com/2025/05/create-a-replica-of-this-image-dont-change-anything-ai-trend-takes-off/ https://dailyai.com/2025/05/create-a-replica-of-this-image-d...
- mudkipdev 6mo agoPeople have made toki pona translation models before, not exclusively trained though
- zhichuanxun 6mo ago[dead]
- zephyrwhimsy 6mo ago[flagged]
- areys 6mo ago[flagged]
- moonu 6mo agoThis comment seems ai-written
- hughw 6mo agoTiny LLM is an oxymoron, just sayin.
- armanified 6mo agoTrue, but most would ignore LM if it weren't LLM.
- uxcolumbo 6mo agoHow about: LLMs are on a spectrum and this one is on the tiny side?
- neurworlds 6mo agoCool project. I'm working on something where multiple LLM agents share a world and interact with each other autonomously. One thing that surprised me is how much the "world" matters — same model, same prompt, but put it in a system with resource constraints, other agents, and persistent memory, the behavior changes dramatically. Made me realize we spend too much time optimizing the model and not enough thinking about the environment it operates in.
- rahen 6mo agoI don't mean to be 'that guy', but after a quick review, this really feels like low-effort AI slop to me. There is nothing wrong using AI tools to write code, but nothing here seems to have taken more than a generic 'write me a small LLM in PyTorch' prompt, or any specific human understanding. The bar for what constitutes an engineering feat on HN seems to have shifted significantly.
- zhainya 5mo agoI don't really understand the point of this project or how it demystifies anything. Click the browser demo and I get a generic AI chat screen. Is the readme the part that "demystifies" something? I feel like I am living in a bizarro world. Is this all AI? Are all the comments here from bots?
- thomasfl 6mo agoIs there some documentation for this? The code is probably the simplest (Not So) Large Language Model implementation possible, but it is not straight forward to understand for developers not familiar with multi-head attention, ReLU FFN, LayerNorm and learned positional embeddings. This projects shares similarities with Minix. Minix is still used at universities as an educational tool for teaching operating system design. Minix is the operating system that taught Linus Torvalds how to design (monolithic) operating systems. Similarly having students adding capabilities to GuppyLM is a good way to learn LLM design.
- achenatx 6mo agogive the code to an LLM and have a discussion about it.
- dominotw 6mo agodoes this work? there is no more need for writing high level docs?
- bigmadshoe 6mo agoLLMs can tell you what the code does but not why the developer chose to do it that way. Also, large codebases are harder to understand. But projects like these are simple to discuss with an LLM.
- stronglikedan 6mo ago> LLMs can tell you what the code does but not why the developer chose to do it that way. Do LLMs not take comments into consideration? (Serious question - I'm just getting into this stuff)
- dr_hooo 6mo agoThey do (it's just text), if they are there...
- textai2026 6mo ago[dead]
- agdexai 6mo ago[dead]
- Vektorceraptor 6mo agoHaha, funny name :)
- adamsilvacons 6mo ago[flagged]
- nobodyandproud 6mo agoThanks. Tinkering is how I learn and this is what I’ve been looking for.
- Leomuck 6mo agoWow that is such a cool idea! And honestly very much needed. LLMs seem to be this blackbox nobody understands. So I love every effort to make that whole thing less mysterious. I will definitely have a look at dabbling with this, may it not be a goldfish LLM :)
- rpdaiml 6mo agoThis is a nice idea. A tiny implementation can be way more useful for learning than yet another wrapper around a big model, especially if it keeps the training loop and inference path small enough to read end to end.
- CaseFlatline 6mo agoI am trying to find how the synthetic data was created (looking through the repo) and didn't find it. Maybe I am missing it - Would love to see the prompts and process on that aspect of the training data generation!
- vunderba 6mo agoIt's here: https://github.com/arman-bd/guppylm/blob/main/guppylm/generate_data.py https://github.com/arman-bd/guppylm/blob/main/guppylm/genera... Uses a sort of mad-libs templatized style to generate all the permutations.
- jzer0cool 6mo agoDoes this work by just training once with next token prediction? Want to understand better how it creates fluent sentences if anyone can provide insights.
- solsafe_dev 6mo ago[dead]
- solsafe_dev 6mo ago[dead]
- winter_blue 6mo agoThis is amazing work. Thank you.
- maxothex 6mo ago[dead]
- techpulselab 6mo ago[dead]
- jbethune 6mo agoForked. Very cool. I appreciate the simplicity and documentation.
- meidad_g 6mo ago[flagged]
- EmilioOldenziel 6mo agoBuilding it yourself is always the best test if you really understand how it works.
- BiraIgnacio 6mo agoNice work and thanks for sharing it! Now, I ask, have LLMs ben demystified to you? :D I am still impressed how much (for the most part) trivial statistics and a lot of compute can do.
- bharat1010 6mo agoThis is such a smart way to demystify LLMs. I really like that GuppyLM makes the whole pipeline feel approachable..great work
- zephyrwhimsy 6mo ago[flagged]
- ergocoder 6mo agoIt's just so amazing that 5 years ago it would be extremely to build a conversational bot like this. But right now people make it a hobby, and that thing can run on a laptop. This is just so wild.
- zephyrwhimsy 6mo ago[flagged]
- hahooh 6mo agohaha funny, but really cool project. why fish tho lol.
- tombelieber 5mo agolooking forward to try it, great job
- aimemobe 5mo ago[flagged]
- AIOperator2026 5mo ago[dead]