rasbt

8 ms·

1. 1.▲
   by rasbt 2y ago

   Thanks for sharing!
2. 2.▲
   Show HN: New LLM Pre-Training and Post-Training Paradigms(sebastianraschka.com)

   2 points by rasbt 2y ago|0 comments
3. 3.▲
   by rasbt 2y ago

   I disagree here: Setting up a large-scale pretraining run is super complex if you have to manage your distributed computing platform, but looking at how the training data looks like and is fed into an LLM is not that complex. If you are dev
4. 4.▲
   by rasbt 2y ago

   Agreed, understanding how a method works and how it would be done helps with developing an intuition for its limitations -- what it can and what it can't do
5. 5.▲
   by rasbt 2y ago

   thanks for mentioning, that makes me super happy to hear!
6. 6.▲
   by rasbt 2y ago

   I wouldn't pretrain from scratch, but continued pretraining is pretty popular for adapating LLMs to recent and/or custom data. (Sometimes this is referred to 'finetuning', however, not to be confused with 'instructi
7. 7.▲
   by rasbt 2y ago

   Thanks, glad that this is helpful!
8. 8.▲
   Developing an LLM: Building, Training, Finetuning (A 1h Video Explainer)(youtube.com)

   43 points by rasbt 2y ago|12 comments
9. 9.▲
   Evaluating LLMs locally, on a laptop, with Llama 3 and Ollama(github.com)

   2 points by rasbt 2y ago|0 comments
10. 10.▲
    Understanding the LLM Development Cycle: Building, Training, Finetuning(magazine.sebastianraschka.com)

    3 points by rasbt 2y ago|0 comments
11. 11.▲
    The latest major open LLM releases: Mixtral, Llama 3, Phi-3, and OpenELM(magazine.sebastianraschka.com)

    5 points by rasbt 2y ago|0 comments
12. 12.▲
    Finetuning an LLM-Based Spam Classifier with LoRA from Scratch(github.com)

    14 points by rasbt 2y ago|0 comments
13. 13.▲
    Finetune a GPT Model for Spam Detection on Your Laptop in Just 5 Minutes(github.com)

    3 points by rasbt 2y ago|0 comments
14. 14.▲
    Insights from Finetuning LLMs for Classification Tasks(github.com)

    2 points by rasbt 2y ago|0 comments
15. 15.▲
    Tips for LLM Pretraining and Evaluating Reward Models(sebastianraschka.com)

    2 points by rasbt 2y ago|0 comments
16. 16.▲
    by rasbt 2y ago

    Quoting from the readme, it embraces other executers, including torch.compile and also works with multiple GPUs: > Thunder is a source-to-source compiler for PyTorch. It makes PyTorch programs faster by combining and using different hard
17. 17.▲
    Comparing 5 ways to implement Multihead Attention in PyTorch(github.com)

    3 points by rasbt 3y ago|0 comments
18. 18.▲
    AI Research in Feb 2024 – LoRA Successor, "Small" LLMs, Transparent LLM Research(sebastianraschka.com)

    3 points by rasbt 3y ago|0 comments
19. 19.▲
    by rasbt 3y ago

    Yes, it's somewhat similar to the 2B model as it uses the same vocabulary size.
20. 20.▲
    by rasbt 3y ago

    Yes, it's definitely unfair to count it as a 7B model. In that case, we could call Llama 2, which is 6.6B parameters, a 6B (or even 5B) parameter model.
21. 21.▲
    by rasbt 3y ago

    Yes, it's 8.5B params if you account for weight tying, and 9.3B if you count the embedding layer and output layer weights separately as shown in the 2nd figure in the article. In the paper, I think they justified 7B by only counting th
22. 22.▲
    by rasbt 3y ago

    > Gemma is a +9B model Yes that's correct. It's 9.3B parameters if you count the embedding layer and final projection layer separately. However, since they used weight tying, the adjusted count is 8.5B as discussed in the artic
23. 23.▲
    Understanding, using, and finetuning Gemma(lightning.ai)

    118 points by rasbt 3y ago|48 comments
24. 24.▲
    by rasbt 3y ago

    Thanks for the feedback. Clarifying definitely wouldn't hurt. Added a paragraph and new figure at the top of the DoRA section: https://magazine.sebastianraschka.com/i/141797214/introducin... I haven't tr
25. 25.▲
    by rasbt 3y ago

    Not sure, but in general, it looks like ZipLoRA is only useful in specific contexts like when you have two different tasks you want to optimize for (like style and content in a vision context). DoRA is more general, it's basically norm
26. 26.▲
    by rasbt 3y ago

    Thanks, fixed!
27. 27.▲
    Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch(magazine.sebastianraschka.com)

    96 points by rasbt 3y ago|10 comments
28. 28.▲
    AI Research Papers in Jan 2024: Model Merging, Mixtures of Experts, Smaller LLMs(magazine.sebastianraschka.com)

    20 points by rasbt 3y ago|0 comments
29. 29.▲
    by rasbt 3y ago

    thanks for the kind words!
30. 30.▲
    by rasbt 3y ago

    I'd say that I've finished all of my previous books, and I have no intention of doing anything different here. Of course, there's always the chance that I get run over by a bus or equivalent, but in that case, I assume that M