markusheimerl

6 ms·

1. 1.▲
   by markusheimerl 3mo ago

   https://github.com/markusheimerl/gpt/blob/main/train.c - in this file, search for the line "const int batch_size = 15;" - reduce this number
2. 2.▲
   by markusheimerl 3mo ago

   Sure it could be extended to support LoRA finetuning but this implementation has the goal to be as lean and efficient as possible for a pre-training stack as you can be.
3. 3.▲
   by markusheimerl 3mo ago

   the data gets downloaded via curl from huggingface - sure you can make your own data, simply dump all text you want the model to be trained on into "corpus.txt" and skip "make data". As the tokenizer adds substantial com
4. 4.▲
   by markusheimerl 3mo ago

   I did run it as a test on the NVIDIA Jetson Orin Nano Super Dev. Kit once - so yea it works on arm like a charm ;)
5. 5.▲
   by markusheimerl 3mo ago

   Reduce batch size in train.c
6. 6.▲
   by markusheimerl 3mo ago

   I deleted the numerical checks a while back after confirming the backward pass is correct to keep the code base lean - running https://github.com/markusheimerl/gpt/blob/main/transformer/a... is also
7. 7.▲
   Tiny hackable CUDA language model implementation(github.com)

   81 points by markusheimerl 4mo ago|13 comments
8. 8.▲
   Stick in Bike Wheel OOP(imgflip.com)

   1 points by markusheimerl 2y ago|0 comments