6 ms·
Llama 3 implemented in pure NumPy
- ulam2 2y agoI'll consider superintelligence achieved if AI can do such work faithfully.
- sebzim4500 2y agoWhat? Lots of people could produce this repo, it hardly counts as superintelligence.
- deleted 2y ago[deleted]
- Olesya000 2y ago[dead]
- deleted 2y ago[deleted]
- Scene_Cast2 2y agoThe rotary embeddings bit is neat. I wonder if a complex representation would simplify vs complexify things (readability, performance, expressive power).
- johndough 2y agoSome implementations use a complex rotary encoding, but it makes it a bit harder to port to platforms or frameworks which do not support complex numbers natively.
- 6gvONxR4sf7o 2y agoThe tensor cores that do the bulk of the flops on the bulk of the gpus people use are just various sizes of floats, i think. We're in a funny position where progress in models and progress in hardware are kind of linked. As far as expressive power goes, it shouldn't make a difference for the models in common use, but I could totally imagine models where it improves readability.
- johndough 2y agoWhat is the difference to the llama.np repository credited in the README? https://github.com/hscspring/llama.np https://github.com/hscspring/llama.np
- aeyes 2y agoWell, it supports Llama3. But the other question I have is about the license. The tokenizer.py file is identical, and the rest is very similar - just making minor adjustments here and there. Can they just take this Apache 2 licensed code, change it a bit and offer it as MIT? They are clearly not the original author.
- Scaevolus 2y agoUnfortunately, licenses are only worth as much as your lawyers.
- yjftsjthsd-h 2y agoDMCA takedowns are free.
- not2b 2y agoA less aggressive approach would be to file an issue and let the maintainer correct the license issue.
- kolinko 2y agoObligatory Recmo’s Llama1 implementation in numpy :) https://github.com/recmo/cria https://github.com/recmo/cria
- joennlae 2y agoTrainable Llama-like transformer (with backpropagation) in numpy only (~600 lines) https://github.com/joennlae/tensorli https://github.com/joennlae/tensorli
- Zambyte 2y agoThe description says GPT-like, but is is just a GPT, right?
- lnyan 2y ago`import jax.numpy as np`, then we also get a jax implemention after certain modifications: e.g. remove in-place index assignment, replace unsupported functions, etc
- cl3misch 2y ago...which should be much faster also on CPU, I assume.
- ffriend 2y agoJAX requires a bit more work to maintain fixed-size buffers as required by XLA, especially in case of caching and rotary embeddings. But yeah, overall the code can be pretty similar [1]. [1]: https://github.com/dfdx/fabrique/blob/main/fabrique/llama/model.py https://github.com/dfdx/fabrique/blob/main/fabrique/llama/mo...
- xchip 2y agoNice but the tricky part is the training data.
- whereismyacc 2y agothere are a lot of tricky parts.
- swader999 2y agoThe tricky part is getting big enough that no one can successfully sue you for using "your" training data.
- buildbot 2y agoCool, instant cuda acceleration via cupy! `import cupy as np`
- AI_hacker 2y agoHow does the performance of llama3.np compare to other implementations, especially considering it's a pure NumPy implementation?
- rhdunn 2y agoFrom the TinyStories dataset card [1] the dataset is generated by GPT-3.5 and GPT-4. Reading the discussions in the community tab [2] it looks like there are a lot of incomplete or misspelled words, incorrect grammar, and even Chinese characters in the dataset. As such, I'd be weary of using that dataset to train or evaluate models. [1] https://huggingface.co/datasets/roneneldan/TinyStories https://huggingface.co/datasets/roneneldan/TinyStories [2] https://huggingface.co/datasets/roneneldan/TinyStories/discussions https://huggingface.co/datasets/roneneldan/TinyStories/discu...
- nwoli 2y agoIt’s just used for checking that the implementation is correct. The dataset is just a toy dataset it doesn’t matter if it has misspelled words
- ffriend 2y agoIt's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.py https://github.com/meta-llama/llama3/blob/main/llama/model.p...
- blt 2y agothe simplicity of the transformer is quite refreshing. especially in vision where the Vision Transformer with linear patch encodings replaces complex intertwined decisions about filter size, striding, pooling, #filters, depth, etc., with the simpler decision of how to allocate your FLOPS between dimensionality, #heads, and #layers.
- blharr 2y agoSo is this the case that the information is in the data set? Or the code is very well defined to be so small? As an outsider it's surprising that such a capable model can be so "simple".
- jacobn 2y agoThe training code is presumably quite a bit more complex than what they've open sourced, but part of the beauty of the GPT-based LLMs is their structural simplicity. Now, that simplicity can be deceiving - there are a lot of conceptual interconnectedness within these models. They've been put together "just so" if you will. If you look at the source code to nanoGPT and compare it to Llama3, the most remarkable thing (when you look past the superficial name changes) is just how similar they are. If I recall correctly the primary differences are: - The MLP: Llama3 uses SwiGLU vs the more "traditional" x = x + proj(gelu(expand(x))) in GPT2 - The token encoders, which is arguably external to the model - Attention: Llama3 uses Grouped Query Attention, vs full Multi-Head Attention in GPT2 - Normalization: Llama3 uses RMSNorm, vs LayerNorm for GPT2 They were published more than five years apart. On the one hand progress has been breathtaking, truly astounding. On the other hand, it's almost exactly the same model. Goes to show just how much is in the training data.
- 2y ago
- dang 2y agoWe changed the URL from https://github.com/likejazz/llama3.np https://github.com/likejazz/llama3.np to the article it points to, which gives more background.
- threatripper 2y ago> np.sin(freqs) Didn't we drop 2 pi somewhere?