6 ms·

Show HN: Explore large language models with 512MB of RAM

138 points by jncraton 3y ago

GaggiX 3y ago
In case anyone is wondering, this repo uses various fine-tuned Flan-T5 models.
- rolisz 3y ago
  And Flan-T5 models are awesome! Fastchat is a finetuned 3b parameter Flan-T5 model that can perform really well, comparably to llama 13b
- jncraton 3y ago
  That's correct. The current base model is an int8 quantization of LaMini-Flan-T5-248M described here: https://github.com/mbzuai-nlp/lamini-lm https://github.com/mbzuai-nlp/lamini-lm I shared more details over on Reddit: https://www.reddit.com/r/LocalLLaMA/comments/14btk3a/explore_large_language_models_on_any_computer/ https://www.reddit.com/r/LocalLLaMA/comments/14btk3a/explore...
behnamoh 3y ago
Nothing new here, just yet another wrapper around the current language models, and weak ones at that.
- rolisz 3y ago
  Flan-T5 models perform really well, considering their size. In my experiments and in some recent papers, they are very close to 13b Llama for example: https://twitter.com/YiTayML/status/1668302949276356609 https://twitter.com/YiTayML/status/1668302949276356609
  - yieldcrv 3y ago
    its like the bar keeps getting lower and lower
1024core 3y ago
>>> lm.do("If I have 7 apples then eat 5, how many apples do I have?") 'You have 8 apples.' >>> lm.set_max_ram('4gb') 4.0 >>> lm.do("If I have 7 apples then eat 5, how many apples do I have?") 'I have 2 apples left.' It's funny how it switched from "You" to "I" when the memory was increased.
- george_808 3y ago
  From what I can tell, this doesn’t re-seed the random number generator, so we shouldn’t expect deterministic results. A better test would be to examine the model’s logits, or probabilities of the next token, across different ram sizes.
xrd 3y ago
At first glance i would be surprised if this works at all. The readme says it loads a significant amount of data the first time, 250mb. None of the LLM weights I know of are less than several gigabytes in size. It says it only requires 512mb of RAM. None of the interesting LLMs I know of run in less than 6gb of VRAM. It says it uses no API keys, which is great, but that means inference is local, which I can't imagine works with the above constraints.
- nattaylor 3y ago
  At 512gb RAM, it uses https://huggingface.co/jncraton/LaMini-Flan-T5-248M-ct2-int8 https://huggingface.co/jncraton/LaMini-Flan-T5-248M-ct2-int8 which says "This model is one of our LaMini-LM model series in paper "LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions". This model is a fine-tuned version of google/flan-t5-base on LaMini-instruction dataset that contains 2.58M samples for instruction fine-tuning"
- Translationaut 3y ago
  Those minified models are still equal or bigger compared to the initial "attention is all you need" transformer.
znagengast 3y ago
Isn't it crazy that the entirety of human knowledge can be condensed down to fit on an SD card.
- PaulHoule 3y ago
  A CD-ROM
- cjtrowbridge 3y ago
  That's not what this is.
- lionkor 3y ago
  Isn't it crazy that the entirety of human stupidity can be condensed down to "isnt it crazy that <wildly wrong fact>"
jb1991 3y ago
Shoot! I only have 511MB of RAM. Time to upgrade and get that AI!
Animats 3y ago
>>> lm.do("What is the population of Chicago") 'As of 2021, the population of Chicago is approximately 8.4 million.' >>> lm.do("What is the population of Shenzhen") 'As of 2021, the population of Shenzhen is approximately 1.3 million people.' >>> lm.do("What is the wavelength of blue light.") 'The wavelength of blue light is approximately 299,792,458 meters per second.' >>> lm.do("What is YCombinator") 'YCombinator is a programming language used to combine two or more languages into a single program.' >>> lm.do("What is asphalt made of?") 'Asphalt is made of sand, gravel, and other materials.' >>> lm.do("What is the square root of 2") '2.' >>> lm.do("How do I get to New York City from California?") 'You can get to New York City from California by taking a bus or train.' >>> lm.do("How can I unlock a lock without a key") 'You can use a combination of keys and a password to unlock a lock without a key.' >>> lm.do("How long should rice be cooked.") 'The recommended cooking time for rice depends on the type of rice, but generally it should be cooked for about 8-10 minutes per pound.' 100% wrong. This is an automated version of the Dunning-Kruger effect. You can ask it anything, and get back a confident wrong answer. So far, it hasn't replied to any question of mine with an indication that it doesn't know. It's a nice demonstration of the hallucination problem with LLMs. With a small data set, the results are usually bogus, but that's not detected.
- Translationaut 3y ago
  Have you also tried the bigger models? The smaller models are good for assisted generation: https://huggingface.co/blog/assisted-generation https://huggingface.co/blog/assisted-generation Those models of LaMini-Flan-T5 are trained to follow instructions and not to recognize the truth content. You could train a transformer like Ernie or Vega (which lead superglue) on such challenging factual data. But don't expect mathematical correct results only from the model. Therefore you have langchain with other APIs.
- Blahah 3y ago
  Yeah these aren't meant to know facts, just to parse language. Good for understanding simple instructions to automate tasks using external tools
  - 3y ago
btdmaster 3y ago
Really cool, I didn't know CPU/traditional RAM was enough already. Though, surprising results from repl.it: lm.classify("unabridged", "positive", "negative")=="unabridged is"
- jncraton 3y ago
  Thanks for pointing that out. Classification is half-baked at the moment. It should ultimately be restricting output to only appropriate labels, but right now it is simply sampling.
quickthrower2 3y ago
So does this download a hugging-face model and run it locally? Is the hughing face library doing the inference or something? Can’t see as much code as I expected.
angeleye 3y ago
[flagged]

138 points by jncraton 3y ago

GaggiX 3y ago

In case anyone is wondering, this repo uses various fine-tuned Flan-T5 models.

rolisz 3y ago
And Flan-T5 models are awesome! Fastchat is a finetuned 3b parameter Flan-T5 model that can perform really well, comparably to llama 13b
jncraton 3y ago
That's correct. The current base model is an int8 quantization of LaMini-Flan-T5-248M described here: https://github.com/mbzuai-nlp/lamini-lm https://github.com/mbzuai-nlp/lamini-lm I shared more details over on Reddit: https://www.reddit.com/r/LocalLLaMA/comments/14btk3a/explore_large_language_models_on_any_computer/ https://www.reddit.com/r/LocalLLaMA/comments/14btk3a/explore...

behnamoh 3y ago

Nothing new here, just yet another wrapper around the current language models, and weak ones at that.

rolisz 3y ago
Flan-T5 models perform really well, considering their size. In my experiments and in some recent papers, they are very close to 13b Llama for example: https://twitter.com/YiTayML/status/1668302949276356609 https://twitter.com/YiTayML/status/1668302949276356609
- yieldcrv 3y ago
  its like the bar keeps getting lower and lower

1024core 3y ago

>>> lm.do("If I have 7 apples then eat 5, how many apples do I have?") 'You have 8 apples.' >>> lm.set_max_ram('4gb') 4.0 >>> lm.do("If I have 7 apples then eat 5, how many apples do I have?") 'I have 2 apples left.' It's funny how it switched from "You" to "I" when the memory was increased.

george_808 3y ago
From what I can tell, this doesn’t re-seed the random number generator, so we shouldn’t expect deterministic results. A better test would be to examine the model’s logits, or probabilities of the next token, across different ram sizes.

xrd 3y ago

At first glance i would be surprised if this works at all. The readme says it loads a significant amount of data the first time, 250mb. None of the LLM weights I know of are less than several gigabytes in size. It says it only requires 512mb of RAM. None of the interesting LLMs I know of run in less than 6gb of VRAM. It says it uses no API keys, which is great, but that means inference is local, which I can't imagine works with the above constraints.

nattaylor 3y ago
At 512gb RAM, it uses https://huggingface.co/jncraton/LaMini-Flan-T5-248M-ct2-int8 https://huggingface.co/jncraton/LaMini-Flan-T5-248M-ct2-int8 which says "This model is one of our LaMini-LM model series in paper "LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions". This model is a fine-tuned version of google/flan-t5-base on LaMini-instruction dataset that contains 2.58M samples for instruction fine-tuning"
Translationaut 3y ago
Those minified models are still equal or bigger compared to the initial "attention is all you need" transformer.

znagengast 3y ago

Isn't it crazy that the entirety of human knowledge can be condensed down to fit on an SD card.

PaulHoule 3y ago
A CD-ROM
cjtrowbridge 3y ago
That's not what this is.
lionkor 3y ago
Isn't it crazy that the entirety of human stupidity can be condensed down to "isnt it crazy that <wildly wrong fact>"

jb1991 3y ago

Shoot! I only have 511MB of RAM. Time to upgrade and get that AI!

Animats 3y ago

>>> lm.do("What is the population of Chicago") 'As of 2021, the population of Chicago is approximately 8.4 million.' >>> lm.do("What is the population of Shenzhen") 'As of 2021, the population of Shenzhen is approximately 1.3 million people.' >>> lm.do("What is the wavelength of blue light.") 'The wavelength of blue light is approximately 299,792,458 meters per second.' >>> lm.do("What is YCombinator") 'YCombinator is a programming language used to combine two or more languages into a single program.' >>> lm.do("What is asphalt made of?") 'Asphalt is made of sand, gravel, and other materials.' >>> lm.do("What is the square root of 2") '2.' >>> lm.do("How do I get to New York City from California?") 'You can get to New York City from California by taking a bus or train.' >>> lm.do("How can I unlock a lock without a key") 'You can use a combination of keys and a password to unlock a lock without a key.' >>> lm.do("How long should rice be cooked.") 'The recommended cooking time for rice depends on the type of rice, but generally it should be cooked for about 8-10 minutes per pound.' 100% wrong. This is an automated version of the Dunning-Kruger effect. You can ask it anything, and get back a confident wrong answer. So far, it hasn't replied to any question of mine with an indication that it doesn't know. It's a nice demonstration of the hallucination problem with LLMs. With a small data set, the results are usually bogus, but that's not detected.

Translationaut 3y ago
Have you also tried the bigger models? The smaller models are good for assisted generation: https://huggingface.co/blog/assisted-generation https://huggingface.co/blog/assisted-generation Those models of LaMini-Flan-T5 are trained to follow instructions and not to recognize the truth content. You could train a transformer like Ernie or Vega (which lead superglue) on such challenging factual data. But don't expect mathematical correct results only from the model. Therefore you have langchain with other APIs.
Blahah 3y ago
Yeah these aren't meant to know facts, just to parse language. Good for understanding simple instructions to automate tasks using external tools
- 3y ago

btdmaster 3y ago

Really cool, I didn't know CPU/traditional RAM was enough already. Though, surprising results from repl.it: lm.classify("unabridged", "positive", "negative")=="unabridged is"

jncraton 3y ago
Thanks for pointing that out. Classification is half-baked at the moment. It should ultimately be restricting output to only appropriate labels, but right now it is simply sampling.

quickthrower2 3y ago

So does this download a hugging-face model and run it locally? Is the hughing face library doing the inference or something? Can’t see as much code as I expected.

angeleye 3y ago

[flagged]