6 ms·

This type of article (or press release, or whatever you want to call it) is exactly what makes the future so interesting. The cat is out of the bag, the genie

This type of article (or press release, or whatever you want to call it) is exactly what makes the future so interesting. The cat is out of the bag, the genie is out of the bottle, the confetti has left the cannon[0]. It's tempting to see a world dominated by Google Bard, ChatGPT, Bing Search, etc. And no doubt, they will be huge players, with services that are far more powerful than anything that can be run on the edge. But. BUT. The things that we can do on the edge are incredible now. Just imagine a year from now, or two. These earth-shattering models, which seem to be upending a whole industry, will soon have equivalents that run on the edge. Without services spying on your data. Without censorship on what the model can/cannot say. Because it's all local. When was the last time this happened? There will be players who publish weights for models that are free to use. The moment that torrent magnet link is published, it's out in the wild. And smart people will package them as "one click installers" for people who aren't tech-savvy. This is already happening. So every time you're amazed by something chat-gpt4 says, remember that soon this will be in your pocket. [0] the "confetti" idiom brought to you by chat-gpt4.

jazzkingrt 3y ago
Serious question: is it typical to describe client-side computing as "on the edge"? I thought running something on the edge referred to running it in close network proximity to the user, rather than users having control and running things themselves.
- capableweb 3y ago
  Yes, "edge computing" can refer to both computing done as close to the user as possible geographically, or even on the device itself. If someone says "I wanna do edge computing" it's not clear enough to know if they just want to have servers they control as close to the user as possible, or do the computing on the device itself. I think Apple would say "edge computing" is on the actual device while CloudFlare would say "edge computing" is on their infrastructure, but distributed to be physically closer to the end user.
  - iamerroragent 3y ago
    I guess I've been out of the loop for a bit and didn't realize that "edge computing" became a term since cloud computing took off. It is kind of cyclical then is not? By that I mean computers used to be shared and to log into it through a terminal. Then the PC came around. Then about 15 years ago Cloud computing became the rage (really an extension or more sophisticated system than the first time shared computers) Now we're back to local computing. I even see more self hosting and moving away from cloud due to costs. All that rant is to say is it's interesting. Side note, getting this AI to be localized as much as possible I imagine will be really useful in the medical industry because it helps alleviate HIPAA requirements.
    - nordsieck 3y ago
      > It is kind of cyclical then is not? > By that I mean computers used to be shared and to log into it through a terminal. > Then the PC came around. > Then about 15 years ago Cloud computing became the rage (really an extension or more sophisticated system than the first time shared computers) There's a really neat article called "The Eternal Mainframe"[1] that you might be interested. It explores this idea in greater depth. --- 1. http://www.winestockwebdesign.com/Essays/Eternal_Mainframe.html http://www.winestockwebdesign.com/Essays/Eternal_Mainframe.h...
      - iamerroragent 3y ago
        Thanks, that was an interesting read! I wonder if the author's perspective has changed with regards to freedom to compute. Social Media is often used as an example of privacy invasion though I've failed to see why concerns over Facebook handling your private data is worrying when they don't have a product you need to have. Email on the other hand, is pretty much a necessity today so privacy concerns are vital there imo. Of course you can host your own server whereas you can't host your own Facebook.
        macintux 3y ago
        > I've failed to see why concerns over Facebook handling your private data is worrying when they don't have a product you need to have. I have at least two concerns. 1) Viscerally, it would be intensely creepy if someone started following me everywhere. Why would I want a company doing the same? 2) We are in an era of diminishing freedoms, and it's impossible to prove that data Facebook has on us can't be turned against us in the court system in the future. Women's menstrual cycles are being weaponized, for Pete's sake.
- wsgeorge 3y ago
  I believe this has been extended to mean "on device", which is interesting. See Gerganov's article on Github [0]. I wrote about this here [1] where I made a contrast between the core and the edge. I think the term maps well to this meaning. What I find more interesting is that in the classic "close network proximity", some parts of the world may not have benefited as much from that trend since the closest nodes of a global delivery network could be several countries away. [0] https://github.com/ggerganov/llama.cpp/discussions/205 https://github.com/ggerganov/llama.cpp/discussions/205 [1] https://medium.com/sort-of-like-a-tech-diary/consumer-ai-is-ripe-for-centralization-f3251dfa460e https://medium.com/sort-of-like-a-tech-diary/consumer-ai-is-...
  - TeMPOraL 3y ago
    > I believe this has been extended to mean "on device", which is interesting. I don't like the connotations this carries. This is almost openly talking about reaching all the way into peoples' hardware to run your software, for your benefit, on them, without their knowledge, consent or control...
    - wsgeorge 3y ago
      I see. Hadn't considered this. Yes, I see how that might be a concern. What I think is important in this AI Spring is that we make it possible for people to run their own models on their own hardware too, without having to submit anything to a large, centralised model for inference.
- dragonwriter 3y ago
  > Serious question: is it typical to describe client-side computing as “on the edge”? Somewhat; its consistent with, e.g., Google’s “Edge TPU” designation for its client-side neural processors. > I thought running something on the edge referred to running it in close network proximity to the user Typically, but on the client device is the limit-case of “close network proximity to the user”, so the use is consistent.
- layer8 3y ago
  “Edge computing” arguably implies there’s a network you are connected t, that you’re on the edge of, so I wouldn’t apply the term to applications that can function completely offline. With edge computing there’s usually still a notion of having some sort of internet integration, like IoT devices.
- dannyobrien 3y ago
  I've used "edge" in this context for around 15 years[1], and I've always intended it to mean "at the edge of the network", which can include being on the other side of the world to a user. [1] from https://www.oblomovka.com/wp/2007/08/ https://www.oblomovka.com/wp/2007/08/ at least
- aargh_aargh 3y ago
  Because of the ambiguity of the term "on the edge" that is used to refer to both close network proximity and the device closest to the user, as evidenced by this thread, I would suggest to use a new term, at least in the context of A.I. The AI running on the device closest to the user should be called a "terminator".
lioeters 3y ago
Yes, yes, and yes. I'm waiting for an actually open AI that can run on the edge, purely on commodity hardware like our laptops and phones - it's inevitable. I imagine this "cat out of the bag" situation, the democratization and commodification of powerful technology accessible and affordable to the public, is similar to what's happening with single-board computers and microcontrollers like Raspberry Pi, Arduino, ESP32. It might be similar to what happened with mobile phones, but there the power was quite restricted. The (mostly) duopoly of iOS and Android, with devices and apps locked down in various ways. Sure we can "jail break" and "root" our phone, but that's not for the general public. Maybe solar energy production is going through a similar process, with panels and batteries becoming more efficient and affordable every year. Certainly, it reminds one of the history of personal computers, the way such a powerful general-purpose tool became ubiquitous and local.
- akiselev 3y ago
  After using ChatGPT 4 extensively for a few days, I think we're probably only a few years away from the first generation of truly conversational assistants ala Jarvis in Iron Man. Between LangChain and existing voice recognition software, we've already 95% of the way there, it just needs to be packaged up into a UI/UX that makes sense. These local models are absolutely critical for that to happen though. I'm hitting daily situations where I have to reconsider my use of ChatGPT because what I'm asking would leak very private personal information or somehow trip its morality filter. Just swapped in a 2TB nvme for a separate /home mount and reinstalled Arch just so I could have enough disk space to test a dozen models locally. I'm so ready!
  - alfor 3y ago
    GPT-4 training was finished last summer. Karpathy is working on a JARVIS there. I think they already have something quite good for internal use, maybe trying to make it "safe" before releasing it. My guess: we will have a competent JARVIS(minus the holograms) this year.
  - meghan_rain 3y ago
    I pray to the AI gods that OpenAI will fail at calibrating the censorship layer and will continue to overcensor, which in turn will hopefully lead to many usecases requiring local LLMs, which in turn would increase the incentive to build them.
simon83 3y ago
Google: "confetti has left the cannon" > No results found for "confetti has left the cannon". I'm amazed that a "stochastic parrot" can come up with such a beautiful idiom.
- athom 3y ago
  Try looking up "pinkie pie party cannon"
- barking_biscuit 3y ago
  Out of distribution generations are a thing.
  - simon83 3y ago
    I understand that all of this is based on some fundamental mathematics, a couple of algorithms here, and some statistical analysis there. And I find it absolutely amazing that we can use all of that knowledge and encode it into something that resembles intelligence. This makes me think if our brains and the concept of intelligence are really as special and mysterious as we assume.
- visarga 3y ago
  That name aged like milk. First of all, what you said. And second - a parrot can make more parrots without human help, language models can't make GPU chips. Insulting for both LLMs and parrots.
  - groestl 3y ago
    > language models can't make GPU chips Not yet. I'm sure somebody is taking a hard look already.
    - dougabug 3y ago
      I would assume some of them can write Verilog code
hiAndrewQuinn 3y ago
I for one dream of a future without maps. I want to walk through a distant forest to find an ancient, unconnected ESP-32 in the bark of a tree containing a tiny specialized AI that can only tell me about things relevant to the area, how far to walk upstream to the nearest town. And only if I can find it and scan an RFID tag to wake it up.
- vinc 3y ago
  A beautiful dream! > I like to think (right now please!) of a cybernetic forest filled with pines and electronics where deer stroll peacefully past computers as if they were flowers with spinning blossoms.
deleted 3y ago
[deleted]
t_minus_2 3y ago
The cat is out of the bag,The genie is out of the bottle,The confetti has left the cannon,The ship has sailed,The horse has bolted,The toothpaste is out of the tube,The beans have been spilled,The train has left the station,The die is cast,The bell has been run.
- pmoriarty 3y ago
  The cookie has crumbled. The mirror has shattered. The poop has hit the propeller. Pandora's box has opened.
cjf101 3y ago
Yes, this is true. But, I worry about how long it will take for the utility of "GPT-4" on my phone to be close enough to whatever is only possible through models running on large cloud platforms to make that choice relatively drawback free. Is the curve of what this class of algorithms can provide sigmoid? If so, then yeah, eventually researchers should be able to democratize it sufficiently that the choice to use versions that can run on private hardware rational. But if the utility increases linearly or better over time/scale, the future will belong to whoever owns the biggest datacenters.
hintymad 3y ago
I'd go one step further if it is not happening yet: smaller companies should really pool their resources to train open LLMs. Say, form a consortium and work with the open source community to build ChatGPT-equivalent. Companies will be crazy to assume that they can hand their future to the APIs offered by a handful of companies during this monumental technological paradigm shift in history. That is, a real OpenAI with a open government body.
matchagaucho 3y ago
An LLM running locally providing type-ahead completions seems inevitable.
yieldcrv 3y ago
> And smart people will package them as "one click installers" for people who aren't tech-savvy. This is already happening. Any projects I can follow? Because I haven't seen any one click installers yet that didn't begin with "first install a package manager on the command line"
- slickdork 3y ago
  Not an llm but this 1 click installer for stable diffusion is literally a 1 click installer. Impressively works. https://github.com/cmdr2/stable-diffusion-ui https://github.com/cmdr2/stable-diffusion-ui
  - yieldcrv 3y ago
    > In the terminal, run ./start.sh (or bash start.sh) smh. well, very close! its interesting that they just got Mac M1/M2 support at all, 2 weeks ago. for SD I've been using DiffusionBee since maybe October last year. I expect LLMs to have something in a few weeks, just want to know about it so I can tell other people that need it that way.
- 2bitencryption 3y ago
  I was mostly referring to this project, which has some 1-click installers: https://github.com/oobabooga/text-generation-webui#alternative-one-click-installers https://github.com/oobabooga/text-generation-webui#alternati... Though I have not tried those 1-click installers, instead I have been manually running it. That project is based on the concept of this Stable Diffusion project: https://github.com/AUTOMATIC1111/stable-diffusion-webui https://github.com/AUTOMATIC1111/stable-diffusion-webui Which is a few months ahead (because the Stable Diffusion tech happened a few months earlier) and is definitely at a point where anyone can easily run it, locally or on a hosted environment. I expect this "text-generation-webui" (or something like it) will be just as easy to use in the near, near future.
- nathanasmith 3y ago
  Here's alpaca running in electron. Not exactly one click but close. https://github.com/ItsPi3141/alpaca-electron https://github.com/ItsPi3141/alpaca-electron
slowmovintarget 3y ago
> Without services spying on your data. Without censorship on what the model can/cannot say. Because it's all local... Wouldn't that be nice? It would also be contrary to all experience of the outcomes and pulls of corporations in modern society. The "local" LLMs will be on the fringe more than at the edge, because the ones that work the best and attract the most money will be the ones controlled by walled-garden "ecosystems." I really hope it's different. I really hope there are local models. Actual personal assistants actually designed to assist their users and not the people that provide the access.
wing-_-nuts 3y ago
>So every time you're amazed by something chat-gpt4 says, remember that soon this will be in your pocket. I want to believe you, but I'm ignorant of the hardware requirements for these things. How soon do you think we'd be able to run something reasonably gpt4-like on, say, a 4090?
- LightMachine 3y ago
  I feel like no less than 10 years if the singularity doesn't kick in before that. Hardware and energy isn't progressing as fast as we'd like, and that is the main bottleneck. As in, imagine a world where we actually had the same computing power required to train (not run) GPT-4 in 1s in a phone? That kind of world is way beyond AGI and the cure of cancer IMO. Which is great, because it gives us a very objective goal to achieve these things. Sadly, I don't think we're nowhere near that. What was even the total energy consumption of GPT-4 training? Very hard to imagine computers will get that much better anytime soon. IIRC we have some kind of data that the a smartphone today has the power of the best supercomputer... of 30 years ago, right? Don't remember the source, sadly.
  - colordrops 3y ago
    Your comment makes me wonder if it's not a coincidence that as we seem to be the hitting a limit in hardware power that human level intelligence begins to emerge.
    - pmoriarty 3y ago
      Such hardware problems might be overcome with new computer architectures and/or substrates, like DNA computing, quantum computing, etc... Current AI's could help us overcome such limits.
  - waboremo 3y ago
    I don't think 10 years for training such large models on phones is entirely feasible, if only because phones are mainly concerned with power drain and ergonomics before anything else. Nobody is going to buy an iPhone 15 if it's the size of a brick and lasts 2 hours just because you're training&running models on it. The focus instead should be on running expansive models locally on your desktop system at home. This is a better focus in two ways: one the power issue is a non-concern (and AMD has already stated power usage will reach 500-700W average by 2025), two once you have it running locally the avenues of use open up such as being able to access your local model through other devices without the heavy burden on those devices.
xnx 3y ago
This is a shocking turn of events given there's no edge equivalent of the previous most powerful information tools (web-scale search). It does seem like it will still be a challenge to continuously collect, validate, and train on fresh information. Large orgs like Google/YouTube/TikTok/Microsoft still seem to have a huge advantage there.

jazzkingrt 3y ago

Serious question: is it typical to describe client-side computing as "on the edge"? I thought running something on the edge referred to running it in close network proximity to the user, rather than users having control and running things themselves.

capableweb 3y ago
Yes, "edge computing" can refer to both computing done as close to the user as possible geographically, or even on the device itself. If someone says "I wanna do edge computing" it's not clear enough to know if they just want to have servers they control as close to the user as possible, or do the computing on the device itself. I think Apple would say "edge computing" is on the actual device while CloudFlare would say "edge computing" is on their infrastructure, but distributed to be physically closer to the end user.
- iamerroragent 3y ago
  I guess I've been out of the loop for a bit and didn't realize that "edge computing" became a term since cloud computing took off. It is kind of cyclical then is not? By that I mean computers used to be shared and to log into it through a terminal. Then the PC came around. Then about 15 years ago Cloud computing became the rage (really an extension or more sophisticated system than the first time shared computers) Now we're back to local computing. I even see more self hosting and moving away from cloud due to costs. All that rant is to say is it's interesting. Side note, getting this AI to be localized as much as possible I imagine will be really useful in the medical industry because it helps alleviate HIPAA requirements.
  - nordsieck 3y ago
    > It is kind of cyclical then is not? > By that I mean computers used to be shared and to log into it through a terminal. > Then the PC came around. > Then about 15 years ago Cloud computing became the rage (really an extension or more sophisticated system than the first time shared computers) There's a really neat article called "The Eternal Mainframe"[1] that you might be interested. It explores this idea in greater depth. --- 1. http://www.winestockwebdesign.com/Essays/Eternal_Mainframe.html http://www.winestockwebdesign.com/Essays/Eternal_Mainframe.h...
    - iamerroragent 3y ago
      Thanks, that was an interesting read! I wonder if the author's perspective has changed with regards to freedom to compute. Social Media is often used as an example of privacy invasion though I've failed to see why concerns over Facebook handling your private data is worrying when they don't have a product you need to have. Email on the other hand, is pretty much a necessity today so privacy concerns are vital there imo. Of course you can host your own server whereas you can't host your own Facebook.
      - macintux 3y ago
        > I've failed to see why concerns over Facebook handling your private data is worrying when they don't have a product you need to have. I have at least two concerns. 1) Viscerally, it would be intensely creepy if someone started following me everywhere. Why would I want a company doing the same? 2) We are in an era of diminishing freedoms, and it's impossible to prove that data Facebook has on us can't be turned against us in the court system in the future. Women's menstrual cycles are being weaponized, for Pete's sake.
wsgeorge 3y ago
I believe this has been extended to mean "on device", which is interesting. See Gerganov's article on Github [0]. I wrote about this here [1] where I made a contrast between the core and the edge. I think the term maps well to this meaning. What I find more interesting is that in the classic "close network proximity", some parts of the world may not have benefited as much from that trend since the closest nodes of a global delivery network could be several countries away. [0] https://github.com/ggerganov/llama.cpp/discussions/205 https://github.com/ggerganov/llama.cpp/discussions/205 [1] https://medium.com/sort-of-like-a-tech-diary/consumer-ai-is-ripe-for-centralization-f3251dfa460e https://medium.com/sort-of-like-a-tech-diary/consumer-ai-is-...
- TeMPOraL 3y ago
  > I believe this has been extended to mean "on device", which is interesting. I don't like the connotations this carries. This is almost openly talking about reaching all the way into peoples' hardware to run your software, for your benefit, on them, without their knowledge, consent or control...
  - wsgeorge 3y ago
    I see. Hadn't considered this. Yes, I see how that might be a concern. What I think is important in this AI Spring is that we make it possible for people to run their own models on their own hardware too, without having to submit anything to a large, centralised model for inference.
dragonwriter 3y ago
> Serious question: is it typical to describe client-side computing as “on the edge”? Somewhat; its consistent with, e.g., Google’s “Edge TPU” designation for its client-side neural processors. > I thought running something on the edge referred to running it in close network proximity to the user Typically, but on the client device is the limit-case of “close network proximity to the user”, so the use is consistent.
layer8 3y ago
“Edge computing” arguably implies there’s a network you are connected t, that you’re on the edge of, so I wouldn’t apply the term to applications that can function completely offline. With edge computing there’s usually still a notion of having some sort of internet integration, like IoT devices.
dannyobrien 3y ago
I've used "edge" in this context for around 15 years[1], and I've always intended it to mean "at the edge of the network", which can include being on the other side of the world to a user. [1] from https://www.oblomovka.com/wp/2007/08/ https://www.oblomovka.com/wp/2007/08/ at least
aargh_aargh 3y ago
Because of the ambiguity of the term "on the edge" that is used to refer to both close network proximity and the device closest to the user, as evidenced by this thread, I would suggest to use a new term, at least in the context of A.I. The AI running on the device closest to the user should be called a "terminator".

lioeters 3y ago

Yes, yes, and yes. I'm waiting for an actually open AI that can run on the edge, purely on commodity hardware like our laptops and phones - it's inevitable. I imagine this "cat out of the bag" situation, the democratization and commodification of powerful technology accessible and affordable to the public, is similar to what's happening with single-board computers and microcontrollers like Raspberry Pi, Arduino, ESP32. It might be similar to what happened with mobile phones, but there the power was quite restricted. The (mostly) duopoly of iOS and Android, with devices and apps locked down in various ways. Sure we can "jail break" and "root" our phone, but that's not for the general public. Maybe solar energy production is going through a similar process, with panels and batteries becoming more efficient and affordable every year. Certainly, it reminds one of the history of personal computers, the way such a powerful general-purpose tool became ubiquitous and local.

akiselev 3y ago
After using ChatGPT 4 extensively for a few days, I think we're probably only a few years away from the first generation of truly conversational assistants ala Jarvis in Iron Man. Between LangChain and existing voice recognition software, we've already 95% of the way there, it just needs to be packaged up into a UI/UX that makes sense. These local models are absolutely critical for that to happen though. I'm hitting daily situations where I have to reconsider my use of ChatGPT because what I'm asking would leak very private personal information or somehow trip its morality filter. Just swapped in a 2TB nvme for a separate /home mount and reinstalled Arch just so I could have enough disk space to test a dozen models locally. I'm so ready!
- alfor 3y ago
  GPT-4 training was finished last summer. Karpathy is working on a JARVIS there. I think they already have something quite good for internal use, maybe trying to make it "safe" before releasing it. My guess: we will have a competent JARVIS(minus the holograms) this year.
- meghan_rain 3y ago
  I pray to the AI gods that OpenAI will fail at calibrating the censorship layer and will continue to overcensor, which in turn will hopefully lead to many usecases requiring local LLMs, which in turn would increase the incentive to build them.

simon83 3y ago

Google: "confetti has left the cannon" > No results found for "confetti has left the cannon". I'm amazed that a "stochastic parrot" can come up with such a beautiful idiom.

athom 3y ago
Try looking up "pinkie pie party cannon"
barking_biscuit 3y ago
Out of distribution generations are a thing.
- simon83 3y ago
  I understand that all of this is based on some fundamental mathematics, a couple of algorithms here, and some statistical analysis there. And I find it absolutely amazing that we can use all of that knowledge and encode it into something that resembles intelligence. This makes me think if our brains and the concept of intelligence are really as special and mysterious as we assume.
visarga 3y ago
That name aged like milk. First of all, what you said. And second - a parrot can make more parrots without human help, language models can't make GPU chips. Insulting for both LLMs and parrots.
- groestl 3y ago
  > language models can't make GPU chips Not yet. I'm sure somebody is taking a hard look already.
  - dougabug 3y ago
    I would assume some of them can write Verilog code

hiAndrewQuinn 3y ago

I for one dream of a future without maps. I want to walk through a distant forest to find an ancient, unconnected ESP-32 in the bark of a tree containing a tiny specialized AI that can only tell me about things relevant to the area, how far to walk upstream to the nearest town. And only if I can find it and scan an RFID tag to wake it up.

vinc 3y ago
A beautiful dream! > I like to think (right now please!) of a cybernetic forest filled with pines and electronics where deer stroll peacefully past computers as if they were flowers with spinning blossoms.

deleted 3y ago

[deleted]

t_minus_2 3y ago

The cat is out of the bag,The genie is out of the bottle,The confetti has left the cannon,The ship has sailed,The horse has bolted,The toothpaste is out of the tube,The beans have been spilled,The train has left the station,The die is cast,The bell has been run.

pmoriarty 3y ago
The cookie has crumbled. The mirror has shattered. The poop has hit the propeller. Pandora's box has opened.

cjf101 3y ago

Yes, this is true. But, I worry about how long it will take for the utility of "GPT-4" on my phone to be close enough to whatever is only possible through models running on large cloud platforms to make that choice relatively drawback free. Is the curve of what this class of algorithms can provide sigmoid? If so, then yeah, eventually researchers should be able to democratize it sufficiently that the choice to use versions that can run on private hardware rational. But if the utility increases linearly or better over time/scale, the future will belong to whoever owns the biggest datacenters.

hintymad 3y ago

I'd go one step further if it is not happening yet: smaller companies should really pool their resources to train open LLMs. Say, form a consortium and work with the open source community to build ChatGPT-equivalent. Companies will be crazy to assume that they can hand their future to the APIs offered by a handful of companies during this monumental technological paradigm shift in history. That is, a real OpenAI with a open government body.

matchagaucho 3y ago

An LLM running locally providing type-ahead completions seems inevitable.

yieldcrv 3y ago

> And smart people will package them as "one click installers" for people who aren't tech-savvy. This is already happening. Any projects I can follow? Because I haven't seen any one click installers yet that didn't begin with "first install a package manager on the command line"

slickdork 3y ago
Not an llm but this 1 click installer for stable diffusion is literally a 1 click installer. Impressively works. https://github.com/cmdr2/stable-diffusion-ui https://github.com/cmdr2/stable-diffusion-ui
- yieldcrv 3y ago
  > In the terminal, run ./start.sh (or bash start.sh) smh. well, very close! its interesting that they just got Mac M1/M2 support at all, 2 weeks ago. for SD I've been using DiffusionBee since maybe October last year. I expect LLMs to have something in a few weeks, just want to know about it so I can tell other people that need it that way.
2bitencryption 3y ago
I was mostly referring to this project, which has some 1-click installers: https://github.com/oobabooga/text-generation-webui#alternative-one-click-installers https://github.com/oobabooga/text-generation-webui#alternati... Though I have not tried those 1-click installers, instead I have been manually running it. That project is based on the concept of this Stable Diffusion project: https://github.com/AUTOMATIC1111/stable-diffusion-webui https://github.com/AUTOMATIC1111/stable-diffusion-webui Which is a few months ahead (because the Stable Diffusion tech happened a few months earlier) and is definitely at a point where anyone can easily run it, locally or on a hosted environment. I expect this "text-generation-webui" (or something like it) will be just as easy to use in the near, near future.
nathanasmith 3y ago
Here's alpaca running in electron. Not exactly one click but close. https://github.com/ItsPi3141/alpaca-electron https://github.com/ItsPi3141/alpaca-electron

slowmovintarget 3y ago

> Without services spying on your data. Without censorship on what the model can/cannot say. Because it's all local... Wouldn't that be nice? It would also be contrary to all experience of the outcomes and pulls of corporations in modern society. The "local" LLMs will be on the fringe more than at the edge, because the ones that work the best and attract the most money will be the ones controlled by walled-garden "ecosystems." I really hope it's different. I really hope there are local models. Actual personal assistants actually designed to assist their users and not the people that provide the access.

wing-_-nuts 3y ago

>So every time you're amazed by something chat-gpt4 says, remember that soon this will be in your pocket. I want to believe you, but I'm ignorant of the hardware requirements for these things. How soon do you think we'd be able to run something reasonably gpt4-like on, say, a 4090?

LightMachine 3y ago
I feel like no less than 10 years if the singularity doesn't kick in before that. Hardware and energy isn't progressing as fast as we'd like, and that is the main bottleneck. As in, imagine a world where we actually had the same computing power required to train (not run) GPT-4 in 1s in a phone? That kind of world is way beyond AGI and the cure of cancer IMO. Which is great, because it gives us a very objective goal to achieve these things. Sadly, I don't think we're nowhere near that. What was even the total energy consumption of GPT-4 training? Very hard to imagine computers will get that much better anytime soon. IIRC we have some kind of data that the a smartphone today has the power of the best supercomputer... of 30 years ago, right? Don't remember the source, sadly.
- colordrops 3y ago
  Your comment makes me wonder if it's not a coincidence that as we seem to be the hitting a limit in hardware power that human level intelligence begins to emerge.
  - pmoriarty 3y ago
    Such hardware problems might be overcome with new computer architectures and/or substrates, like DNA computing, quantum computing, etc... Current AI's could help us overcome such limits.
- waboremo 3y ago
  I don't think 10 years for training such large models on phones is entirely feasible, if only because phones are mainly concerned with power drain and ergonomics before anything else. Nobody is going to buy an iPhone 15 if it's the size of a brick and lasts 2 hours just because you're training&running models on it. The focus instead should be on running expansive models locally on your desktop system at home. This is a better focus in two ways: one the power issue is a non-concern (and AMD has already stated power usage will reach 500-700W average by 2025), two once you have it running locally the avenues of use open up such as being able to access your local model through other devices without the heavy burden on those devices.

xnx 3y ago

This is a shocking turn of events given there's no edge equivalent of the previous most powerful information tools (web-scale search). It does seem like it will still be a challenge to continuously collect, validate, and train on fresh information. Large orgs like Google/YouTube/TikTok/Microsoft still seem to have a huge advantage there.