6 ms·

I’m struggling to understand the point of this. It appears to be a more simplified way of getting a local LLM running on your machine, but I expect less technic

I’m struggling to understand the point of this. It appears to be a more simplified way of getting a local LLM running on your machine, but I expect less technically inclined users would default to using the AI built into Windows while the more technical users will leverage llama.cpp to run whatever models they are interested in. Who is the target audience for this solution?

papichulo2023 3y ago
Does windows uses the pc's gpu or just cpu or cloud?
- robotnikman 3y ago
  If they are talking about the Bing AI, just using whatever OpenAI has in the cloud
  - McAtNite 3y ago
    I’m referring to CoPilot which for your average non technical user who doesn’t care whether something is local or not has the huge benefit of not requiring the purchase an expensive GPU.
    - zamadatix 3y ago
      Never underestimate people's interest in running something which lets them generate crass jokes about their friends or smutty conversation when hosted solutions like CoPilot could never allow such non-puritan morals. If this delivers on being the easiest way to run local models quickly then many people will be interested.
seydor 3y ago
Windows users who haven't bought an Nvidia card yet
SirMaster 3y ago
This lets you run Mistral or Llama 2, so whomever has an RTX card and wants to run either of those models? And perhaps they will add more models in the future?
- McAtNite 3y ago
  I suppose I’m just struggling to see the value add. Ollama already makes it dead simple to get a local LLM running, and this appears to be a more limited vendor locked equivalent. From my point of view the only person who would be likely to use this would be the small slice of people who are willing to purchase an expensive GPU, know enough about LLMs to not want to use CoPilot, but don’t know enough about them to know of the already existing solutions.
  - SirMaster 3y ago
    I just looked up Ollama and it doesn't look like it supports Windows. (At least not yet)
    - McAtNite 3y ago
      Oh my apologies for the wild goose chase. I thought they had added support for Windows already. Should be possible to run it through WSL, but I suppose that’s a solid point for Nvidia in this discussion.
      - SirMaster 3y ago
        I think there's a market for a user who is not very computer savvy who at least understands how to use LLMs and would potentially run a chat one on their GPU especially if it's just a few clicks to turn on.
  - kkielhofner 3y ago
    With all due respect this comment has fairly strong (and infamous) HN Dropbox thread vibes. It's an Nvidia "product", published and promoted via their usual channels. This is co-sign/official support from Nvidia vs "Here's an obscure name from a dizzying array of indistinguishable implementations pointing to some random open source project website and Github repo where your eyes will glaze over in seconds". Completely different but wider and significantly less sophisticated audience. The story link is on The Verge and because this is Nvidia it will also get immediately featured in every other tech publication, website, subreddit, forum, twitter account, youtube channel, etc. This will get more installs and usage in the next 72 hours than the entire Llama/open LLM ecosystem has had in its history.
fortran77 3y ago
It seems really clear to me! I downloaded it, pointed it to my documents folder, and started running it. It's nothing like the "AI built into Windows" and it's much easier than dealing with rolling my own.
operator-name 3y ago
This is a tech demo for TensorRT, which is ment to greatly improve inference time for compatible models.
dkarras 3y ago
>It appears to be a more simplified way of getting a local LLM running on your machine No, it answers questions from the documents you provide. Off the shelf local LLMs don't do this by default. You need a RAG stack on top of it or fine tune with your own content.
- westurner 3y ago
  From "Artificial intelligence is ineffective and potentially harmful for fact checking" (2023) https://news.ycombinator.com/item?id=37226233 https://news.ycombinator.com/item?id=37226233 : pdfgpt, knowledge_gpt, elasticsearch : > Are LLM tools better or worse than e.g. meilisearch or elasticsearch for searching with snippets over a set of document resources? > How does search compare to generating things with citations? pdfGPT: https://github.com/bhaskatripathi/pdfGPT https://github.com/bhaskatripathi/pdfGPT : > PDF GPT allows you to chat with the contents of your PDF file by using GPT capabilities. GH "pdfgpt" topic: https://github.com/topics/pdfgpt https://github.com/topics/pdfgpt knowledge_gpt: https://github.com/mmz-001/knowledge_gpt https://github.com/mmz-001/knowledge_gpt From https://news.ycombinator.com/item?id=39112014 https://news.ycombinator.com/item?id=39112014 : paperai neuml/paperai: https://github.com/neuml/paperai https://github.com/neuml/paperai : > Semantic search and workflows for medical/scientific papers RAG: https://news.ycombinator.com/item?id=38370452 https://news.ycombinator.com/item?id=38370452 Google Desktop (2004-2011): https://en.wikipedia.org/wiki/Google_Desktop https://en.wikipedia.org/wiki/Google_Desktop : > Google Desktop was a computer program with desktop search capabilities, created by Google for Linux, Apple Mac OS X, and Microsoft Windows systems. It allowed text searches of a user's email messages, computer files, music, photos, chats, Web pages viewed, and the ability to display "Google Gadgets" on the user's desktop in a Sidebar GNOME/tracker-miners: https://gitlab.gnome.org/GNOME/tracker-miners https://gitlab.gnome.org/GNOME/tracker-miners src/miners/fs: https://gitlab.gnome.org/GNOME/tracker-miners/-/tree/master/src/miners/fs https://gitlab.gnome.org/GNOME/tracker-miners/-/tree/master/... SPARQL + SQLite: https://gitlab.gnome.org/GNOME/tracker-miners/-/blob/master/examples/python/query-async.py https://gitlab.gnome.org/GNOME/tracker-miners/-/blob/master/... https://news.ycombinator.com/item?id=38355385 https://news.ycombinator.com/item?id=38355385 : LocalAI, braintrust-proxy; promptfoo, chainforge, mixtral
joenot443 3y ago
The immediate value prop here is the ability to load up documents to train your model on the fly. 6mos ago I was looking for a tool to do exactly this and ended up deciding to wait. Amazing how fast this wave of innovation is happening.
brucethemoose2 3y ago
> the more technical users will leverage llama.cpp to run whatever models they are interested in. Llama.cpp is much slower, and does not have built-in RAG. TRT-LLM is a finicky deployment grade framework, and TBH having it packaged into a one click install with llama index is very cool. The RAG in particular is beyond what most local LLM UIs do out-of-the-box.

papichulo2023 3y ago

Does windows uses the pc's gpu or just cpu or cloud?

robotnikman 3y ago
If they are talking about the Bing AI, just using whatever OpenAI has in the cloud
- McAtNite 3y ago
  I’m referring to CoPilot which for your average non technical user who doesn’t care whether something is local or not has the huge benefit of not requiring the purchase an expensive GPU.
  - zamadatix 3y ago
    Never underestimate people's interest in running something which lets them generate crass jokes about their friends or smutty conversation when hosted solutions like CoPilot could never allow such non-puritan morals. If this delivers on being the easiest way to run local models quickly then many people will be interested.

seydor 3y ago

Windows users who haven't bought an Nvidia card yet

SirMaster 3y ago

This lets you run Mistral or Llama 2, so whomever has an RTX card and wants to run either of those models? And perhaps they will add more models in the future?

McAtNite 3y ago
I suppose I’m just struggling to see the value add. Ollama already makes it dead simple to get a local LLM running, and this appears to be a more limited vendor locked equivalent. From my point of view the only person who would be likely to use this would be the small slice of people who are willing to purchase an expensive GPU, know enough about LLMs to not want to use CoPilot, but don’t know enough about them to know of the already existing solutions.
- SirMaster 3y ago
  I just looked up Ollama and it doesn't look like it supports Windows. (At least not yet)
  - McAtNite 3y ago
    Oh my apologies for the wild goose chase. I thought they had added support for Windows already. Should be possible to run it through WSL, but I suppose that’s a solid point for Nvidia in this discussion.
    - SirMaster 3y ago
      I think there's a market for a user who is not very computer savvy who at least understands how to use LLMs and would potentially run a chat one on their GPU especially if it's just a few clicks to turn on.
- kkielhofner 3y ago
  With all due respect this comment has fairly strong (and infamous) HN Dropbox thread vibes. It's an Nvidia "product", published and promoted via their usual channels. This is co-sign/official support from Nvidia vs "Here's an obscure name from a dizzying array of indistinguishable implementations pointing to some random open source project website and Github repo where your eyes will glaze over in seconds". Completely different but wider and significantly less sophisticated audience. The story link is on The Verge and because this is Nvidia it will also get immediately featured in every other tech publication, website, subreddit, forum, twitter account, youtube channel, etc. This will get more installs and usage in the next 72 hours than the entire Llama/open LLM ecosystem has had in its history.

fortran77 3y ago

It seems really clear to me! I downloaded it, pointed it to my documents folder, and started running it. It's nothing like the "AI built into Windows" and it's much easier than dealing with rolling my own.

operator-name 3y ago

This is a tech demo for TensorRT, which is ment to greatly improve inference time for compatible models.

dkarras 3y ago

>It appears to be a more simplified way of getting a local LLM running on your machine No, it answers questions from the documents you provide. Off the shelf local LLMs don't do this by default. You need a RAG stack on top of it or fine tune with your own content.

westurner 3y ago
From "Artificial intelligence is ineffective and potentially harmful for fact checking" (2023) https://news.ycombinator.com/item?id=37226233 https://news.ycombinator.com/item?id=37226233 : pdfgpt, knowledge_gpt, elasticsearch : > Are LLM tools better or worse than e.g. meilisearch or elasticsearch for searching with snippets over a set of document resources? > How does search compare to generating things with citations? pdfGPT: https://github.com/bhaskatripathi/pdfGPT https://github.com/bhaskatripathi/pdfGPT : > PDF GPT allows you to chat with the contents of your PDF file by using GPT capabilities. GH "pdfgpt" topic: https://github.com/topics/pdfgpt https://github.com/topics/pdfgpt knowledge_gpt: https://github.com/mmz-001/knowledge_gpt https://github.com/mmz-001/knowledge_gpt From https://news.ycombinator.com/item?id=39112014 https://news.ycombinator.com/item?id=39112014 : paperai neuml/paperai: https://github.com/neuml/paperai https://github.com/neuml/paperai : > Semantic search and workflows for medical/scientific papers RAG: https://news.ycombinator.com/item?id=38370452 https://news.ycombinator.com/item?id=38370452 Google Desktop (2004-2011): https://en.wikipedia.org/wiki/Google_Desktop https://en.wikipedia.org/wiki/Google_Desktop : > Google Desktop was a computer program with desktop search capabilities, created by Google for Linux, Apple Mac OS X, and Microsoft Windows systems. It allowed text searches of a user's email messages, computer files, music, photos, chats, Web pages viewed, and the ability to display "Google Gadgets" on the user's desktop in a Sidebar GNOME/tracker-miners: https://gitlab.gnome.org/GNOME/tracker-miners https://gitlab.gnome.org/GNOME/tracker-miners src/miners/fs: https://gitlab.gnome.org/GNOME/tracker-miners/-/tree/master/src/miners/fs https://gitlab.gnome.org/GNOME/tracker-miners/-/tree/master/... SPARQL + SQLite: https://gitlab.gnome.org/GNOME/tracker-miners/-/blob/master/examples/python/query-async.py https://gitlab.gnome.org/GNOME/tracker-miners/-/blob/master/... https://news.ycombinator.com/item?id=38355385 https://news.ycombinator.com/item?id=38355385 : LocalAI, braintrust-proxy; promptfoo, chainforge, mixtral

joenot443 3y ago

The immediate value prop here is the ability to load up documents to train your model on the fly. 6mos ago I was looking for a tool to do exactly this and ended up deciding to wait. Amazing how fast this wave of innovation is happening.

brucethemoose2 3y ago

> the more technical users will leverage llama.cpp to run whatever models they are interested in. Llama.cpp is much slower, and does not have built-in RAG. TRT-LLM is a finicky deployment grade framework, and TBH having it packaged into a one click install with llama index is very cool. The RAG in particular is beyond what most local LLM UIs do out-of-the-box.