5 ms·

While a lot of people are concerned with local model performance, I wonder how feasible is it now to run a local indexed web search? Surely running an old schoo

While a lot of people are concerned with local model performance, I wonder how feasible is it now to run a local indexed web search? Surely running an old school Google is possible with the beefy AI rigs today. I know the problem will be crawling which would be bottlenecked by the ISP but I use Google to search SO, Wikipedia, programming language docs, Github issues, and AWS docs. I think a feasible workflow would be to build a set of sites of most interest to you and then prioritize those in crawling. While typing this out I remembered https://en.wikipedia.org/wiki/Google_Search_Appliance https://en.wikipedia.org/wiki/Google_Search_Appliance which I never personally used but shows feasibility for the idea. I'm pretty sure one of the newly-announced Macbooks is more than up to the task of matching GSA's offering.

Jakob 5d ago
Until 10 years ago, i used Dash for that. It’s still around https://kapeli.com/dash https://kapeli.com/dash It’s instant, works offline, auto-updates, and includes all the websites you listed, and allows for custom ones too.
- skinfaxi 5d ago
  What do you use now?
  - ormax3 4d ago
    https://devdocs.io/ https://devdocs.io/
CobrastanJorji 5d ago
Plausible. Figure 100 GB each of search index for Stack Overflow, Wikipedia, and GitHub issues, then add a dozen more for docs of all your favorite techs. So maybe half a terabyte. Download and build updated dumps of those once every week or two, and it'd work pretty well. Impractical, but possible.
orbital-decay 5d ago
https://yacy.net/ https://yacy.net/
Jskewel 5d ago
Impossible. The majority of websites firewall automated crawler traffic (because of the rise of the bots), only making exceptions for the largest search engines. There is no possibility of starting a new crawler.
- Cakez0r 5d ago
  It would be interesting to see a decentralised, residential collective that builds and publishes an index. There are surely enough interested people on HN alone that would be willing to run software at home to scrape a small slice of the internet.
  - nairboon 5d ago
    Something like YaCy?
    - Cakez0r 4d ago
      Just like that! I will give it a try
  - nektro 4d ago
    residential proxies are one of the leading causes of what's killing the internet right now
- mitxela 5d ago
  The majority of websites try to do that but they do not catch as much traffic as they think they do. A starting point for a scraper is to run it on your home connection in an undetectable web driver framework such as zendriver.
- dwedge 4d ago
  This was a knee jerk response to the first paragraph. They weren't talking about a general crawler, but a subset of Wikipedia, stack overflow, programming docs and github. You can download archives of all of those except github, and github could be queried using the api or GH cli
tingletech 5d ago
SearXNG configured as in the OpenWebUI docs is pretty cool. My "Hello World" with a new agent framework is teaching it to use SearXNG. Hook this in as a tool and the agent can answer a lot of questions. SearXNG is more of a metasearch, the dude who wrote it pops in on here and is working on a cool sounding project that is more like a local personal search engine, I forget the name, but I've been meaning to check it out. There is also Common Crawl. https://docs.openwebui.com/features/chat-conversations/web-search/providers/searxng/ https://docs.openwebui.com/features/chat-conversations/web-s... https://github.com/brian-learns/xng-agent https://github.com/brian-learns/xng-agent
- rolisz 5d ago
  He's working on Hister now. I really like it.
- nijave 4d ago
  SearXNG has been working pretty well for me. I had an agent write the MCP then do a couple passes comparing to server side LLM web tools and exa and tweaking and it works pretty well. I also added scrapling for fetch which covers pretty much everything but sometimes is a bit context heavy.
nairboon 5d ago
It's still kind of an open problem. There are partial solutions, but not yet really an integrated one. There is Hister: https://hister.org/ https://hister.org/ that builds some sort of drive-by index of what you're browsing anyway. And there are "true p2p" solutions like YaCy: https://yacy.net/ https://yacy.net/ but it takes ages to crawl the open web (and lots of storage). There's still a lot of optimization to do in this space.
numpad0 4d ago
Wayback Machine full archive is like less than 50PB. Let's say you could strip multimedia and remove every patterned data to compress that into about a petabyte. The per-bit cheapest disk right now is consumer Seagate 24TB(SI; 21.8TiB usable) at ~$500, or around $1200k for just the disks. Doable if you had couple million dollars to burn. Cheaper than private jets new.
- nijave 4d ago
  Seems like the most practical path is some kind of distributed peer to peer contraption where you can allocate some storage and optionally participate in crawling. I imagine if set some constraints you could get index size down quite a bit but I still suspect it'd be hard keeping up with content churn. Edit: Looks like enwiki bz2 is coming in around 46Gi which isn't too bad considering the amount of content it contains.
8organicbits 4d ago
For some of that, you don't need to crawl. Wikipedia offers database dumps which you can download in one go. Lots of programming docs are managed in repos, so you can clone the repo instead. Even stackoverflow seems to have a snapshot dump (https://archive.org/details/stackexchange https://archive.org/details/stackexchange).

Jakob 5d ago

Until 10 years ago, i used Dash for that. It’s still around https://kapeli.com/dash https://kapeli.com/dash It’s instant, works offline, auto-updates, and includes all the websites you listed, and allows for custom ones too.

skinfaxi 5d ago
What do you use now?
- ormax3 4d ago
  https://devdocs.io/ https://devdocs.io/

CobrastanJorji 5d ago

Plausible. Figure 100 GB each of search index for Stack Overflow, Wikipedia, and GitHub issues, then add a dozen more for docs of all your favorite techs. So maybe half a terabyte. Download and build updated dumps of those once every week or two, and it'd work pretty well. Impractical, but possible.

orbital-decay 5d ago

https://yacy.net/ https://yacy.net/

Jskewel 5d ago

Impossible. The majority of websites firewall automated crawler traffic (because of the rise of the bots), only making exceptions for the largest search engines. There is no possibility of starting a new crawler.

Cakez0r 5d ago
It would be interesting to see a decentralised, residential collective that builds and publishes an index. There are surely enough interested people on HN alone that would be willing to run software at home to scrape a small slice of the internet.
- nairboon 5d ago
  Something like YaCy?
  - Cakez0r 4d ago
    Just like that! I will give it a try
- nektro 4d ago
  residential proxies are one of the leading causes of what's killing the internet right now
mitxela 5d ago
The majority of websites try to do that but they do not catch as much traffic as they think they do. A starting point for a scraper is to run it on your home connection in an undetectable web driver framework such as zendriver.
dwedge 4d ago
This was a knee jerk response to the first paragraph. They weren't talking about a general crawler, but a subset of Wikipedia, stack overflow, programming docs and github. You can download archives of all of those except github, and github could be queried using the api or GH cli

tingletech 5d ago

SearXNG configured as in the OpenWebUI docs is pretty cool. My "Hello World" with a new agent framework is teaching it to use SearXNG. Hook this in as a tool and the agent can answer a lot of questions. SearXNG is more of a metasearch, the dude who wrote it pops in on here and is working on a cool sounding project that is more like a local personal search engine, I forget the name, but I've been meaning to check it out. There is also Common Crawl. https://docs.openwebui.com/features/chat-conversations/web-search/providers/searxng/ https://docs.openwebui.com/features/chat-conversations/web-s... https://github.com/brian-learns/xng-agent https://github.com/brian-learns/xng-agent

rolisz 5d ago
He's working on Hister now. I really like it.
nijave 4d ago
SearXNG has been working pretty well for me. I had an agent write the MCP then do a couple passes comparing to server side LLM web tools and exa and tweaking and it works pretty well. I also added scrapling for fetch which covers pretty much everything but sometimes is a bit context heavy.

nairboon 5d ago

It's still kind of an open problem. There are partial solutions, but not yet really an integrated one. There is Hister: https://hister.org/ https://hister.org/ that builds some sort of drive-by index of what you're browsing anyway. And there are "true p2p" solutions like YaCy: https://yacy.net/ https://yacy.net/ but it takes ages to crawl the open web (and lots of storage). There's still a lot of optimization to do in this space.

numpad0 4d ago

Wayback Machine full archive is like less than 50PB. Let's say you could strip multimedia and remove every patterned data to compress that into about a petabyte. The per-bit cheapest disk right now is consumer Seagate 24TB(SI; 21.8TiB usable) at ~$500, or around $1200k for just the disks. Doable if you had couple million dollars to burn. Cheaper than private jets new.

nijave 4d ago
Seems like the most practical path is some kind of distributed peer to peer contraption where you can allocate some storage and optionally participate in crawling. I imagine if set some constraints you could get index size down quite a bit but I still suspect it'd be hard keeping up with content churn. Edit: Looks like enwiki bz2 is coming in around 46Gi which isn't too bad considering the amount of content it contains.

8organicbits 4d ago

For some of that, you don't need to crawl. Wikipedia offers database dumps which you can download in one go. Lots of programming docs are managed in repos, so you can clone the repo instead. Even stackoverflow seems to have a snapshot dump (https://archive.org/details/stackexchange https://archive.org/details/stackexchange).