6 ms·

Matadisco – Decentralized Data Discovery

64 points by biggestfan 6mo ago

mentalgear 6mo ago
Decentralised Web Indexes are an important avenue if we truly want to ever break free from the private indexes used and dominated by Google and chatbots that in-transparently return results based on their opaque preferences.
- stingraycharles 6mo ago
  Those indexes cost an incredible amount of money to build, maintain and keep improving. It’s worth it for these orgs as these are considered a competitive edge and the “secret sauce” of these orgs. I find it hard to believe the same amount of investment will be made into a decentralized web index, precisely for the reason that it cannot be monetized.
  - dietr1ch 6mo ago
    The keep improving part hasn't done so well in 10 years already? Maybe this year the new force-fed AI answers got a bit useful, but many times the risk of hallucination means you still have to go and read a more credible source.
  - johnebgd 6mo ago
    This. Unless in 2026, we suddenly see Costco sell personal quantum computers for sub $2k, with practically unlimited high speed crystal storage, all cooled/powered by unobtainium fusion reactors that are made with 3D printers at your neighborhood coop, then we either must accept private entities that do this kind of indexing at scale or we should ask government to offer it on taxpayer expense.
    - somat 6mo ago
      On a pedantic note, that is unimportant to the discussion at hand, but I want to talk about it anyway. quantum computers have theoretical gains in some specific domains but would not do anything for this one. Anyhow I have doubts about even those specific domains, my gut says that quantum computing is an inherently analog process and we moved off analog computers for a reason. Bonus points if you know why that was.
  - somat 6mo ago
    No they don't. Sure, if you are trying to capture the entire web with hourly latency for billions of users, that takes significant resources. But if your goal is to catalog the subsection of the web you are interested in with weekly latency for yourself and share that catalog with like minded individuals. any body can do that. Don't make gods out of basic web indexing.
    - deleted 6mo ago
      [deleted]
  - gessha 6mo ago
    Are you familiar with any systems that use collaborative filtering to distribute the work and the benefits among users?
  - mentalgear 6mo ago
    Doesn't have to be expensive: You can have decentralised cohorts of same interests groups that provide common indexes, with each client verifying the index of every n item they use, reporting bad indexes/cohort users to avoid abuse. Also there are state-funded open search indexes, like the European Search Index. So it can be done realisticly, even though the massive private search index lobby push as hard as they can to deny it.
dopidopHN2 6mo ago
[dead]
thenthenthen 6mo ago
The demo shows a bunch of Sentinel image tiles for me.
tomnicholas1 6mo ago
Awesome to see this project here - it was partly inspired by my blog post (original is linked from the OP, but there's a slightly newer version on my personal site here[0]). [0]: https://tom-nicholas.com/blog/2025/science-needs-a-social-network/ https://tom-nicholas.com/blog/2025/science-needs-a-social-ne...

64 points by biggestfan 6mo ago

Decentralised Web Indexes are an important avenue if we truly want to ever break free from the private indexes used and dominated by Google and chatbots that in-transparently return results based on their opaque preferences.

stingraycharles 6mo ago
Those indexes cost an incredible amount of money to build, maintain and keep improving. It’s worth it for these orgs as these are considered a competitive edge and the “secret sauce” of these orgs. I find it hard to believe the same amount of investment will be made into a decentralized web index, precisely for the reason that it cannot be monetized.
- dietr1ch 6mo ago
  The keep improving part hasn't done so well in 10 years already? Maybe this year the new force-fed AI answers got a bit useful, but many times the risk of hallucination means you still have to go and read a more credible source.
- johnebgd 6mo ago
  This. Unless in 2026, we suddenly see Costco sell personal quantum computers for sub $2k, with practically unlimited high speed crystal storage, all cooled/powered by unobtainium fusion reactors that are made with 3D printers at your neighborhood coop, then we either must accept private entities that do this kind of indexing at scale or we should ask government to offer it on taxpayer expense.
  - somat 6mo ago
    On a pedantic note, that is unimportant to the discussion at hand, but I want to talk about it anyway. quantum computers have theoretical gains in some specific domains but would not do anything for this one. Anyhow I have doubts about even those specific domains, my gut says that quantum computing is an inherently analog process and we moved off analog computers for a reason. Bonus points if you know why that was.
- somat 6mo ago
  No they don't. Sure, if you are trying to capture the entire web with hourly latency for billions of users, that takes significant resources. But if your goal is to catalog the subsection of the web you are interested in with weekly latency for yourself and share that catalog with like minded individuals. any body can do that. Don't make gods out of basic web indexing.
  - deleted 6mo ago
    [deleted]
- gessha 6mo ago
  Are you familiar with any systems that use collaborative filtering to distribute the work and the benefits among users?
- mentalgear 6mo ago
  Doesn't have to be expensive: You can have decentralised cohorts of same interests groups that provide common indexes, with each client verifying the index of every n item they use, reporting bad indexes/cohort users to avoid abuse. Also there are state-funded open search indexes, like the European Search Index. So it can be done realisticly, even though the massive private search index lobby push as hard as they can to deny it.

dopidopHN2 6mo ago

[dead]

thenthenthen 6mo ago

The demo shows a bunch of Sentinel image tiles for me.

tomnicholas1 6mo ago

Awesome to see this project here - it was partly inspired by my blog post (original is linked from the OP, but there's a slightly newer version on my personal site here[0]). [0]: https://tom-nicholas.com/blog/2025/science-needs-a-social-network/ https://tom-nicholas.com/blog/2025/science-needs-a-social-ne...