6 ms·

Why couldn't several coordinating specialized search engines share their data via something like "charge the downloader" S3 buckets? Then you get an org like St

Why couldn't several coordinating specialized search engines share their data via something like "charge the downloader" S3 buckets? Then you get an org like StackExchange who could provide indexed data from their site and the algorithms to search the data the most efficiently, GitHub can do the same for their specific zone of speciality, Amazon, etc. Then anyone who wants to use the data can either copy it to their own S3 buckets to pay just once, or can use it with some sort of pay-as-you-go method. Anyone who runs a search engine can use the algorithms as a guide for the specific searches they are interested in for their site, or can just make their own. You could trust the other indexers not to give you bad data, because you'd have some sort of legal agreement and technical standards that would ensure that they couldn't/wouldn't "poison the well" somehow with the data they provide. Further, if a bad actor was providing faulty data, the other actors would notice and kick them out of the group or just stop using their data. It would have to be fully open source, I agree with the other parts of Drew's essay here, but I think we could share the index/data somehow if we got together and tried to think about it. We just need a standard for how we share the data.

cptskippy 6y ago
So you're proposing Snowflake for search?
- lambda_obrien 6y ago
  It appears to be the case in a technical workflow sense, from the little I just read of Snowflake, but my proposal would be a much more open system than one under control of a single vendor. It'd be more like a set of standards for interoperation and a common data center so that the data is accessible under one roof. Maybe each entity could do specialized search as a service and the search aggregators would pay by providing some infrastructure to run the crawlers. I don't personally think any system specification is impossible unless it goes against since mathematical law, so a really fast, distributed query system where there are a few hundred specialized providers for a single query is feasible. Imagine the aggregator does initial analysis to determine the category of search, like programming, news, or restaurant reviews, then sends the users query to a set of specialized providers that supply an index for that category, then fuse the results with some further analysis of the metadata returned. Then the user can also include or exclude the specialized providers at will. You could also eliminate the aggregator as a service as simply make it a user application on the desktop, allowing for even more user control and maybe caching or something.
ricardo81 6y ago
There's Common Crawl for the crawling aspect, about 3.2 billion pages last time I looked. One of the issues with that kind of detachment of jobs is crawl data freshness.
- lambda_obrien 6y ago
  I'm thinking more like search indexers with like 100k pages, in a specialized category like 3d printing or basketball. I can index things like those on my home PC, practically.
phire 6y ago
I think your idea has merit. Though it would require development of the "charge to download" S3 buckets and infrastructure to support payments. There is also an economic issue where you have to calculate the download cost to also cover storage costs.
- lambda_obrien 6y ago
  I think you can already set an S3 bucket to charge the downloader for bandwidth, that's what I was talking about, the storage part is harder. The storage costs could be borne via some sharing agreements between the commercial interests and the pay customers where the infrastructure could be provided for smaller indexers (storage, compute) and in exchange the provider can use the data freely for their own services. Or, we could have a micropayments system that aggregates towards the end of the month, something open source and free to use, maybe a blockchain, this is actually one place that tokens on a blockchain could work as a semi-decentralized payment system between the parties in agreement in my "vision" or whatever you want to call it.

cptskippy 6y ago

So you're proposing Snowflake for search?

lambda_obrien 6y ago
It appears to be the case in a technical workflow sense, from the little I just read of Snowflake, but my proposal would be a much more open system than one under control of a single vendor. It'd be more like a set of standards for interoperation and a common data center so that the data is accessible under one roof. Maybe each entity could do specialized search as a service and the search aggregators would pay by providing some infrastructure to run the crawlers. I don't personally think any system specification is impossible unless it goes against since mathematical law, so a really fast, distributed query system where there are a few hundred specialized providers for a single query is feasible. Imagine the aggregator does initial analysis to determine the category of search, like programming, news, or restaurant reviews, then sends the users query to a set of specialized providers that supply an index for that category, then fuse the results with some further analysis of the metadata returned. Then the user can also include or exclude the specialized providers at will. You could also eliminate the aggregator as a service as simply make it a user application on the desktop, allowing for even more user control and maybe caching or something.

ricardo81 6y ago

There's Common Crawl for the crawling aspect, about 3.2 billion pages last time I looked. One of the issues with that kind of detachment of jobs is crawl data freshness.

lambda_obrien 6y ago
I'm thinking more like search indexers with like 100k pages, in a specialized category like 3d printing or basketball. I can index things like those on my home PC, practically.

phire 6y ago

I think your idea has merit. Though it would require development of the "charge to download" S3 buckets and infrastructure to support payments. There is also an economic issue where you have to calculate the download cost to also cover storage costs.

lambda_obrien 6y ago
I think you can already set an S3 bucket to charge the downloader for bandwidth, that's what I was talking about, the storage part is harder. The storage costs could be borne via some sharing agreements between the commercial interests and the pay customers where the infrastructure could be provided for smaller indexers (storage, compute) and in exchange the provider can use the data freely for their own services. Or, we could have a micropayments system that aggregates towards the end of the month, something open source and free to use, maybe a blockchain, this is actually one place that tokens on a blockchain could work as a semi-decentralized payment system between the parties in agreement in my "vision" or whatever you want to call it.