6 ms·

What are some of the niche search engines build on Google's index affected by this?

by throwaway_20357 8mo ago

doublerabbit 8mo ago
Kagi
- pell 8mo ago
  I think Kagi buys search engine results from SERP vendors who typically scrape Google’s results and offer an API experience on top of it.
- deleted 8mo ago
  [deleted]
- marginalia_nu 8mo ago
  They published this the other day: https://blog.kagi.com/waiting-dawn-search https://blog.kagi.com/waiting-dawn-search Which saw some discussion on HN.
  - embedding-shape 8mo ago
    > some discussion ~450 score, ~247 comments and still on /best ("Most-upvoted stories of the last 48 hours"): https://news.ycombinator.com/item?id=46708678 https://news.ycombinator.com/item?id=46708678 - "Waiting for dawn in search: Search index, Google rulings and impact on Kagi"
- echelon 8mo ago
  No wonder Kagi is angry. Google is a monopoly across several broad categories. They're also a taxation enterprise. Google Search took over as the URL bar for 91% of all web users across all devices. Since this intercepts trademarks and brand names, Google gets to tax all businesses unfairly. Tell your legislators in the US and the EU that Google shouldn't be able to sell ads against registered trademarks (+/- some edit distance). They re-engineered the web to be a taxation system for all businesses across all categories. Searching for Claude -> Ads in first place Searching for ChatGPT -> Ads in first place Searching for iPhone -> Ads in first place This is inexcusable. Only searches for "ChatGPT versus", "iPhone reviews", or "Nintendo game comparison" should allow ads. And one could argue that the "URL Bar" shouldn't auto suggest these either when a trademark is in the URL bar. If Google won't play fair, we have to kill 50% of their search revenue for being egregiously evil. If you own a trademark, Google shouldn't be able to sell ads against you. -- Google's really bad. Ideally we'd get an antitrust breakup. They're worse than Ma Bell. I wouldn't even split Google into multiple companies by division - I'd force them to be multiple copies of the same exact entity that then have to compete with each other: Bell Systems -> {BellSouth, Bell Atlantic, Southwestern Bell, ...} Google -> {GoogleA, GoogleB, GoogleC, ...} They'd each have cloud, search, browser, and YouTube. But new brand names for new parent companies. That would create all-out war and lead to incredible consumer wins.
  - marginalia_nu 8mo ago
    Could probably argued that search access is an essential facility[1], though it doesn't appear antitrust law has anywhere near the same sort of enforcement it did in the past. [1] https://en.wikipedia.org/wiki/Essential_facilities_doctrine https://en.wikipedia.org/wiki/Essential_facilities_doctrine
  - onetokeoverthe 8mo ago
    [dead]
  - throwaway290 8mo ago
    what stops Kagi from indexing internet and makes them pay some guys to scrape search results from Google? one guy at Marginalia can do it and entire dev team at a PAID search engine can't?
    - DangitBobby 8mo ago
      As we've seen here on HN on the AI boom, it's not wonderful when a bunch of companies all use bots to scrape the entire web. Many sites only allow Google scrapers in robots.txt and the public will fight you hard if you scrape them without permission. It's just one of those things where it would be better for everyone if search engines could pay for access to the work that's done only once.
      - echelon 8mo ago
        > Many sites only allow Google scrapers in robots.txt and the public will fight you hard if you scrape them without permission. This just lets a monopoly replace the website instead of distributing power and fostering open source. The same monopoly that was already bleeding off the web's utility and taxing it.
    - mrweasel 8mo ago
      I don't know about others, but we have special rules for Google, Bing, and a few others, rate-limiting them less than some random bot. The problem is scrapers (mostly AI scrapers from what we can tell). They will pound a site into the ground and not care and they are becoming increasingly good at hiding their tracks. The only reasonable way to deal with them is to rate-limit every IP by default and then lifting some of those restrictions on known, well behaving bots. Now we will lift those restrictions if asked, and frequently look at statistics to lift the restrictions from search engines we might have missed, but it's an up hill battle if you're new and unknown.
- nemosaltat 8mo ago
  > Kagi This seems to be true, but more indirectly. From Kagi’s blog [0] which is a follow up to a Kagi blog post from last year [1]. [0]> Google: Google does not offer a public search API. The only available path is an ad-syndication bundle with no changes to result presentation - the model Startpage uses. Ad syndication is a non-starter for Kagi’s ad-free subscription model.[^1] [0]> The current interim approach (current as of Jan 21, 2026) [0]> Because direct licensing isn’t available to us on compatible terms, we - like many others - use third-party API providers for SERP-style results (SERP meaning search engine results page). These providers serve major enterprises (according to their websites) including Nvidia, Adobe, Samsung, Stanford, DeepMind, Uber, and the United Nations. I’m an avid Kagi user, and it seems like Kagi and some other notable interested parties have _already_ been unable to do get what they want/need with Google’s index. [0]> The fact that we - and companies like Stanford, Nvidia, Adobe, and the United Nations - have had to rely on third-party vendors is a symptom of the closed ecosystem, not a preference. Hopefully someone here can clarify for me, or enumerate some of these “third-party vendors” who seem like they will/might/could be directly affected by this. [0] antibabelic > relevant https://blog.kagi.com/waiting-dawn-search https://blog.kagi.com/waiting-dawn-search [1] https://blog.kagi.com/dawn-new-era-search https://blog.kagi.com/dawn-new-era-search > [^1]: A note on Google’s existing APIs: Google offers PSE, designed for adding search boxes to websites. It can return web results, but with reduced scope and terms tailored for that narrow use case. More recently, Google offers Grounding with Google Search through Vertex AI, intended for grounding LLM responses. Neither is general-purpose index access. Programmable Search Engine is not designed for building competitive search. Grounding with Google Search is priced at $35 per 1,000 requests - economically unviable for search at scale, and structured as an AI add-on rather than standalone index syndication. These are not the FRAND terms the market needs
  - tpetry 8mo ago
    I believe they try to indirectly say they are using SerpApi or a similar product that scrapes Google search results to use them. And other big ones use it too so it must be ok... That must be the reason why they limit the searches you can do in the starter plan. Every SerpApi call costs money.
- monooso 8mo ago
  Kagi does not use Google's search index. From their post which made the front page of HN yesterday [1]: > Google does not offer a public search API. The only available path is an ad-syndication bundle with no changes to result presentation - the model Startpage uses. Ad syndication is a non-starter for Kagi’s ad-free subscription model. [1]: https://news.ycombinator.com/item?id=46708678 https://news.ycombinator.com/item?id=46708678
  - jsnell 8mo ago
    They then go on to say that they pay a 3rd party company to scrape Google results (and serve those scraped results to their users). So their search engine is indeed based on unauthorized and uncompensated use of Google's index. But since they're not using/paying for a supported API but just taking what they want, they indeed are unlikely to be impacted by this API turndown.
    - ipaddr 8mo ago
      They get results from another provider who has authorized access. Google doesn't provide search results to unauthorized requests as many on tor have experienced.
      - direwolf20 8mo ago
        Residential proxies are also cheaper than you might realize.
      - jsnell 8mo ago
        No. They pay SerpApi to scrape Google. And SerpApi is currently being sued by Google for unauthorized scraping. Kagi did make comments for years implying that they had a deal with Google for search results, but their latest blog post makes it clear that is not true and was never true.
    - DangitBobby 8mo ago
      Congrats on saying that in the most one-sided way possible. Google makes it literally impossible for them to pay for access to search results to make the product they want (customizable subscription search with no ads), and Google also is the de-facto globally sanctioned crawler because they are the only search engine anyone gives a shit about, and also sites need to be indexed by them to survive. In short, Google owns the river and sells the boats, and the public built a wall around it. Google is in a monopoly position in search.
deleted 8mo ago
[deleted]

What are some of the niche search engines build on Google's index affected by this?

doublerabbit 8mo ago

Kagi

pell 8mo ago
I think Kagi buys search engine results from SERP vendors who typically scrape Google’s results and offer an API experience on top of it.
deleted 8mo ago
[deleted]
marginalia_nu 8mo ago
They published this the other day: https://blog.kagi.com/waiting-dawn-search https://blog.kagi.com/waiting-dawn-search Which saw some discussion on HN.
- embedding-shape 8mo ago
  > some discussion ~450 score, ~247 comments and still on /best ("Most-upvoted stories of the last 48 hours"): https://news.ycombinator.com/item?id=46708678 https://news.ycombinator.com/item?id=46708678 - "Waiting for dawn in search: Search index, Google rulings and impact on Kagi"
echelon 8mo ago
No wonder Kagi is angry. Google is a monopoly across several broad categories. They're also a taxation enterprise. Google Search took over as the URL bar for 91% of all web users across all devices. Since this intercepts trademarks and brand names, Google gets to tax all businesses unfairly. Tell your legislators in the US and the EU that Google shouldn't be able to sell ads against registered trademarks (+/- some edit distance). They re-engineered the web to be a taxation system for all businesses across all categories. Searching for Claude -> Ads in first place Searching for ChatGPT -> Ads in first place Searching for iPhone -> Ads in first place This is inexcusable. Only searches for "ChatGPT versus", "iPhone reviews", or "Nintendo game comparison" should allow ads. And one could argue that the "URL Bar" shouldn't auto suggest these either when a trademark is in the URL bar. If Google won't play fair, we have to kill 50% of their search revenue for being egregiously evil. If you own a trademark, Google shouldn't be able to sell ads against you. -- Google's really bad. Ideally we'd get an antitrust breakup. They're worse than Ma Bell. I wouldn't even split Google into multiple companies by division - I'd force them to be multiple copies of the same exact entity that then have to compete with each other: Bell Systems -> {BellSouth, Bell Atlantic, Southwestern Bell, ...} Google -> {GoogleA, GoogleB, GoogleC, ...} They'd each have cloud, search, browser, and YouTube. But new brand names for new parent companies. That would create all-out war and lead to incredible consumer wins.
- marginalia_nu 8mo ago
  Could probably argued that search access is an essential facility[1], though it doesn't appear antitrust law has anywhere near the same sort of enforcement it did in the past. [1] https://en.wikipedia.org/wiki/Essential_facilities_doctrine https://en.wikipedia.org/wiki/Essential_facilities_doctrine
- onetokeoverthe 8mo ago
  [dead]
- throwaway290 8mo ago
  what stops Kagi from indexing internet and makes them pay some guys to scrape search results from Google? one guy at Marginalia can do it and entire dev team at a PAID search engine can't?
  - DangitBobby 8mo ago
    As we've seen here on HN on the AI boom, it's not wonderful when a bunch of companies all use bots to scrape the entire web. Many sites only allow Google scrapers in robots.txt and the public will fight you hard if you scrape them without permission. It's just one of those things where it would be better for everyone if search engines could pay for access to the work that's done only once.
    - echelon 8mo ago
      > Many sites only allow Google scrapers in robots.txt and the public will fight you hard if you scrape them without permission. This just lets a monopoly replace the website instead of distributing power and fostering open source. The same monopoly that was already bleeding off the web's utility and taxing it.
  - mrweasel 8mo ago
    I don't know about others, but we have special rules for Google, Bing, and a few others, rate-limiting them less than some random bot. The problem is scrapers (mostly AI scrapers from what we can tell). They will pound a site into the ground and not care and they are becoming increasingly good at hiding their tracks. The only reasonable way to deal with them is to rate-limit every IP by default and then lifting some of those restrictions on known, well behaving bots. Now we will lift those restrictions if asked, and frequently look at statistics to lift the restrictions from search engines we might have missed, but it's an up hill battle if you're new and unknown.
nemosaltat 8mo ago
> Kagi This seems to be true, but more indirectly. From Kagi’s blog [0] which is a follow up to a Kagi blog post from last year [1]. [0]> Google: Google does not offer a public search API. The only available path is an ad-syndication bundle with no changes to result presentation - the model Startpage uses. Ad syndication is a non-starter for Kagi’s ad-free subscription model.[^1] [0]> The current interim approach (current as of Jan 21, 2026) [0]> Because direct licensing isn’t available to us on compatible terms, we - like many others - use third-party API providers for SERP-style results (SERP meaning search engine results page). These providers serve major enterprises (according to their websites) including Nvidia, Adobe, Samsung, Stanford, DeepMind, Uber, and the United Nations. I’m an avid Kagi user, and it seems like Kagi and some other notable interested parties have _already_ been unable to do get what they want/need with Google’s index. [0]> The fact that we - and companies like Stanford, Nvidia, Adobe, and the United Nations - have had to rely on third-party vendors is a symptom of the closed ecosystem, not a preference. Hopefully someone here can clarify for me, or enumerate some of these “third-party vendors” who seem like they will/might/could be directly affected by this. [0] antibabelic > relevant https://blog.kagi.com/waiting-dawn-search https://blog.kagi.com/waiting-dawn-search [1] https://blog.kagi.com/dawn-new-era-search https://blog.kagi.com/dawn-new-era-search > [^1]: A note on Google’s existing APIs: Google offers PSE, designed for adding search boxes to websites. It can return web results, but with reduced scope and terms tailored for that narrow use case. More recently, Google offers Grounding with Google Search through Vertex AI, intended for grounding LLM responses. Neither is general-purpose index access. Programmable Search Engine is not designed for building competitive search. Grounding with Google Search is priced at $35 per 1,000 requests - economically unviable for search at scale, and structured as an AI add-on rather than standalone index syndication. These are not the FRAND terms the market needs
- tpetry 8mo ago
  I believe they try to indirectly say they are using SerpApi or a similar product that scrapes Google search results to use them. And other big ones use it too so it must be ok... That must be the reason why they limit the searches you can do in the starter plan. Every SerpApi call costs money.
monooso 8mo ago
Kagi does not use Google's search index. From their post which made the front page of HN yesterday [1]: > Google does not offer a public search API. The only available path is an ad-syndication bundle with no changes to result presentation - the model Startpage uses. Ad syndication is a non-starter for Kagi’s ad-free subscription model. [1]: https://news.ycombinator.com/item?id=46708678 https://news.ycombinator.com/item?id=46708678
- jsnell 8mo ago
  They then go on to say that they pay a 3rd party company to scrape Google results (and serve those scraped results to their users). So their search engine is indeed based on unauthorized and uncompensated use of Google's index. But since they're not using/paying for a supported API but just taking what they want, they indeed are unlikely to be impacted by this API turndown.
  - ipaddr 8mo ago
    They get results from another provider who has authorized access. Google doesn't provide search results to unauthorized requests as many on tor have experienced.
    - direwolf20 8mo ago
      Residential proxies are also cheaper than you might realize.
    - jsnell 8mo ago
      No. They pay SerpApi to scrape Google. And SerpApi is currently being sued by Google for unauthorized scraping. Kagi did make comments for years implying that they had a deal with Google for search results, but their latest blog post makes it clear that is not true and was never true.
  - DangitBobby 8mo ago
    Congrats on saying that in the most one-sided way possible. Google makes it literally impossible for them to pay for access to search results to make the product they want (customizable subscription search with no ads), and Google also is the de-facto globally sanctioned crawler because they are the only search engine anyone gives a shit about, and also sites need to be indexed by them to survive. In short, Google owns the river and sells the boats, and the public built a wall around it. Google is in a monopoly position in search.

deleted 8mo ago

[deleted]