7 ms·

Web search on the Anthropic API

272 points by cmogni1 1y ago

benjamoon 1y ago
Good that it has an “allowed domain” list, makes it really useable. The OpenAI Responses api web search doesn’t let you limit domains currently so can’t make good use of it for client stuff.
minimaxir 1y ago
The web search functionality is also available in the backend Workbench (click the wrench Tools icon) https://console.anthropic.com/workbench/ https://console.anthropic.com/workbench/ The API request notably includes the exact text it cites from its sources (https://docs.anthropic.com/en/docs/build-with-claude/tool-use/web-search-tool https://docs.anthropic.com/en/docs/build-with-claude/tool-us...), which is nifty. Cost-wise it's interesting. $10/1000 queries is much cheaper for heavy use than Google's Gemini (1500 free per day then $35/1000) when you'd expect Google to be the cheaper option. https://ai.google.dev/gemini-api/docs/grounding https://ai.google.dev/gemini-api/docs/grounding
- handfuloflight 1y ago
  So the price is just the $0.01 per query? Are they not charging for the tokens loaded into context from the various sources?
  - minimaxir 1y ago
    The query cost is in addition to tokens used. It is unclear if the tokens ingested from the search query count as addititional input tokens. > Web search is available on the Anthropic API for $10 per 1,000 searches, plus standard token costs for search-generated content. > Each web search counts as one use, regardless of the number of results returned. If an error occurs during web search, the web search will not be billed.
    - stephpang 1y ago
      Hi, stephanie from Anthropic here. Thanks for the feedback! We've updated the docs to hopefully make it a little more clear but yes search results do count towards input tokens https://docs.anthropic.com/en/docs/build-with-claude/tool-use/web-search-tool#usage-and-pricing https://docs.anthropic.com/en/docs/build-with-claude/tool-us...
      - minimaxir 1y ago
        Thanks for the update! > Web search results in the conversation are counted as input tokens on subsequent completion requests during the current turn or on subsequent conversation turns. Yes, that's clear.
- istjohn 1y ago
  Well also Google has put onerous conditions on their service: - If you show users text generated by Gemini using Google Search (grounded Gemini), you must display a provided widget with suggested search terms that links directly to Google Search results on google.com. - You may not modify the text generated by grounded Gemini before displaying it to your users. - You may not store grounded responses more than 30 days, except for user histories, which can retain responses for up to 6 months. https://ai.google.dev/gemini-api/terms#grounding-with-google-search https://ai.google.dev/gemini-api/terms#grounding-with-google... https://ai.google.dev/gemini-api/docs/grounding/search-suggestions https://ai.google.dev/gemini-api/docs/grounding/search-sugge...
  - miohtama 1y ago
    Google obviously does not want to cannibalise their golden goose. However it's inevitable that Google search will start to suffer because people need it less and less with LLMs.
aaronscott 1y ago
It would be nice if the search provider could be configured. I would like to use this with Kagi.
- lemming 1y ago
  I would really love this too. However I think that the only solution for that is to give it a Kagi search tool, in combination with a web scraping tool, and a loop while it figures out whether it's got the information it needs to answer the question.
cmogni1 1y ago
I think the most interesting thing to me is they have multi-hop search & query refinement built in based on prior context/searches. I'm curious how well this works. I've built a lot of LLM applications with web browsing in it. Allow/block lists are easy to implement with most web search APIs, but multi-hop gets really hairy (and expensive) to do well because it usually requires context from the URLs themselves. The thing I'm still not seeing here that makes LLM web browsing particularly difficult is the mismatch between search result relevance vs LLM relevance. Getting a diverse list of links is great when searching Google because there is less context per query, but what I really need from an out-of-the-box LLM web browsing API is reranking based on the richer context provided by a message thread/prompt. For example, writing an article about the side effects of Accutane should err on the side of pulling in research articles first for higher quality information and not blog posts. It's possible to do this reranking decently well with LLMs (I do it in my "agents" that I've written), but I haven't seen this highlighted from anyone thus far, including in this announcement.
- simple10 1y ago
  That's been my experience as well. Web search built into the API is great for convenience, but it would be ideal to be able to provide detailed search and reranking params. Would be interesting to see comparisons for custom web search RAG vs API. I'm assuming that many of the search "params" of the API could be controlled via prompting?
- peterldowns 1y ago
  > For example, writing an article about the side effects of Accutane should err on the side of pulling in research articles first for higher quality information and not blog posts. Interesting, I'm taking isotretinoin right now and I've found it's more interesting and useful to me to read "real" experiences (from reddit and blogs) than research papers.
  - TechDebtDevin 1y ago
    Wear lots of (mineral) sunscreen, and drink lots and lots of water. La Roche Posey lotions are what I used, and continue to use with tretinoin. Sunscreen is the most important.
simonw 1y ago
I couldn't see anything in the documentation about whether or not it's allowed to permanently store the results coming back from search. Presumably this is using Brave under the hood, same as Claude's search feature via the Anthropic apps?
- minimaxir 1y ago
  Given the context/use of encrypted_index and encrypted_context, I suspect search results are temporarily cached.
  - simonw 1y ago
    Right, but are there any restrictions on what I can do with them? Google Gemini has some: https://ai.google.dev/gemini-api/docs/grounding/search-suggestions#requirements https://ai.google.dev/gemini-api/docs/grounding/search-sugge... OpenAI has some rules too: https://platform.openai.com/docs/guides/tools-web-search#output-and-citations https://platform.openai.com/docs/guides/tools-web-search#out... > "When displaying web results or information contained in web results to end users, inline citations must be made clearly visible and clickable in your user interface." I'm used to search APIs coming with BIG sets of rules on how you can use the results. I'd be surprised but happy if Anthropic didn't have any. The Brave Search API is a great example of this: https://brave.com/search/api/ https://brave.com/search/api/ They have a special, much more expensive tier called "Data w/ storage rights" which is $45 CPM, compared to $5 CPM for the tier that doesn't include those storage rights.
    - istjohn 1y ago
      Google's restrictions are outlandish: "[You] will not modify, or intersperse any other content with, the Grounded Results or Search Suggestions..."
      - minimaxir 1y ago
        The API response actually contains the full HTML to include.
lemming 1y ago
I'm also interested to know if there are other limitations with this. Gemini, for example, has a built-in web search tool, but it can't be used in combination with other tools, which is a little annoying. o3/o4-mini can't use the search tool at all over the API, which is even more annoying.
potlee 1y ago
If you use your own search tool, you would have to pay for input tokens again every time the model decides to search. This would be a big discount if they only charging once for all output as output tokens but seems unclear from the blog post
- stephpang 1y ago
  Thanks for the feedback, just updated our docs to hopefully make this a little clearer. Search results count towards input tokens on every subsequent iteration https://docs.anthropic.com/en/docs/build-with-claude/tool-use/web-search-tool#usage-and-pricing https://docs.anthropic.com/en/docs/build-with-claude/tool-us...
  - potlee 1y ago
    Thanks for addressing it. Still sounds like a significant discount if only the search results and not all messages count are input tokens on subsequent iterations!
omneity 1y ago
Related: For those who want to build their own AI search for free and connect it to any model they want, I created a browser MCP that interfaces with major public search engines [0], a SERP MCP if you want, with support for multiple pages of results. The rate limits of the upstream engines are fine for personal use, and the benefit is it uses the same browser you do, so results are customized to your search habits out-of-the-box (or you could use a blank browser profile). 0: https://herd.garden/trails/@omneity/serp https://herd.garden/trails/@omneity/serp
jarbus 1y ago
Is search really that costly to run? $10/1000 searches seems really pricey. I'm wondering if these costs will come down in a few years.
- tuyguntn 1y ago
  they will come down, because up until recently consumers were not paying directly for searches, with the LLM which has a cutoff date in the past and hallucinations, search got popular paid API. Popularity will grow even more, hence competition will increase and prices will change eventually
  - AznHisoka 1y ago
    I dont think that will be true. What competition? Google, Bing, and.. Kagi? (And only one of those have a far superior index/algo than the others)
- jsnell 1y ago
  Yes. The Bing Search API is priced at $15/1k queries in the cheapest tier, Brave API is $9 at the non-toy tier, Google's pricing for a general search API is unknown but their Search grounding in Gemini costs $35/1k queries. Search API prices have been going up, not down, over time. The opposite of LLMs, which have gotten 1000x cheaper over the last two years.
  - jwr 1y ago
    > Google's pricing for a general search API As I discovered recently, and much to my surprise, Google does not offer a "general search API", at least not officially. There is a "custom search" API that sounds like web search, but isn't: it offers a subset of the index, which is not immediately apparent. Confusing and misleading labeling there. Bing offers something a bit better, but I recently ended up trying the Kagi API, and it is the best thing I found so far. Expensive ($25/1000), but works well.
    - camkego 1y ago
zhyder 1y ago
Now all the big 3 LLM providers provide web search grounding in their APIs, but how do they compare in ranking quality of the retrieved web search results? Anyone run benchmarks here? Clearly web search ranking is hard after decades of content spam that's been SEO optimized (and we get to look forward to increasing AI spam dominating the web in the future). The best LLM provider in the future could be the one with just the best web search ranking, just like what allowed Google to initially win in search.
- RainbowcityKun 1y ago
  Right now, most LLMs with web search grounding are still in Stage 1: they can retrieve content, but their ability to assess quality, trustworthiness, and semantic ranking is still very limited. The LLMs can access the web, but they can't yet understand it in a structured, evaluative way. What’s missing is a layer of engineered relevance modeling, capable of filtering not just based on keywords or citations, but on deeper truth alignment and human utility. And yes, as you mentioned, we may even see the rise of LLM-targeted SEO—content optimized not for human readers, but to game LLM attention and summarization heuristics. That's a whole new arms race. The next leap won’t be about just accessing more data, but about curating and interpreting it meaningfully.
  - simianwords 1y ago
    >Right now, most LLMs with web search grounding are still in Stage 1: they can retrieve content, but their ability to assess quality, trustworthiness, and semantic ranking is still very limited. Why do you think it is limited? Imagine you show a link with details to an LLM and ask it if it is trustworthy or high quality w.r.t the query, why can't it answer it?
    - RainbowcityKun 1y ago
      What I mean is that more powerful engineering capabilities are needed to provide LLM with processing of search results.
metalrain 1y ago
It's a good reminder that AI chats won't make web searches obsolete, just embed them at deeper in the stack. Maybe Google search revenue moves from ads to more towards B2B deals for search API use.
simianwords 1y ago
Can any one answer this question: are they using custom home made web index? Or are they using bing/google api? Also I'm quite sure that they don't use vector embeddings for web search, its purely on text space. I think the same holds for all LLM web search tools. They all seem to work well -- maybe we don't need embeddings for RAG and grepping works well enough?
elisson22 1y ago
Regarding the costs, do we have a clear indication as to how much it costs to the company to perform tasks from a power consumption perspective? Or is it negligible?
gyani 1y ago
Bing is shutting down API access as well and going to "Grounding with Bing Search" at $35/1k https://azure.microsoft.com/en-us/updates?id=492574 https://azure.microsoft.com/en-us/updates?id=492574
throwaway314155 1y ago
Setting the allowed url to "youtube.com" results in tool usage failing (up to max_calls times). Does this mean there are certain sites that the search tool simply can't access?
- alberduris 1y ago
  Yes, it seems so. I’ve tried every possible way to get it to specifically search for YouTube videos and there’s just no way. Even if you search for videos on a topic with hundreds of highly relevant YouTube videos, and explicitly ask it to find YouTube videos, it still says it found other relevant stuff, but not YouTube videos.
yuta2912 1y ago
I Noticed a huge amount of latency using the web search on the api , is it me or with everyone ? Generally when u see other providers like open ai's web search it takes usually 7-8 secs and same for pplx sonar api. Any comments?

272 points by cmogni1 1y ago

Good that it has an “allowed domain” list, makes it really useable. The OpenAI Responses api web search doesn’t let you limit domains currently so can’t make good use of it for client stuff.

minimaxir 1y ago

The web search functionality is also available in the backend Workbench (click the wrench Tools icon) https://console.anthropic.com/workbench/ https://console.anthropic.com/workbench/ The API request notably includes the exact text it cites from its sources (https://docs.anthropic.com/en/docs/build-with-claude/tool-use/web-search-tool https://docs.anthropic.com/en/docs/build-with-claude/tool-us...), which is nifty. Cost-wise it's interesting. $10/1000 queries is much cheaper for heavy use than Google's Gemini (1500 free per day then $35/1000) when you'd expect Google to be the cheaper option. https://ai.google.dev/gemini-api/docs/grounding https://ai.google.dev/gemini-api/docs/grounding

handfuloflight 1y ago
So the price is just the $0.01 per query? Are they not charging for the tokens loaded into context from the various sources?
- minimaxir 1y ago
  The query cost is in addition to tokens used. It is unclear if the tokens ingested from the search query count as addititional input tokens. > Web search is available on the Anthropic API for $10 per 1,000 searches, plus standard token costs for search-generated content. > Each web search counts as one use, regardless of the number of results returned. If an error occurs during web search, the web search will not be billed.
  - stephpang 1y ago
    Hi, stephanie from Anthropic here. Thanks for the feedback! We've updated the docs to hopefully make it a little more clear but yes search results do count towards input tokens https://docs.anthropic.com/en/docs/build-with-claude/tool-use/web-search-tool#usage-and-pricing https://docs.anthropic.com/en/docs/build-with-claude/tool-us...
    - minimaxir 1y ago
      Thanks for the update! > Web search results in the conversation are counted as input tokens on subsequent completion requests during the current turn or on subsequent conversation turns. Yes, that's clear.
istjohn 1y ago
Well also Google has put onerous conditions on their service: - If you show users text generated by Gemini using Google Search (grounded Gemini), you must display a provided widget with suggested search terms that links directly to Google Search results on google.com. - You may not modify the text generated by grounded Gemini before displaying it to your users. - You may not store grounded responses more than 30 days, except for user histories, which can retain responses for up to 6 months. https://ai.google.dev/gemini-api/terms#grounding-with-google-search https://ai.google.dev/gemini-api/terms#grounding-with-google... https://ai.google.dev/gemini-api/docs/grounding/search-suggestions https://ai.google.dev/gemini-api/docs/grounding/search-sugge...
- miohtama 1y ago
  Google obviously does not want to cannibalise their golden goose. However it's inevitable that Google search will start to suffer because people need it less and less with LLMs.

aaronscott 1y ago

It would be nice if the search provider could be configured. I would like to use this with Kagi.

lemming 1y ago
I would really love this too. However I think that the only solution for that is to give it a Kagi search tool, in combination with a web scraping tool, and a loop while it figures out whether it's got the information it needs to answer the question.

cmogni1 1y ago

I think the most interesting thing to me is they have multi-hop search & query refinement built in based on prior context/searches. I'm curious how well this works. I've built a lot of LLM applications with web browsing in it. Allow/block lists are easy to implement with most web search APIs, but multi-hop gets really hairy (and expensive) to do well because it usually requires context from the URLs themselves. The thing I'm still not seeing here that makes LLM web browsing particularly difficult is the mismatch between search result relevance vs LLM relevance. Getting a diverse list of links is great when searching Google because there is less context per query, but what I really need from an out-of-the-box LLM web browsing API is reranking based on the richer context provided by a message thread/prompt. For example, writing an article about the side effects of Accutane should err on the side of pulling in research articles first for higher quality information and not blog posts. It's possible to do this reranking decently well with LLMs (I do it in my "agents" that I've written), but I haven't seen this highlighted from anyone thus far, including in this announcement.

simple10 1y ago
That's been my experience as well. Web search built into the API is great for convenience, but it would be ideal to be able to provide detailed search and reranking params. Would be interesting to see comparisons for custom web search RAG vs API. I'm assuming that many of the search "params" of the API could be controlled via prompting?
peterldowns 1y ago
> For example, writing an article about the side effects of Accutane should err on the side of pulling in research articles first for higher quality information and not blog posts. Interesting, I'm taking isotretinoin right now and I've found it's more interesting and useful to me to read "real" experiences (from reddit and blogs) than research papers.
- TechDebtDevin 1y ago
  Wear lots of (mineral) sunscreen, and drink lots and lots of water. La Roche Posey lotions are what I used, and continue to use with tretinoin. Sunscreen is the most important.

simonw 1y ago

I couldn't see anything in the documentation about whether or not it's allowed to permanently store the results coming back from search. Presumably this is using Brave under the hood, same as Claude's search feature via the Anthropic apps?

minimaxir 1y ago
Given the context/use of encrypted_index and encrypted_context, I suspect search results are temporarily cached.
- simonw 1y ago
  Right, but are there any restrictions on what I can do with them? Google Gemini has some: https://ai.google.dev/gemini-api/docs/grounding/search-suggestions#requirements https://ai.google.dev/gemini-api/docs/grounding/search-sugge... OpenAI has some rules too: https://platform.openai.com/docs/guides/tools-web-search#output-and-citations https://platform.openai.com/docs/guides/tools-web-search#out... > "When displaying web results or information contained in web results to end users, inline citations must be made clearly visible and clickable in your user interface." I'm used to search APIs coming with BIG sets of rules on how you can use the results. I'd be surprised but happy if Anthropic didn't have any. The Brave Search API is a great example of this: https://brave.com/search/api/ https://brave.com/search/api/ They have a special, much more expensive tier called "Data w/ storage rights" which is $45 CPM, compared to $5 CPM for the tier that doesn't include those storage rights.
  - istjohn 1y ago
    Google's restrictions are outlandish: "[You] will not modify, or intersperse any other content with, the Grounded Results or Search Suggestions..."
    - minimaxir 1y ago
      The API response actually contains the full HTML to include.

lemming 1y ago

I'm also interested to know if there are other limitations with this. Gemini, for example, has a built-in web search tool, but it can't be used in combination with other tools, which is a little annoying. o3/o4-mini can't use the search tool at all over the API, which is even more annoying.

potlee 1y ago

If you use your own search tool, you would have to pay for input tokens again every time the model decides to search. This would be a big discount if they only charging once for all output as output tokens but seems unclear from the blog post

stephpang 1y ago
Thanks for the feedback, just updated our docs to hopefully make this a little clearer. Search results count towards input tokens on every subsequent iteration https://docs.anthropic.com/en/docs/build-with-claude/tool-use/web-search-tool#usage-and-pricing https://docs.anthropic.com/en/docs/build-with-claude/tool-us...
- potlee 1y ago
  Thanks for addressing it. Still sounds like a significant discount if only the search results and not all messages count are input tokens on subsequent iterations!

omneity 1y ago

Related: For those who want to build their own AI search for free and connect it to any model they want, I created a browser MCP that interfaces with major public search engines [0], a SERP MCP if you want, with support for multiple pages of results. The rate limits of the upstream engines are fine for personal use, and the benefit is it uses the same browser you do, so results are customized to your search habits out-of-the-box (or you could use a blank browser profile). 0: https://herd.garden/trails/@omneity/serp https://herd.garden/trails/@omneity/serp

jarbus 1y ago

Is search really that costly to run? $10/1000 searches seems really pricey. I'm wondering if these costs will come down in a few years.

tuyguntn 1y ago
they will come down, because up until recently consumers were not paying directly for searches, with the LLM which has a cutoff date in the past and hallucinations, search got popular paid API. Popularity will grow even more, hence competition will increase and prices will change eventually
- AznHisoka 1y ago
  I dont think that will be true. What competition? Google, Bing, and.. Kagi? (And only one of those have a far superior index/algo than the others)
jsnell 1y ago
Yes. The Bing Search API is priced at $15/1k queries in the cheapest tier, Brave API is $9 at the non-toy tier, Google's pricing for a general search API is unknown but their Search grounding in Gemini costs $35/1k queries. Search API prices have been going up, not down, over time. The opposite of LLMs, which have gotten 1000x cheaper over the last two years.
- jwr 1y ago
  > Google's pricing for a general search API As I discovered recently, and much to my surprise, Google does not offer a "general search API", at least not officially. There is a "custom search" API that sounds like web search, but isn't: it offers a subset of the index, which is not immediately apparent. Confusing and misleading labeling there. Bing offers something a bit better, but I recently ended up trying the Kagi API, and it is the best thing I found so far. Expensive ($25/1000), but works well.
  - camkego 1y ago

zhyder 1y ago

Now all the big 3 LLM providers provide web search grounding in their APIs, but how do they compare in ranking quality of the retrieved web search results? Anyone run benchmarks here? Clearly web search ranking is hard after decades of content spam that's been SEO optimized (and we get to look forward to increasing AI spam dominating the web in the future). The best LLM provider in the future could be the one with just the best web search ranking, just like what allowed Google to initially win in search.

RainbowcityKun 1y ago
Right now, most LLMs with web search grounding are still in Stage 1: they can retrieve content, but their ability to assess quality, trustworthiness, and semantic ranking is still very limited. The LLMs can access the web, but they can't yet understand it in a structured, evaluative way. What’s missing is a layer of engineered relevance modeling, capable of filtering not just based on keywords or citations, but on deeper truth alignment and human utility. And yes, as you mentioned, we may even see the rise of LLM-targeted SEO—content optimized not for human readers, but to game LLM attention and summarization heuristics. That's a whole new arms race. The next leap won’t be about just accessing more data, but about curating and interpreting it meaningfully.
- simianwords 1y ago
  >Right now, most LLMs with web search grounding are still in Stage 1: they can retrieve content, but their ability to assess quality, trustworthiness, and semantic ranking is still very limited. Why do you think it is limited? Imagine you show a link with details to an LLM and ask it if it is trustworthy or high quality w.r.t the query, why can't it answer it?
  - RainbowcityKun 1y ago
    What I mean is that more powerful engineering capabilities are needed to provide LLM with processing of search results.

metalrain 1y ago

It's a good reminder that AI chats won't make web searches obsolete, just embed them at deeper in the stack. Maybe Google search revenue moves from ads to more towards B2B deals for search API use.

simianwords 1y ago

Can any one answer this question: are they using custom home made web index? Or are they using bing/google api? Also I'm quite sure that they don't use vector embeddings for web search, its purely on text space. I think the same holds for all LLM web search tools. They all seem to work well -- maybe we don't need embeddings for RAG and grepping works well enough?

elisson22 1y ago

Regarding the costs, do we have a clear indication as to how much it costs to the company to perform tasks from a power consumption perspective? Or is it negligible?

gyani 1y ago

Bing is shutting down API access as well and going to "Grounding with Bing Search" at $35/1k https://azure.microsoft.com/en-us/updates?id=492574 https://azure.microsoft.com/en-us/updates?id=492574

throwaway314155 1y ago

Setting the allowed url to "youtube.com" results in tool usage failing (up to max_calls times). Does this mean there are certain sites that the search tool simply can't access?

alberduris 1y ago
Yes, it seems so. I’ve tried every possible way to get it to specifically search for YouTube videos and there’s just no way. Even if you search for videos on a topic with hundreds of highly relevant YouTube videos, and explicitly ask it to find YouTube videos, it still says it found other relevant stuff, but not YouTube videos.

yuta2912 1y ago

I Noticed a huge amount of latency using the web search on the api , is it me or with everyone ? Generally when u see other providers like open ai's web search it takes usually 7-8 secs and same for pplx sonar api. Any comments?