6 ms·

Congrats on the launch! While search with LLMs is quite popular nowadays, it is still hard to get it right. As a smoke test, I tried the following queries, and

Congrats on the launch! While search with LLMs is quite popular nowadays, it is still hard to get it right. As a smoke test, I tried the following queries, and they returned the same result. Good job! international relations Turkey international relations about the country with the capital city of Ankara Both return info from this link: https://www.state.gov/secretary-blinkens-call-with-foreign-minister-fidan-on-the-situation-in-syria/ https://www.state.gov/secretary-blinkens-call-with-foreign-m...

sdesol 2y ago
Well if you ask GPT-4o the following: Give me keywords to search for based on this sentence "international relations about the country with the capital city of Ankara" You get the following: - Turkey international relations - Ankara diplomacy - Turkey foreign policy - Turkey global partnerships - Turkey international politics - Turkey geopolitical strategy - Turkey foreign affairs - Turkey global relations - Turkey NATO relations (if relevant to your topic) - Ankara as a diplomatic hub So it is not unsurprising that the same link was returned
- enisberk 2y ago
  This is a demo from a small startup dedicated to enhancing government transparency, which I greatly appreciate. As a result, my expectations are aligned with this goal, and I refer to this as a smoke test. Achieving accuracy with RAG and LLMs is a challenging task that requires balancing precision and recall. For instance, when you type "Ankara" into GPT-4o, it provides information about Turkey. However, searching "Ankara" in their product does not yield articles related to Turkey.
  - sdesol 2y ago
    > Achieving accuracy with RAG and LLMs is a challenging task that requires balancing precision and recall The challenge is domain knowledge and not tech in my opinion. There are dozens if not hundreds of companies providing RAG and LLM, but the challenge is, like you pointed out, what should you do if you encounter something like "Ankara". For BestBuy, this might not mean much, unless there is a BestBuy in Turkey. For a government related site, cities and geography is important, so trying to extract additional meaning from Ankara is probably important.
sand1929 2y ago
Thank you! It's been interesting to watch HN playing around with it. This community definitely phrases its search queries differently from how many government affairs professionals would (especially to try smoke tests like yours), so I'm glad it's holding up :)
fl0id 2y ago
which category did you select? if you select custom, it just says to contact them. If I want to search for another region, why should I select UK or US?

sdesol 2y ago

Well if you ask GPT-4o the following: Give me keywords to search for based on this sentence "international relations about the country with the capital city of Ankara" You get the following: - Turkey international relations - Ankara diplomacy - Turkey foreign policy - Turkey global partnerships - Turkey international politics - Turkey geopolitical strategy - Turkey foreign affairs - Turkey global relations - Turkey NATO relations (if relevant to your topic) - Ankara as a diplomatic hub So it is not unsurprising that the same link was returned

enisberk 2y ago
This is a demo from a small startup dedicated to enhancing government transparency, which I greatly appreciate. As a result, my expectations are aligned with this goal, and I refer to this as a smoke test. Achieving accuracy with RAG and LLMs is a challenging task that requires balancing precision and recall. For instance, when you type "Ankara" into GPT-4o, it provides information about Turkey. However, searching "Ankara" in their product does not yield articles related to Turkey.
- sdesol 2y ago
  > Achieving accuracy with RAG and LLMs is a challenging task that requires balancing precision and recall The challenge is domain knowledge and not tech in my opinion. There are dozens if not hundreds of companies providing RAG and LLM, but the challenge is, like you pointed out, what should you do if you encounter something like "Ankara". For BestBuy, this might not mean much, unless there is a BestBuy in Turkey. For a government related site, cities and geography is important, so trying to extract additional meaning from Ankara is probably important.

sand1929 2y ago

Thank you! It's been interesting to watch HN playing around with it. This community definitely phrases its search queries differently from how many government affairs professionals would (especially to try smoke tests like yours), so I'm glad it's holding up :)

fl0id 2y ago

which category did you select? if you select custom, it just says to contact them. If I want to search for another region, why should I select UK or US?