7 ms·

Show HN: Cleverb.ee – open-source agent that writes a cited research report

67 points by nickwatson 1y ago

smallnix 1y ago
How do you do the citing? Reverse-RAG post processing?
- nickwatson 1y ago
  Good question — it's pretty straightforward right now: I pass the collected content chunks (with their original URLs attached) into Gemini 2.5 Pro, asking it to synthesize a balanced report and to inline citations throughout. So it's not doing anything fancy like dynamic retrieval or classic RAG architecture. Basically: - The agent gathers sources (webpages, PDFs, Reddit, etc.) - Summarises each as it goes (using a cheaper model) - Then hands a bundle of summarised + raw content to Gemini 2.5 Pro - Gemini 2.5 Pro writes the final report, embedding links directly as citations with [1], [2], etc style citations throughout. Reverse-RAG is something I for sure want to implement. Once I can afford a better computer to run this with at scale. Even an 8B model will take overnight to summarize an average piece of content for me right now! But I'm also keeping an eye on the pace of which AI moves in the larger LLM space. The size and abilities of likes of Gemini 2.5 Pro context windows are pretty crazy these days! Thanks for the question.
  - iamandoni 1y ago
    Do you take any measures to prevent link hallucination? And content grounding / attribution verification?
    - nickwatson 1y ago
      At the moment the measures taken are: - Full content analysis by Primary LLM (Default is Gemini 2.5 Pro) with link hard-coded alongside each piece of content with structured output for better parsing. - Temperature right down (0.2), strict instructions to synthesize, precise prompts to attribute links exactly and without modification. What I hope to introduce: - Hard-coded parsing of links mentioned in final report to verify with the link map created throughout the research journey - Optional, "double-checking" LLM review of synthesized content to ensure no drift. - RAG enhancements for token-efficient verification and subsequent user questions (post-research) Do you have any further suggestions? Right now I hope to strike the delicate balance between token efficiency, with enhanced grounding as optional settings in the future. I have a big task list of things, and this is one of them. I will ensure to re-prioritize alongside user requests for the different features. Of course, being open source, contributions are highly welcome. I would love to see large community involvement. Collaboration benefits everyone. P.s. I have spent hundreds of dollars in tests. I'd say for every 1 hour of building, about 3 hours of testing have gone into this, debugging, optimizing quality, ensuring guard-rails are in place. If you go to the repo, also check out the config/prompts.py file - it will give you a little more insight into what is going on (there are code checks as well, but generally it gives you an idea).
nickwatson 1y ago
Hi HN I built *cleverb.ee* to solve my own research pain: too many tabs, too much messy sourcing. Gemini and OpenAI deep research tools weren't getting the balanced/unbiased quality I desired. *What it does*: • Reads webpages, PDFs, Reddit posts, PubMed abstracts, YouTube transcripts. • Plans with Gemini 2.5 Pro, acts with Gemini 2.5 Flash, summarises with Gemini 2.0 Flash (or you can use any Local LLM or Claude) • Outputs a fact-checked, cited Markdown report with live token usage tracking. *Tech*: Python + Playwright + LangChain with MCP tool support. AGPL-3.0 licensed. *Why open source?*: I wanted full transparency at every agent step and easy pluggable data sources. Quick install: ```bash git clone https://github.com/SureScaleAI/cleverbee https://github.com/SureScaleAI/cleverbee cd cleverbee && bash setup.sh Would love feedback — especially what critical research sources you’d want integrated next!
- Quanttek 1y ago
  Can you specify ? The default heavy reliance on Reddit and YouTube, rather than trusted publications (e.g. Scientific American, NYTimes) and scientific publications, is worrying given widespread misinformation in certain scientific fields (e.g. nutrition, health, economics)
  - nickwatson 1y ago
    I never said "heavy reliance" on Reddit/YouTube. It actually is requested to use discernment to recognize poor, or biased sources and opinions, and label them as such (see the example report on Coffee which I shared previously in another comment). Most the time it has only sought out one or two post/youtube videos as it can recognize the low credibility value. It comes loaded with a PubMed MCP tool and the beauty of it being open source is you can exclude or limit the sources as much as you want, or add in new sources - that's why I wanted to open it up, to allow for critique over methodologies and allow for improved, balanced research from experts. It is also requested to evaluate the source and whether or not they have "some benefit to gain" from the article, to ensure it balances this into the research, also.
- kleiba 1y ago
  > Gemini and OpenAI deep research tools weren't getting the balanced/unbiased quality I desired. Could you elaborate, please?
cantaloupe 1y ago
I browsed the GitHub and website for a bit, but didn’t see any examples! It would be useful to share the output for a common question that can be substantiated with reliable sources, like “Is coffee good for me?”. Even better if you can show a comparison to other deep research tools. From the copy it seems like Cleverbee could pull from more diverse sources (e.g. YouTube) and fewer unreliable sources (e.g. product blogs). Show that off!
- nickwatson 1y ago
  I think that's a very good idea, thanks! I'm actually working on "The Beehive" at the moment, where the app can push the research to a hive on the website, so people can share their research/discoveries. My client's paid work takes priority, but I hope to do it over the course of this week. P.s. Running report now for you, "Is coffee good for me?" to show you this example ;)
- nickwatson 1y ago
  Done! https://docs.google.com/document/d/1bGVkI3xaBP1AvRxkB4GKeL0NrQhQ5ypBKKM5p5_TC8E/edit?usp=sharing https://docs.google.com/document/d/1bGVkI3xaBP1AvRxkB4GKeL0N...
  - endianswap 1y ago
    lol what > Reference to failed web browser attempt for Rush University
    - nickwatson 1y ago
      Good spot! I already am on the case with that one. The Rush University website takes a long time to load (check it out), and the script recognized the article had loaded but something on the website was causing it to hang (waiting for networkidle status) so it terminated the parsing early and worked with the content it had. So in a nutshell, it still parsed but with a warning/error that this happened. I'm optimizing this now to exclude the wording from the report, or make a note.
    - 1y ago
semi-extrinsic 1y ago
I think it's really unfortunate that this type of thing gets called "research". I get that it fits with what has unfortunately become modern day usage - "Karen did her own research before becoming a flat-earther" - but I really wish the AI companies would've had better faith in their future solutions than to call this research. There's gotta be quite a few actual researchers at these companies who are shaking their heads. To spare others the lookup, here's from the Oxford dictionary. Emphasis on the word "new": To study a subject in detail, especially in order to discover new information or reach a new understanding. Example: They are carrying out/conducting/doing some fascinating research into/on the language of dolphins.
- gnuly 1y ago
  this case should simply be called search, no? to me research takes a long time, and not just an hour or so.
  - kleiba 1y ago
    You overlook the fact that the system integrates various sources into a coherent report. This in my opinion makes it more than just mere search.
  - pcthrowaway 1y ago
    Well the models often do an initial search and then a follow-up search. So it's a re-search
    - nickwatson 1y ago
      It does it several times, so maybe re-re-re-search works?
- yard2010 1y ago
  New is a matter of perspective.
- nickwatson 1y ago
  I understand what you're saying. I believe any kind of research will nearly always begin with learning and understanding of the knowledge that is already out there. Almost every subject has been learned this way, whether at school from a teacher or text-book, or reading papers. The Oxford dictionary definition says the same, "to study a subject in detail". This is what AI is doing - I see it as a "power suit" for distilling information much faster, without the cognitive bias that many of us will carry. Learning is an important part of research, and this must come with discernment over credibility of existing research, including identifying where the gaps are. This kind of critical thinking allows for another level, experiments, surveys, etc to uncover things even further. If you were to study the language of dolphins today, where would you start? Would you jump into the ocean and start trying to talk with them, or would you look up what is already discovered? Would you study their behaviors, patterns, etc? What drove me to do this project is exactly the example you mentioned, the flat-earther type who look up an article on some kind of free hosting website or Sandra from accounts social media page and taken as the be-all-and-end-all of knowledge. It comes without bias recognition or critical thinking skills. This is where I'm hopeful to level the playing field, and ensure unbiased, balanced information is uncovered.
nickwatson 1y ago
Thanks for all the thoughtful feedback today, everyone. I’m logging the ideas (grounding, source ranking, etc.) and will open issues tonight. Heading offline now but I’ll circle back tomorrow. Feel free to keep the questions coming!
dackdel 1y ago
this looks useful!!!!
devmor 1y ago
This is not research. This is a search engine.

67 points by nickwatson 1y ago

smallnix 1y ago

How do you do the citing? Reverse-RAG post processing?

nickwatson 1y ago
Good question — it's pretty straightforward right now: I pass the collected content chunks (with their original URLs attached) into Gemini 2.5 Pro, asking it to synthesize a balanced report and to inline citations throughout. So it's not doing anything fancy like dynamic retrieval or classic RAG architecture. Basically: - The agent gathers sources (webpages, PDFs, Reddit, etc.) - Summarises each as it goes (using a cheaper model) - Then hands a bundle of summarised + raw content to Gemini 2.5 Pro - Gemini 2.5 Pro writes the final report, embedding links directly as citations with [1], [2], etc style citations throughout. Reverse-RAG is something I for sure want to implement. Once I can afford a better computer to run this with at scale. Even an 8B model will take overnight to summarize an average piece of content for me right now! But I'm also keeping an eye on the pace of which AI moves in the larger LLM space. The size and abilities of likes of Gemini 2.5 Pro context windows are pretty crazy these days! Thanks for the question.
- iamandoni 1y ago
  Do you take any measures to prevent link hallucination? And content grounding / attribution verification?
  - nickwatson 1y ago
    At the moment the measures taken are: - Full content analysis by Primary LLM (Default is Gemini 2.5 Pro) with link hard-coded alongside each piece of content with structured output for better parsing. - Temperature right down (0.2), strict instructions to synthesize, precise prompts to attribute links exactly and without modification. What I hope to introduce: - Hard-coded parsing of links mentioned in final report to verify with the link map created throughout the research journey - Optional, "double-checking" LLM review of synthesized content to ensure no drift. - RAG enhancements for token-efficient verification and subsequent user questions (post-research) Do you have any further suggestions? Right now I hope to strike the delicate balance between token efficiency, with enhanced grounding as optional settings in the future. I have a big task list of things, and this is one of them. I will ensure to re-prioritize alongside user requests for the different features. Of course, being open source, contributions are highly welcome. I would love to see large community involvement. Collaboration benefits everyone. P.s. I have spent hundreds of dollars in tests. I'd say for every 1 hour of building, about 3 hours of testing have gone into this, debugging, optimizing quality, ensuring guard-rails are in place. If you go to the repo, also check out the config/prompts.py file - it will give you a little more insight into what is going on (there are code checks as well, but generally it gives you an idea).

nickwatson 1y ago

Hi HN I built *cleverb.ee* to solve my own research pain: too many tabs, too much messy sourcing. Gemini and OpenAI deep research tools weren't getting the balanced/unbiased quality I desired. *What it does*: • Reads webpages, PDFs, Reddit posts, PubMed abstracts, YouTube transcripts. • Plans with Gemini 2.5 Pro, acts with Gemini 2.5 Flash, summarises with Gemini 2.0 Flash (or you can use any Local LLM or Claude) • Outputs a fact-checked, cited Markdown report with live token usage tracking. *Tech*: Python + Playwright + LangChain with MCP tool support. AGPL-3.0 licensed. *Why open source?*: I wanted full transparency at every agent step and easy pluggable data sources. Quick install: ```bash git clone https://github.com/SureScaleAI/cleverbee https://github.com/SureScaleAI/cleverbee cd cleverbee && bash setup.sh Would love feedback — especially what critical research sources you’d want integrated next!

Quanttek 1y ago
Can you specify ? The default heavy reliance on Reddit and YouTube, rather than trusted publications (e.g. Scientific American, NYTimes) and scientific publications, is worrying given widespread misinformation in certain scientific fields (e.g. nutrition, health, economics)
- nickwatson 1y ago
  I never said "heavy reliance" on Reddit/YouTube. It actually is requested to use discernment to recognize poor, or biased sources and opinions, and label them as such (see the example report on Coffee which I shared previously in another comment). Most the time it has only sought out one or two post/youtube videos as it can recognize the low credibility value. It comes loaded with a PubMed MCP tool and the beauty of it being open source is you can exclude or limit the sources as much as you want, or add in new sources - that's why I wanted to open it up, to allow for critique over methodologies and allow for improved, balanced research from experts. It is also requested to evaluate the source and whether or not they have "some benefit to gain" from the article, to ensure it balances this into the research, also.
kleiba 1y ago
> Gemini and OpenAI deep research tools weren't getting the balanced/unbiased quality I desired. Could you elaborate, please?

cantaloupe 1y ago

I browsed the GitHub and website for a bit, but didn’t see any examples! It would be useful to share the output for a common question that can be substantiated with reliable sources, like “Is coffee good for me?”. Even better if you can show a comparison to other deep research tools. From the copy it seems like Cleverbee could pull from more diverse sources (e.g. YouTube) and fewer unreliable sources (e.g. product blogs). Show that off!

nickwatson 1y ago
I think that's a very good idea, thanks! I'm actually working on "The Beehive" at the moment, where the app can push the research to a hive on the website, so people can share their research/discoveries. My client's paid work takes priority, but I hope to do it over the course of this week. P.s. Running report now for you, "Is coffee good for me?" to show you this example ;)
nickwatson 1y ago
Done! https://docs.google.com/document/d/1bGVkI3xaBP1AvRxkB4GKeL0NrQhQ5ypBKKM5p5_TC8E/edit?usp=sharing https://docs.google.com/document/d/1bGVkI3xaBP1AvRxkB4GKeL0N...
- endianswap 1y ago
  lol what > Reference to failed web browser attempt for Rush University
  - nickwatson 1y ago
    Good spot! I already am on the case with that one. The Rush University website takes a long time to load (check it out), and the script recognized the article had loaded but something on the website was causing it to hang (waiting for networkidle status) so it terminated the parsing early and worked with the content it had. So in a nutshell, it still parsed but with a warning/error that this happened. I'm optimizing this now to exclude the wording from the report, or make a note.
  - 1y ago

semi-extrinsic 1y ago

I think it's really unfortunate that this type of thing gets called "research". I get that it fits with what has unfortunately become modern day usage - "Karen did her own research before becoming a flat-earther" - but I really wish the AI companies would've had better faith in their future solutions than to call this research. There's gotta be quite a few actual researchers at these companies who are shaking their heads. To spare others the lookup, here's from the Oxford dictionary. Emphasis on the word "new": To study a subject in detail, especially in order to discover new information or reach a new understanding. Example: They are carrying out/conducting/doing some fascinating research into/on the language of dolphins.

gnuly 1y ago
this case should simply be called search, no? to me research takes a long time, and not just an hour or so.
- kleiba 1y ago
  You overlook the fact that the system integrates various sources into a coherent report. This in my opinion makes it more than just mere search.
- pcthrowaway 1y ago
  Well the models often do an initial search and then a follow-up search. So it's a re-search
  - nickwatson 1y ago
    It does it several times, so maybe re-re-re-search works?
yard2010 1y ago
New is a matter of perspective.
nickwatson 1y ago
I understand what you're saying. I believe any kind of research will nearly always begin with learning and understanding of the knowledge that is already out there. Almost every subject has been learned this way, whether at school from a teacher or text-book, or reading papers. The Oxford dictionary definition says the same, "to study a subject in detail". This is what AI is doing - I see it as a "power suit" for distilling information much faster, without the cognitive bias that many of us will carry. Learning is an important part of research, and this must come with discernment over credibility of existing research, including identifying where the gaps are. This kind of critical thinking allows for another level, experiments, surveys, etc to uncover things even further. If you were to study the language of dolphins today, where would you start? Would you jump into the ocean and start trying to talk with them, or would you look up what is already discovered? Would you study their behaviors, patterns, etc? What drove me to do this project is exactly the example you mentioned, the flat-earther type who look up an article on some kind of free hosting website or Sandra from accounts social media page and taken as the be-all-and-end-all of knowledge. It comes without bias recognition or critical thinking skills. This is where I'm hopeful to level the playing field, and ensure unbiased, balanced information is uncovered.

nickwatson 1y ago

Thanks for all the thoughtful feedback today, everyone. I’m logging the ideas (grounding, source ranking, etc.) and will open issues tonight. Heading offline now but I’ll circle back tomorrow. Feel free to keep the questions coming!

dackdel 1y ago

this looks useful!!!!

devmor 1y ago

This is not research. This is a search engine.